Note: This is test shard 1 of 3.
[==========] Running 2 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 2 tests from CatalogManagerConfigurations/MasterStressTest
[ RUN ] CatalogManagerConfigurations/MasterStressTest.Test/0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240430 20:11:17.594071 18116 test_util.cc:268] Using random seed: -117296209
2024-04-30T20:11:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-04-30T20:11:17Z Disabled control of system clock
I20240430 20:11:17.628244 18116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNG3HqZ/build/asan/bin/kudu
/tmp/dist-test-taskNG3HqZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.177.62:36441
--webserver_interface=127.17.177.62
--webserver_port=0
--builtin_ntp_servers=127.17.177.20:42079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.177.62:36441,127.17.177.61:34777,127.17.177.60:37225
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20240430 20:11:17.962261 18124 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240430 20:11:17.962641 18124 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240430 20:11:17.962738 18124 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240430 20:11:17.970438 18124 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240430 20:11:17.970579 18124 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240430 20:11:17.970675 18124 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240430 20:11:17.970736 18124 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240430 20:11:17.980255 18124 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.177.20:42079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/master-0/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.17.177.62:36441,127.17.177.61:34777,127.17.177.60:37225
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.177.62:36441
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.177.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type FASTDEBUG
built by None at 30 Apr 2024 19:43:22 UTC on efd1337a2aa3
build id 63
ASAN enabled
I20240430 20:11:17.981665 18124 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240430 20:11:17.983510 18124 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240430 20:11:17.995445 18130 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 20:11:17.996130 18132 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 20:11:17.997133 18129 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240430 20:11:17.997900 18124 server_base.cc:1034] running on GCE node
I20240430 20:11:17.998972 18124 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240430 20:11:18.000706 18124 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240430 20:11:18.002008 18124 hybrid_clock.cc:648] HybridClock initialized: now 1714507878001937 us; error 60 us; skew 500 ppm
I20240430 20:11:18.002398 18124 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240430 20:11:18.005002 18124 webserver.cc:460] Webserver started at http://127.17.177.62:33023/ using document root <none> and password file <none>
I20240430 20:11:18.005795 18124 fs_manager.cc:359] Metadata directory not provided
I20240430 20:11:18.005940 18124 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240430 20:11:18.006251 18124 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240430 20:11:18.009393 18124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/master-0/data/instance:
uuid: "88b3f98793354c7aaa5c106e8ec31ed8"
format_stamp: "Formatted at 2024-04-30 20:11:17 on dist-test-slave-54l7"
I20240430 20:11:18.010217 18124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/master-0/wal/instance:
uuid: "88b3f98793354c7aaa5c106e8ec31ed8"
format_stamp: "Formatted at 2024-04-30 20:11:17 on dist-test-slave-54l7"
I20240430 20:11:18.016557 18124 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.002s
I20240430 20:11:18.020632 18138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240430 20:11:18.022420 18124 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20240430 20:11:18.022670 18124 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/master-0/data,/tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/master-0/wal
uuid: "88b3f98793354c7aaa5c106e8ec31ed8"
format_stamp: "Formatted at 2024-04-30 20:11:17 on dist-test-slave-54l7"
I20240430 20:11:18.022898 18124 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240430 20:11:18.041337 18124 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240430 20:11:18.042320 18124 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240430 20:11:18.042697 18124 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240430 20:11:18.062234 18124 rpc_server.cc:307] RPC server started. Bound to: 127.17.177.62:36441
I20240430 20:11:18.062254 18190 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.177.62:36441 every 8 connection(s)
I20240430 20:11:18.063822 18124 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/master-0/data/info.pb
I20240430 20:11:18.066354 18116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNG3HqZ/build/asan/bin/kudu as pid 18124
I20240430 20:11:18.066726 18116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/master-0/wal/instance
I20240430 20:11:18.069187 18191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:18.070281 18116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNG3HqZ/build/asan/bin/kudu
/tmp/dist-test-taskNG3HqZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.177.61:34777
--webserver_interface=127.17.177.61
--webserver_port=0
--builtin_ntp_servers=127.17.177.20:42079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.177.62:36441,127.17.177.61:34777,127.17.177.60:37225
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20240430 20:11:18.077297 18191 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.177.62" port: 36441 } has no permanent_uuid. Determining permanent_uuid...
I20240430 20:11:18.094009 18191 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.177.61" port: 34777 } has no permanent_uuid. Determining permanent_uuid...
W20240430 20:11:18.095415 18140 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.177.61:34777: connect: Connection refused (error 111)
W20240430 20:11:18.097011 18191 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.17.177.61:34777: Network error: Client connection negotiation failed: client connection to 127.17.177.61:34777: connect: Connection refused (error 111)
I20240430 20:11:18.146379 18191 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.177.61" port: 34777 } attempt: 1
W20240430 20:11:18.148819 18191 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.17.177.61:34777: Network error: Client connection negotiation failed: client connection to 127.17.177.61:34777: connect: Connection refused (error 111)
I20240430 20:11:18.217176 18191 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.177.61" port: 34777 } attempt: 2
W20240430 20:11:18.219174 18191 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.17.177.61:34777: Network error: Client connection negotiation failed: client connection to 127.17.177.61:34777: connect: Connection refused (error 111)
I20240430 20:11:18.310470 18191 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.177.61" port: 34777 } attempt: 3
W20240430 20:11:18.312460 18191 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.17.177.61:34777: Network error: Client connection negotiation failed: client connection to 127.17.177.61:34777: connect: Connection refused (error 111)
W20240430 20:11:18.417363 18193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240430 20:11:18.417702 18193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240430 20:11:18.417832 18193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240430 20:11:18.425436 18193 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240430 20:11:18.425570 18193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240430 20:11:18.425616 18193 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240430 20:11:18.425706 18193 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240430 20:11:18.435163 18193 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.177.20:42079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/master-1/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.17.177.62:36441,127.17.177.61:34777,127.17.177.60:37225
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.177.61:34777
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.17.177.61
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type FASTDEBUG
built by None at 30 Apr 2024 19:43:22 UTC on efd1337a2aa3
build id 63
ASAN enabled
I20240430 20:11:18.436582 18193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240430 20:11:18.438395 18193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240430 20:11:18.450009 18201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 20:11:18.450554 18204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 20:11:18.451014 18202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240430 20:11:18.452713 18193 server_base.cc:1034] running on GCE node
I20240430 20:11:18.453588 18193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240430 20:11:18.455153 18193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240430 20:11:18.455689 18191 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.177.61" port: 34777 } attempt: 4
I20240430 20:11:18.456400 18193 hybrid_clock.cc:648] HybridClock initialized: now 1714507878456321 us; error 49 us; skew 500 ppm
I20240430 20:11:18.456841 18193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
W20240430 20:11:18.457583 18191 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.17.177.61:34777: Network error: Client connection negotiation failed: client connection to 127.17.177.61:34777: connect: Connection refused (error 111)
I20240430 20:11:18.459517 18193 webserver.cc:460] Webserver started at http://127.17.177.61:40629/ using document root <none> and password file <none>
I20240430 20:11:18.460242 18193 fs_manager.cc:359] Metadata directory not provided
I20240430 20:11:18.460359 18193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240430 20:11:18.460695 18193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240430 20:11:18.463804 18193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/master-1/data/instance:
uuid: "89df84ccf5394c9bb60d4e4ed872fc2e"
format_stamp: "Formatted at 2024-04-30 20:11:18 on dist-test-slave-54l7"
I20240430 20:11:18.464601 18193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/master-1/wal/instance:
uuid: "89df84ccf5394c9bb60d4e4ed872fc2e"
format_stamp: "Formatted at 2024-04-30 20:11:18 on dist-test-slave-54l7"
I20240430 20:11:18.470924 18193 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20240430 20:11:18.474546 18210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240430 20:11:18.476001 18193 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240430 20:11:18.476202 18193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/master-1/data,/tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/master-1/wal
uuid: "89df84ccf5394c9bb60d4e4ed872fc2e"
format_stamp: "Formatted at 2024-04-30 20:11:18 on dist-test-slave-54l7"
I20240430 20:11:18.476392 18193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240430 20:11:18.502821 18193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240430 20:11:18.503748 18193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240430 20:11:18.504089 18193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240430 20:11:18.523175 18193 rpc_server.cc:307] RPC server started. Bound to: 127.17.177.61:34777
I20240430 20:11:18.523193 18261 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.177.61:34777 every 8 connection(s)
I20240430 20:11:18.524639 18193 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/master-1/data/info.pb
I20240430 20:11:18.528817 18262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:18.530351 18116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNG3HqZ/build/asan/bin/kudu as pid 18193
I20240430 20:11:18.530548 18116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/master-1/wal/instance
I20240430 20:11:18.531648 18116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNG3HqZ/build/asan/bin/kudu
/tmp/dist-test-taskNG3HqZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.177.60:37225
--webserver_interface=127.17.177.60
--webserver_port=0
--builtin_ntp_servers=127.17.177.20:42079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.177.62:36441,127.17.177.61:34777,127.17.177.60:37225
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20240430 20:11:18.535151 18262 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.177.62" port: 36441 } has no permanent_uuid. Determining permanent_uuid...
I20240430 20:11:18.549670 18262 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.177.61" port: 34777 } has no permanent_uuid. Determining permanent_uuid...
I20240430 20:11:18.559872 18262 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.177.60" port: 37225 } has no permanent_uuid. Determining permanent_uuid...
W20240430 20:11:18.560999 18211 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.177.60:37225: connect: Connection refused (error 111)
W20240430 20:11:18.562563 18262 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.17.177.60:37225: Network error: Client connection negotiation failed: client connection to 127.17.177.60:37225: connect: Connection refused (error 111)
I20240430 20:11:18.611941 18262 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.177.60" port: 37225 } attempt: 1
W20240430 20:11:18.613886 18262 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.17.177.60:37225: Network error: Client connection negotiation failed: client connection to 127.17.177.60:37225: connect: Connection refused (error 111)
I20240430 20:11:18.682126 18262 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.177.60" port: 37225 } attempt: 2
W20240430 20:11:18.684103 18262 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.17.177.60:37225: Network error: Client connection negotiation failed: client connection to 127.17.177.60:37225: connect: Connection refused (error 111)
I20240430 20:11:18.756831 18191 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.177.61" port: 34777 } attempt: 5
I20240430 20:11:18.763602 18191 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.177.60" port: 37225 } has no permanent_uuid. Determining permanent_uuid...
W20240430 20:11:18.765257 18191 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.17.177.60:37225: Network error: Client connection negotiation failed: client connection to 127.17.177.60:37225: connect: Connection refused (error 111)
I20240430 20:11:18.775327 18262 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.177.60" port: 37225 } attempt: 3
W20240430 20:11:18.777390 18262 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.17.177.60:37225: Network error: Client connection negotiation failed: client connection to 127.17.177.60:37225: connect: Connection refused (error 111)
I20240430 20:11:18.816488 18191 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.177.60" port: 37225 } attempt: 1
W20240430 20:11:18.818343 18191 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.17.177.60:37225: Network error: Client connection negotiation failed: client connection to 127.17.177.60:37225: connect: Connection refused (error 111)
W20240430 20:11:18.871138 18264 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240430 20:11:18.871488 18264 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240430 20:11:18.871629 18264 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240430 20:11:18.879460 18264 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240430 20:11:18.879601 18264 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240430 20:11:18.879686 18264 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240430 20:11:18.879776 18264 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240430 20:11:18.886619 18191 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.177.60" port: 37225 } attempt: 2
W20240430 20:11:18.888398 18191 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.17.177.60:37225: Network error: Client connection negotiation failed: client connection to 127.17.177.60:37225: connect: Connection refused (error 111)
I20240430 20:11:18.889396 18264 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.177.20:42079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/master-2/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.17.177.62:36441,127.17.177.61:34777,127.17.177.60:37225
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.177.60:37225
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.17.177.60
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type FASTDEBUG
built by None at 30 Apr 2024 19:43:22 UTC on efd1337a2aa3
build id 63
ASAN enabled
I20240430 20:11:18.890866 18264 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240430 20:11:18.892684 18264 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240430 20:11:18.904101 18272 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 20:11:18.904654 18275 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 20:11:18.905201 18273 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240430 20:11:18.905827 18264 server_base.cc:1034] running on GCE node
I20240430 20:11:18.906704 18264 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240430 20:11:18.908263 18264 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240430 20:11:18.909498 18264 hybrid_clock.cc:648] HybridClock initialized: now 1714507878909427 us; error 55 us; skew 500 ppm
I20240430 20:11:18.909847 18264 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240430 20:11:18.912235 18264 webserver.cc:460] Webserver started at http://127.17.177.60:34267/ using document root <none> and password file <none>
I20240430 20:11:18.912983 18264 fs_manager.cc:359] Metadata directory not provided
I20240430 20:11:18.913115 18264 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240430 20:11:18.913417 18264 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240430 20:11:18.916180 18264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/master-2/data/instance:
uuid: "b632d6ef4c104a058a6db208024cab33"
format_stamp: "Formatted at 2024-04-30 20:11:18 on dist-test-slave-54l7"
I20240430 20:11:18.916918 18264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/master-2/wal/instance:
uuid: "b632d6ef4c104a058a6db208024cab33"
format_stamp: "Formatted at 2024-04-30 20:11:18 on dist-test-slave-54l7"
I20240430 20:11:18.920626 18262 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.177.60" port: 37225 } attempt: 4
W20240430 20:11:18.922395 18262 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.17.177.60:37225: Network error: Client connection negotiation failed: client connection to 127.17.177.60:37225: connect: Connection refused (error 111)
I20240430 20:11:18.922427 18264 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20240430 20:11:18.926113 18281 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240430 20:11:18.927564 18264 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240430 20:11:18.927747 18264 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/master-2/data,/tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/master-2/wal
uuid: "b632d6ef4c104a058a6db208024cab33"
format_stamp: "Formatted at 2024-04-30 20:11:18 on dist-test-slave-54l7"
I20240430 20:11:18.927974 18264 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240430 20:11:18.953819 18264 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240430 20:11:18.954761 18264 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240430 20:11:18.955072 18264 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240430 20:11:18.972630 18264 rpc_server.cc:307] RPC server started. Bound to: 127.17.177.60:37225
I20240430 20:11:18.972652 18332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.177.60:37225 every 8 connection(s)
I20240430 20:11:18.974104 18264 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/master-2/data/info.pb
I20240430 20:11:18.977955 18333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:18.979290 18116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNG3HqZ/build/asan/bin/kudu as pid 18264
I20240430 20:11:18.979507 18116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/master-2/wal/instance
I20240430 20:11:18.985492 18333 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.177.62" port: 36441 } has no permanent_uuid. Determining permanent_uuid...
I20240430 20:11:18.994638 18191 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.177.60" port: 37225 } attempt: 3
I20240430 20:11:19.000023 18333 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.177.61" port: 34777 } has no permanent_uuid. Determining permanent_uuid...
I20240430 20:11:19.008946 18333 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.177.60" port: 37225 } has no permanent_uuid. Determining permanent_uuid...
I20240430 20:11:19.010111 18191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: Bootstrap starting.
I20240430 20:11:19.014254 18191 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:19.016057 18191 log.cc:824] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: Log is configured to *not* fsync() on all Append() calls
I20240430 20:11:19.019652 18333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: Bootstrap starting.
I20240430 20:11:19.022511 18333 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:19.023828 18333 log.cc:824] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: Log is configured to *not* fsync() on all Append() calls
I20240430 20:11:19.025104 18191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: No bootstrap required, opened a new log
I20240430 20:11:19.026419 18333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: No bootstrap required, opened a new log
I20240430 20:11:19.031749 18191 raft_consensus.cc:348] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b3f98793354c7aaa5c106e8ec31ed8" member_type: VOTER last_known_addr { host: "127.17.177.62" port: 36441 } } peers { permanent_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e" member_type: VOTER last_known_addr { host: "127.17.177.61" port: 34777 } } peers { permanent_uuid: "b632d6ef4c104a058a6db208024cab33" member_type: VOTER last_known_addr { host: "127.17.177.60" port: 37225 } }
I20240430 20:11:19.032047 18191 raft_consensus.cc:374] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:19.032178 18191 raft_consensus.cc:729] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b3f98793354c7aaa5c106e8ec31ed8, State: Initialized, Role: FOLLOWER
I20240430 20:11:19.032802 18191 consensus_queue.cc:260] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b3f98793354c7aaa5c106e8ec31ed8" member_type: VOTER last_known_addr { host: "127.17.177.62" port: 36441 } } peers { permanent_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e" member_type: VOTER last_known_addr { host: "127.17.177.61" port: 34777 } } peers { permanent_uuid: "b632d6ef4c104a058a6db208024cab33" member_type: VOTER last_known_addr { host: "127.17.177.60" port: 37225 } }
I20240430 20:11:19.033258 18333 raft_consensus.cc:348] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b3f98793354c7aaa5c106e8ec31ed8" member_type: VOTER last_known_addr { host: "127.17.177.62" port: 36441 } } peers { permanent_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e" member_type: VOTER last_known_addr { host: "127.17.177.61" port: 34777 } } peers { permanent_uuid: "b632d6ef4c104a058a6db208024cab33" member_type: VOTER last_known_addr { host: "127.17.177.60" port: 37225 } }
I20240430 20:11:19.033640 18333 raft_consensus.cc:374] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:19.033784 18333 raft_consensus.cc:729] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b632d6ef4c104a058a6db208024cab33, State: Initialized, Role: FOLLOWER
I20240430 20:11:19.033721 18343 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b3f98793354c7aaa5c106e8ec31ed8" member_type: VOTER last_known_addr { host: "127.17.177.62" port: 36441 } } peers { permanent_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e" member_type: VOTER last_known_addr { host: "127.17.177.61" port: 34777 } } peers { permanent_uuid: "b632d6ef4c104a058a6db208024cab33" member_type: VOTER last_known_addr { host: "127.17.177.60" port: 37225 } } }
I20240430 20:11:19.034067 18343 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8 [sys.catalog]: This master's current role is: FOLLOWER
I20240430 20:11:19.034442 18333 consensus_queue.cc:260] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b3f98793354c7aaa5c106e8ec31ed8" member_type: VOTER last_known_addr { host: "127.17.177.62" port: 36441 } } peers { permanent_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e" member_type: VOTER last_known_addr { host: "127.17.177.61" port: 34777 } } peers { permanent_uuid: "b632d6ef4c104a058a6db208024cab33" member_type: VOTER last_known_addr { host: "127.17.177.60" port: 37225 } }
I20240430 20:11:19.034802 18191 sys_catalog.cc:564] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8 [sys.catalog]: configured and running, proceeding with master startup.
I20240430 20:11:19.035370 18344 sys_catalog.cc:455] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b3f98793354c7aaa5c106e8ec31ed8" member_type: VOTER last_known_addr { host: "127.17.177.62" port: 36441 } } peers { permanent_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e" member_type: VOTER last_known_addr { host: "127.17.177.61" port: 34777 } } peers { permanent_uuid: "b632d6ef4c104a058a6db208024cab33" member_type: VOTER last_known_addr { host: "127.17.177.60" port: 37225 } } }
I20240430 20:11:19.035691 18344 sys_catalog.cc:458] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33 [sys.catalog]: This master's current role is: FOLLOWER
I20240430 20:11:19.036302 18333 sys_catalog.cc:564] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33 [sys.catalog]: configured and running, proceeding with master startup.
W20240430 20:11:19.045706 18144 tablet.cc:2369] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240430 20:11:19.052418 18359 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240430 20:11:19.052556 18359 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240430 20:11:19.051308 18124 master_runner.cc:186] Error getting master registration for 127.17.177.61:34777: OK, instance_id { permanent_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e" instance_seqno: 1714507878517643 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20240430 20:11:19.060639 18124 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20240430 20:11:19.063900 18367 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240430 20:11:19.064038 18367 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240430 20:11:19.072571 18344 raft_consensus.cc:482] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:19.072845 18344 raft_consensus.cc:504] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b3f98793354c7aaa5c106e8ec31ed8" member_type: VOTER last_known_addr { host: "127.17.177.62" port: 36441 } } peers { permanent_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e" member_type: VOTER last_known_addr { host: "127.17.177.61" port: 34777 } } peers { permanent_uuid: "b632d6ef4c104a058a6db208024cab33" member_type: VOTER last_known_addr { host: "127.17.177.60" port: 37225 } }
I20240430 20:11:19.076690 18166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b632d6ef4c104a058a6db208024cab33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3f98793354c7aaa5c106e8ec31ed8" is_pre_election: true
I20240430 20:11:19.077216 18166 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b632d6ef4c104a058a6db208024cab33 in term 0.
I20240430 20:11:19.078027 18282 leader_election.cc:302] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b3f98793354c7aaa5c106e8ec31ed8, b632d6ef4c104a058a6db208024cab33; no voters:
I20240430 20:11:19.074698 18344 leader_election.cc:288] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88b3f98793354c7aaa5c106e8ec31ed8 (127.17.177.62:36441), 89df84ccf5394c9bb60d4e4ed872fc2e (127.17.177.61:34777)
I20240430 20:11:19.078786 18344 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:19.078924 18344 raft_consensus.cc:482] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:19.079054 18344 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:19.069192 18264 master_runner.cc:186] Error getting master registration for 127.17.177.61:34777: OK, instance_id { permanent_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e" instance_seqno: 1714507878517643 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20240430 20:11:19.079253 18264 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240430 20:11:19.083480 18344 raft_consensus.cc:504] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b3f98793354c7aaa5c106e8ec31ed8" member_type: VOTER last_known_addr { host: "127.17.177.62" port: 36441 } } peers { permanent_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e" member_type: VOTER last_known_addr { host: "127.17.177.61" port: 34777 } } peers { permanent_uuid: "b632d6ef4c104a058a6db208024cab33" member_type: VOTER last_known_addr { host: "127.17.177.60" port: 37225 } }
I20240430 20:11:19.085083 18344 leader_election.cc:288] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33 [CANDIDATE]: Term 1 election: Requested vote from peers 88b3f98793354c7aaa5c106e8ec31ed8 (127.17.177.62:36441), 89df84ccf5394c9bb60d4e4ed872fc2e (127.17.177.61:34777)
I20240430 20:11:19.087502 18166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b632d6ef4c104a058a6db208024cab33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b3f98793354c7aaa5c106e8ec31ed8"
I20240430 20:11:19.087709 18166 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:19.091122 18166 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b632d6ef4c104a058a6db208024cab33 in term 1.
I20240430 20:11:19.091576 18282 leader_election.cc:302] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b3f98793354c7aaa5c106e8ec31ed8, b632d6ef4c104a058a6db208024cab33; no voters:
I20240430 20:11:19.092016 18370 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:19.092249 18370 raft_consensus.cc:686] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33 [term 1 LEADER]: Becoming Leader. State: Replica: b632d6ef4c104a058a6db208024cab33, State: Running, Role: LEADER
I20240430 20:11:19.092952 18370 consensus_queue.cc:237] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b3f98793354c7aaa5c106e8ec31ed8" member_type: VOTER last_known_addr { host: "127.17.177.62" port: 36441 } } peers { permanent_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e" member_type: VOTER last_known_addr { host: "127.17.177.61" port: 34777 } } peers { permanent_uuid: "b632d6ef4c104a058a6db208024cab33" member_type: VOTER last_known_addr { host: "127.17.177.60" port: 37225 } }
I20240430 20:11:19.097543 18344 sys_catalog.cc:455] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b632d6ef4c104a058a6db208024cab33. Latest consensus state: current_term: 1 leader_uuid: "b632d6ef4c104a058a6db208024cab33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b3f98793354c7aaa5c106e8ec31ed8" member_type: VOTER last_known_addr { host: "127.17.177.62" port: 36441 } } peers { permanent_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e" member_type: VOTER last_known_addr { host: "127.17.177.61" port: 34777 } } peers { permanent_uuid: "b632d6ef4c104a058a6db208024cab33" member_type: VOTER last_known_addr { host: "127.17.177.60" port: 37225 } } }
I20240430 20:11:19.097867 18344 sys_catalog.cc:458] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33 [sys.catalog]: This master's current role is: LEADER
I20240430 20:11:19.107628 18372 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240430 20:11:19.111851 18372 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240430 20:11:19.124693 18166 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8 [term 1 FOLLOWER]: Refusing update from remote peer b632d6ef4c104a058a6db208024cab33: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240430 20:11:19.125850 18344 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3f98793354c7aaa5c106e8ec31ed8" member_type: VOTER last_known_addr { host: "127.17.177.62" port: 36441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:19.144559 18343 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b632d6ef4c104a058a6db208024cab33. Latest consensus state: current_term: 1 leader_uuid: "b632d6ef4c104a058a6db208024cab33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b3f98793354c7aaa5c106e8ec31ed8" member_type: VOTER last_known_addr { host: "127.17.177.62" port: 36441 } } peers { permanent_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e" member_type: VOTER last_known_addr { host: "127.17.177.61" port: 34777 } } peers { permanent_uuid: "b632d6ef4c104a058a6db208024cab33" member_type: VOTER last_known_addr { host: "127.17.177.60" port: 37225 } } }
I20240430 20:11:19.144968 18343 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8 [sys.catalog]: This master's current role is: FOLLOWER
I20240430 20:11:19.149638 18373 mvcc.cc:205] Tried to move back new op lower bound from 7022624272883286016 to 7022624272773148672. Current Snapshot: MvccSnapshot[applied={T|T < 7022624272883286016}]
I20240430 20:11:19.151057 18374 mvcc.cc:205] Tried to move back new op lower bound from 7022624272883286016 to 7022624272773148672. Current Snapshot: MvccSnapshot[applied={T|T < 7022624272883286016}]
I20240430 20:11:19.157348 18343 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b632d6ef4c104a058a6db208024cab33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b3f98793354c7aaa5c106e8ec31ed8" member_type: VOTER last_known_addr { host: "127.17.177.62" port: 36441 } } peers { permanent_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e" member_type: VOTER last_known_addr { host: "127.17.177.61" port: 34777 } } peers { permanent_uuid: "b632d6ef4c104a058a6db208024cab33" member_type: VOTER last_known_addr { host: "127.17.177.60" port: 37225 } } }
I20240430 20:11:19.157615 18343 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8 [sys.catalog]: This master's current role is: FOLLOWER
I20240430 20:11:19.165236 18344 sys_catalog.cc:455] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b632d6ef4c104a058a6db208024cab33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b3f98793354c7aaa5c106e8ec31ed8" member_type: VOTER last_known_addr { host: "127.17.177.62" port: 36441 } } peers { permanent_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e" member_type: VOTER last_known_addr { host: "127.17.177.61" port: 34777 } } peers { permanent_uuid: "b632d6ef4c104a058a6db208024cab33" member_type: VOTER last_known_addr { host: "127.17.177.60" port: 37225 } } }
I20240430 20:11:19.165532 18344 sys_catalog.cc:458] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33 [sys.catalog]: This master's current role is: LEADER
I20240430 20:11:19.167402 18372 catalog_manager.cc:1347] Generated new cluster ID: 889e0faede1f41d1ab74633d6826c0d4
I20240430 20:11:19.167516 18372 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240430 20:11:19.205683 18372 catalog_manager.cc:1370] Generated new certificate authority record
I20240430 20:11:19.207624 18372 catalog_manager.cc:1504] Loading token signing keys...
I20240430 20:11:19.222012 18262 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.177.60" port: 37225 } attempt: 5
I20240430 20:11:19.233556 18372 catalog_manager.cc:5857] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: Generated new TSK 0
I20240430 20:11:19.234835 18372 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240430 20:11:19.239472 18262 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89df84ccf5394c9bb60d4e4ed872fc2e: Bootstrap starting.
I20240430 20:11:19.250046 18262 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 89df84ccf5394c9bb60d4e4ed872fc2e: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:19.251511 18116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNG3HqZ/build/asan/bin/kudu
/tmp/dist-test-taskNG3HqZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.177.1:0
--local_ip_for_outbound_sockets=127.17.177.1
--webserver_interface=127.17.177.1
--webserver_port=0
--tserver_master_addrs=127.17.177.62:36441,127.17.177.61:34777,127.17.177.60:37225
--builtin_ntp_servers=127.17.177.20:42079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20240430 20:11:19.251719 18262 log.cc:824] T 00000000000000000000000000000000 P 89df84ccf5394c9bb60d4e4ed872fc2e: Log is configured to *not* fsync() on all Append() calls
I20240430 20:11:19.261186 18262 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89df84ccf5394c9bb60d4e4ed872fc2e: No bootstrap required, opened a new log
I20240430 20:11:19.271440 18262 raft_consensus.cc:348] T 00000000000000000000000000000000 P 89df84ccf5394c9bb60d4e4ed872fc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b3f98793354c7aaa5c106e8ec31ed8" member_type: VOTER last_known_addr { host: "127.17.177.62" port: 36441 } } peers { permanent_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e" member_type: VOTER last_known_addr { host: "127.17.177.61" port: 34777 } } peers { permanent_uuid: "b632d6ef4c104a058a6db208024cab33" member_type: VOTER last_known_addr { host: "127.17.177.60" port: 37225 } }
I20240430 20:11:19.271979 18262 raft_consensus.cc:374] T 00000000000000000000000000000000 P 89df84ccf5394c9bb60d4e4ed872fc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:19.272150 18262 raft_consensus.cc:729] T 00000000000000000000000000000000 P 89df84ccf5394c9bb60d4e4ed872fc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89df84ccf5394c9bb60d4e4ed872fc2e, State: Initialized, Role: FOLLOWER
I20240430 20:11:19.273075 18262 consensus_queue.cc:260] T 00000000000000000000000000000000 P 89df84ccf5394c9bb60d4e4ed872fc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b3f98793354c7aaa5c106e8ec31ed8" member_type: VOTER last_known_addr { host: "127.17.177.62" port: 36441 } } peers { permanent_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e" member_type: VOTER last_known_addr { host: "127.17.177.61" port: 34777 } } peers { permanent_uuid: "b632d6ef4c104a058a6db208024cab33" member_type: VOTER last_known_addr { host: "127.17.177.60" port: 37225 } }
I20240430 20:11:19.283627 18262 sys_catalog.cc:564] T 00000000000000000000000000000000 P 89df84ccf5394c9bb60d4e4ed872fc2e [sys.catalog]: configured and running, proceeding with master startup.
I20240430 20:11:19.283672 18236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b632d6ef4c104a058a6db208024cab33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e"
I20240430 20:11:19.284363 18237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b632d6ef4c104a058a6db208024cab33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e" is_pre_election: true
I20240430 20:11:19.285445 18380 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89df84ccf5394c9bb60d4e4ed872fc2e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b3f98793354c7aaa5c106e8ec31ed8" member_type: VOTER last_known_addr { host: "127.17.177.62" port: 36441 } } peers { permanent_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e" member_type: VOTER last_known_addr { host: "127.17.177.61" port: 34777 } } peers { permanent_uuid: "b632d6ef4c104a058a6db208024cab33" member_type: VOTER last_known_addr { host: "127.17.177.60" port: 37225 } } }
I20240430 20:11:19.285745 18380 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89df84ccf5394c9bb60d4e4ed872fc2e [sys.catalog]: This master's current role is: FOLLOWER
I20240430 20:11:19.286545 18236 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 89df84ccf5394c9bb60d4e4ed872fc2e [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:19.290031 18236 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 89df84ccf5394c9bb60d4e4ed872fc2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b632d6ef4c104a058a6db208024cab33 in term 1.
I20240430 20:11:19.291713 18235 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 89df84ccf5394c9bb60d4e4ed872fc2e [term 1 FOLLOWER]: Refusing update from remote peer b632d6ef4c104a058a6db208024cab33: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240430 20:11:19.292944 18370 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e" member_type: VOTER last_known_addr { host: "127.17.177.61" port: 34777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:19.362108 18388 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89df84ccf5394c9bb60d4e4ed872fc2e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b632d6ef4c104a058a6db208024cab33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b3f98793354c7aaa5c106e8ec31ed8" member_type: VOTER last_known_addr { host: "127.17.177.62" port: 36441 } } peers { permanent_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e" member_type: VOTER last_known_addr { host: "127.17.177.61" port: 34777 } } peers { permanent_uuid: "b632d6ef4c104a058a6db208024cab33" member_type: VOTER last_known_addr { host: "127.17.177.60" port: 37225 } } }
I20240430 20:11:19.362545 18388 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89df84ccf5394c9bb60d4e4ed872fc2e [sys.catalog]: This master's current role is: FOLLOWER
I20240430 20:11:19.362102 18380 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89df84ccf5394c9bb60d4e4ed872fc2e [sys.catalog]: SysCatalogTable state changed. Reason: New leader b632d6ef4c104a058a6db208024cab33. Latest consensus state: current_term: 1 leader_uuid: "b632d6ef4c104a058a6db208024cab33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b3f98793354c7aaa5c106e8ec31ed8" member_type: VOTER last_known_addr { host: "127.17.177.62" port: 36441 } } peers { permanent_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e" member_type: VOTER last_known_addr { host: "127.17.177.61" port: 34777 } } peers { permanent_uuid: "b632d6ef4c104a058a6db208024cab33" member_type: VOTER last_known_addr { host: "127.17.177.60" port: 37225 } } }
I20240430 20:11:19.363103 18380 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89df84ccf5394c9bb60d4e4ed872fc2e [sys.catalog]: This master's current role is: FOLLOWER
I20240430 20:11:19.423391 18396 catalog_manager.cc:1259] Loaded cluster ID: 889e0faede1f41d1ab74633d6826c0d4
I20240430 20:11:19.423592 18396 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 89df84ccf5394c9bb60d4e4ed872fc2e: loading cluster ID for follower catalog manager: success
I20240430 20:11:19.430270 18396 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 89df84ccf5394c9bb60d4e4ed872fc2e: acquiring CA information for follower catalog manager: success
I20240430 20:11:19.437672 18396 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 89df84ccf5394c9bb60d4e4ed872fc2e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240430 20:11:19.746937 18398 raft_consensus.cc:482] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b632d6ef4c104a058a6db208024cab33)
I20240430 20:11:19.747193 18398 raft_consensus.cc:504] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b3f98793354c7aaa5c106e8ec31ed8" member_type: VOTER last_known_addr { host: "127.17.177.62" port: 36441 } } peers { permanent_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e" member_type: VOTER last_known_addr { host: "127.17.177.61" port: 34777 } } peers { permanent_uuid: "b632d6ef4c104a058a6db208024cab33" member_type: VOTER last_known_addr { host: "127.17.177.60" port: 37225 } }
I20240430 20:11:19.749285 18398 leader_election.cc:288] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 89df84ccf5394c9bb60d4e4ed872fc2e (127.17.177.61:34777), b632d6ef4c104a058a6db208024cab33 (127.17.177.60:37225)
I20240430 20:11:19.749661 18235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88b3f98793354c7aaa5c106e8ec31ed8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e" is_pre_election: true
I20240430 20:11:19.750818 18308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88b3f98793354c7aaa5c106e8ec31ed8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b632d6ef4c104a058a6db208024cab33" is_pre_election: true
I20240430 20:11:19.752041 18140 leader_election.cc:302] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88b3f98793354c7aaa5c106e8ec31ed8; no voters: 89df84ccf5394c9bb60d4e4ed872fc2e, b632d6ef4c104a058a6db208024cab33
I20240430 20:11:19.752496 18398 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240430 20:11:19.789139 18379 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240430 20:11:19.789588 18379 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240430 20:11:19.789793 18379 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240430 20:11:19.796487 18380 raft_consensus.cc:482] T 00000000000000000000000000000000 P 89df84ccf5394c9bb60d4e4ed872fc2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b632d6ef4c104a058a6db208024cab33)
I20240430 20:11:19.796749 18380 raft_consensus.cc:504] T 00000000000000000000000000000000 P 89df84ccf5394c9bb60d4e4ed872fc2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b3f98793354c7aaa5c106e8ec31ed8" member_type: VOTER last_known_addr { host: "127.17.177.62" port: 36441 } } peers { permanent_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e" member_type: VOTER last_known_addr { host: "127.17.177.61" port: 34777 } } peers { permanent_uuid: "b632d6ef4c104a058a6db208024cab33" member_type: VOTER last_known_addr { host: "127.17.177.60" port: 37225 } }
I20240430 20:11:19.798677 18380 leader_election.cc:288] T 00000000000000000000000000000000 P 89df84ccf5394c9bb60d4e4ed872fc2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 88b3f98793354c7aaa5c106e8ec31ed8 (127.17.177.62:36441), b632d6ef4c104a058a6db208024cab33 (127.17.177.60:37225)
I20240430 20:11:19.799130 18166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "88b3f98793354c7aaa5c106e8ec31ed8" is_pre_election: true
I20240430 20:11:19.799224 18308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b632d6ef4c104a058a6db208024cab33" is_pre_election: true
I20240430 20:11:19.799393 18166 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89df84ccf5394c9bb60d4e4ed872fc2e in term 1.
I20240430 20:11:19.799916 18211 leader_election.cc:302] T 00000000000000000000000000000000 P 89df84ccf5394c9bb60d4e4ed872fc2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b3f98793354c7aaa5c106e8ec31ed8, 89df84ccf5394c9bb60d4e4ed872fc2e; no voters:
I20240430 20:11:19.800817 18380 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 89df84ccf5394c9bb60d4e4ed872fc2e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240430 20:11:19.800961 18380 raft_consensus.cc:482] T 00000000000000000000000000000000 P 89df84ccf5394c9bb60d4e4ed872fc2e [term 1 FOLLOWER]: Starting leader election (detected failure of leader b632d6ef4c104a058a6db208024cab33)
I20240430 20:11:19.801066 18380 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 89df84ccf5394c9bb60d4e4ed872fc2e [term 1 FOLLOWER]: Advancing to term 2
W20240430 20:11:19.804363 18379 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240430 20:11:19.804647 18379 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.177.1
I20240430 20:11:19.804489 18380 raft_consensus.cc:504] T 00000000000000000000000000000000 P 89df84ccf5394c9bb60d4e4ed872fc2e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b3f98793354c7aaa5c106e8ec31ed8" member_type: VOTER last_known_addr { host: "127.17.177.62" port: 36441 } } peers { permanent_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e" member_type: VOTER last_known_addr { host: "127.17.177.61" port: 34777 } } peers { permanent_uuid: "b632d6ef4c104a058a6db208024cab33" member_type: VOTER last_known_addr { host: "127.17.177.60" port: 37225 } }
I20240430 20:11:19.806432 18166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "88b3f98793354c7aaa5c106e8ec31ed8"
I20240430 20:11:19.806711 18166 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8 [term 1 FOLLOWER]: Advancing to term 2
I20240430 20:11:19.807461 18308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b632d6ef4c104a058a6db208024cab33"
I20240430 20:11:19.808161 18380 leader_election.cc:288] T 00000000000000000000000000000000 P 89df84ccf5394c9bb60d4e4ed872fc2e [CANDIDATE]: Term 2 election: Requested vote from peers 88b3f98793354c7aaa5c106e8ec31ed8 (127.17.177.62:36441), b632d6ef4c104a058a6db208024cab33 (127.17.177.60:37225)
I20240430 20:11:19.809681 18166 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89df84ccf5394c9bb60d4e4ed872fc2e in term 2.
I20240430 20:11:19.810199 18211 leader_election.cc:302] T 00000000000000000000000000000000 P 89df84ccf5394c9bb60d4e4ed872fc2e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88b3f98793354c7aaa5c106e8ec31ed8, 89df84ccf5394c9bb60d4e4ed872fc2e; no voters: b632d6ef4c104a058a6db208024cab33
I20240430 20:11:19.811161 18380 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 89df84ccf5394c9bb60d4e4ed872fc2e [term 2 FOLLOWER]: Leader election won for term 2
I20240430 20:11:19.811517 18380 raft_consensus.cc:686] T 00000000000000000000000000000000 P 89df84ccf5394c9bb60d4e4ed872fc2e [term 2 LEADER]: Becoming Leader. State: Replica: 89df84ccf5394c9bb60d4e4ed872fc2e, State: Running, Role: LEADER
I20240430 20:11:19.812072 18380 consensus_queue.cc:237] T 00000000000000000000000000000000 P 89df84ccf5394c9bb60d4e4ed872fc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b3f98793354c7aaa5c106e8ec31ed8" member_type: VOTER last_known_addr { host: "127.17.177.62" port: 36441 } } peers { permanent_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e" member_type: VOTER last_known_addr { host: "127.17.177.61" port: 34777 } } peers { permanent_uuid: "b632d6ef4c104a058a6db208024cab33" member_type: VOTER last_known_addr { host: "127.17.177.60" port: 37225 } }
I20240430 20:11:19.813774 18388 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89df84ccf5394c9bb60d4e4ed872fc2e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 89df84ccf5394c9bb60d4e4ed872fc2e. Latest consensus state: current_term: 2 leader_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b3f98793354c7aaa5c106e8ec31ed8" member_type: VOTER last_known_addr { host: "127.17.177.62" port: 36441 } } peers { permanent_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e" member_type: VOTER last_known_addr { host: "127.17.177.61" port: 34777 } } peers { permanent_uuid: "b632d6ef4c104a058a6db208024cab33" member_type: VOTER last_known_addr { host: "127.17.177.60" port: 37225 } } }
I20240430 20:11:19.814085 18388 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89df84ccf5394c9bb60d4e4ed872fc2e [sys.catalog]: This master's current role is: LEADER
I20240430 20:11:19.814841 18402 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240430 20:11:19.816334 18166 raft_consensus.cc:1226] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8 [term 2 FOLLOWER]: Rejecting Update request from peer b632d6ef4c104a058a6db208024cab33 for earlier term 1. Current term is 2. Ops: []
I20240430 20:11:19.817046 18370 consensus_queue.cc:1046] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88b3f98793354c7aaa5c106e8ec31ed8" member_type: VOTER last_known_addr { host: "127.17.177.62" port: 36441 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240430 20:11:19.817642 18370 raft_consensus.cc:3032] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33 [term 1 LEADER]: Stepping down as leader of term 1
I20240430 20:11:19.817757 18370 raft_consensus.cc:729] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b632d6ef4c104a058a6db208024cab33, State: Running, Role: LEADER
I20240430 20:11:19.818046 18370 consensus_queue.cc:260] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b3f98793354c7aaa5c106e8ec31ed8" member_type: VOTER last_known_addr { host: "127.17.177.62" port: 36441 } } peers { permanent_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e" member_type: VOTER last_known_addr { host: "127.17.177.61" port: 34777 } } peers { permanent_uuid: "b632d6ef4c104a058a6db208024cab33" member_type: VOTER last_known_addr { host: "127.17.177.60" port: 37225 } }
I20240430 20:11:19.818578 18370 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33 [term 1 FOLLOWER]: Advancing to term 2
I20240430 20:11:19.819283 18402 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240430 20:11:19.820601 18402 catalog_manager.cc:1259] Loaded cluster ID: 889e0faede1f41d1ab74633d6826c0d4
I20240430 20:11:19.820762 18402 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240430 20:11:19.820850 18379 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.177.20:42079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.177.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.177.1
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.17.177.62:36441,127.17.177.61:34777,127.17.177.60:37225
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.177.1
--log_dir=/tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type FASTDEBUG
built by None at 30 Apr 2024 19:43:22 UTC on efd1337a2aa3
build id 63
ASAN enabled
I20240430 20:11:19.822404 18379 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240430 20:11:19.822712 18402 catalog_manager.cc:1504] Loading token signing keys...
I20240430 20:11:19.823917 18402 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 89df84ccf5394c9bb60d4e4ed872fc2e: Loaded TSK: 0
I20240430 20:11:19.824461 18379 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240430 20:11:19.825117 18402 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20240430 20:11:19.837647 18405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 20:11:19.838236 18406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 20:11:19.838800 18408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240430 20:11:19.839759 18379 server_base.cc:1034] running on GCE node
I20240430 20:11:19.840410 18379 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240430 20:11:19.841504 18379 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240430 20:11:19.842693 18379 hybrid_clock.cc:648] HybridClock initialized: now 1714507879842629 us; error 40 us; skew 500 ppm
I20240430 20:11:19.843016 18379 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240430 20:11:19.850008 18379 webserver.cc:460] Webserver started at http://127.17.177.1:33921/ using document root <none> and password file <none>
I20240430 20:11:19.850888 18379 fs_manager.cc:359] Metadata directory not provided
I20240430 20:11:19.851054 18379 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240430 20:11:19.851370 18379 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240430 20:11:19.854060 18379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/data/instance:
uuid: "2efb542aa55e439b99ccb98d1c5c884a"
format_stamp: "Formatted at 2024-04-30 20:11:19 on dist-test-slave-54l7"
I20240430 20:11:19.854862 18379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/instance:
uuid: "2efb542aa55e439b99ccb98d1c5c884a"
format_stamp: "Formatted at 2024-04-30 20:11:19 on dist-test-slave-54l7"
I20240430 20:11:19.860855 18379 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20240430 20:11:19.864684 18414 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240430 20:11:19.866147 18379 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240430 20:11:19.866308 18379 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal
uuid: "2efb542aa55e439b99ccb98d1c5c884a"
format_stamp: "Formatted at 2024-04-30 20:11:19 on dist-test-slave-54l7"
I20240430 20:11:19.866494 18379 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240430 20:11:19.893003 18379 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240430 20:11:19.893872 18379 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240430 20:11:19.894189 18379 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240430 20:11:19.895782 18379 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240430 20:11:19.897403 18379 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240430 20:11:19.897476 18379 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240430 20:11:19.897578 18379 ts_tablet_manager.cc:590] Registered 0 tablets
I20240430 20:11:19.897636 18379 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240430 20:11:19.941228 18379 rpc_server.cc:307] RPC server started. Bound to: 127.17.177.1:41217
I20240430 20:11:19.941228 18527 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.177.1:41217 every 8 connection(s)
I20240430 20:11:19.942699 18379 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/data/info.pb
I20240430 20:11:19.943161 18116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNG3HqZ/build/asan/bin/kudu as pid 18379
I20240430 20:11:19.943399 18116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/instance
I20240430 20:11:19.952817 18116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNG3HqZ/build/asan/bin/kudu
/tmp/dist-test-taskNG3HqZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.177.2:0
--local_ip_for_outbound_sockets=127.17.177.2
--webserver_interface=127.17.177.2
--webserver_port=0
--tserver_master_addrs=127.17.177.62:36441,127.17.177.61:34777,127.17.177.60:37225
--builtin_ntp_servers=127.17.177.20:42079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20240430 20:11:20.001394 18528 heartbeater.cc:347] Connected to a master server at 127.17.177.62:36441
I20240430 20:11:20.001761 18528 heartbeater.cc:464] Registering TS with master...
I20240430 20:11:20.003091 18528 heartbeater.cc:511] Master 127.17.177.62:36441 requested a full tablet report, sending...
I20240430 20:11:20.006049 18155 ts_manager.cc:194] Registered new tserver with Master: 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:20.011142 18529 heartbeater.cc:347] Connected to a master server at 127.17.177.60:37225
I20240430 20:11:20.011329 18529 heartbeater.cc:464] Registering TS with master...
I20240430 20:11:20.011899 18529 heartbeater.cc:511] Master 127.17.177.60:37225 requested a full tablet report, sending...
I20240430 20:11:20.012208 18531 heartbeater.cc:347] Connected to a master server at 127.17.177.61:34777
I20240430 20:11:20.012362 18531 heartbeater.cc:464] Registering TS with master...
I20240430 20:11:20.012842 18531 heartbeater.cc:511] Master 127.17.177.61:34777 requested a full tablet report, sending...
I20240430 20:11:20.013883 18298 ts_manager.cc:194] Registered new tserver with Master: 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:20.016541 18227 ts_manager.cc:194] Registered new tserver with Master: 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:20.018231 18227 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.177.1:43429
I20240430 20:11:20.054466 18359 catalog_manager.cc:1259] Loaded cluster ID: 889e0faede1f41d1ab74633d6826c0d4
I20240430 20:11:20.054570 18359 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: loading cluster ID for follower catalog manager: success
I20240430 20:11:20.057618 18359 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: acquiring CA information for follower catalog manager: success
I20240430 20:11:20.059358 18359 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240430 20:11:20.066571 18367 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240430 20:11:20.193290 18166 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8 [term 2 FOLLOWER]: Refusing update from remote peer 89df84ccf5394c9bb60d4e4ed872fc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240430 20:11:20.194125 18388 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 89df84ccf5394c9bb60d4e4ed872fc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3f98793354c7aaa5c106e8ec31ed8" member_type: VOTER last_known_addr { host: "127.17.177.62" port: 36441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240430 20:11:20.195657 18398 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 89df84ccf5394c9bb60d4e4ed872fc2e. Latest consensus state: current_term: 2 leader_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b3f98793354c7aaa5c106e8ec31ed8" member_type: VOTER last_known_addr { host: "127.17.177.62" port: 36441 } } peers { permanent_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e" member_type: VOTER last_known_addr { host: "127.17.177.61" port: 34777 } } peers { permanent_uuid: "b632d6ef4c104a058a6db208024cab33" member_type: VOTER last_known_addr { host: "127.17.177.60" port: 37225 } } }
I20240430 20:11:20.195972 18398 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8 [sys.catalog]: This master's current role is: FOLLOWER
I20240430 20:11:20.197546 18380 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89df84ccf5394c9bb60d4e4ed872fc2e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b3f98793354c7aaa5c106e8ec31ed8" member_type: VOTER last_known_addr { host: "127.17.177.62" port: 36441 } } peers { permanent_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e" member_type: VOTER last_known_addr { host: "127.17.177.61" port: 34777 } } peers { permanent_uuid: "b632d6ef4c104a058a6db208024cab33" member_type: VOTER last_known_addr { host: "127.17.177.60" port: 37225 } } }
I20240430 20:11:20.197839 18380 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89df84ccf5394c9bb60d4e4ed872fc2e [sys.catalog]: This master's current role is: LEADER
I20240430 20:11:20.197825 18398 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b3f98793354c7aaa5c106e8ec31ed8" member_type: VOTER last_known_addr { host: "127.17.177.62" port: 36441 } } peers { permanent_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e" member_type: VOTER last_known_addr { host: "127.17.177.61" port: 34777 } } peers { permanent_uuid: "b632d6ef4c104a058a6db208024cab33" member_type: VOTER last_known_addr { host: "127.17.177.60" port: 37225 } } }
I20240430 20:11:20.198069 18398 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8 [sys.catalog]: This master's current role is: FOLLOWER
I20240430 20:11:20.198937 18308 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33 [term 2 FOLLOWER]: Refusing update from remote peer 89df84ccf5394c9bb60d4e4ed872fc2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240430 20:11:20.199422 18370 raft_consensus.cc:482] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:20.199609 18370 raft_consensus.cc:504] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b3f98793354c7aaa5c106e8ec31ed8" member_type: VOTER last_known_addr { host: "127.17.177.62" port: 36441 } } peers { permanent_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e" member_type: VOTER last_known_addr { host: "127.17.177.61" port: 34777 } } peers { permanent_uuid: "b632d6ef4c104a058a6db208024cab33" member_type: VOTER last_known_addr { host: "127.17.177.60" port: 37225 } }
I20240430 20:11:20.200289 18370 leader_election.cc:288] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 88b3f98793354c7aaa5c106e8ec31ed8 (127.17.177.62:36441), 89df84ccf5394c9bb60d4e4ed872fc2e (127.17.177.61:34777)
I20240430 20:11:20.200749 18166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b632d6ef4c104a058a6db208024cab33" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "88b3f98793354c7aaa5c106e8ec31ed8" is_pre_election: true
I20240430 20:11:20.200881 18380 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 89df84ccf5394c9bb60d4e4ed872fc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b632d6ef4c104a058a6db208024cab33" member_type: VOTER last_known_addr { host: "127.17.177.60" port: 37225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240430 20:11:20.201133 18235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b632d6ef4c104a058a6db208024cab33" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e" is_pre_election: true
I20240430 20:11:20.201813 18283 leader_election.cc:302] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b632d6ef4c104a058a6db208024cab33; no voters: 88b3f98793354c7aaa5c106e8ec31ed8, 89df84ccf5394c9bb60d4e4ed872fc2e
I20240430 20:11:20.202140 18370 raft_consensus.cc:2726] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240430 20:11:20.203421 18370 sys_catalog.cc:455] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 89df84ccf5394c9bb60d4e4ed872fc2e. Latest consensus state: current_term: 2 leader_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b3f98793354c7aaa5c106e8ec31ed8" member_type: VOTER last_known_addr { host: "127.17.177.62" port: 36441 } } peers { permanent_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e" member_type: VOTER last_known_addr { host: "127.17.177.61" port: 34777 } } peers { permanent_uuid: "b632d6ef4c104a058a6db208024cab33" member_type: VOTER last_known_addr { host: "127.17.177.60" port: 37225 } } }
I20240430 20:11:20.203712 18370 sys_catalog.cc:458] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33 [sys.catalog]: This master's current role is: FOLLOWER
I20240430 20:11:20.203646 18403 sys_catalog.cc:455] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b3f98793354c7aaa5c106e8ec31ed8" member_type: VOTER last_known_addr { host: "127.17.177.62" port: 36441 } } peers { permanent_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e" member_type: VOTER last_known_addr { host: "127.17.177.61" port: 34777 } } peers { permanent_uuid: "b632d6ef4c104a058a6db208024cab33" member_type: VOTER last_known_addr { host: "127.17.177.60" port: 37225 } } }
I20240430 20:11:20.204453 18403 sys_catalog.cc:458] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33 [sys.catalog]: This master's current role is: FOLLOWER
W20240430 20:11:20.346134 18535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240430 20:11:20.346472 18535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240430 20:11:20.346724 18535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240430 20:11:20.355484 18535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240430 20:11:20.355715 18535 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.177.2
I20240430 20:11:20.365336 18535 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.177.20:42079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.177.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.177.2
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.17.177.62:36441,127.17.177.61:34777,127.17.177.60:37225
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.177.2
--log_dir=/tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type FASTDEBUG
built by None at 30 Apr 2024 19:43:22 UTC on efd1337a2aa3
build id 63
ASAN enabled
I20240430 20:11:20.366850 18535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240430 20:11:20.368666 18535 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240430 20:11:20.382256 18545 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 20:11:20.382738 18548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 20:11:20.383306 18546 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240430 20:11:20.383980 18535 server_base.cc:1034] running on GCE node
I20240430 20:11:20.384661 18535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240430 20:11:20.385723 18535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240430 20:11:20.386963 18535 hybrid_clock.cc:648] HybridClock initialized: now 1714507880386906 us; error 47 us; skew 500 ppm
I20240430 20:11:20.387408 18535 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240430 20:11:20.390391 18535 webserver.cc:460] Webserver started at http://127.17.177.2:40755/ using document root <none> and password file <none>
I20240430 20:11:20.391206 18535 fs_manager.cc:359] Metadata directory not provided
I20240430 20:11:20.391337 18535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240430 20:11:20.391646 18535 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240430 20:11:20.394368 18535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/data/instance:
uuid: "5e1c6397f09747f091534faf93542710"
format_stamp: "Formatted at 2024-04-30 20:11:20 on dist-test-slave-54l7"
I20240430 20:11:20.395298 18535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/instance:
uuid: "5e1c6397f09747f091534faf93542710"
format_stamp: "Formatted at 2024-04-30 20:11:20 on dist-test-slave-54l7"
I20240430 20:11:20.400784 18535 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20240430 20:11:20.404754 18554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240430 20:11:20.406311 18535 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240430 20:11:20.406493 18535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal
uuid: "5e1c6397f09747f091534faf93542710"
format_stamp: "Formatted at 2024-04-30 20:11:20 on dist-test-slave-54l7"
I20240430 20:11:20.406735 18535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240430 20:11:20.424165 18535 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240430 20:11:20.425087 18535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240430 20:11:20.425413 18535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240430 20:11:20.426512 18535 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240430 20:11:20.428265 18535 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240430 20:11:20.428350 18535 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240430 20:11:20.428488 18535 ts_tablet_manager.cc:590] Registered 0 tablets
I20240430 20:11:20.428542 18535 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240430 20:11:20.465198 18535 rpc_server.cc:307] RPC server started. Bound to: 127.17.177.2:46027
I20240430 20:11:20.465232 18667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.177.2:46027 every 8 connection(s)
I20240430 20:11:20.466559 18535 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/data/info.pb
I20240430 20:11:20.471549 18116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNG3HqZ/build/asan/bin/kudu as pid 18535
I20240430 20:11:20.471823 18116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/instance
I20240430 20:11:20.480420 18116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNG3HqZ/build/asan/bin/kudu
/tmp/dist-test-taskNG3HqZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.177.3:0
--local_ip_for_outbound_sockets=127.17.177.3
--webserver_interface=127.17.177.3
--webserver_port=0
--tserver_master_addrs=127.17.177.62:36441,127.17.177.61:34777,127.17.177.60:37225
--builtin_ntp_servers=127.17.177.20:42079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20240430 20:11:20.491248 18668 heartbeater.cc:347] Connected to a master server at 127.17.177.62:36441
I20240430 20:11:20.491695 18668 heartbeater.cc:464] Registering TS with master...
I20240430 20:11:20.493098 18668 heartbeater.cc:511] Master 127.17.177.62:36441 requested a full tablet report, sending...
I20240430 20:11:20.495245 18670 heartbeater.cc:347] Connected to a master server at 127.17.177.61:34777
I20240430 20:11:20.495375 18670 heartbeater.cc:464] Registering TS with master...
I20240430 20:11:20.495597 18155 ts_manager.cc:194] Registered new tserver with Master: 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:20.495875 18670 heartbeater.cc:511] Master 127.17.177.61:34777 requested a full tablet report, sending...
I20240430 20:11:20.497043 18227 ts_manager.cc:194] Registered new tserver with Master: 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:20.497805 18227 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.177.2:54539
I20240430 20:11:20.501291 18669 heartbeater.cc:347] Connected to a master server at 127.17.177.60:37225
I20240430 20:11:20.501441 18669 heartbeater.cc:464] Registering TS with master...
I20240430 20:11:20.501762 18669 heartbeater.cc:511] Master 127.17.177.60:37225 requested a full tablet report, sending...
I20240430 20:11:20.503108 18298 ts_manager.cc:194] Registered new tserver with Master: 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:20.520648 18531 heartbeater.cc:503] Master 127.17.177.61:34777 was elected leader, sending a full tablet report...
I20240430 20:11:20.697932 18398 raft_consensus.cc:482] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 89df84ccf5394c9bb60d4e4ed872fc2e)
I20240430 20:11:20.698199 18398 raft_consensus.cc:504] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b3f98793354c7aaa5c106e8ec31ed8" member_type: VOTER last_known_addr { host: "127.17.177.62" port: 36441 } } peers { permanent_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e" member_type: VOTER last_known_addr { host: "127.17.177.61" port: 34777 } } peers { permanent_uuid: "b632d6ef4c104a058a6db208024cab33" member_type: VOTER last_known_addr { host: "127.17.177.60" port: 37225 } }
I20240430 20:11:20.699123 18398 leader_election.cc:288] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 89df84ccf5394c9bb60d4e4ed872fc2e (127.17.177.61:34777), b632d6ef4c104a058a6db208024cab33 (127.17.177.60:37225)
I20240430 20:11:20.699779 18235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88b3f98793354c7aaa5c106e8ec31ed8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e" is_pre_election: true
I20240430 20:11:20.699772 18308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88b3f98793354c7aaa5c106e8ec31ed8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b632d6ef4c104a058a6db208024cab33" is_pre_election: true
I20240430 20:11:20.700495 18140 leader_election.cc:302] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88b3f98793354c7aaa5c106e8ec31ed8; no voters: 89df84ccf5394c9bb60d4e4ed872fc2e, b632d6ef4c104a058a6db208024cab33
I20240430 20:11:20.700819 18398 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240430 20:11:20.703608 18403 raft_consensus.cc:482] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 89df84ccf5394c9bb60d4e4ed872fc2e)
I20240430 20:11:20.703794 18403 raft_consensus.cc:504] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b3f98793354c7aaa5c106e8ec31ed8" member_type: VOTER last_known_addr { host: "127.17.177.62" port: 36441 } } peers { permanent_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e" member_type: VOTER last_known_addr { host: "127.17.177.61" port: 34777 } } peers { permanent_uuid: "b632d6ef4c104a058a6db208024cab33" member_type: VOTER last_known_addr { host: "127.17.177.60" port: 37225 } }
I20240430 20:11:20.704551 18403 leader_election.cc:288] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 88b3f98793354c7aaa5c106e8ec31ed8 (127.17.177.62:36441), 89df84ccf5394c9bb60d4e4ed872fc2e (127.17.177.61:34777)
I20240430 20:11:20.705137 18235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b632d6ef4c104a058a6db208024cab33" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e" is_pre_election: true
I20240430 20:11:20.705325 18166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b632d6ef4c104a058a6db208024cab33" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "88b3f98793354c7aaa5c106e8ec31ed8" is_pre_election: true
I20240430 20:11:20.705546 18166 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b632d6ef4c104a058a6db208024cab33 in term 2.
I20240430 20:11:20.705922 18282 leader_election.cc:302] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88b3f98793354c7aaa5c106e8ec31ed8, b632d6ef4c104a058a6db208024cab33; no voters: 89df84ccf5394c9bb60d4e4ed872fc2e
I20240430 20:11:20.706223 18403 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240430 20:11:20.706331 18403 raft_consensus.cc:482] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 89df84ccf5394c9bb60d4e4ed872fc2e)
I20240430 20:11:20.706419 18403 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33 [term 2 FOLLOWER]: Advancing to term 3
I20240430 20:11:20.708303 18403 raft_consensus.cc:504] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b3f98793354c7aaa5c106e8ec31ed8" member_type: VOTER last_known_addr { host: "127.17.177.62" port: 36441 } } peers { permanent_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e" member_type: VOTER last_known_addr { host: "127.17.177.61" port: 34777 } } peers { permanent_uuid: "b632d6ef4c104a058a6db208024cab33" member_type: VOTER last_known_addr { host: "127.17.177.60" port: 37225 } }
I20240430 20:11:20.708878 18403 leader_election.cc:288] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33 [CANDIDATE]: Term 3 election: Requested vote from peers 88b3f98793354c7aaa5c106e8ec31ed8 (127.17.177.62:36441), 89df84ccf5394c9bb60d4e4ed872fc2e (127.17.177.61:34777)
I20240430 20:11:20.709488 18166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b632d6ef4c104a058a6db208024cab33" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "88b3f98793354c7aaa5c106e8ec31ed8"
I20240430 20:11:20.709754 18166 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8 [term 2 FOLLOWER]: Advancing to term 3
I20240430 20:11:20.712790 18166 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b632d6ef4c104a058a6db208024cab33 in term 3.
I20240430 20:11:20.713196 18235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b632d6ef4c104a058a6db208024cab33" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e"
I20240430 20:11:20.713279 18282 leader_election.cc:302] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b3f98793354c7aaa5c106e8ec31ed8, b632d6ef4c104a058a6db208024cab33; no voters:
I20240430 20:11:20.713586 18403 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33 [term 3 FOLLOWER]: Leader election won for term 3
I20240430 20:11:20.714107 18403 raft_consensus.cc:686] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33 [term 3 LEADER]: Becoming Leader. State: Replica: b632d6ef4c104a058a6db208024cab33, State: Running, Role: LEADER
I20240430 20:11:20.714416 18403 consensus_queue.cc:237] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b3f98793354c7aaa5c106e8ec31ed8" member_type: VOTER last_known_addr { host: "127.17.177.62" port: 36441 } } peers { permanent_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e" member_type: VOTER last_known_addr { host: "127.17.177.61" port: 34777 } } peers { permanent_uuid: "b632d6ef4c104a058a6db208024cab33" member_type: VOTER last_known_addr { host: "127.17.177.60" port: 37225 } }
I20240430 20:11:20.715438 18676 sys_catalog.cc:455] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b632d6ef4c104a058a6db208024cab33. Latest consensus state: current_term: 3 leader_uuid: "b632d6ef4c104a058a6db208024cab33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b3f98793354c7aaa5c106e8ec31ed8" member_type: VOTER last_known_addr { host: "127.17.177.62" port: 36441 } } peers { permanent_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e" member_type: VOTER last_known_addr { host: "127.17.177.61" port: 34777 } } peers { permanent_uuid: "b632d6ef4c104a058a6db208024cab33" member_type: VOTER last_known_addr { host: "127.17.177.60" port: 37225 } } }
I20240430 20:11:20.715616 18676 sys_catalog.cc:458] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33 [sys.catalog]: This master's current role is: LEADER
I20240430 20:11:20.716362 18677 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240430 20:11:20.717895 18677 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240430 20:11:20.719003 18677 catalog_manager.cc:1259] Loaded cluster ID: 889e0faede1f41d1ab74633d6826c0d4
I20240430 20:11:20.719087 18677 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240430 20:11:20.719163 18678 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89df84ccf5394c9bb60d4e4ed872fc2e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b3f98793354c7aaa5c106e8ec31ed8" member_type: VOTER last_known_addr { host: "127.17.177.62" port: 36441 } } peers { permanent_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e" member_type: VOTER last_known_addr { host: "127.17.177.61" port: 34777 } } peers { permanent_uuid: "b632d6ef4c104a058a6db208024cab33" member_type: VOTER last_known_addr { host: "127.17.177.60" port: 37225 } } }
I20240430 20:11:20.719426 18678 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89df84ccf5394c9bb60d4e4ed872fc2e [sys.catalog]: This master's current role is: LEADER
I20240430 20:11:20.719854 18308 raft_consensus.cc:1226] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33 [term 3 LEADER]: Rejecting Update request from peer 89df84ccf5394c9bb60d4e4ed872fc2e for earlier term 2. Current term is 3. Ops: []
I20240430 20:11:20.721083 18677 catalog_manager.cc:1504] Loading token signing keys...
I20240430 20:11:20.722456 18677 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: Loaded TSK: 0
I20240430 20:11:20.723562 18677 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240430 20:11:20.724236 18681 consensus_queue.cc:1046] T 00000000000000000000000000000000 P 89df84ccf5394c9bb60d4e4ed872fc2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b632d6ef4c104a058a6db208024cab33" member_type: VOTER last_known_addr { host: "127.17.177.60" port: 37225 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240430 20:11:20.724838 18681 consensus_queue.cc:1225] T 00000000000000000000000000000000 P 89df84ccf5394c9bb60d4e4ed872fc2e [LEADER]: Peer b632d6ef4c104a058a6db208024cab33 log is divergent from this leader: its last log entry 3.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20240430 20:11:20.725013 18679 raft_consensus.cc:3032] T 00000000000000000000000000000000 P 89df84ccf5394c9bb60d4e4ed872fc2e [term 2 LEADER]: Stepping down as leader of term 2
I20240430 20:11:20.725106 18679 raft_consensus.cc:729] T 00000000000000000000000000000000 P 89df84ccf5394c9bb60d4e4ed872fc2e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 89df84ccf5394c9bb60d4e4ed872fc2e, State: Running, Role: LEADER
I20240430 20:11:20.725479 18679 consensus_queue.cc:260] T 00000000000000000000000000000000 P 89df84ccf5394c9bb60d4e4ed872fc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b3f98793354c7aaa5c106e8ec31ed8" member_type: VOTER last_known_addr { host: "127.17.177.62" port: 36441 } } peers { permanent_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e" member_type: VOTER last_known_addr { host: "127.17.177.61" port: 34777 } } peers { permanent_uuid: "b632d6ef4c104a058a6db208024cab33" member_type: VOTER last_known_addr { host: "127.17.177.60" port: 37225 } }
I20240430 20:11:20.726075 18679 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 89df84ccf5394c9bb60d4e4ed872fc2e [term 2 FOLLOWER]: Advancing to term 3
W20240430 20:11:20.730098 18680 catalog_manager.cc:1409] Failed waiting for node to catch up after master election: Illegal state: Node 89df84ccf5394c9bb60d4e4ed872fc2e not leader. Raft Consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b3f98793354c7aaa5c106e8ec31ed8" member_type: VOTER last_known_addr { host: "127.17.177.62" port: 36441 } } peers { permanent_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e" member_type: VOTER last_known_addr { host: "127.17.177.61" port: 34777 } } peers { permanent_uuid: "b632d6ef4c104a058a6db208024cab33" member_type: VOTER last_known_addr { host: "127.17.177.60" port: 37225 } } }
W20240430 20:11:20.855968 18675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240430 20:11:20.856343 18675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240430 20:11:20.856519 18675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240430 20:11:20.864542 18675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240430 20:11:20.864800 18675 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.177.3
I20240430 20:11:20.874897 18675 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.177.20:42079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.177.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.177.3
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.17.177.62:36441,127.17.177.61:34777,127.17.177.60:37225
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.177.3
--log_dir=/tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type FASTDEBUG
built by None at 30 Apr 2024 19:43:22 UTC on efd1337a2aa3
build id 63
ASAN enabled
I20240430 20:11:20.876395 18675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240430 20:11:20.878249 18675 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240430 20:11:20.891924 18686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 20:11:20.892308 18689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 20:11:20.892832 18687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240430 20:11:20.893569 18675 server_base.cc:1034] running on GCE node
I20240430 20:11:20.894237 18675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240430 20:11:20.895255 18675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240430 20:11:20.896489 18675 hybrid_clock.cc:648] HybridClock initialized: now 1714507880896414 us; error 46 us; skew 500 ppm
I20240430 20:11:20.896777 18675 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240430 20:11:20.899216 18675 webserver.cc:460] Webserver started at http://127.17.177.3:46509/ using document root <none> and password file <none>
I20240430 20:11:20.899971 18675 fs_manager.cc:359] Metadata directory not provided
I20240430 20:11:20.900085 18675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240430 20:11:20.900347 18675 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240430 20:11:20.903182 18675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/data/instance:
uuid: "3c51c33d5616482cafb68e26688820f4"
format_stamp: "Formatted at 2024-04-30 20:11:20 on dist-test-slave-54l7"
I20240430 20:11:20.904044 18675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/instance:
uuid: "3c51c33d5616482cafb68e26688820f4"
format_stamp: "Formatted at 2024-04-30 20:11:20 on dist-test-slave-54l7"
I20240430 20:11:20.910084 18675 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20240430 20:11:20.914041 18695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240430 20:11:20.915563 18675 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240430 20:11:20.915726 18675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal
uuid: "3c51c33d5616482cafb68e26688820f4"
format_stamp: "Formatted at 2024-04-30 20:11:20 on dist-test-slave-54l7"
I20240430 20:11:20.915900 18675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240430 20:11:20.945482 18675 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240430 20:11:20.946511 18675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240430 20:11:20.946951 18675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240430 20:11:20.948084 18675 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240430 20:11:20.949677 18675 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240430 20:11:20.949753 18675 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240430 20:11:20.949846 18675 ts_tablet_manager.cc:590] Registered 0 tablets
I20240430 20:11:20.949898 18675 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240430 20:11:20.986080 18675 rpc_server.cc:307] RPC server started. Bound to: 127.17.177.3:44867
I20240430 20:11:20.986097 18808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.177.3:44867 every 8 connection(s)
I20240430 20:11:20.987807 18675 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/data/info.pb
I20240430 20:11:20.995918 18116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNG3HqZ/build/asan/bin/kudu as pid 18675
I20240430 20:11:20.996146 18116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/instance
I20240430 20:11:21.000370 18670 heartbeater.cc:503] Master 127.17.177.61:34777 was elected leader, sending a full tablet report...
I20240430 20:11:21.020551 18810 heartbeater.cc:347] Connected to a master server at 127.17.177.60:37225
I20240430 20:11:21.021512 18810 heartbeater.cc:464] Registering TS with master...
I20240430 20:11:21.025357 18810 heartbeater.cc:511] Master 127.17.177.60:37225 requested a full tablet report, sending...
I20240430 20:11:21.027122 18298 ts_manager.cc:194] Registered new tserver with Master: 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:21.027179 18809 heartbeater.cc:347] Connected to a master server at 127.17.177.62:36441
I20240430 20:11:21.027303 18809 heartbeater.cc:464] Registering TS with master...
I20240430 20:11:21.027722 18809 heartbeater.cc:511] Master 127.17.177.62:36441 requested a full tablet report, sending...
I20240430 20:11:21.028594 18298 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.177.3:51039
I20240430 20:11:21.028687 18155 ts_manager.cc:194] Registered new tserver with Master: 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:21.029302 18811 heartbeater.cc:347] Connected to a master server at 127.17.177.61:34777
I20240430 20:11:21.029444 18811 heartbeater.cc:464] Registering TS with master...
I20240430 20:11:21.029919 18811 heartbeater.cc:511] Master 127.17.177.61:34777 requested a full tablet report, sending...
I20240430 20:11:21.030956 18226 ts_manager.cc:194] Registered new tserver with Master: 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:21.040913 18116 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240430 20:11:21.071786 18298 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51068:
name: "default.Table_bb7889b3f37b4b3b9a07300f92edbcea"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240430 20:11:21.073474 18297 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51068:
name: "default.Table_ccf13d54d878437a898e7cac83ae5dba"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:21.073810 18298 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bb7889b3f37b4b3b9a07300f92edbcea in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240430 20:11:21.074079 18297 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ccf13d54d878437a898e7cac83ae5dba in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:21.075100 18296 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51068:
name: "default.Table_3a260044be774c57b36753acff663173"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240430 20:11:21.075224 18295 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51068:
name: "default.Table_b8cdd08adc0044d5a17cb39a3b20576d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:21.075814 18296 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3a260044be774c57b36753acff663173 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:21.076009 18293 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51068:
name: "default.Table_977297e4a31548ea91da9f99813678f5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:21.076978 18293 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_977297e4a31548ea91da9f99813678f5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:21.077282 18291 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51068:
name: "default.Table_9dd8b028e2574db496d4437fc3c891d2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240430 20:11:21.077538 18294 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51068:
name: "default.Table_2e245f45131c4abbb4742f14ca0231ea"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:21.077948 18291 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9dd8b028e2574db496d4437fc3c891d2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240430 20:11:21.078141 18294 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2e245f45131c4abbb4742f14ca0231ea in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:21.079203 18292 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51068:
name: "default.Table_6a3bec20b3f34ffd9c83b3640d02060a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:21.079766 18292 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6a3bec20b3f34ffd9c83b3640d02060a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240430 20:11:21.081214 18295 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b8cdd08adc0044d5a17cb39a3b20576d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:21.088573 18290 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51068:
name: "default.Table_6063ac99fc404967b6f23f9567c97ee4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240430 20:11:21.089123 18235 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 89df84ccf5394c9bb60d4e4ed872fc2e [term 3 FOLLOWER]: Refusing update from remote peer b632d6ef4c104a058a6db208024cab33: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 7. (index mismatch)
W20240430 20:11:21.089321 18290 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6063ac99fc404967b6f23f9567c97ee4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:21.090884 18676 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e" member_type: VOTER last_known_addr { host: "127.17.177.61" port: 34777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240430 20:11:21.090555 18289 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51068:
name: "default.Table_a01951ca625b47d98243806e31973cce"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:21.091558 18289 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a01951ca625b47d98243806e31973cce in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:21.094033 18166 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8 [term 3 FOLLOWER]: Refusing update from remote peer b632d6ef4c104a058a6db208024cab33: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 7. (index mismatch)
I20240430 20:11:21.096706 18676 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b3f98793354c7aaa5c106e8ec31ed8" member_type: VOTER last_known_addr { host: "127.17.177.62" port: 36441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240430 20:11:21.101392 18679 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89df84ccf5394c9bb60d4e4ed872fc2e [sys.catalog]: SysCatalogTable state changed. Reason: New leader b632d6ef4c104a058a6db208024cab33. Latest consensus state: current_term: 3 leader_uuid: "b632d6ef4c104a058a6db208024cab33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b3f98793354c7aaa5c106e8ec31ed8" member_type: VOTER last_known_addr { host: "127.17.177.62" port: 36441 } } peers { permanent_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e" member_type: VOTER last_known_addr { host: "127.17.177.61" port: 34777 } } peers { permanent_uuid: "b632d6ef4c104a058a6db208024cab33" member_type: VOTER last_known_addr { host: "127.17.177.60" port: 37225 } } }
I20240430 20:11:21.101760 18679 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89df84ccf5394c9bb60d4e4ed872fc2e [sys.catalog]: This master's current role is: FOLLOWER
I20240430 20:11:21.110206 18403 sys_catalog.cc:455] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "b632d6ef4c104a058a6db208024cab33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b3f98793354c7aaa5c106e8ec31ed8" member_type: VOTER last_known_addr { host: "127.17.177.62" port: 36441 } } peers { permanent_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e" member_type: VOTER last_known_addr { host: "127.17.177.61" port: 34777 } } peers { permanent_uuid: "b632d6ef4c104a058a6db208024cab33" member_type: VOTER last_known_addr { host: "127.17.177.60" port: 37225 } } }
I20240430 20:11:21.110632 18403 sys_catalog.cc:458] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33 [sys.catalog]: This master's current role is: LEADER
I20240430 20:11:21.118294 18298 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51068:
table { table_name: "default.Table_bb7889b3f37b4b3b9a07300f92edbcea" } new_table_name: "default.Table_88d5674c038a499682559e6c50aa9a70" modify_external_catalogs: true
I20240430 20:11:21.121431 18398 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b632d6ef4c104a058a6db208024cab33. Latest consensus state: current_term: 3 leader_uuid: "b632d6ef4c104a058a6db208024cab33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b3f98793354c7aaa5c106e8ec31ed8" member_type: VOTER last_known_addr { host: "127.17.177.62" port: 36441 } } peers { permanent_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e" member_type: VOTER last_known_addr { host: "127.17.177.61" port: 34777 } } peers { permanent_uuid: "b632d6ef4c104a058a6db208024cab33" member_type: VOTER last_known_addr { host: "127.17.177.60" port: 37225 } } }
I20240430 20:11:21.121805 18398 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8 [sys.catalog]: This master's current role is: FOLLOWER
I20240430 20:11:21.131208 18679 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89df84ccf5394c9bb60d4e4ed872fc2e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "b632d6ef4c104a058a6db208024cab33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b3f98793354c7aaa5c106e8ec31ed8" member_type: VOTER last_known_addr { host: "127.17.177.62" port: 36441 } } peers { permanent_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e" member_type: VOTER last_known_addr { host: "127.17.177.61" port: 34777 } } peers { permanent_uuid: "b632d6ef4c104a058a6db208024cab33" member_type: VOTER last_known_addr { host: "127.17.177.60" port: 37225 } } }
I20240430 20:11:21.131570 18679 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89df84ccf5394c9bb60d4e4ed872fc2e [sys.catalog]: This master's current role is: FOLLOWER
I20240430 20:11:21.158443 18676 sys_catalog.cc:455] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "b632d6ef4c104a058a6db208024cab33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b3f98793354c7aaa5c106e8ec31ed8" member_type: VOTER last_known_addr { host: "127.17.177.62" port: 36441 } } peers { permanent_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e" member_type: VOTER last_known_addr { host: "127.17.177.61" port: 34777 } } peers { permanent_uuid: "b632d6ef4c104a058a6db208024cab33" member_type: VOTER last_known_addr { host: "127.17.177.60" port: 37225 } } }
I20240430 20:11:21.158612 18291 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51068:
table { table_name: "default.Table_2e245f45131c4abbb4742f14ca0231ea" } new_table_name: "default.Table_5dd79a4e3c8545ae8a3105d9c29f27cc" modify_external_catalogs: true
I20240430 20:11:21.158861 18676 sys_catalog.cc:458] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33 [sys.catalog]: This master's current role is: LEADER
I20240430 20:11:21.161481 18294 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51068:
table { table_name: "default.Table_977297e4a31548ea91da9f99813678f5" } new_table_name: "default.Table_3c81228276ba40fe8b78578cdd253c0d" modify_external_catalogs: true
I20240430 20:11:21.163422 18296 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51068:
table { table_name: "default.Table_ccf13d54d878437a898e7cac83ae5dba" } new_table_name: "default.Table_2829200b322c4201b8351098eac2ddae" modify_external_catalogs: true
I20240430 20:11:21.163581 18297 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51068:
table { table_name: "default.Table_3a260044be774c57b36753acff663173" } new_table_name: "default.Table_4312aac764824e93904ef77f93fb3578" modify_external_catalogs: true
I20240430 20:11:21.165413 18293 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51068:
table { table_name: "default.Table_9dd8b028e2574db496d4437fc3c891d2" } modify_external_catalogs: true
I20240430 20:11:21.165716 18293 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51068:
table { table_name: "default.Table_9dd8b028e2574db496d4437fc3c891d2" } modify_external_catalogs: true
I20240430 20:11:21.166285 18289 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51068:
table { table_name: "default.Table_6a3bec20b3f34ffd9c83b3640d02060a" } modify_external_catalogs: true
I20240430 20:11:21.166561 18289 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51068:
table { table_name: "default.Table_6a3bec20b3f34ffd9c83b3640d02060a" } modify_external_catalogs: true
I20240430 20:11:21.166885 18290 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51068:
table { table_name: "default.Table_b8cdd08adc0044d5a17cb39a3b20576d" } modify_external_catalogs: true
I20240430 20:11:21.167125 18290 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51068:
table { table_name: "default.Table_b8cdd08adc0044d5a17cb39a3b20576d" } modify_external_catalogs: true
I20240430 20:11:21.169675 18295 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51068:
table { table_name: "default.Table_a01951ca625b47d98243806e31973cce" } modify_external_catalogs: true
I20240430 20:11:21.169945 18295 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51068:
table { table_name: "default.Table_a01951ca625b47d98243806e31973cce" } modify_external_catalogs: true
I20240430 20:11:21.176205 18292 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51068:
table { table_name: "default.Table_6063ac99fc404967b6f23f9567c97ee4" } modify_external_catalogs: true
I20240430 20:11:21.176497 18292 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51068:
table { table_name: "default.Table_6063ac99fc404967b6f23f9567c97ee4" } modify_external_catalogs: true
W20240430 20:11:21.183377 18298 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e49de8fd5081451693734b6d91522298 consensus state has no leader
W20240430 20:11:21.199282 18297 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0810eb04f1dd4d4bae41845453a623ab has no consensus state
W20240430 20:11:21.199342 18296 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2b6be4a072094c02b14bddfea18c8600 has no consensus state
W20240430 20:11:21.200052 18294 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ef7a73d6307c4b099e967e7e70bf4f06 has no consensus state
W20240430 20:11:21.200558 18291 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 05b5a0b272ca41509c3291336079ef81 has no consensus state
I20240430 20:11:21.215088 18289 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet 0b57d38db9174d91916db0547714a1f9
I20240430 20:11:21.215401 18290 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet c217f9ca9412402d99f1c6908f5eb2d8
I20240430 20:11:21.216332 18295 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet df91b392b5574858846ad486c3cf7ad6
I20240430 20:11:21.216794 18293 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet 79ae75740c644edb96b0616d30f7f2bd
I20240430 20:11:21.226495 18292 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet ea6368482d6b4370b3e07ebd90676e54
I20240430 20:11:21.243788 18398 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "b632d6ef4c104a058a6db208024cab33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b3f98793354c7aaa5c106e8ec31ed8" member_type: VOTER last_known_addr { host: "127.17.177.62" port: 36441 } } peers { permanent_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e" member_type: VOTER last_known_addr { host: "127.17.177.61" port: 34777 } } peers { permanent_uuid: "b632d6ef4c104a058a6db208024cab33" member_type: VOTER last_known_addr { host: "127.17.177.60" port: 37225 } } }
I20240430 20:11:21.244189 18398 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8 [sys.catalog]: This master's current role is: FOLLOWER
W20240430 20:11:21.247552 18283 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet e49de8fd5081451693734b6d91522298 (table default.Table_88d5674c038a499682559e6c50aa9a70 [id=a5d580cffa2447d78fc9f2c8e31264f1], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet e49de8fd5081451693734b6d91522298 consensus state has no leader
I20240430 20:11:21.256199 18603 tablet_service.cc:1467] Processing CreateTablet for tablet e49de8fd5081451693734b6d91522298 (DEFAULT_TABLE table=default.Table_88d5674c038a499682559e6c50aa9a70 [id=a5d580cffa2447d78fc9f2c8e31264f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:21.257344 18602 tablet_service.cc:1467] Processing CreateTablet for tablet 05b5a0b272ca41509c3291336079ef81 (DEFAULT_TABLE table=default.Table_5dd79a4e3c8545ae8a3105d9c29f27cc [id=50ddbed249404afcbd875ca1f9a04101]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:21.259956 18602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05b5a0b272ca41509c3291336079ef81. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:21.257826 18601 tablet_service.cc:1467] Processing CreateTablet for tablet 2b6be4a072094c02b14bddfea18c8600 (DEFAULT_TABLE table=default.Table_2829200b322c4201b8351098eac2ddae [id=a6a27dcbd0e648a8bf29a8cd5b3a2a11]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:21.263746 18601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b6be4a072094c02b14bddfea18c8600. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:21.258283 18600 tablet_service.cc:1467] Processing CreateTablet for tablet 0810eb04f1dd4d4bae41845453a623ab (DEFAULT_TABLE table=default.Table_4312aac764824e93904ef77f93fb3578 [id=e63582736e544bffba71f7bd54bc0cbc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:21.264848 18600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0810eb04f1dd4d4bae41845453a623ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:21.264703 18742 tablet_service.cc:1467] Processing CreateTablet for tablet 2b6be4a072094c02b14bddfea18c8600 (DEFAULT_TABLE table=default.Table_2829200b322c4201b8351098eac2ddae [id=a6a27dcbd0e648a8bf29a8cd5b3a2a11]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:21.264554 18741 tablet_service.cc:1467] Processing CreateTablet for tablet 0810eb04f1dd4d4bae41845453a623ab (DEFAULT_TABLE table=default.Table_4312aac764824e93904ef77f93fb3578 [id=e63582736e544bffba71f7bd54bc0cbc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:21.258720 18599 tablet_service.cc:1467] Processing CreateTablet for tablet ef7a73d6307c4b099e967e7e70bf4f06 (DEFAULT_TABLE table=default.Table_3c81228276ba40fe8b78578cdd253c0d [id=820d7f76444f42178fdb54864a998903]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:21.264636 18744 tablet_service.cc:1467] Processing CreateTablet for tablet e49de8fd5081451693734b6d91522298 (DEFAULT_TABLE table=default.Table_bb7889b3f37b4b3b9a07300f92edbcea [id=a5d580cffa2447d78fc9f2c8e31264f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:21.265708 18599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef7a73d6307c4b099e967e7e70bf4f06. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:21.266012 18740 tablet_service.cc:1467] Processing CreateTablet for tablet ef7a73d6307c4b099e967e7e70bf4f06 (DEFAULT_TABLE table=default.Table_3c81228276ba40fe8b78578cdd253c0d [id=820d7f76444f42178fdb54864a998903]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:21.266350 18741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0810eb04f1dd4d4bae41845453a623ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:21.264377 18743 tablet_service.cc:1467] Processing CreateTablet for tablet 05b5a0b272ca41509c3291336079ef81 (DEFAULT_TABLE table=default.Table_5dd79a4e3c8545ae8a3105d9c29f27cc [id=50ddbed249404afcbd875ca1f9a04101]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:21.266906 18603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e49de8fd5081451693734b6d91522298. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:21.267107 18744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e49de8fd5081451693734b6d91522298. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:21.268426 18742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b6be4a072094c02b14bddfea18c8600. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:21.268803 18740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef7a73d6307c4b099e967e7e70bf4f06. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:21.269413 18743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05b5a0b272ca41509c3291336079ef81. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:21.269953 18116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNG3HqZ/build/asan/bin/kudu pid 18264
I20240430 20:11:21.272091 18328 generic_service.cc:196] Checking for leaks (request via RPC)
I20240430 20:11:21.275576 18462 tablet_service.cc:1467] Processing CreateTablet for tablet 05b5a0b272ca41509c3291336079ef81 (DEFAULT_TABLE table=default.Table_5dd79a4e3c8545ae8a3105d9c29f27cc [id=50ddbed249404afcbd875ca1f9a04101]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:21.275604 18463 tablet_service.cc:1467] Processing CreateTablet for tablet e49de8fd5081451693734b6d91522298 (DEFAULT_TABLE table=default.Table_88d5674c038a499682559e6c50aa9a70 [id=a5d580cffa2447d78fc9f2c8e31264f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:21.277233 18461 tablet_service.cc:1467] Processing CreateTablet for tablet 2b6be4a072094c02b14bddfea18c8600 (DEFAULT_TABLE table=default.Table_2829200b322c4201b8351098eac2ddae [id=a6a27dcbd0e648a8bf29a8cd5b3a2a11]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:21.277365 18459 tablet_service.cc:1467] Processing CreateTablet for tablet 0810eb04f1dd4d4bae41845453a623ab (DEFAULT_TABLE table=default.Table_4312aac764824e93904ef77f93fb3578 [id=e63582736e544bffba71f7bd54bc0cbc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:21.278070 18461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b6be4a072094c02b14bddfea18c8600. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:21.278523 18459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0810eb04f1dd4d4bae41845453a623ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:21.280978 18458 tablet_service.cc:1467] Processing CreateTablet for tablet ef7a73d6307c4b099e967e7e70bf4f06 (DEFAULT_TABLE table=default.Table_3c81228276ba40fe8b78578cdd253c0d [id=820d7f76444f42178fdb54864a998903]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:21.281504 18458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef7a73d6307c4b099e967e7e70bf4f06. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:21.282294 18463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e49de8fd5081451693734b6d91522298. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:21.296085 18462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05b5a0b272ca41509c3291336079ef81. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:21.319339 18877 tablet_bootstrap.cc:492] T ef7a73d6307c4b099e967e7e70bf4f06 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:21.323418 18878 tablet_bootstrap.cc:492] T 0810eb04f1dd4d4bae41845453a623ab P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:21.326505 18878 tablet_bootstrap.cc:654] T 0810eb04f1dd4d4bae41845453a623ab P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:21.328051 18878 log.cc:824] T 0810eb04f1dd4d4bae41845453a623ab P 2efb542aa55e439b99ccb98d1c5c884a: Log is configured to *not* fsync() on all Append() calls
I20240430 20:11:21.328974 18877 tablet_bootstrap.cc:654] T ef7a73d6307c4b099e967e7e70bf4f06 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:21.330724 18877 log.cc:824] T ef7a73d6307c4b099e967e7e70bf4f06 P 5e1c6397f09747f091534faf93542710: Log is configured to *not* fsync() on all Append() calls
I20240430 20:11:21.331779 18876 tablet_bootstrap.cc:492] T e49de8fd5081451693734b6d91522298 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:21.338219 18876 tablet_bootstrap.cc:654] T e49de8fd5081451693734b6d91522298 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:21.339763 18876 log.cc:824] T e49de8fd5081451693734b6d91522298 P 3c51c33d5616482cafb68e26688820f4: Log is configured to *not* fsync() on all Append() calls
I20240430 20:11:21.340616 18877 tablet_bootstrap.cc:492] T ef7a73d6307c4b099e967e7e70bf4f06 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:21.340996 18877 ts_tablet_manager.cc:1367] T ef7a73d6307c4b099e967e7e70bf4f06 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.023s user 0.009s sys 0.007s
I20240430 20:11:21.342682 18878 tablet_bootstrap.cc:492] T 0810eb04f1dd4d4bae41845453a623ab P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:21.343070 18878 ts_tablet_manager.cc:1367] T 0810eb04f1dd4d4bae41845453a623ab P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.020s user 0.012s sys 0.005s
I20240430 20:11:21.354079 18878 raft_consensus.cc:348] T 0810eb04f1dd4d4bae41845453a623ab P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:21.354737 18878 raft_consensus.cc:374] T 0810eb04f1dd4d4bae41845453a623ab P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:21.354887 18878 raft_consensus.cc:729] T 0810eb04f1dd4d4bae41845453a623ab P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:21.355829 18878 consensus_queue.cc:260] T 0810eb04f1dd4d4bae41845453a623ab P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:21.357453 18877 raft_consensus.cc:348] T ef7a73d6307c4b099e967e7e70bf4f06 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:21.358112 18877 raft_consensus.cc:374] T ef7a73d6307c4b099e967e7e70bf4f06 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:21.358240 18877 raft_consensus.cc:729] T ef7a73d6307c4b099e967e7e70bf4f06 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:21.359176 18877 consensus_queue.cc:260] T ef7a73d6307c4b099e967e7e70bf4f06 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:21.362797 18876 tablet_bootstrap.cc:492] T e49de8fd5081451693734b6d91522298 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:21.363261 18876 ts_tablet_manager.cc:1367] T e49de8fd5081451693734b6d91522298 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.032s user 0.019s sys 0.008s
I20240430 20:11:21.368214 18878 ts_tablet_manager.cc:1398] T 0810eb04f1dd4d4bae41845453a623ab P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.025s user 0.020s sys 0.003s
I20240430 20:11:21.368499 18529 heartbeater.cc:503] Master 127.17.177.60:37225 was elected leader, sending a full tablet report...
I20240430 20:11:21.375685 18878 tablet_bootstrap.cc:492] T ef7a73d6307c4b099e967e7e70bf4f06 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:21.377840 18878 tablet_bootstrap.cc:654] T ef7a73d6307c4b099e967e7e70bf4f06 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:21.379401 18669 heartbeater.cc:503] Master 127.17.177.60:37225 was elected leader, sending a full tablet report...
I20240430 20:11:21.381534 18877 ts_tablet_manager.cc:1398] T ef7a73d6307c4b099e967e7e70bf4f06 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.040s user 0.015s sys 0.014s
I20240430 20:11:21.382114 18877 tablet_bootstrap.cc:492] T 0810eb04f1dd4d4bae41845453a623ab P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:21.384425 18877 tablet_bootstrap.cc:654] T 0810eb04f1dd4d4bae41845453a623ab P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:21.386862 18876 raft_consensus.cc:348] T e49de8fd5081451693734b6d91522298 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:21.387179 18878 tablet_bootstrap.cc:492] T ef7a73d6307c4b099e967e7e70bf4f06 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:21.387377 18878 ts_tablet_manager.cc:1367] T ef7a73d6307c4b099e967e7e70bf4f06 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20240430 20:11:21.387409 18876 raft_consensus.cc:374] T e49de8fd5081451693734b6d91522298 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:21.387604 18876 raft_consensus.cc:729] T e49de8fd5081451693734b6d91522298 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:21.388446 18876 consensus_queue.cc:260] T e49de8fd5081451693734b6d91522298 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:21.388660 18878 raft_consensus.cc:348] T ef7a73d6307c4b099e967e7e70bf4f06 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:21.388944 18878 raft_consensus.cc:374] T ef7a73d6307c4b099e967e7e70bf4f06 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:21.389072 18878 raft_consensus.cc:729] T ef7a73d6307c4b099e967e7e70bf4f06 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:21.390202 18878 consensus_queue.cc:260] T ef7a73d6307c4b099e967e7e70bf4f06 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:21.392690 18878 ts_tablet_manager.cc:1398] T ef7a73d6307c4b099e967e7e70bf4f06 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.005s user 0.000s sys 0.003s
I20240430 20:11:21.392861 18877 tablet_bootstrap.cc:492] T 0810eb04f1dd4d4bae41845453a623ab P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:21.392992 18878 tablet_bootstrap.cc:492] T 2b6be4a072094c02b14bddfea18c8600 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:21.393011 18877 ts_tablet_manager.cc:1367] T 0810eb04f1dd4d4bae41845453a623ab P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.006s
I20240430 20:11:21.394023 18877 raft_consensus.cc:348] T 0810eb04f1dd4d4bae41845453a623ab P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:21.394243 18877 raft_consensus.cc:374] T 0810eb04f1dd4d4bae41845453a623ab P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:21.394358 18877 raft_consensus.cc:729] T 0810eb04f1dd4d4bae41845453a623ab P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:21.394670 18877 consensus_queue.cc:260] T 0810eb04f1dd4d4bae41845453a623ab P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:21.395242 18877 ts_tablet_manager.cc:1398] T 0810eb04f1dd4d4bae41845453a623ab P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240430 20:11:21.395531 18877 tablet_bootstrap.cc:492] T 2b6be4a072094c02b14bddfea18c8600 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:21.396143 18878 tablet_bootstrap.cc:654] T 2b6be4a072094c02b14bddfea18c8600 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:21.397334 18877 tablet_bootstrap.cc:654] T 2b6be4a072094c02b14bddfea18c8600 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:21.405551 18877 tablet_bootstrap.cc:492] T 2b6be4a072094c02b14bddfea18c8600 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:21.405699 18878 tablet_bootstrap.cc:492] T 2b6be4a072094c02b14bddfea18c8600 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:21.405755 18877 ts_tablet_manager.cc:1367] T 2b6be4a072094c02b14bddfea18c8600 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.008s
I20240430 20:11:21.405879 18878 ts_tablet_manager.cc:1367] T 2b6be4a072094c02b14bddfea18c8600 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.001s
I20240430 20:11:21.406564 18878 raft_consensus.cc:348] T 2b6be4a072094c02b14bddfea18c8600 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:21.406554 18877 raft_consensus.cc:348] T 2b6be4a072094c02b14bddfea18c8600 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:21.406826 18878 raft_consensus.cc:374] T 2b6be4a072094c02b14bddfea18c8600 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:21.406836 18877 raft_consensus.cc:374] T 2b6be4a072094c02b14bddfea18c8600 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:21.406914 18877 raft_consensus.cc:729] T 2b6be4a072094c02b14bddfea18c8600 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:21.406914 18878 raft_consensus.cc:729] T 2b6be4a072094c02b14bddfea18c8600 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:21.406965 18876 ts_tablet_manager.cc:1398] T e49de8fd5081451693734b6d91522298 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.044s user 0.018s sys 0.002s
I20240430 20:11:21.407198 18877 consensus_queue.cc:260] T 2b6be4a072094c02b14bddfea18c8600 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:21.407204 18878 consensus_queue.cc:260] T 2b6be4a072094c02b14bddfea18c8600 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:21.407717 18877 ts_tablet_manager.cc:1398] T 2b6be4a072094c02b14bddfea18c8600 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240430 20:11:21.408066 18877 tablet_bootstrap.cc:492] T e49de8fd5081451693734b6d91522298 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:21.409001 18878 ts_tablet_manager.cc:1398] T 2b6be4a072094c02b14bddfea18c8600 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240430 20:11:21.409287 18878 tablet_bootstrap.cc:492] T e49de8fd5081451693734b6d91522298 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:21.409715 18877 tablet_bootstrap.cc:654] T e49de8fd5081451693734b6d91522298 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:21.410966 18878 tablet_bootstrap.cc:654] T e49de8fd5081451693734b6d91522298 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:21.412680 18877 tablet_bootstrap.cc:492] T e49de8fd5081451693734b6d91522298 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:21.412840 18877 ts_tablet_manager.cc:1367] T e49de8fd5081451693734b6d91522298 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240430 20:11:21.413950 18877 raft_consensus.cc:348] T e49de8fd5081451693734b6d91522298 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:21.414202 18877 raft_consensus.cc:374] T e49de8fd5081451693734b6d91522298 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:21.414297 18877 raft_consensus.cc:729] T e49de8fd5081451693734b6d91522298 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:21.414610 18877 consensus_queue.cc:260] T e49de8fd5081451693734b6d91522298 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:21.415429 18877 ts_tablet_manager.cc:1398] T e49de8fd5081451693734b6d91522298 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20240430 20:11:21.415707 18877 tablet_bootstrap.cc:492] T 05b5a0b272ca41509c3291336079ef81 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:21.418730 18877 tablet_bootstrap.cc:654] T 05b5a0b272ca41509c3291336079ef81 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:21.421953 18878 tablet_bootstrap.cc:492] T e49de8fd5081451693734b6d91522298 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:21.422154 18878 ts_tablet_manager.cc:1367] T e49de8fd5081451693734b6d91522298 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.007s
I20240430 20:11:21.422931 18810 heartbeater.cc:503] Master 127.17.177.60:37225 was elected leader, sending a full tablet report...
I20240430 20:11:21.424361 18878 raft_consensus.cc:348] T e49de8fd5081451693734b6d91522298 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:21.424679 18878 raft_consensus.cc:374] T e49de8fd5081451693734b6d91522298 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:21.424839 18876 tablet_bootstrap.cc:492] T ef7a73d6307c4b099e967e7e70bf4f06 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:21.424836 18878 raft_consensus.cc:729] T e49de8fd5081451693734b6d91522298 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:21.425251 18878 consensus_queue.cc:260] T e49de8fd5081451693734b6d91522298 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:21.426677 18876 tablet_bootstrap.cc:654] T ef7a73d6307c4b099e967e7e70bf4f06 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:21.427093 18877 tablet_bootstrap.cc:492] T 05b5a0b272ca41509c3291336079ef81 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:21.427263 18877 ts_tablet_manager.cc:1367] T 05b5a0b272ca41509c3291336079ef81 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.005s
I20240430 20:11:21.427572 18878 ts_tablet_manager.cc:1398] T e49de8fd5081451693734b6d91522298 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.005s user 0.003s sys 0.001s
I20240430 20:11:21.427963 18878 tablet_bootstrap.cc:492] T 05b5a0b272ca41509c3291336079ef81 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:21.428050 18877 raft_consensus.cc:348] T 05b5a0b272ca41509c3291336079ef81 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:21.428258 18877 raft_consensus.cc:374] T 05b5a0b272ca41509c3291336079ef81 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:21.428329 18877 raft_consensus.cc:729] T 05b5a0b272ca41509c3291336079ef81 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:21.428562 18877 consensus_queue.cc:260] T 05b5a0b272ca41509c3291336079ef81 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:21.430083 18878 tablet_bootstrap.cc:654] T 05b5a0b272ca41509c3291336079ef81 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:21.430920 18877 ts_tablet_manager.cc:1398] T 05b5a0b272ca41509c3291336079ef81 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20240430 20:11:21.443586 18876 tablet_bootstrap.cc:492] T ef7a73d6307c4b099e967e7e70bf4f06 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:21.443740 18878 tablet_bootstrap.cc:492] T 05b5a0b272ca41509c3291336079ef81 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:21.443774 18876 ts_tablet_manager.cc:1367] T ef7a73d6307c4b099e967e7e70bf4f06 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.001s
I20240430 20:11:21.443931 18878 ts_tablet_manager.cc:1367] T 05b5a0b272ca41509c3291336079ef81 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.009s
I20240430 20:11:21.444656 18876 raft_consensus.cc:348] T ef7a73d6307c4b099e967e7e70bf4f06 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:21.444680 18878 raft_consensus.cc:348] T 05b5a0b272ca41509c3291336079ef81 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:21.444869 18876 raft_consensus.cc:374] T ef7a73d6307c4b099e967e7e70bf4f06 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:21.444912 18878 raft_consensus.cc:374] T 05b5a0b272ca41509c3291336079ef81 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:21.444978 18878 raft_consensus.cc:729] T 05b5a0b272ca41509c3291336079ef81 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:21.444973 18876 raft_consensus.cc:729] T ef7a73d6307c4b099e967e7e70bf4f06 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:21.445240 18878 consensus_queue.cc:260] T 05b5a0b272ca41509c3291336079ef81 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:21.445248 18876 consensus_queue.cc:260] T ef7a73d6307c4b099e967e7e70bf4f06 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:21.445840 18876 ts_tablet_manager.cc:1398] T ef7a73d6307c4b099e967e7e70bf4f06 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240430 20:11:21.446115 18876 tablet_bootstrap.cc:492] T 05b5a0b272ca41509c3291336079ef81 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:21.447575 18876 tablet_bootstrap.cc:654] T 05b5a0b272ca41509c3291336079ef81 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:21.453610 18878 ts_tablet_manager.cc:1398] T 05b5a0b272ca41509c3291336079ef81 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20240430 20:11:21.457849 18886 raft_consensus.cc:482] T e49de8fd5081451693734b6d91522298 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:21.458144 18886 raft_consensus.cc:504] T e49de8fd5081451693734b6d91522298 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:21.459633 18876 tablet_bootstrap.cc:492] T 05b5a0b272ca41509c3291336079ef81 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:21.459815 18876 ts_tablet_manager.cc:1367] T 05b5a0b272ca41509c3291336079ef81 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.011s
I20240430 20:11:21.460378 18886 leader_election.cc:288] T e49de8fd5081451693734b6d91522298 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:21.460500 18876 raft_consensus.cc:348] T 05b5a0b272ca41509c3291336079ef81 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:21.460748 18876 raft_consensus.cc:374] T 05b5a0b272ca41509c3291336079ef81 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:21.460837 18876 raft_consensus.cc:729] T 05b5a0b272ca41509c3291336079ef81 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:21.461129 18876 consensus_queue.cc:260] T 05b5a0b272ca41509c3291336079ef81 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:21.461683 18876 ts_tablet_manager.cc:1398] T 05b5a0b272ca41509c3291336079ef81 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240430 20:11:21.462025 18876 tablet_bootstrap.cc:492] T 0810eb04f1dd4d4bae41845453a623ab P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:21.463855 18876 tablet_bootstrap.cc:654] T 0810eb04f1dd4d4bae41845453a623ab P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
W20240430 20:11:21.470913 18671 tablet.cc:2369] T 05b5a0b272ca41509c3291336079ef81 P 5e1c6397f09747f091534faf93542710: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240430 20:11:21.470939 18623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e49de8fd5081451693734b6d91522298" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:21.471634 18623 raft_consensus.cc:2454] T e49de8fd5081451693734b6d91522298 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:21.472437 18696 leader_election.cc:302] T e49de8fd5081451693734b6d91522298 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:21.473923 18483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e49de8fd5081451693734b6d91522298" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
W20240430 20:11:21.474431 18536 tablet.cc:2369] T 05b5a0b272ca41509c3291336079ef81 P 2efb542aa55e439b99ccb98d1c5c884a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240430 20:11:21.474659 18483 raft_consensus.cc:2454] T e49de8fd5081451693734b6d91522298 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:21.476403 18886 raft_consensus.cc:2781] T e49de8fd5081451693734b6d91522298 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:21.476579 18886 raft_consensus.cc:482] T e49de8fd5081451693734b6d91522298 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:21.476706 18886 raft_consensus.cc:3037] T e49de8fd5081451693734b6d91522298 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:21.484496 18886 raft_consensus.cc:504] T e49de8fd5081451693734b6d91522298 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:21.489959 18483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e49de8fd5081451693734b6d91522298" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:21.490237 18483 raft_consensus.cc:3037] T e49de8fd5081451693734b6d91522298 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:21.493878 18483 raft_consensus.cc:2454] T e49de8fd5081451693734b6d91522298 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:21.494486 18699 leader_election.cc:302] T e49de8fd5081451693734b6d91522298 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters:
I20240430 20:11:21.495652 18623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e49de8fd5081451693734b6d91522298" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:21.495887 18623 raft_consensus.cc:3037] T e49de8fd5081451693734b6d91522298 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:21.498436 18886 leader_election.cc:288] T e49de8fd5081451693734b6d91522298 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Requested vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:21.498734 18886 raft_consensus.cc:2781] T e49de8fd5081451693734b6d91522298 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:21.498879 18886 raft_consensus.cc:686] T e49de8fd5081451693734b6d91522298 P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Becoming Leader. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Running, Role: LEADER
I20240430 20:11:21.499452 18886 consensus_queue.cc:237] T e49de8fd5081451693734b6d91522298 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:21.499631 18623 raft_consensus.cc:2454] T e49de8fd5081451693734b6d91522298 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
W20240430 20:11:21.504736 18815 tablet.cc:2369] T 05b5a0b272ca41509c3291336079ef81 P 3c51c33d5616482cafb68e26688820f4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240430 20:11:21.508217 18876 tablet_bootstrap.cc:492] T 0810eb04f1dd4d4bae41845453a623ab P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:21.508388 18876 ts_tablet_manager.cc:1367] T 0810eb04f1dd4d4bae41845453a623ab P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.046s user 0.021s sys 0.022s
I20240430 20:11:21.509260 18876 raft_consensus.cc:348] T 0810eb04f1dd4d4bae41845453a623ab P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:21.509481 18876 raft_consensus.cc:374] T 0810eb04f1dd4d4bae41845453a623ab P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:21.509569 18876 raft_consensus.cc:729] T 0810eb04f1dd4d4bae41845453a623ab P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:21.509805 18876 consensus_queue.cc:260] T 0810eb04f1dd4d4bae41845453a623ab P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:21.510293 18876 ts_tablet_manager.cc:1398] T 0810eb04f1dd4d4bae41845453a623ab P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240430 20:11:21.510622 18876 tablet_bootstrap.cc:492] T 2b6be4a072094c02b14bddfea18c8600 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:21.512230 18876 tablet_bootstrap.cc:654] T 2b6be4a072094c02b14bddfea18c8600 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:21.515602 18876 tablet_bootstrap.cc:492] T 2b6be4a072094c02b14bddfea18c8600 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:21.515762 18876 ts_tablet_manager.cc:1367] T 2b6be4a072094c02b14bddfea18c8600 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20240430 20:11:21.516448 18876 raft_consensus.cc:348] T 2b6be4a072094c02b14bddfea18c8600 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:21.516671 18876 raft_consensus.cc:374] T 2b6be4a072094c02b14bddfea18c8600 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:21.516741 18876 raft_consensus.cc:729] T 2b6be4a072094c02b14bddfea18c8600 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:21.517014 18876 consensus_queue.cc:260] T 2b6be4a072094c02b14bddfea18c8600 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:21.517503 18876 ts_tablet_manager.cc:1398] T 2b6be4a072094c02b14bddfea18c8600 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240430 20:11:21.526821 18295 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51068:
name: "default.Table_04b0c2ff3bf7472ebe765e992d123db9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:21.527546 18295 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_04b0c2ff3bf7472ebe765e992d123db9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:21.528947 18289 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51068:
name: "default.Table_a5725f043dd44bdebe796c50ea387a1c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:21.529460 18289 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a5725f043dd44bdebe796c50ea387a1c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:21.530754 18290 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51068:
name: "default.Table_d82824c897a94dd0ba45955f9473b626"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:21.531251 18290 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d82824c897a94dd0ba45955f9473b626 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:21.532116 18291 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51068:
name: "default.Table_a8d5ac73c6d047698d87b3d1ba34e66f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:21.532519 18291 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a8d5ac73c6d047698d87b3d1ba34e66f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:21.534978 18292 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51068:
name: "default.Table_0b4b99c1c03a4e779046063159ce6861"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:21.535465 18292 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0b4b99c1c03a4e779046063159ce6861 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:21.537166 18294 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51068:
name: "default.Table_2e9c7c683df84a41942d31600a265641"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:21.537597 18294 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2e9c7c683df84a41942d31600a265641 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:21.542956 18297 master_service.cc:946] ReplaceTablet: received request to replace tablet 2b6be4a072094c02b14bddfea18c8600 from {username='slave'} at 127.0.0.1:51068
I20240430 20:11:21.543090 18298 master_service.cc:946] ReplaceTablet: received request to replace tablet 05b5a0b272ca41509c3291336079ef81 from {username='slave'} at 127.0.0.1:51068
I20240430 20:11:21.543336 18296 master_service.cc:946] ReplaceTablet: received request to replace tablet 0810eb04f1dd4d4bae41845453a623ab from {username='slave'} at 127.0.0.1:51068
I20240430 20:11:21.543859 18116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNG3HqZ/build/asan/bin/kudu with pid 18264
W20240430 20:11:21.567725 18818 connection.cc:535] client connection to 127.17.177.60:37225 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240430 20:11:21.568259 18810 heartbeater.cc:644] Failed to heartbeat to 127.17.177.60:37225 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv got EOF from 127.17.177.60:37225 (error 108)
I20240430 20:11:21.568445 18883 raft_consensus.cc:482] T 2b6be4a072094c02b14bddfea18c8600 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:21.568643 18883 raft_consensus.cc:504] T 2b6be4a072094c02b14bddfea18c8600 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
W20240430 20:11:21.568931 18529 heartbeater.cc:644] Failed to heartbeat to 127.17.177.60:37225 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv got EOF from 127.17.177.60:37225 (error 108)
W20240430 20:11:21.568924 18818 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.17.177.60:37225)
W20240430 20:11:21.569409 18818 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.17.177.60:37225)
W20240430 20:11:21.569450 18139 connection.cc:535] server connection from 127.0.0.1:59936 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240430 20:11:21.568640 18669 heartbeater.cc:644] Failed to heartbeat to 127.17.177.60:37225 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv got EOF from 127.17.177.60:37225 (error 108)
W20240430 20:11:21.570017 18818 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.17.177.60:37225)
I20240430 20:11:21.570894 18883 leader_election.cc:288] T 2b6be4a072094c02b14bddfea18c8600 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:21.571079 18883 raft_consensus.cc:482] T ef7a73d6307c4b099e967e7e70bf4f06 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:21.571221 18883 raft_consensus.cc:504] T ef7a73d6307c4b099e967e7e70bf4f06 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:21.571916 18882 raft_consensus.cc:482] T ef7a73d6307c4b099e967e7e70bf4f06 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:21.572095 18882 raft_consensus.cc:504] T ef7a73d6307c4b099e967e7e70bf4f06 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:21.572335 18883 leader_election.cc:288] T ef7a73d6307c4b099e967e7e70bf4f06 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
W20240430 20:11:21.573343 18818 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.17.177.60:37225)
I20240430 20:11:21.574188 18882 leader_election.cc:288] T ef7a73d6307c4b099e967e7e70bf4f06 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:21.594372 18483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b6be4a072094c02b14bddfea18c8600" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:21.594506 18482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef7a73d6307c4b099e967e7e70bf4f06" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:21.594732 18483 raft_consensus.cc:2454] T 2b6be4a072094c02b14bddfea18c8600 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 0.
I20240430 20:11:21.594835 18482 raft_consensus.cc:2454] T ef7a73d6307c4b099e967e7e70bf4f06 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 0.
I20240430 20:11:21.594722 18764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef7a73d6307c4b099e967e7e70bf4f06" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:21.595214 18764 raft_consensus.cc:2454] T ef7a73d6307c4b099e967e7e70bf4f06 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 0.
I20240430 20:11:21.595141 18762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef7a73d6307c4b099e967e7e70bf4f06" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:21.595408 18762 raft_consensus.cc:2454] T ef7a73d6307c4b099e967e7e70bf4f06 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 0.
I20240430 20:11:21.595484 18558 leader_election.cc:302] T 2b6be4a072094c02b14bddfea18c8600 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:21.595209 18763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b6be4a072094c02b14bddfea18c8600" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:21.595777 18763 raft_consensus.cc:2454] T 2b6be4a072094c02b14bddfea18c8600 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 0.
I20240430 20:11:21.595876 18417 leader_election.cc:302] T ef7a73d6307c4b099e967e7e70bf4f06 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters:
I20240430 20:11:21.595925 18557 leader_election.cc:302] T ef7a73d6307c4b099e967e7e70bf4f06 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:21.596129 18901 raft_consensus.cc:2781] T 2b6be4a072094c02b14bddfea18c8600 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:21.596226 18882 raft_consensus.cc:2781] T ef7a73d6307c4b099e967e7e70bf4f06 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:21.596232 18901 raft_consensus.cc:482] T 2b6be4a072094c02b14bddfea18c8600 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:21.596329 18882 raft_consensus.cc:482] T ef7a73d6307c4b099e967e7e70bf4f06 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:21.596405 18901 raft_consensus.cc:3037] T 2b6be4a072094c02b14bddfea18c8600 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:21.596421 18882 raft_consensus.cc:3037] T ef7a73d6307c4b099e967e7e70bf4f06 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:21.596539 18883 raft_consensus.cc:2781] T ef7a73d6307c4b099e967e7e70bf4f06 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:21.596632 18883 raft_consensus.cc:482] T ef7a73d6307c4b099e967e7e70bf4f06 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:21.596743 18883 raft_consensus.cc:3037] T ef7a73d6307c4b099e967e7e70bf4f06 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:21.599179 18882 raft_consensus.cc:504] T ef7a73d6307c4b099e967e7e70bf4f06 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:21.599753 18901 raft_consensus.cc:504] T 2b6be4a072094c02b14bddfea18c8600 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:21.600023 18882 leader_election.cc:288] T ef7a73d6307c4b099e967e7e70bf4f06 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Requested vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:21.599699 18883 raft_consensus.cc:504] T ef7a73d6307c4b099e967e7e70bf4f06 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:21.600817 18482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b6be4a072094c02b14bddfea18c8600" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:21.601023 18482 raft_consensus.cc:3037] T 2b6be4a072094c02b14bddfea18c8600 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:21.601272 18762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b6be4a072094c02b14bddfea18c8600" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:21.601495 18762 raft_consensus.cc:3037] T 2b6be4a072094c02b14bddfea18c8600 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:21.601078 18901 leader_election.cc:288] T 2b6be4a072094c02b14bddfea18c8600 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Requested vote from peers 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:21.602406 18763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef7a73d6307c4b099e967e7e70bf4f06" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:21.602628 18763 raft_consensus.cc:3037] T ef7a73d6307c4b099e967e7e70bf4f06 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:21.603260 18623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef7a73d6307c4b099e967e7e70bf4f06" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:21.603554 18623 raft_consensus.cc:2379] T ef7a73d6307c4b099e967e7e70bf4f06 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2efb542aa55e439b99ccb98d1c5c884a in current term 1: Already voted for candidate 5e1c6397f09747f091534faf93542710 in this term.
I20240430 20:11:21.603960 18482 raft_consensus.cc:2454] T 2b6be4a072094c02b14bddfea18c8600 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
I20240430 20:11:21.604107 18622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef7a73d6307c4b099e967e7e70bf4f06" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:21.604437 18622 raft_consensus.cc:2379] T ef7a73d6307c4b099e967e7e70bf4f06 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 2efb542aa55e439b99ccb98d1c5c884a in current term 1: Already voted for candidate 5e1c6397f09747f091534faf93542710 in this term.
I20240430 20:11:21.604615 18762 raft_consensus.cc:2454] T 2b6be4a072094c02b14bddfea18c8600 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
I20240430 20:11:21.605024 18558 leader_election.cc:302] T 2b6be4a072094c02b14bddfea18c8600 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:21.605592 18901 raft_consensus.cc:2781] T 2b6be4a072094c02b14bddfea18c8600 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:21.605633 18763 raft_consensus.cc:2454] T ef7a73d6307c4b099e967e7e70bf4f06 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 1.
I20240430 20:11:21.605768 18883 leader_election.cc:288] T ef7a73d6307c4b099e967e7e70bf4f06 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Requested vote from peers 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:21.606223 18417 leader_election.cc:302] T ef7a73d6307c4b099e967e7e70bf4f06 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters:
I20240430 20:11:21.606528 18882 raft_consensus.cc:2781] T ef7a73d6307c4b099e967e7e70bf4f06 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:21.606504 18763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef7a73d6307c4b099e967e7e70bf4f06" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:21.606947 18763 raft_consensus.cc:2379] T ef7a73d6307c4b099e967e7e70bf4f06 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 5e1c6397f09747f091534faf93542710 in current term 1: Already voted for candidate 2efb542aa55e439b99ccb98d1c5c884a in this term.
I20240430 20:11:21.607726 18482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef7a73d6307c4b099e967e7e70bf4f06" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:21.608613 18882 raft_consensus.cc:686] T ef7a73d6307c4b099e967e7e70bf4f06 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Becoming Leader. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Running, Role: LEADER
I20240430 20:11:21.608986 18901 raft_consensus.cc:686] T 2b6be4a072094c02b14bddfea18c8600 P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Becoming Leader. State: Replica: 5e1c6397f09747f091534faf93542710, State: Running, Role: LEADER
I20240430 20:11:21.609491 18882 consensus_queue.cc:237] T ef7a73d6307c4b099e967e7e70bf4f06 P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:21.609503 18901 consensus_queue.cc:237] T 2b6be4a072094c02b14bddfea18c8600 P 5e1c6397f09747f091534faf93542710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:21.616690 18558 leader_election.cc:302] T ef7a73d6307c4b099e967e7e70bf4f06 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:21.617035 18901 raft_consensus.cc:2726] T ef7a73d6307c4b099e967e7e70bf4f06 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240430 20:11:21.619350 18886 raft_consensus.cc:482] T 05b5a0b272ca41509c3291336079ef81 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:21.619555 18886 raft_consensus.cc:504] T 05b5a0b272ca41509c3291336079ef81 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:21.620174 18886 leader_election.cc:288] T 05b5a0b272ca41509c3291336079ef81 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:21.620652 18622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05b5a0b272ca41509c3291336079ef81" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:21.620688 18482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05b5a0b272ca41509c3291336079ef81" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:21.620832 18622 raft_consensus.cc:2454] T 05b5a0b272ca41509c3291336079ef81 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:21.620875 18482 raft_consensus.cc:2454] T 05b5a0b272ca41509c3291336079ef81 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:21.621279 18699 leader_election.cc:302] T 05b5a0b272ca41509c3291336079ef81 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:21.621598 18886 raft_consensus.cc:2781] T 05b5a0b272ca41509c3291336079ef81 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:21.621680 18886 raft_consensus.cc:482] T 05b5a0b272ca41509c3291336079ef81 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:21.621757 18886 raft_consensus.cc:3037] T 05b5a0b272ca41509c3291336079ef81 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:21.623401 18882 raft_consensus.cc:482] T 0810eb04f1dd4d4bae41845453a623ab P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:21.623557 18882 raft_consensus.cc:504] T 0810eb04f1dd4d4bae41845453a623ab P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:21.623723 18886 raft_consensus.cc:504] T 05b5a0b272ca41509c3291336079ef81 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:21.624111 18882 leader_election.cc:288] T 0810eb04f1dd4d4bae41845453a623ab P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:21.624339 18886 leader_election.cc:288] T 05b5a0b272ca41509c3291336079ef81 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Requested vote from peers 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:21.624575 18622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0810eb04f1dd4d4bae41845453a623ab" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:21.624783 18622 raft_consensus.cc:2454] T 0810eb04f1dd4d4bae41845453a623ab P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 0.
I20240430 20:11:21.624915 18763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0810eb04f1dd4d4bae41845453a623ab" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:21.624989 18482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05b5a0b272ca41509c3291336079ef81" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:21.625128 18763 raft_consensus.cc:2454] T 0810eb04f1dd4d4bae41845453a623ab P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 0.
I20240430 20:11:21.625171 18482 raft_consensus.cc:3037] T 05b5a0b272ca41509c3291336079ef81 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:21.625294 18415 leader_election.cc:302] T 0810eb04f1dd4d4bae41845453a623ab P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:21.625620 18882 raft_consensus.cc:2781] T 0810eb04f1dd4d4bae41845453a623ab P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:21.625715 18882 raft_consensus.cc:482] T 0810eb04f1dd4d4bae41845453a623ab P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:21.625829 18882 raft_consensus.cc:3037] T 0810eb04f1dd4d4bae41845453a623ab P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:21.625982 18622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05b5a0b272ca41509c3291336079ef81" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:21.626158 18622 raft_consensus.cc:3037] T 05b5a0b272ca41509c3291336079ef81 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:21.627776 18482 raft_consensus.cc:2454] T 05b5a0b272ca41509c3291336079ef81 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:21.628726 18882 raft_consensus.cc:504] T 0810eb04f1dd4d4bae41845453a623ab P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:21.629021 18622 raft_consensus.cc:2454] T 05b5a0b272ca41509c3291336079ef81 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:21.629531 18699 leader_election.cc:302] T 05b5a0b272ca41509c3291336079ef81 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters:
I20240430 20:11:21.629654 18622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0810eb04f1dd4d4bae41845453a623ab" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:21.629824 18886 raft_consensus.cc:2781] T 05b5a0b272ca41509c3291336079ef81 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:21.629889 18622 raft_consensus.cc:3037] T 0810eb04f1dd4d4bae41845453a623ab P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:21.630139 18882 leader_election.cc:288] T 0810eb04f1dd4d4bae41845453a623ab P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Requested vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:21.630636 18763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0810eb04f1dd4d4bae41845453a623ab" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:21.630805 18886 raft_consensus.cc:686] T 05b5a0b272ca41509c3291336079ef81 P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Becoming Leader. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Running, Role: LEADER
I20240430 20:11:21.630887 18763 raft_consensus.cc:3037] T 0810eb04f1dd4d4bae41845453a623ab P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:21.631120 18886 consensus_queue.cc:237] T 05b5a0b272ca41509c3291336079ef81 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:21.632570 18622 raft_consensus.cc:2454] T 0810eb04f1dd4d4bae41845453a623ab P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 1.
I20240430 20:11:21.632911 18415 leader_election.cc:302] T 0810eb04f1dd4d4bae41845453a623ab P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:21.633316 18882 raft_consensus.cc:2781] T 0810eb04f1dd4d4bae41845453a623ab P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:21.633481 18882 raft_consensus.cc:686] T 0810eb04f1dd4d4bae41845453a623ab P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Becoming Leader. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Running, Role: LEADER
I20240430 20:11:21.633725 18763 raft_consensus.cc:2454] T 0810eb04f1dd4d4bae41845453a623ab P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 1.
I20240430 20:11:21.633778 18882 consensus_queue.cc:237] T 0810eb04f1dd4d4bae41845453a623ab P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:22.038954 18913 raft_consensus.cc:482] T 00000000000000000000000000000000 P 89df84ccf5394c9bb60d4e4ed872fc2e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader b632d6ef4c104a058a6db208024cab33)
I20240430 20:11:22.039175 18913 raft_consensus.cc:504] T 00000000000000000000000000000000 P 89df84ccf5394c9bb60d4e4ed872fc2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b3f98793354c7aaa5c106e8ec31ed8" member_type: VOTER last_known_addr { host: "127.17.177.62" port: 36441 } } peers { permanent_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e" member_type: VOTER last_known_addr { host: "127.17.177.61" port: 34777 } } peers { permanent_uuid: "b632d6ef4c104a058a6db208024cab33" member_type: VOTER last_known_addr { host: "127.17.177.60" port: 37225 } }
I20240430 20:11:22.039883 18913 leader_election.cc:288] T 00000000000000000000000000000000 P 89df84ccf5394c9bb60d4e4ed872fc2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 88b3f98793354c7aaa5c106e8ec31ed8 (127.17.177.62:36441), b632d6ef4c104a058a6db208024cab33 (127.17.177.60:37225)
I20240430 20:11:22.040441 18165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 32 } } ignore_live_leader: false dest_uuid: "88b3f98793354c7aaa5c106e8ec31ed8" is_pre_election: true
W20240430 20:11:22.041739 18211 leader_election.cc:334] T 00000000000000000000000000000000 P 89df84ccf5394c9bb60d4e4ed872fc2e [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer b632d6ef4c104a058a6db208024cab33 (127.17.177.60:37225): Network error: Client connection negotiation failed: client connection to 127.17.177.60:37225: connect: Connection refused (error 111)
I20240430 20:11:22.041867 18211 leader_election.cc:302] T 00000000000000000000000000000000 P 89df84ccf5394c9bb60d4e4ed872fc2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89df84ccf5394c9bb60d4e4ed872fc2e; no voters: 88b3f98793354c7aaa5c106e8ec31ed8, b632d6ef4c104a058a6db208024cab33
I20240430 20:11:22.042138 18913 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 89df84ccf5394c9bb60d4e4ed872fc2e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240430 20:11:22.043115 18916 raft_consensus.cc:482] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader b632d6ef4c104a058a6db208024cab33)
I20240430 20:11:22.043262 18916 raft_consensus.cc:504] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b3f98793354c7aaa5c106e8ec31ed8" member_type: VOTER last_known_addr { host: "127.17.177.62" port: 36441 } } peers { permanent_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e" member_type: VOTER last_known_addr { host: "127.17.177.61" port: 34777 } } peers { permanent_uuid: "b632d6ef4c104a058a6db208024cab33" member_type: VOTER last_known_addr { host: "127.17.177.60" port: 37225 } }
I20240430 20:11:22.043867 18916 leader_election.cc:288] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 89df84ccf5394c9bb60d4e4ed872fc2e (127.17.177.61:34777), b632d6ef4c104a058a6db208024cab33 (127.17.177.60:37225)
I20240430 20:11:22.044304 18235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88b3f98793354c7aaa5c106e8ec31ed8" candidate_term: 4 candidate_status { last_received { term: 3 index: 34 } } ignore_live_leader: false dest_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e" is_pre_election: true
I20240430 20:11:22.044548 18235 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 89df84ccf5394c9bb60d4e4ed872fc2e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88b3f98793354c7aaa5c106e8ec31ed8 in term 3.
I20240430 20:11:22.045068 18140 leader_election.cc:302] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b3f98793354c7aaa5c106e8ec31ed8, 89df84ccf5394c9bb60d4e4ed872fc2e; no voters:
I20240430 20:11:22.045292 18916 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240430 20:11:22.045397 18916 raft_consensus.cc:482] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8 [term 3 FOLLOWER]: Starting leader election (detected failure of leader b632d6ef4c104a058a6db208024cab33)
I20240430 20:11:22.045496 18916 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8 [term 3 FOLLOWER]: Advancing to term 4
W20240430 20:11:22.045704 18139 leader_election.cc:334] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer b632d6ef4c104a058a6db208024cab33 (127.17.177.60:37225): Network error: Client connection negotiation failed: client connection to 127.17.177.60:37225: connect: Connection refused (error 111)
I20240430 20:11:22.047968 18916 raft_consensus.cc:504] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b3f98793354c7aaa5c106e8ec31ed8" member_type: VOTER last_known_addr { host: "127.17.177.62" port: 36441 } } peers { permanent_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e" member_type: VOTER last_known_addr { host: "127.17.177.61" port: 34777 } } peers { permanent_uuid: "b632d6ef4c104a058a6db208024cab33" member_type: VOTER last_known_addr { host: "127.17.177.60" port: 37225 } }
I20240430 20:11:22.048554 18916 leader_election.cc:288] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8 [CANDIDATE]: Term 4 election: Requested vote from peers 89df84ccf5394c9bb60d4e4ed872fc2e (127.17.177.61:34777), b632d6ef4c104a058a6db208024cab33 (127.17.177.60:37225)
I20240430 20:11:22.048883 18235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88b3f98793354c7aaa5c106e8ec31ed8" candidate_term: 4 candidate_status { last_received { term: 3 index: 34 } } ignore_live_leader: false dest_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e"
I20240430 20:11:22.049057 18235 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 89df84ccf5394c9bb60d4e4ed872fc2e [term 3 FOLLOWER]: Advancing to term 4
I20240430 20:11:22.050963 18235 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 89df84ccf5394c9bb60d4e4ed872fc2e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b3f98793354c7aaa5c106e8ec31ed8 in term 4.
I20240430 20:11:22.051353 18140 leader_election.cc:302] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b3f98793354c7aaa5c106e8ec31ed8, 89df84ccf5394c9bb60d4e4ed872fc2e; no voters:
I20240430 20:11:22.051651 18916 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8 [term 4 FOLLOWER]: Leader election won for term 4
W20240430 20:11:22.051770 18139 leader_election.cc:334] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer b632d6ef4c104a058a6db208024cab33 (127.17.177.60:37225): Network error: Client connection negotiation failed: client connection to 127.17.177.60:37225: connect: Connection refused (error 111)
I20240430 20:11:22.052081 18916 raft_consensus.cc:686] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8 [term 4 LEADER]: Becoming Leader. State: Replica: 88b3f98793354c7aaa5c106e8ec31ed8, State: Running, Role: LEADER
I20240430 20:11:22.052507 18916 consensus_queue.cc:237] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 3.34, Last appended by leader: 34, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b3f98793354c7aaa5c106e8ec31ed8" member_type: VOTER last_known_addr { host: "127.17.177.62" port: 36441 } } peers { permanent_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e" member_type: VOTER last_known_addr { host: "127.17.177.61" port: 34777 } } peers { permanent_uuid: "b632d6ef4c104a058a6db208024cab33" member_type: VOTER last_known_addr { host: "127.17.177.60" port: 37225 } }
I20240430 20:11:22.053663 18919 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88b3f98793354c7aaa5c106e8ec31ed8. Latest consensus state: current_term: 4 leader_uuid: "88b3f98793354c7aaa5c106e8ec31ed8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b3f98793354c7aaa5c106e8ec31ed8" member_type: VOTER last_known_addr { host: "127.17.177.62" port: 36441 } } peers { permanent_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e" member_type: VOTER last_known_addr { host: "127.17.177.61" port: 34777 } } peers { permanent_uuid: "b632d6ef4c104a058a6db208024cab33" member_type: VOTER last_known_addr { host: "127.17.177.60" port: 37225 } } }
I20240430 20:11:22.053864 18919 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8 [sys.catalog]: This master's current role is: LEADER
I20240430 20:11:22.254191 18924 consensus_queue.cc:1035] T e49de8fd5081451693734b6d91522298 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:22.263355 18924 consensus_queue.cc:1035] T e49de8fd5081451693734b6d91522298 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:22.343781 18116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNG3HqZ/build/asan/bin/kudu
/tmp/dist-test-taskNG3HqZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.177.60:37225
--webserver_interface=127.17.177.60
--webserver_port=34267
--builtin_ntp_servers=127.17.177.20:42079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.177.62:36441,127.17.177.61:34777,127.17.177.60:37225
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20240430 20:11:22.425100 18930 consensus_queue.cc:1035] T 05b5a0b272ca41509c3291336079ef81 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:22.431299 18930 consensus_queue.cc:1035] T 05b5a0b272ca41509c3291336079ef81 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:22.438787 18925 consensus_queue.cc:1035] T 0810eb04f1dd4d4bae41845453a623ab P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:22.457028 18925 consensus_queue.cc:1035] T 0810eb04f1dd4d4bae41845453a623ab P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240430 20:11:22.506520 18139 consensus_peers.cc:485] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8 -> Peer b632d6ef4c104a058a6db208024cab33 (127.17.177.60:37225): Couldn't send request to peer b632d6ef4c104a058a6db208024cab33. Status: Network error: Client connection negotiation failed: client connection to 127.17.177.60:37225: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240430 20:11:22.546931 18931 consensus_queue.cc:1035] T 2b6be4a072094c02b14bddfea18c8600 P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:22.556897 18944 consensus_queue.cc:1035] T 2b6be4a072094c02b14bddfea18c8600 P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240430 20:11:22.625190 18941 consensus_queue.cc:1035] T ef7a73d6307c4b099e967e7e70bf4f06 P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:22.632220 18941 consensus_queue.cc:1035] T ef7a73d6307c4b099e967e7e70bf4f06 P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:22.644335 18235 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 89df84ccf5394c9bb60d4e4ed872fc2e [term 4 FOLLOWER]: Refusing update from remote peer 88b3f98793354c7aaa5c106e8ec31ed8: Log matching property violated. Preceding OpId in replica: term: 3 index: 32. Preceding OpId from leader: term: 4 index: 35. (index mismatch)
I20240430 20:11:22.645006 18919 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e" member_type: VOTER last_known_addr { host: "127.17.177.61" port: 34777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 30, Time since last communication: 0.000s
I20240430 20:11:22.648181 18950 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89df84ccf5394c9bb60d4e4ed872fc2e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88b3f98793354c7aaa5c106e8ec31ed8. Latest consensus state: current_term: 4 leader_uuid: "88b3f98793354c7aaa5c106e8ec31ed8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b3f98793354c7aaa5c106e8ec31ed8" member_type: VOTER last_known_addr { host: "127.17.177.62" port: 36441 } } peers { permanent_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e" member_type: VOTER last_known_addr { host: "127.17.177.61" port: 34777 } } peers { permanent_uuid: "b632d6ef4c104a058a6db208024cab33" member_type: VOTER last_known_addr { host: "127.17.177.60" port: 37225 } } }
I20240430 20:11:22.648449 18950 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89df84ccf5394c9bb60d4e4ed872fc2e [sys.catalog]: This master's current role is: FOLLOWER
I20240430 20:11:22.656036 18950 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89df84ccf5394c9bb60d4e4ed872fc2e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "88b3f98793354c7aaa5c106e8ec31ed8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b3f98793354c7aaa5c106e8ec31ed8" member_type: VOTER last_known_addr { host: "127.17.177.62" port: 36441 } } peers { permanent_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e" member_type: VOTER last_known_addr { host: "127.17.177.61" port: 34777 } } peers { permanent_uuid: "b632d6ef4c104a058a6db208024cab33" member_type: VOTER last_known_addr { host: "127.17.177.60" port: 37225 } } }
I20240430 20:11:22.656292 18950 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89df84ccf5394c9bb60d4e4ed872fc2e [sys.catalog]: This master's current role is: FOLLOWER
I20240430 20:11:22.658211 18953 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "88b3f98793354c7aaa5c106e8ec31ed8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b3f98793354c7aaa5c106e8ec31ed8" member_type: VOTER last_known_addr { host: "127.17.177.62" port: 36441 } } peers { permanent_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e" member_type: VOTER last_known_addr { host: "127.17.177.61" port: 34777 } } peers { permanent_uuid: "b632d6ef4c104a058a6db208024cab33" member_type: VOTER last_known_addr { host: "127.17.177.60" port: 37225 } } }
I20240430 20:11:22.658419 18953 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8 [sys.catalog]: This master's current role is: LEADER
I20240430 20:11:22.699165 18920 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240430 20:11:22.701633 18920 catalog_manager.cc:669] Loaded metadata for table default.Table_04b0c2ff3bf7472ebe765e992d123db9 [id=28a79ab51f6741ae9c5b40ff5220e0b6]
I20240430 20:11:22.702083 18920 catalog_manager.cc:669] Loaded metadata for table default.Table_0b4b99c1c03a4e779046063159ce6861 [id=43d7bdc84bab4c54bac58dd1dd0f74ac]
I20240430 20:11:22.702473 18920 catalog_manager.cc:669] Loaded metadata for table default.Table_5dd79a4e3c8545ae8a3105d9c29f27cc [id=50ddbed249404afcbd875ca1f9a04101]
I20240430 20:11:22.703079 18920 catalog_manager.cc:669] Loaded metadata for table default.Table_3c81228276ba40fe8b78578cdd253c0d [id=820d7f76444f42178fdb54864a998903]
I20240430 20:11:22.703405 18920 catalog_manager.cc:669] Loaded metadata for table default.Table_a5725f043dd44bdebe796c50ea387a1c [id=86c59536bafd4214ac2d93d9496ead5d]
I20240430 20:11:22.703905 18920 catalog_manager.cc:669] Loaded metadata for table default.Table_a8d5ac73c6d047698d87b3d1ba34e66f [id=9c80193cde2346c982cf84ae9612e2f7]
W20240430 20:11:22.704125 18934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
I20240430 20:11:22.704393 18920 catalog_manager.cc:669] Loaded metadata for table default.Table_88d5674c038a499682559e6c50aa9a70 [id=a5d580cffa2447d78fc9f2c8e31264f1]
W20240430 20:11:22.704444 18934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240430 20:11:22.704566 18934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240430 20:11:22.704792 18920 catalog_manager.cc:669] Loaded metadata for table default.Table_2829200b322c4201b8351098eac2ddae [id=a6a27dcbd0e648a8bf29a8cd5b3a2a11]
I20240430 20:11:22.705128 18920 catalog_manager.cc:669] Loaded metadata for table default.Table_2e9c7c683df84a41942d31600a265641 [id=aca6fb02776a40bbac312191adbdc136]
I20240430 20:11:22.705598 18920 catalog_manager.cc:669] Loaded metadata for table default.Table_d82824c897a94dd0ba45955f9473b626 [id=d794640d21a843a185038673f7eb2b00]
I20240430 20:11:22.705912 18920 catalog_manager.cc:669] Loaded metadata for table default.Table_4312aac764824e93904ef77f93fb3578 [id=e63582736e544bffba71f7bd54bc0cbc]
I20240430 20:11:22.708079 18920 tablet_loader.cc:96] loaded metadata for tablet 05b5a0b272ca41509c3291336079ef81 (table default.Table_5dd79a4e3c8545ae8a3105d9c29f27cc [id=50ddbed249404afcbd875ca1f9a04101])
I20240430 20:11:22.708464 18920 tablet_loader.cc:96] loaded metadata for tablet 0810eb04f1dd4d4bae41845453a623ab (table default.Table_4312aac764824e93904ef77f93fb3578 [id=e63582736e544bffba71f7bd54bc0cbc])
I20240430 20:11:22.708878 18920 tablet_loader.cc:96] loaded metadata for tablet 27b21459592649dca1089a7a29021206 (table default.Table_a8d5ac73c6d047698d87b3d1ba34e66f [id=9c80193cde2346c982cf84ae9612e2f7])
I20240430 20:11:22.709170 18920 tablet_loader.cc:96] loaded metadata for tablet 2b6be4a072094c02b14bddfea18c8600 (table default.Table_2829200b322c4201b8351098eac2ddae [id=a6a27dcbd0e648a8bf29a8cd5b3a2a11])
I20240430 20:11:22.709443 18920 tablet_loader.cc:96] loaded metadata for tablet 44699c84ebac49ef8f3765d4069e71ca (table default.Table_0b4b99c1c03a4e779046063159ce6861 [id=43d7bdc84bab4c54bac58dd1dd0f74ac])
I20240430 20:11:22.709787 18920 tablet_loader.cc:96] loaded metadata for tablet a9e8a1052a2a42bf8dbb117c7a7b9c18 (table default.Table_2e9c7c683df84a41942d31600a265641 [id=aca6fb02776a40bbac312191adbdc136])
I20240430 20:11:22.710139 18920 tablet_loader.cc:96] loaded metadata for tablet ca95311f67c040f28b45356fcb284d82 (table default.Table_d82824c897a94dd0ba45955f9473b626 [id=d794640d21a843a185038673f7eb2b00])
I20240430 20:11:22.710525 18920 tablet_loader.cc:96] loaded metadata for tablet e449fe0854824e70872d1990531126ee (table default.Table_04b0c2ff3bf7472ebe765e992d123db9 [id=28a79ab51f6741ae9c5b40ff5220e0b6])
I20240430 20:11:22.710841 18920 tablet_loader.cc:96] loaded metadata for tablet e49de8fd5081451693734b6d91522298 (table default.Table_88d5674c038a499682559e6c50aa9a70 [id=a5d580cffa2447d78fc9f2c8e31264f1])
I20240430 20:11:22.711316 18920 tablet_loader.cc:96] loaded metadata for tablet ef7a73d6307c4b099e967e7e70bf4f06 (table default.Table_3c81228276ba40fe8b78578cdd253c0d [id=820d7f76444f42178fdb54864a998903])
I20240430 20:11:22.711653 18920 tablet_loader.cc:96] loaded metadata for tablet fbdcb69bfd374d41b8ca66c5f90deaee (table default.Table_a5725f043dd44bdebe796c50ea387a1c [id=86c59536bafd4214ac2d93d9496ead5d])
I20240430 20:11:22.711896 18920 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20240430 20:11:22.712544 18934 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240430 20:11:22.712659 18934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240430 20:11:22.712735 18934 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240430 20:11:22.712814 18934 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240430 20:11:22.713312 18920 catalog_manager.cc:1259] Loaded cluster ID: 889e0faede1f41d1ab74633d6826c0d4
I20240430 20:11:22.713390 18920 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240430 20:11:22.714726 18920 catalog_manager.cc:1504] Loading token signing keys...
I20240430 20:11:22.715664 18920 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: Loaded TSK: 0
I20240430 20:11:22.716535 18920 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240430 20:11:22.723498 18155 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_04b0c2ff3bf7472ebe765e992d123db9" } modify_external_catalogs: true
I20240430 20:11:22.723871 18155 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_04b0c2ff3bf7472ebe765e992d123db9" } modify_external_catalogs: true
I20240430 20:11:22.724339 18150 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_3c81228276ba40fe8b78578cdd253c0d" } new_table_name: "default.Table_50a4a93d5e7b4ec5b84f5c758cde8e9c" modify_external_catalogs: true
I20240430 20:11:22.725936 18156 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_a5725f043dd44bdebe796c50ea387a1c" } modify_external_catalogs: true
I20240430 20:11:22.726142 18156 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_a5725f043dd44bdebe796c50ea387a1c" } modify_external_catalogs: true
I20240430 20:11:22.726488 18934 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.177.20:42079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/master-2/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.17.177.62:36441,127.17.177.61:34777,127.17.177.60:37225
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.177.60:37225
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.17.177.60
--webserver_port=34267
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type FASTDEBUG
built by None at 30 Apr 2024 19:43:22 UTC on efd1337a2aa3
build id 63
ASAN enabled
I20240430 20:11:22.727277 18154 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_5dd79a4e3c8545ae8a3105d9c29f27cc" } new_table_name: "default.Table_0c91d730e7054760b34c115293ff75e7" modify_external_catalogs: true
I20240430 20:11:22.728780 18934 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240430 20:11:22.728804 18152 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_4312aac764824e93904ef77f93fb3578" } new_table_name: "default.Table_f29ab93660ac426ba743cb11c68b4deb" modify_external_catalogs: true
I20240430 20:11:22.730185 18151 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_2829200b322c4201b8351098eac2ddae" } new_table_name: "default.Table_c06b54647f004ff7a7f7beafee702b4c" modify_external_catalogs: true
I20240430 20:11:22.731087 18934 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240430 20:11:22.731794 18149 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54524:
name: "default.Table_0b4b99c1c03a4e779046063159ce6861"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:22.732576 18149 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0b4b99c1c03a4e779046063159ce6861 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:22.733317 18149 master_service.cc:946] ReplaceTablet: received request to replace tablet 2b6be4a072094c02b14bddfea18c8600 from {username='slave'} at 127.0.0.1:54524
I20240430 20:11:22.734617 18148 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54524:
name: "default.Table_d82824c897a94dd0ba45955f9473b626"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:22.735103 18148 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d82824c897a94dd0ba45955f9473b626 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:22.735381 18148 master_service.cc:946] ReplaceTablet: received request to replace tablet 0810eb04f1dd4d4bae41845453a623ab from {username='slave'} at 127.0.0.1:54524
I20240430 20:11:22.736533 18147 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54524:
name: "default.Table_a8d5ac73c6d047698d87b3d1ba34e66f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:22.737088 18147 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a8d5ac73c6d047698d87b3d1ba34e66f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:22.737385 18147 master_service.cc:946] ReplaceTablet: received request to replace tablet 05b5a0b272ca41509c3291336079ef81 from {username='slave'} at 127.0.0.1:54524
I20240430 20:11:22.738353 18146 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54524:
name: "default.Table_2e9c7c683df84a41942d31600a265641"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:22.738857 18146 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2e9c7c683df84a41942d31600a265641 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:22.739203 18146 master_service.cc:946] ReplaceTablet: received request to replace tablet 05b5a0b272ca41509c3291336079ef81 from {username='slave'} at 127.0.0.1:54524
I20240430 20:11:22.741722 18155 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet e449fe0854824e70872d1990531126ee
I20240430 20:11:22.741947 18156 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet fbdcb69bfd374d41b8ca66c5f90deaee
I20240430 20:11:22.742146 18155 master_service.cc:946] ReplaceTablet: received request to replace tablet e49de8fd5081451693734b6d91522298 from {username='slave'} at 127.0.0.1:54524
W20240430 20:11:22.742805 18150 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ef7a73d6307c4b099e967e7e70bf4f06 consensus state has no leader
W20240430 20:11:22.748188 18964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 20:11:22.752295 18965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240430 20:11:22.755897 18150 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54524:
name: "default.Table_613940f4d9a94605b3486079b72aa571"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240430 20:11:22.756260 18156 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54524:
name: "default.Table_68ed98ea02c44ee1920dabf781e01e04"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:22.756541 18150 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_613940f4d9a94605b3486079b72aa571 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240430 20:11:22.756824 18156 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_68ed98ea02c44ee1920dabf781e01e04 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240430 20:11:22.757398 18154 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 05b5a0b272ca41509c3291336079ef81 consensus state has no leader
W20240430 20:11:22.758651 18969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240430 20:11:22.758808 18934 server_base.cc:1034] running on GCE node
I20240430 20:11:22.759945 18934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240430 20:11:22.760039 18154 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54524:
name: "default.Table_33059b681e564888992fa98876243716"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:22.760584 18154 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_33059b681e564888992fa98876243716 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240430 20:11:22.761507 18152 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0810eb04f1dd4d4bae41845453a623ab consensus state has no leader
I20240430 20:11:22.762013 18152 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54524:
name: "default.Table_8c0a41d37b0243799236894c186433b6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:22.762701 18152 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8c0a41d37b0243799236894c186433b6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:22.765578 18934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
W20240430 20:11:22.766242 18151 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2b6be4a072094c02b14bddfea18c8600 consensus state has no leader
I20240430 20:11:22.766655 18151 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_88d5674c038a499682559e6c50aa9a70" } new_table_name: "default.Table_e970b787d0a24a49a55b99b8238490d8" modify_external_catalogs: true
I20240430 20:11:22.769004 18934 hybrid_clock.cc:648] HybridClock initialized: now 1714507882768256 us; error 714 us; skew 500 ppm
I20240430 20:11:22.769465 18934 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240430 20:11:22.770093 18149 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: Sending DeleteTablet for 3 replicas of tablet 2b6be4a072094c02b14bddfea18c8600
I20240430 20:11:22.770442 18147 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: Sending DeleteTablet for 3 replicas of tablet 05b5a0b272ca41509c3291336079ef81
I20240430 20:11:22.771090 18155 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: Sending DeleteTablet for 3 replicas of tablet e49de8fd5081451693734b6d91522298
W20240430 20:11:22.772738 18142 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet ef7a73d6307c4b099e967e7e70bf4f06 (table default.Table_50a4a93d5e7b4ec5b84f5c758cde8e9c [id=820d7f76444f42178fdb54864a998903], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet ef7a73d6307c4b099e967e7e70bf4f06 consensus state has no leader
I20240430 20:11:22.770089 18148 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: Sending DeleteTablet for 3 replicas of tablet 0810eb04f1dd4d4bae41845453a623ab
I20240430 20:11:22.777640 18149 catalog_manager.cc:6554] ReplaceTablet: tablet 2b6be4a072094c02b14bddfea18c8600 deleted and replaced by tablet 3525f5a5051d4fb28cd9c95ca6ed6c48
I20240430 20:11:22.778333 18147 catalog_manager.cc:6554] ReplaceTablet: tablet 05b5a0b272ca41509c3291336079ef81 deleted and replaced by tablet da1c17aa2b4d4cc4a50ff958b11ac6d4
I20240430 20:11:22.782472 18155 catalog_manager.cc:6554] ReplaceTablet: tablet e49de8fd5081451693734b6d91522298 deleted and replaced by tablet e5d1707ba0b34cc4b1cda39e36ccbd2f
I20240430 20:11:22.787806 18148 catalog_manager.cc:6554] ReplaceTablet: tablet 0810eb04f1dd4d4bae41845453a623ab deleted and replaced by tablet 727326b84b41443894245c8a1d0fed20
I20240430 20:11:22.838479 18599 tablet_service.cc:1514] Processing DeleteTablet for tablet e49de8fd5081451693734b6d91522298 with delete_type TABLET_DATA_DELETED (replaced by tablet e5d1707ba0b34cc4b1cda39e36ccbd2f) from {username='slave'} at 127.0.0.1:40202
I20240430 20:11:22.838887 18461 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b6be4a072094c02b14bddfea18c8600 with delete_type TABLET_DATA_DELETED (replaced by tablet 3525f5a5051d4fb28cd9c95ca6ed6c48) from {username='slave'} at 127.0.0.1:41678
I20240430 20:11:22.840046 18150 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_50a4a93d5e7b4ec5b84f5c758cde8e9c" } modify_external_catalogs: true
I20240430 20:11:22.840246 18155 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54524:
name: "default.Table_25ea07a841144dab969f3baf7ee7c0dc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240430 20:11:22.840574 18148 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54524:
name: "default.Table_a4227ca7f32c4ff88677bcecedac89d9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:22.841004 18155 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_25ea07a841144dab969f3baf7ee7c0dc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240430 20:11:22.841063 18148 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a4227ca7f32c4ff88677bcecedac89d9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:22.841461 18147 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54524:
name: "default.Table_b78b1b2c95694d1faee32321f845047b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:22.841907 18147 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b78b1b2c95694d1faee32321f845047b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:22.840274 18150 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_50a4a93d5e7b4ec5b84f5c758cde8e9c" } modify_external_catalogs: true
W20240430 20:11:22.845477 18151 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e5d1707ba0b34cc4b1cda39e36ccbd2f has no consensus state
I20240430 20:11:22.845697 18154 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_0c91d730e7054760b34c115293ff75e7" } modify_external_catalogs: true
I20240430 20:11:22.845863 18154 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_0c91d730e7054760b34c115293ff75e7" } modify_external_catalogs: true
I20240430 20:11:22.845868 18602 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b6be4a072094c02b14bddfea18c8600 with delete_type TABLET_DATA_DELETED (replaced by tablet 3525f5a5051d4fb28cd9c95ca6ed6c48) from {username='slave'} at 127.0.0.1:40202
I20240430 20:11:22.846562 18602 tablet_service.cc:1514] Processing DeleteTablet for tablet 05b5a0b272ca41509c3291336079ef81 with delete_type TABLET_DATA_DELETED (replaced by tablet da1c17aa2b4d4cc4a50ff958b11ac6d4) from {username='slave'} at 127.0.0.1:40202
I20240430 20:11:22.847087 18152 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_f29ab93660ac426ba743cb11c68b4deb" } modify_external_catalogs: true
I20240430 20:11:22.847294 18152 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_f29ab93660ac426ba743cb11c68b4deb" } modify_external_catalogs: true
I20240430 20:11:22.850677 18146 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54524:
name: "default.Table_5feaab34a5cd439abdc705b85c22313b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:22.851213 18146 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5feaab34a5cd439abdc705b85c22313b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:22.852950 18156 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54524:
name: "default.Table_c6fcbdec7e67466da00ab71d5404fe4e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:22.853456 18156 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c6fcbdec7e67466da00ab71d5404fe4e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:22.852952 18149 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54524:
name: "default.Table_195ce26ee51647a8aad6871d72484cba"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:22.855284 18149 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_195ce26ee51647a8aad6871d72484cba in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:22.855497 18602 tablet_service.cc:1514] Processing DeleteTablet for tablet 0810eb04f1dd4d4bae41845453a623ab with delete_type TABLET_DATA_DELETED (replaced by tablet 727326b84b41443894245c8a1d0fed20) from {username='slave'} at 127.0.0.1:40202
I20240430 20:11:22.856403 18602 tablet_service.cc:1467] Processing CreateTablet for tablet ca95311f67c040f28b45356fcb284d82 (DEFAULT_TABLE table=default.Table_d82824c897a94dd0ba45955f9473b626 [id=d794640d21a843a185038673f7eb2b00]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:22.856979 18602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca95311f67c040f28b45356fcb284d82. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:22.863235 18603 tablet_service.cc:1467] Processing CreateTablet for tablet a9e8a1052a2a42bf8dbb117c7a7b9c18 (DEFAULT_TABLE table=default.Table_2e9c7c683df84a41942d31600a265641 [id=aca6fb02776a40bbac312191adbdc136]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:22.863679 18603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9e8a1052a2a42bf8dbb117c7a7b9c18. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:22.865381 18154 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet da1c17aa2b4d4cc4a50ff958b11ac6d4
I20240430 20:11:22.867266 18458 tablet_service.cc:1514] Processing DeleteTablet for tablet 05b5a0b272ca41509c3291336079ef81 with delete_type TABLET_DATA_DELETED (replaced by tablet da1c17aa2b4d4cc4a50ff958b11ac6d4) from {username='slave'} at 127.0.0.1:41678
I20240430 20:11:22.867594 18462 tablet_service.cc:1514] Processing DeleteTablet for tablet e49de8fd5081451693734b6d91522298 with delete_type TABLET_DATA_DELETED (replaced by tablet e5d1707ba0b34cc4b1cda39e36ccbd2f) from {username='slave'} at 127.0.0.1:41678
I20240430 20:11:22.867815 18463 tablet_service.cc:1514] Processing DeleteTablet for tablet 0810eb04f1dd4d4bae41845453a623ab with delete_type TABLET_DATA_DELETED (replaced by tablet 727326b84b41443894245c8a1d0fed20) from {username='slave'} at 127.0.0.1:41678
I20240430 20:11:22.867838 18458 tablet_service.cc:1467] Processing CreateTablet for tablet ca95311f67c040f28b45356fcb284d82 (DEFAULT_TABLE table=default.Table_d82824c897a94dd0ba45955f9473b626 [id=d794640d21a843a185038673f7eb2b00]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:22.868283 18458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca95311f67c040f28b45356fcb284d82. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:22.868933 18152 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet 727326b84b41443894245c8a1d0fed20
I20240430 20:11:22.870039 18601 tablet_service.cc:1467] Processing CreateTablet for tablet 44699c84ebac49ef8f3765d4069e71ca (DEFAULT_TABLE table=default.Table_0b4b99c1c03a4e779046063159ce6861 [id=43d7bdc84bab4c54bac58dd1dd0f74ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:22.870515 18601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44699c84ebac49ef8f3765d4069e71ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:22.870795 18150 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: Sending DeleteTablet for 3 replicas of tablet ef7a73d6307c4b099e967e7e70bf4f06
I20240430 20:11:22.874615 18983 tablet_bootstrap.cc:492] T a9e8a1052a2a42bf8dbb117c7a7b9c18 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:22.876627 18983 tablet_bootstrap.cc:654] T a9e8a1052a2a42bf8dbb117c7a7b9c18 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:22.879262 18463 tablet_service.cc:1467] Processing CreateTablet for tablet a9e8a1052a2a42bf8dbb117c7a7b9c18 (DEFAULT_TABLE table=default.Table_2e9c7c683df84a41942d31600a265641 [id=aca6fb02776a40bbac312191adbdc136]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:22.879434 18600 tablet_service.cc:1467] Processing CreateTablet for tablet 27b21459592649dca1089a7a29021206 (DEFAULT_TABLE table=default.Table_a8d5ac73c6d047698d87b3d1ba34e66f [id=9c80193cde2346c982cf84ae9612e2f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:22.879524 18459 tablet_service.cc:1467] Processing CreateTablet for tablet 27b21459592649dca1089a7a29021206 (DEFAULT_TABLE table=default.Table_a8d5ac73c6d047698d87b3d1ba34e66f [id=9c80193cde2346c982cf84ae9612e2f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:22.879724 18463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9e8a1052a2a42bf8dbb117c7a7b9c18. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:22.879745 18600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27b21459592649dca1089a7a29021206. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:22.880288 18459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27b21459592649dca1089a7a29021206. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:22.881290 18462 tablet_service.cc:1467] Processing CreateTablet for tablet 44699c84ebac49ef8f3765d4069e71ca (DEFAULT_TABLE table=default.Table_0b4b99c1c03a4e779046063159ce6861 [id=43d7bdc84bab4c54bac58dd1dd0f74ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:22.881680 18462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44699c84ebac49ef8f3765d4069e71ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:22.884460 18981 tablet_replica.cc:330] T 2b6be4a072094c02b14bddfea18c8600 P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:22.884518 18985 tablet_bootstrap.cc:492] T ca95311f67c040f28b45356fcb284d82 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:22.885288 18457 tablet_service.cc:1514] Processing DeleteTablet for tablet ef7a73d6307c4b099e967e7e70bf4f06 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:22 UTC) from {username='slave'} at 127.0.0.1:41678
I20240430 20:11:22.886045 18981 raft_consensus.cc:2229] T 2b6be4a072094c02b14bddfea18c8600 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:22.886799 18600 tablet_service.cc:1514] Processing DeleteTablet for tablet ef7a73d6307c4b099e967e7e70bf4f06 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:22 UTC) from {username='slave'} at 127.0.0.1:40202
I20240430 20:11:22.892107 18983 tablet_bootstrap.cc:492] T a9e8a1052a2a42bf8dbb117c7a7b9c18 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:22.892300 18983 ts_tablet_manager.cc:1367] T a9e8a1052a2a42bf8dbb117c7a7b9c18 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.007s
I20240430 20:11:22.893141 18981 raft_consensus.cc:2258] T 2b6be4a072094c02b14bddfea18c8600 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:22.893136 18983 raft_consensus.cc:348] T a9e8a1052a2a42bf8dbb117c7a7b9c18 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:22.893384 18983 raft_consensus.cc:374] T a9e8a1052a2a42bf8dbb117c7a7b9c18 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:22.893499 18983 raft_consensus.cc:729] T a9e8a1052a2a42bf8dbb117c7a7b9c18 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:22.893761 18983 consensus_queue.cc:260] T a9e8a1052a2a42bf8dbb117c7a7b9c18 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:22.894405 18983 ts_tablet_manager.cc:1398] T a9e8a1052a2a42bf8dbb117c7a7b9c18 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240430 20:11:22.894706 18983 tablet_bootstrap.cc:492] T 27b21459592649dca1089a7a29021206 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:22.895009 18981 ts_tablet_manager.cc:1875] T 2b6be4a072094c02b14bddfea18c8600 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:22.896478 18983 tablet_bootstrap.cc:654] T 27b21459592649dca1089a7a29021206 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:22.898905 18742 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b6be4a072094c02b14bddfea18c8600 with delete_type TABLET_DATA_DELETED (replaced by tablet 3525f5a5051d4fb28cd9c95ca6ed6c48) from {username='slave'} at 127.0.0.1:59818
I20240430 20:11:22.898978 18741 tablet_service.cc:1514] Processing DeleteTablet for tablet 05b5a0b272ca41509c3291336079ef81 with delete_type TABLET_DATA_DELETED (replaced by tablet da1c17aa2b4d4cc4a50ff958b11ac6d4) from {username='slave'} at 127.0.0.1:59818
I20240430 20:11:22.899407 18743 tablet_service.cc:1514] Processing DeleteTablet for tablet 0810eb04f1dd4d4bae41845453a623ab with delete_type TABLET_DATA_DELETED (replaced by tablet 727326b84b41443894245c8a1d0fed20) from {username='slave'} at 127.0.0.1:59818
I20240430 20:11:22.899618 18739 tablet_service.cc:1467] Processing CreateTablet for tablet a9e8a1052a2a42bf8dbb117c7a7b9c18 (DEFAULT_TABLE table=default.Table_2e9c7c683df84a41942d31600a265641 [id=aca6fb02776a40bbac312191adbdc136]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:22.899940 18740 tablet_service.cc:1467] Processing CreateTablet for tablet ca95311f67c040f28b45356fcb284d82 (DEFAULT_TABLE table=default.Table_d82824c897a94dd0ba45955f9473b626 [id=d794640d21a843a185038673f7eb2b00]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:22.900197 18739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9e8a1052a2a42bf8dbb117c7a7b9c18. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:22.900429 18744 tablet_service.cc:1514] Processing DeleteTablet for tablet e49de8fd5081451693734b6d91522298 with delete_type TABLET_DATA_DELETED (replaced by tablet e5d1707ba0b34cc4b1cda39e36ccbd2f) from {username='slave'} at 127.0.0.1:59818
I20240430 20:11:22.902843 18740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca95311f67c040f28b45356fcb284d82. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:22.907451 18741 tablet_service.cc:1467] Processing CreateTablet for tablet 44699c84ebac49ef8f3765d4069e71ca (DEFAULT_TABLE table=default.Table_0b4b99c1c03a4e779046063159ce6861 [id=43d7bdc84bab4c54bac58dd1dd0f74ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:22.907984 18741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44699c84ebac49ef8f3765d4069e71ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:22.910794 18743 tablet_service.cc:1514] Processing DeleteTablet for tablet ef7a73d6307c4b099e967e7e70bf4f06 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:22 UTC) from {username='slave'} at 127.0.0.1:59818
I20240430 20:11:22.911336 18744 tablet_service.cc:1467] Processing CreateTablet for tablet 27b21459592649dca1089a7a29021206 (DEFAULT_TABLE table=default.Table_a8d5ac73c6d047698d87b3d1ba34e66f [id=9c80193cde2346c982cf84ae9612e2f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:22.911862 18744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27b21459592649dca1089a7a29021206. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:22.912199 18981 ts_tablet_manager.cc:1888] T 2b6be4a072094c02b14bddfea18c8600 P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:22.912375 18981 log.cc:1196] T 2b6be4a072094c02b14bddfea18c8600 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/2b6be4a072094c02b14bddfea18c8600
I20240430 20:11:22.912948 18981 ts_tablet_manager.cc:1909] T 2b6be4a072094c02b14bddfea18c8600 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:22.914831 18980 tablet_replica.cc:330] T e49de8fd5081451693734b6d91522298 P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:22.915416 18980 raft_consensus.cc:2229] T e49de8fd5081451693734b6d91522298 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:22.915848 18980 raft_consensus.cc:2258] T e49de8fd5081451693734b6d91522298 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:22.917502 18985 tablet_bootstrap.cc:654] T ca95311f67c040f28b45356fcb284d82 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:22.917941 18980 ts_tablet_manager.cc:1875] T e49de8fd5081451693734b6d91522298 P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:22.923405 18142 catalog_manager.cc:4830] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): tablet 2b6be4a072094c02b14bddfea18c8600 (table default.Table_c06b54647f004ff7a7f7beafee702b4c [id=a6a27dcbd0e648a8bf29a8cd5b3a2a11]) successfully deleted
I20240430 20:11:22.924383 18981 tablet_replica.cc:330] T 05b5a0b272ca41509c3291336079ef81 P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:22.924652 18981 raft_consensus.cc:2229] T 05b5a0b272ca41509c3291336079ef81 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:22.924876 18981 raft_consensus.cc:2258] T 05b5a0b272ca41509c3291336079ef81 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:22.926204 18985 tablet_bootstrap.cc:492] T ca95311f67c040f28b45356fcb284d82 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:22.926368 18985 ts_tablet_manager.cc:1367] T ca95311f67c040f28b45356fcb284d82 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.042s user 0.003s sys 0.000s
I20240430 20:11:22.927043 18981 ts_tablet_manager.cc:1875] T 05b5a0b272ca41509c3291336079ef81 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:22.929920 18985 raft_consensus.cc:348] T ca95311f67c040f28b45356fcb284d82 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:22.930122 18985 raft_consensus.cc:374] T ca95311f67c040f28b45356fcb284d82 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:22.930213 18985 raft_consensus.cc:729] T ca95311f67c040f28b45356fcb284d82 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:22.932581 18985 consensus_queue.cc:260] T ca95311f67c040f28b45356fcb284d82 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:22.938113 18981 ts_tablet_manager.cc:1888] T 05b5a0b272ca41509c3291336079ef81 P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:22.940376 18987 tablet_replica.cc:330] T 2b6be4a072094c02b14bddfea18c8600 P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:22.941462 18980 ts_tablet_manager.cc:1888] T e49de8fd5081451693734b6d91522298 P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:22.941476 18987 raft_consensus.cc:2229] T 2b6be4a072094c02b14bddfea18c8600 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:22.941789 18987 raft_consensus.cc:2258] T 2b6be4a072094c02b14bddfea18c8600 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:22.942559 18988 tablet_bootstrap.cc:492] T a9e8a1052a2a42bf8dbb117c7a7b9c18 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:22.943189 18934 webserver.cc:460] Webserver started at http://127.17.177.60:34267/ using document root <none> and password file <none>
I20240430 20:11:22.944370 18988 tablet_bootstrap.cc:654] T a9e8a1052a2a42bf8dbb117c7a7b9c18 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:22.944792 18934 fs_manager.cc:359] Metadata directory not provided
I20240430 20:11:22.945015 18934 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240430 20:11:22.946244 18980 log.cc:1196] T e49de8fd5081451693734b6d91522298 P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/e49de8fd5081451693734b6d91522298
I20240430 20:11:22.946898 18980 ts_tablet_manager.cc:1909] T e49de8fd5081451693734b6d91522298 P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:22.947018 18156 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_a4227ca7f32c4ff88677bcecedac89d9" } modify_external_catalogs: true
I20240430 20:11:22.947228 18152 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_33059b681e564888992fa98876243716" } modify_external_catalogs: true
I20240430 20:11:22.947264 18156 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_a4227ca7f32c4ff88677bcecedac89d9" } modify_external_catalogs: true
I20240430 20:11:22.947438 18152 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_33059b681e564888992fa98876243716" } modify_external_catalogs: true
I20240430 20:11:22.947050 18148 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_e970b787d0a24a49a55b99b8238490d8" } new_table_name: "default.Table_7fec253c8f4a4fb1a66ec3af88a66ba6" modify_external_catalogs: true
I20240430 20:11:22.950457 18988 tablet_bootstrap.cc:492] T a9e8a1052a2a42bf8dbb117c7a7b9c18 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:22.950682 18988 ts_tablet_manager.cc:1367] T a9e8a1052a2a42bf8dbb117c7a7b9c18 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20240430 20:11:22.950927 18987 ts_tablet_manager.cc:1875] T 2b6be4a072094c02b14bddfea18c8600 P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:22.954350 18981 log.cc:1196] T 05b5a0b272ca41509c3291336079ef81 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/05b5a0b272ca41509c3291336079ef81
I20240430 20:11:22.951534 18988 raft_consensus.cc:348] T a9e8a1052a2a42bf8dbb117c7a7b9c18 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:22.954574 18988 raft_consensus.cc:374] T a9e8a1052a2a42bf8dbb117c7a7b9c18 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:22.954699 18988 raft_consensus.cc:729] T a9e8a1052a2a42bf8dbb117c7a7b9c18 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:22.954839 18981 ts_tablet_manager.cc:1909] T 05b5a0b272ca41509c3291336079ef81 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:22.954898 18988 consensus_queue.cc:260] T a9e8a1052a2a42bf8dbb117c7a7b9c18 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:22.955124 18139 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet e49de8fd5081451693734b6d91522298 (table default.Table_e970b787d0a24a49a55b99b8238490d8 [id=a5d580cffa2447d78fc9f2c8e31264f1]) successfully deleted
I20240430 20:11:22.944164 18985 ts_tablet_manager.cc:1398] T ca95311f67c040f28b45356fcb284d82 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.018s user 0.000s sys 0.003s
I20240430 20:11:22.956099 18602 tablet_service.cc:1467] Processing CreateTablet for tablet 17843ea77664447eaaa19666e0445ada (DEFAULT_TABLE table=default.Table_613940f4d9a94605b3486079b72aa571 [id=c0d8dc87d1f9426a85f6176f161595a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:22.956277 18603 tablet_service.cc:1467] Processing CreateTablet for tablet e5d1707ba0b34cc4b1cda39e36ccbd2f (DEFAULT_TABLE table=default.Table_e970b787d0a24a49a55b99b8238490d8 [id=a5d580cffa2447d78fc9f2c8e31264f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:22.956317 18599 tablet_service.cc:1467] Processing CreateTablet for tablet 269144de63db417a9c9423a913d4536a (DEFAULT_TABLE table=default.Table_8c0a41d37b0243799236894c186433b6 [id=b93708ff15ef4c8299967c8c91045c8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:22.956511 18602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17843ea77664447eaaa19666e0445ada. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:22.956713 18601 tablet_service.cc:1467] Processing CreateTablet for tablet 235ea182e02740f68f30c3c34f77012a (DEFAULT_TABLE table=default.Table_33059b681e564888992fa98876243716 [id=2eae27505e2b4218b933f03c397a8495]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:22.956712 18603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5d1707ba0b34cc4b1cda39e36ccbd2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:22.957082 18601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 235ea182e02740f68f30c3c34f77012a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:22.958479 18744 tablet_service.cc:1467] Processing CreateTablet for tablet 17843ea77664447eaaa19666e0445ada (DEFAULT_TABLE table=default.Table_613940f4d9a94605b3486079b72aa571 [id=c0d8dc87d1f9426a85f6176f161595a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:22.958876 18744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17843ea77664447eaaa19666e0445ada. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:22.962152 18740 tablet_service.cc:1467] Processing CreateTablet for tablet f6e23412ad5b4773bb62590e452abc7a (DEFAULT_TABLE table=default.Table_68ed98ea02c44ee1920dabf781e01e04 [id=ee9addb7eb9f4430a57fd9ee4bfaae7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:22.962167 18150 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_68ed98ea02c44ee1920dabf781e01e04" } new_table_name: "default.Table_d4b5e1a26a9449cdbe70f185b1097321" modify_external_catalogs: true
I20240430 20:11:22.962517 18740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6e23412ad5b4773bb62590e452abc7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:22.962502 18600 tablet_service.cc:1467] Processing CreateTablet for tablet f6e23412ad5b4773bb62590e452abc7a (DEFAULT_TABLE table=default.Table_68ed98ea02c44ee1920dabf781e01e04 [id=ee9addb7eb9f4430a57fd9ee4bfaae7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:22.962859 18599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 269144de63db417a9c9423a913d4536a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:22.963676 18739 tablet_service.cc:1467] Processing CreateTablet for tablet 235ea182e02740f68f30c3c34f77012a (DEFAULT_TABLE table=default.Table_33059b681e564888992fa98876243716 [id=2eae27505e2b4218b933f03c397a8495]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:22.964013 18739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 235ea182e02740f68f30c3c34f77012a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:22.965023 18154 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_8c0a41d37b0243799236894c186433b6" } new_table_name: "default.Table_a235489d8a2248c9b257cf1bd72de770" modify_external_catalogs: true
I20240430 20:11:22.965109 18742 tablet_service.cc:1467] Processing CreateTablet for tablet 269144de63db417a9c9423a913d4536a (DEFAULT_TABLE table=default.Table_8c0a41d37b0243799236894c186433b6 [id=b93708ff15ef4c8299967c8c91045c8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:22.965492 18742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 269144de63db417a9c9423a913d4536a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:22.962708 18598 tablet_service.cc:1467] Processing CreateTablet for tablet 3525f5a5051d4fb28cd9c95ca6ed6c48 (DEFAULT_TABLE table=default.Table_c06b54647f004ff7a7f7beafee702b4c [id=a6a27dcbd0e648a8bf29a8cd5b3a2a11]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:22.966073 18600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6e23412ad5b4773bb62590e452abc7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:22.966900 18741 tablet_service.cc:1467] Processing CreateTablet for tablet e5d1707ba0b34cc4b1cda39e36ccbd2f (DEFAULT_TABLE table=default.Table_e970b787d0a24a49a55b99b8238490d8 [id=a5d580cffa2447d78fc9f2c8e31264f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:22.961743 18463 tablet_service.cc:1467] Processing CreateTablet for tablet 269144de63db417a9c9423a913d4536a (DEFAULT_TABLE table=default.Table_8c0a41d37b0243799236894c186433b6 [id=b93708ff15ef4c8299967c8c91045c8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:22.969200 18463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 269144de63db417a9c9423a913d4536a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:22.970835 18147 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_613940f4d9a94605b3486079b72aa571" } new_table_name: "default.Table_4fd51e528ae6409e8987a3be835b54cc" modify_external_catalogs: true
I20240430 20:11:22.972048 18598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3525f5a5051d4fb28cd9c95ca6ed6c48. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:22.972731 18741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5d1707ba0b34cc4b1cda39e36ccbd2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:22.973953 18743 tablet_service.cc:1467] Processing CreateTablet for tablet 3525f5a5051d4fb28cd9c95ca6ed6c48 (DEFAULT_TABLE table=default.Table_c06b54647f004ff7a7f7beafee702b4c [id=a6a27dcbd0e648a8bf29a8cd5b3a2a11]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:22.974072 18156 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet ac672f003c6d448480fa633ea0f750b9
I20240430 20:11:22.974262 18743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3525f5a5051d4fb28cd9c95ca6ed6c48. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:22.962172 18461 tablet_service.cc:1467] Processing CreateTablet for tablet f6e23412ad5b4773bb62590e452abc7a (DEFAULT_TABLE table=default.Table_68ed98ea02c44ee1920dabf781e01e04 [id=ee9addb7eb9f4430a57fd9ee4bfaae7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:22.974690 18461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6e23412ad5b4773bb62590e452abc7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:22.975427 18988 ts_tablet_manager.cc:1398] T a9e8a1052a2a42bf8dbb117c7a7b9c18 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.024s user 0.000s sys 0.002s
I20240430 20:11:22.978552 18988 tablet_bootstrap.cc:492] T ca95311f67c040f28b45356fcb284d82 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:22.962517 18458 tablet_service.cc:1467] Processing CreateTablet for tablet e5d1707ba0b34cc4b1cda39e36ccbd2f (DEFAULT_TABLE table=default.Table_e970b787d0a24a49a55b99b8238490d8 [id=a5d580cffa2447d78fc9f2c8e31264f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:22.979386 18458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5d1707ba0b34cc4b1cda39e36ccbd2f. 1 dirs total, 0 dirs full, 0 dirs failed
W20240430 20:11:22.979503 18148 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e5d1707ba0b34cc4b1cda39e36ccbd2f consensus state has no leader
I20240430 20:11:22.982311 18148 master_service.cc:946] ReplaceTablet: received request to replace tablet a9e8a1052a2a42bf8dbb117c7a7b9c18 from {username='slave'} at 127.0.0.1:54524
I20240430 20:11:22.966331 18462 tablet_service.cc:1467] Processing CreateTablet for tablet 235ea182e02740f68f30c3c34f77012a (DEFAULT_TABLE table=default.Table_33059b681e564888992fa98876243716 [id=2eae27505e2b4218b933f03c397a8495]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:22.982942 18462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 235ea182e02740f68f30c3c34f77012a. 1 dirs total, 0 dirs full, 0 dirs failed
W20240430 20:11:22.983443 18150 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f6e23412ad5b4773bb62590e452abc7a consensus state has no leader
I20240430 20:11:22.983676 18988 tablet_bootstrap.cc:654] T ca95311f67c040f28b45356fcb284d82 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:22.967628 18459 tablet_service.cc:1467] Processing CreateTablet for tablet 17843ea77664447eaaa19666e0445ada (DEFAULT_TABLE table=default.Table_613940f4d9a94605b3486079b72aa571 [id=c0d8dc87d1f9426a85f6176f161595a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:22.967800 18457 tablet_service.cc:1467] Processing CreateTablet for tablet 3525f5a5051d4fb28cd9c95ca6ed6c48 (DEFAULT_TABLE table=default.Table_c06b54647f004ff7a7f7beafee702b4c [id=a6a27dcbd0e648a8bf29a8cd5b3a2a11]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:22.993903 18457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3525f5a5051d4fb28cd9c95ca6ed6c48. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:22.994899 18459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17843ea77664447eaaa19666e0445ada. 1 dirs total, 0 dirs full, 0 dirs failed
W20240430 20:11:22.995854 18154 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 269144de63db417a9c9423a913d4536a consensus state has no leader
I20240430 20:11:22.999594 18983 tablet_bootstrap.cc:492] T 27b21459592649dca1089a7a29021206 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:22.999768 18983 ts_tablet_manager.cc:1367] T 27b21459592649dca1089a7a29021206 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.105s user 0.022s sys 0.022s
I20240430 20:11:23.000541 18983 raft_consensus.cc:348] T 27b21459592649dca1089a7a29021206 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:23.000783 18983 raft_consensus.cc:374] T 27b21459592649dca1089a7a29021206 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:23.000864 18983 raft_consensus.cc:729] T 27b21459592649dca1089a7a29021206 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:23.001125 18983 consensus_queue.cc:260] T 27b21459592649dca1089a7a29021206 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:23.004911 18983 ts_tablet_manager.cc:1398] T 27b21459592649dca1089a7a29021206 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
W20240430 20:11:23.004972 18147 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 17843ea77664447eaaa19666e0445ada consensus state has no leader
I20240430 20:11:23.008154 18983 tablet_bootstrap.cc:492] T ca95311f67c040f28b45356fcb284d82 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:22.995687 18987 ts_tablet_manager.cc:1888] T 2b6be4a072094c02b14bddfea18c8600 P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:23.012249 18987 log.cc:1196] T 2b6be4a072094c02b14bddfea18c8600 P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/2b6be4a072094c02b14bddfea18c8600
I20240430 20:11:23.012876 18987 ts_tablet_manager.cc:1909] T 2b6be4a072094c02b14bddfea18c8600 P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:23.013187 18528 heartbeater.cc:503] Master 127.17.177.62:36441 was elected leader, sending a full tablet report...
I20240430 20:11:23.013936 18988 tablet_bootstrap.cc:492] T ca95311f67c040f28b45356fcb284d82 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:23.014112 18988 ts_tablet_manager.cc:1367] T ca95311f67c040f28b45356fcb284d82 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.036s user 0.004s sys 0.003s
I20240430 20:11:23.014878 18985 tablet_bootstrap.cc:492] T 27b21459592649dca1089a7a29021206 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:23.014894 18141 catalog_manager.cc:4830] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): tablet 2b6be4a072094c02b14bddfea18c8600 (table default.Table_c06b54647f004ff7a7f7beafee702b4c [id=a6a27dcbd0e648a8bf29a8cd5b3a2a11]) successfully deleted
I20240430 20:11:23.014941 18988 raft_consensus.cc:348] T ca95311f67c040f28b45356fcb284d82 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:23.015146 18988 raft_consensus.cc:374] T ca95311f67c040f28b45356fcb284d82 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:23.015264 18988 raft_consensus.cc:729] T ca95311f67c040f28b45356fcb284d82 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:23.015465 18987 tablet_replica.cc:330] T 0810eb04f1dd4d4bae41845453a623ab P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:23.015564 18988 consensus_queue.cc:260] T ca95311f67c040f28b45356fcb284d82 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:23.016113 18988 ts_tablet_manager.cc:1398] T ca95311f67c040f28b45356fcb284d82 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240430 20:11:23.016356 18988 tablet_bootstrap.cc:492] T 44699c84ebac49ef8f3765d4069e71ca P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:23.016458 18985 tablet_bootstrap.cc:654] T 27b21459592649dca1089a7a29021206 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:23.017740 18983 tablet_bootstrap.cc:654] T ca95311f67c040f28b45356fcb284d82 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:23.019520 18988 tablet_bootstrap.cc:654] T 44699c84ebac49ef8f3765d4069e71ca P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:23.015707 18987 raft_consensus.cc:2229] T 0810eb04f1dd4d4bae41845453a623ab P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:23.020788 18987 raft_consensus.cc:2258] T 0810eb04f1dd4d4bae41845453a623ab P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:23.021447 18980 tablet_replica.cc:330] T 2b6be4a072094c02b14bddfea18c8600 P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:23.021734 18980 raft_consensus.cc:2229] T 2b6be4a072094c02b14bddfea18c8600 P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Raft consensus shutting down.
I20240430 20:11:23.022339 18980 raft_consensus.cc:2258] T 2b6be4a072094c02b14bddfea18c8600 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:23.022805 18981 tablet_replica.cc:330] T e49de8fd5081451693734b6d91522298 P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:23.023016 18981 raft_consensus.cc:2229] T e49de8fd5081451693734b6d91522298 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:23.023144 18987 ts_tablet_manager.cc:1875] T 0810eb04f1dd4d4bae41845453a623ab P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:23.023229 18981 raft_consensus.cc:2258] T e49de8fd5081451693734b6d91522298 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:23.023675 18668 heartbeater.cc:503] Master 127.17.177.62:36441 was elected leader, sending a full tablet report...
I20240430 20:11:23.024063 18981 ts_tablet_manager.cc:1875] T e49de8fd5081451693734b6d91522298 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:23.027566 18809 heartbeater.cc:503] Master 127.17.177.62:36441 was elected leader, sending a full tablet report...
I20240430 20:11:23.030452 18934 fs_manager.cc:709] Time spent opening directory manager: real 0.079s user 0.003s sys 0.002s
I20240430 20:11:23.031527 18462 tablet_service.cc:1467] Processing CreateTablet for tablet 6f729a6e8dc14de9a2057cfe09ba9257 (DEFAULT_TABLE table=default.Table_5feaab34a5cd439abdc705b85c22313b [id=1cefefb9abd64daa914f73fb65c4b201]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:23.032007 18462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f729a6e8dc14de9a2057cfe09ba9257. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:23.038285 18742 tablet_service.cc:1467] Processing CreateTablet for tablet 6f729a6e8dc14de9a2057cfe09ba9257 (DEFAULT_TABLE table=default.Table_5feaab34a5cd439abdc705b85c22313b [id=1cefefb9abd64daa914f73fb65c4b201]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:23.038821 18742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f729a6e8dc14de9a2057cfe09ba9257. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:23.039124 18980 ts_tablet_manager.cc:1875] T 2b6be4a072094c02b14bddfea18c8600 P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:23.044265 18981 ts_tablet_manager.cc:1888] T e49de8fd5081451693734b6d91522298 P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:23.044332 18155 master_service.cc:946] ReplaceTablet: received request to replace tablet 05b5a0b272ca41509c3291336079ef81 from {username='slave'} at 127.0.0.1:54524
I20240430 20:11:23.044412 18981 log.cc:1196] T e49de8fd5081451693734b6d91522298 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/e49de8fd5081451693734b6d91522298
I20240430 20:11:23.044736 18154 master_service.cc:946] ReplaceTablet: received request to replace tablet 3525f5a5051d4fb28cd9c95ca6ed6c48 from {username='slave'} at 127.0.0.1:54524
I20240430 20:11:23.044888 18981 ts_tablet_manager.cc:1909] T e49de8fd5081451693734b6d91522298 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:23.045104 18156 master_service.cc:946] ReplaceTablet: received request to replace tablet 235ea182e02740f68f30c3c34f77012a from {username='slave'} at 127.0.0.1:54524
I20240430 20:11:23.045977 18142 catalog_manager.cc:4830] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): tablet e49de8fd5081451693734b6d91522298 (table default.Table_7fec253c8f4a4fb1a66ec3af88a66ba6 [id=a5d580cffa2447d78fc9f2c8e31264f1]) successfully deleted
I20240430 20:11:23.047832 18150 master_service.cc:946] ReplaceTablet: received request to replace tablet ca95311f67c040f28b45356fcb284d82 from {username='slave'} at 127.0.0.1:54524
I20240430 20:11:23.047861 18980 ts_tablet_manager.cc:1888] T 2b6be4a072094c02b14bddfea18c8600 P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:23.048066 18980 log.cc:1196] T 2b6be4a072094c02b14bddfea18c8600 P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/2b6be4a072094c02b14bddfea18c8600
I20240430 20:11:23.048512 18980 ts_tablet_manager.cc:1909] T 2b6be4a072094c02b14bddfea18c8600 P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:23.048724 18997 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240430 20:11:23.048943 18981 tablet_replica.cc:330] T 0810eb04f1dd4d4bae41845453a623ab P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:23.049163 18981 raft_consensus.cc:2229] T 0810eb04f1dd4d4bae41845453a623ab P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Raft consensus shutting down.
I20240430 20:11:23.049489 18980 tablet_replica.cc:330] T 05b5a0b272ca41509c3291336079ef81 P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:23.049585 18139 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet 2b6be4a072094c02b14bddfea18c8600 (table default.Table_c06b54647f004ff7a7f7beafee702b4c [id=a6a27dcbd0e648a8bf29a8cd5b3a2a11]) successfully deleted
I20240430 20:11:23.049677 18980 raft_consensus.cc:2229] T 05b5a0b272ca41509c3291336079ef81 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:23.049675 18981 raft_consensus.cc:2258] T 0810eb04f1dd4d4bae41845453a623ab P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:23.049902 18980 raft_consensus.cc:2258] T 05b5a0b272ca41509c3291336079ef81 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:23.051672 18601 tablet_service.cc:1467] Processing CreateTablet for tablet 6f729a6e8dc14de9a2057cfe09ba9257 (DEFAULT_TABLE table=default.Table_5feaab34a5cd439abdc705b85c22313b [id=1cefefb9abd64daa914f73fb65c4b201]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:23.051931 18987 ts_tablet_manager.cc:1888] T 0810eb04f1dd4d4bae41845453a623ab P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:23.052083 18987 log.cc:1196] T 0810eb04f1dd4d4bae41845453a623ab P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/0810eb04f1dd4d4bae41845453a623ab
I20240430 20:11:23.052094 18601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f729a6e8dc14de9a2057cfe09ba9257. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:23.052476 18987 ts_tablet_manager.cc:1909] T 0810eb04f1dd4d4bae41845453a623ab P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:23.053221 18603 tablet_service.cc:1467] Processing CreateTablet for tablet 4d3117a9a5f243b59a69d960c0a1eee4 (DEFAULT_TABLE table=default.Table_c6fcbdec7e67466da00ab71d5404fe4e [id=ff76c635446a4b4583759831970f784e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:23.053608 18603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d3117a9a5f243b59a69d960c0a1eee4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:23.054060 18740 tablet_service.cc:1467] Processing CreateTablet for tablet f79497bf22ed417d988530f6e69cc07f (DEFAULT_TABLE table=default.Table_b78b1b2c95694d1faee32321f845047b [id=2a3033894afd47ed8d907628a2e9712d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:23.054406 18740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f79497bf22ed417d988530f6e69cc07f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:23.054874 18599 tablet_service.cc:1467] Processing CreateTablet for tablet f79497bf22ed417d988530f6e69cc07f (DEFAULT_TABLE table=default.Table_b78b1b2c95694d1faee32321f845047b [id=2a3033894afd47ed8d907628a2e9712d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:23.055233 18599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f79497bf22ed417d988530f6e69cc07f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:23.055204 18602 tablet_service.cc:1467] Processing CreateTablet for tablet bf925791ddcf4fa7b5a1645c63f0d696 (DEFAULT_TABLE table=default.Table_195ce26ee51647a8aad6871d72484cba [id=b536a31acaea48d78885b0e585d00832]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:23.055644 18602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf925791ddcf4fa7b5a1645c63f0d696. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:23.055629 18463 tablet_service.cc:1467] Processing CreateTablet for tablet f79497bf22ed417d988530f6e69cc07f (DEFAULT_TABLE table=default.Table_b78b1b2c95694d1faee32321f845047b [id=2a3033894afd47ed8d907628a2e9712d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:23.056028 18463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f79497bf22ed417d988530f6e69cc07f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:23.058900 18457 tablet_service.cc:1467] Processing CreateTablet for tablet bf925791ddcf4fa7b5a1645c63f0d696 (DEFAULT_TABLE table=default.Table_195ce26ee51647a8aad6871d72484cba [id=b536a31acaea48d78885b0e585d00832]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:23.059284 18457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf925791ddcf4fa7b5a1645c63f0d696. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:23.059262 18743 tablet_service.cc:1467] Processing CreateTablet for tablet bf925791ddcf4fa7b5a1645c63f0d696 (DEFAULT_TABLE table=default.Table_195ce26ee51647a8aad6871d72484cba [id=b536a31acaea48d78885b0e585d00832]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:23.059633 18743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf925791ddcf4fa7b5a1645c63f0d696. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:23.066254 18459 tablet_service.cc:1467] Processing CreateTablet for tablet 4d3117a9a5f243b59a69d960c0a1eee4 (DEFAULT_TABLE table=default.Table_c6fcbdec7e67466da00ab71d5404fe4e [id=ff76c635446a4b4583759831970f784e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:23.066706 18459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d3117a9a5f243b59a69d960c0a1eee4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:23.075119 18987 tablet_replica.cc:330] T e49de8fd5081451693734b6d91522298 P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:23.075311 18987 raft_consensus.cc:2229] T e49de8fd5081451693734b6d91522298 P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Raft consensus shutting down.
I20240430 20:11:23.075693 18742 tablet_service.cc:1467] Processing CreateTablet for tablet 4d3117a9a5f243b59a69d960c0a1eee4 (DEFAULT_TABLE table=default.Table_c6fcbdec7e67466da00ab71d5404fe4e [id=ff76c635446a4b4583759831970f784e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:23.075888 18987 raft_consensus.cc:2258] T e49de8fd5081451693734b6d91522298 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:23.076113 18742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d3117a9a5f243b59a69d960c0a1eee4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:23.077822 18987 ts_tablet_manager.cc:1875] T e49de8fd5081451693734b6d91522298 P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:23.081476 18980 ts_tablet_manager.cc:1875] T 05b5a0b272ca41509c3291336079ef81 P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:23.083729 18934 fs_manager.cc:725] Time spent opening block manager: real 0.051s user 0.005s sys 0.000s
I20240430 20:11:23.085253 18981 ts_tablet_manager.cc:1875] T 0810eb04f1dd4d4bae41845453a623ab P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:23.086462 18151 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54524:
name: "default.Table_8502447753b04761815bf9602fd28e37"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:23.087177 18151 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8502447753b04761815bf9602fd28e37 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:23.084789 18934 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/master-2/data,/tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/master-2/wal
uuid: "b632d6ef4c104a058a6db208024cab33"
format_stamp: "Formatted at 2024-04-30 20:11:18 on dist-test-slave-54l7"
I20240430 20:11:23.092378 18934 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240430 20:11:23.095850 18980 ts_tablet_manager.cc:1888] T 05b5a0b272ca41509c3291336079ef81 P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:23.096050 18980 log.cc:1196] T 05b5a0b272ca41509c3291336079ef81 P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/05b5a0b272ca41509c3291336079ef81
I20240430 20:11:23.096570 18980 ts_tablet_manager.cc:1909] T 05b5a0b272ca41509c3291336079ef81 P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:23.100091 18980 tablet_replica.cc:330] T 0810eb04f1dd4d4bae41845453a623ab P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:23.100399 18980 raft_consensus.cc:2229] T 0810eb04f1dd4d4bae41845453a623ab P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:23.100621 18980 raft_consensus.cc:2258] T 0810eb04f1dd4d4bae41845453a623ab P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:23.101279 18987 ts_tablet_manager.cc:1888] T e49de8fd5081451693734b6d91522298 P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:23.101466 18987 log.cc:1196] T e49de8fd5081451693734b6d91522298 P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/e49de8fd5081451693734b6d91522298
I20240430 20:11:23.102491 18980 ts_tablet_manager.cc:1875] T 0810eb04f1dd4d4bae41845453a623ab P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:23.103896 18459 tablet_service.cc:1467] Processing CreateTablet for tablet 2067cb2cf76d495c92c1f25c176eb6dc (DEFAULT_TABLE table=default.Table_25ea07a841144dab969f3baf7ee7c0dc [id=b4bd1ac2bb0b4989af99b2f4936ce927]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:23.104331 18459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2067cb2cf76d495c92c1f25c176eb6dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:23.104941 18739 tablet_service.cc:1467] Processing CreateTablet for tablet 2067cb2cf76d495c92c1f25c176eb6dc (DEFAULT_TABLE table=default.Table_25ea07a841144dab969f3baf7ee7c0dc [id=b4bd1ac2bb0b4989af99b2f4936ce927]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:23.105316 18739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2067cb2cf76d495c92c1f25c176eb6dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:23.107064 18601 tablet_service.cc:1467] Processing CreateTablet for tablet 2067cb2cf76d495c92c1f25c176eb6dc (DEFAULT_TABLE table=default.Table_25ea07a841144dab969f3baf7ee7c0dc [id=b4bd1ac2bb0b4989af99b2f4936ce927]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:23.107472 18601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2067cb2cf76d495c92c1f25c176eb6dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:23.108644 18148 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: Sending DeleteTablet for 3 replicas of tablet a9e8a1052a2a42bf8dbb117c7a7b9c18
I20240430 20:11:23.109120 18148 catalog_manager.cc:6554] ReplaceTablet: tablet a9e8a1052a2a42bf8dbb117c7a7b9c18 deleted and replaced by tablet 26a015250eeb4750a42df93840e8d47f
I20240430 20:11:23.109622 18740 tablet_service.cc:1514] Processing DeleteTablet for tablet a9e8a1052a2a42bf8dbb117c7a7b9c18 with delete_type TABLET_DATA_DELETED (replaced by tablet 26a015250eeb4750a42df93840e8d47f) from {username='slave'} at 127.0.0.1:59818
I20240430 20:11:23.110203 18987 ts_tablet_manager.cc:1909] T e49de8fd5081451693734b6d91522298 P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:23.110306 18463 tablet_service.cc:1514] Processing DeleteTablet for tablet a9e8a1052a2a42bf8dbb117c7a7b9c18 with delete_type TABLET_DATA_DELETED (replaced by tablet 26a015250eeb4750a42df93840e8d47f) from {username='slave'} at 127.0.0.1:41678
I20240430 20:11:23.111105 18156 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_195ce26ee51647a8aad6871d72484cba" } new_table_name: "default.Table_917365c2d10941558081a9898db3d7ea" modify_external_catalogs: true
I20240430 20:11:23.112068 18141 catalog_manager.cc:4830] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): tablet e49de8fd5081451693734b6d91522298 (table default.Table_7fec253c8f4a4fb1a66ec3af88a66ba6 [id=a5d580cffa2447d78fc9f2c8e31264f1]) successfully deleted
I20240430 20:11:23.112536 18148 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54524:
name: "default.Table_6d2fc9759c4741f780a8096a79261d75"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:23.113065 18148 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6d2fc9759c4741f780a8096a79261d75 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:23.114166 18602 tablet_service.cc:1514] Processing DeleteTablet for tablet a9e8a1052a2a42bf8dbb117c7a7b9c18 with delete_type TABLET_DATA_DELETED (replaced by tablet 26a015250eeb4750a42df93840e8d47f) from {username='slave'} at 127.0.0.1:40202
I20240430 20:11:23.114480 18980 ts_tablet_manager.cc:1888] T 0810eb04f1dd4d4bae41845453a623ab P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:23.114650 18980 log.cc:1196] T 0810eb04f1dd4d4bae41845453a623ab P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/0810eb04f1dd4d4bae41845453a623ab
I20240430 20:11:23.115092 18980 ts_tablet_manager.cc:1909] T 0810eb04f1dd4d4bae41845453a623ab P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:23.117796 18987 tablet_replica.cc:330] T 05b5a0b272ca41509c3291336079ef81 P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:23.118276 18987 raft_consensus.cc:2229] T 05b5a0b272ca41509c3291336079ef81 P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Raft consensus shutting down.
I20240430 20:11:23.119362 18980 tablet_replica.cc:330] T ef7a73d6307c4b099e967e7e70bf4f06 P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:23.119540 18980 raft_consensus.cc:2229] T ef7a73d6307c4b099e967e7e70bf4f06 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:23.119736 18980 raft_consensus.cc:2258] T ef7a73d6307c4b099e967e7e70bf4f06 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:23.121004 18981 ts_tablet_manager.cc:1888] T 0810eb04f1dd4d4bae41845453a623ab P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:23.121325 18987 raft_consensus.cc:2258] T 05b5a0b272ca41509c3291336079ef81 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:23.121801 18981 log.cc:1196] T 0810eb04f1dd4d4bae41845453a623ab P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/0810eb04f1dd4d4bae41845453a623ab
I20240430 20:11:23.121943 18980 ts_tablet_manager.cc:1875] T ef7a73d6307c4b099e967e7e70bf4f06 P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:23.122262 18154 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: Sending DeleteTablet for 3 replicas of tablet 3525f5a5051d4fb28cd9c95ca6ed6c48
I20240430 20:11:23.122469 18987 ts_tablet_manager.cc:1875] T 05b5a0b272ca41509c3291336079ef81 P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:23.122771 18154 catalog_manager.cc:6554] ReplaceTablet: tablet 3525f5a5051d4fb28cd9c95ca6ed6c48 deleted and replaced by tablet 9750608faeab4be38ddce4e2e4d46964
I20240430 20:11:23.123183 18740 tablet_service.cc:1514] Processing DeleteTablet for tablet 3525f5a5051d4fb28cd9c95ca6ed6c48 with delete_type TABLET_DATA_DELETED (replaced by tablet 9750608faeab4be38ddce4e2e4d46964) from {username='slave'} at 127.0.0.1:59818
I20240430 20:11:23.124147 18601 tablet_service.cc:1514] Processing DeleteTablet for tablet 3525f5a5051d4fb28cd9c95ca6ed6c48 with delete_type TABLET_DATA_DELETED (replaced by tablet 9750608faeab4be38ddce4e2e4d46964) from {username='slave'} at 127.0.0.1:40202
I20240430 20:11:23.123831 18459 tablet_service.cc:1514] Processing DeleteTablet for tablet 3525f5a5051d4fb28cd9c95ca6ed6c48 with delete_type TABLET_DATA_DELETED (replaced by tablet 9750608faeab4be38ddce4e2e4d46964) from {username='slave'} at 127.0.0.1:41678
I20240430 20:11:23.124980 18981 ts_tablet_manager.cc:1909] T 0810eb04f1dd4d4bae41845453a623ab P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:23.127429 18981 tablet_replica.cc:330] T ef7a73d6307c4b099e967e7e70bf4f06 P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:23.127599 18981 raft_consensus.cc:2229] T ef7a73d6307c4b099e967e7e70bf4f06 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Raft consensus shutting down.
I20240430 20:11:23.129787 18985 tablet_bootstrap.cc:492] T 27b21459592649dca1089a7a29021206 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:23.129947 18985 ts_tablet_manager.cc:1367] T 27b21459592649dca1089a7a29021206 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.115s user 0.010s sys 0.010s
I20240430 20:11:23.131470 18985 raft_consensus.cc:348] T 27b21459592649dca1089a7a29021206 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:23.131702 18985 raft_consensus.cc:374] T 27b21459592649dca1089a7a29021206 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:23.131778 18985 raft_consensus.cc:729] T 27b21459592649dca1089a7a29021206 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:23.132022 18985 consensus_queue.cc:260] T 27b21459592649dca1089a7a29021206 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:23.134162 18151 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54524:
name: "default.Table_23b82186d14a44eeb81cad606fb8e9cc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:23.134747 18151 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_23b82186d14a44eeb81cad606fb8e9cc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:23.135089 18985 ts_tablet_manager.cc:1398] T 27b21459592649dca1089a7a29021206 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240430 20:11:23.136179 18980 ts_tablet_manager.cc:1888] T ef7a73d6307c4b099e967e7e70bf4f06 P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:23.136301 18980 log.cc:1196] T ef7a73d6307c4b099e967e7e70bf4f06 P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/ef7a73d6307c4b099e967e7e70bf4f06
I20240430 20:11:23.136216 18154 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54524:
name: "default.Table_32647411b85344b490966d64f135631b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:23.136724 18154 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_32647411b85344b490966d64f135631b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:23.136785 18980 ts_tablet_manager.cc:1909] T ef7a73d6307c4b099e967e7e70bf4f06 P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:23.137667 18985 tablet_bootstrap.cc:492] T 44699c84ebac49ef8f3765d4069e71ca P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:23.138940 18981 raft_consensus.cc:2258] T ef7a73d6307c4b099e967e7e70bf4f06 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:23.140054 18981 ts_tablet_manager.cc:1875] T ef7a73d6307c4b099e967e7e70bf4f06 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:23.140666 18980 tablet_replica.cc:330] T a9e8a1052a2a42bf8dbb117c7a7b9c18 P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:23.140812 18139 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet ef7a73d6307c4b099e967e7e70bf4f06 (table default.Table_50a4a93d5e7b4ec5b84f5c758cde8e9c [id=820d7f76444f42178fdb54864a998903]) successfully deleted
I20240430 20:11:23.140861 18980 raft_consensus.cc:2229] T a9e8a1052a2a42bf8dbb117c7a7b9c18 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:23.141010 18980 raft_consensus.cc:2258] T a9e8a1052a2a42bf8dbb117c7a7b9c18 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:23.142691 18980 ts_tablet_manager.cc:1875] T a9e8a1052a2a42bf8dbb117c7a7b9c18 P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:23.143291 18985 tablet_bootstrap.cc:654] T 44699c84ebac49ef8f3765d4069e71ca P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:23.146914 18150 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: Sending DeleteTablet for 3 replicas of tablet ca95311f67c040f28b45356fcb284d82
I20240430 20:11:23.147495 18150 catalog_manager.cc:6554] ReplaceTablet: tablet ca95311f67c040f28b45356fcb284d82 deleted and replaced by tablet 5809e7c9c2204ba293426c24129fb16e
I20240430 20:11:23.148295 18462 tablet_service.cc:1514] Processing DeleteTablet for tablet ca95311f67c040f28b45356fcb284d82 with delete_type TABLET_DATA_DELETED (replaced by tablet 5809e7c9c2204ba293426c24129fb16e) from {username='slave'} at 127.0.0.1:41678
I20240430 20:11:23.148816 18146 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 17843ea77664447eaaa19666e0445ada (table default.Table_4fd51e528ae6409e8987a3be835b54cc [id=c0d8dc87d1f9426a85f6176f161595a1]). Expected version 1 got 0
I20240430 20:11:23.149109 18146 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 269144de63db417a9c9423a913d4536a (table default.Table_a235489d8a2248c9b257cf1bd72de770 [id=b93708ff15ef4c8299967c8c91045c8f]). Expected version 1 got 0
I20240430 20:11:23.149356 18146 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet f6e23412ad5b4773bb62590e452abc7a (table default.Table_d4b5e1a26a9449cdbe70f185b1097321 [id=ee9addb7eb9f4430a57fd9ee4bfaae7b]). Expected version 1 got 0
I20240430 20:11:23.149533 18146 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet e5d1707ba0b34cc4b1cda39e36ccbd2f (table default.Table_7fec253c8f4a4fb1a66ec3af88a66ba6 [id=a5d580cffa2447d78fc9f2c8e31264f1]). Expected version 3 got 0
I20240430 20:11:23.151546 18987 ts_tablet_manager.cc:1888] T 05b5a0b272ca41509c3291336079ef81 P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:23.151736 18987 log.cc:1196] T 05b5a0b272ca41509c3291336079ef81 P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/05b5a0b272ca41509c3291336079ef81
I20240430 20:11:23.152247 18987 ts_tablet_manager.cc:1909] T 05b5a0b272ca41509c3291336079ef81 P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:23.152657 18149 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 17843ea77664447eaaa19666e0445ada (table default.Table_4fd51e528ae6409e8987a3be835b54cc [id=c0d8dc87d1f9426a85f6176f161595a1]). Expected version 1 got 0
I20240430 20:11:23.152926 18149 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet f6e23412ad5b4773bb62590e452abc7a (table default.Table_d4b5e1a26a9449cdbe70f185b1097321 [id=ee9addb7eb9f4430a57fd9ee4bfaae7b]). Expected version 1 got 0
I20240430 20:11:23.153107 18149 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 269144de63db417a9c9423a913d4536a (table default.Table_a235489d8a2248c9b257cf1bd72de770 [id=b93708ff15ef4c8299967c8c91045c8f]). Expected version 1 got 0
I20240430 20:11:23.153612 18739 tablet_service.cc:1514] Processing DeleteTablet for tablet ca95311f67c040f28b45356fcb284d82 with delete_type TABLET_DATA_DELETED (replaced by tablet 5809e7c9c2204ba293426c24129fb16e) from {username='slave'} at 127.0.0.1:59818
I20240430 20:11:23.154502 18601 tablet_service.cc:1514] Processing DeleteTablet for tablet ca95311f67c040f28b45356fcb284d82 with delete_type TABLET_DATA_DELETED (replaced by tablet 5809e7c9c2204ba293426c24129fb16e) from {username='slave'} at 127.0.0.1:40202
I20240430 20:11:23.154770 18980 ts_tablet_manager.cc:1888] T a9e8a1052a2a42bf8dbb117c7a7b9c18 P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:23.154939 18980 log.cc:1196] T a9e8a1052a2a42bf8dbb117c7a7b9c18 P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/a9e8a1052a2a42bf8dbb117c7a7b9c18
I20240430 20:11:23.155274 18980 ts_tablet_manager.cc:1909] T a9e8a1052a2a42bf8dbb117c7a7b9c18 P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:23.155741 18981 ts_tablet_manager.cc:1888] T ef7a73d6307c4b099e967e7e70bf4f06 P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:23.155930 18981 log.cc:1196] T ef7a73d6307c4b099e967e7e70bf4f06 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/ef7a73d6307c4b099e967e7e70bf4f06
I20240430 20:11:23.156314 18139 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet a9e8a1052a2a42bf8dbb117c7a7b9c18 (table default.Table_2e9c7c683df84a41942d31600a265641 [id=aca6fb02776a40bbac312191adbdc136]) successfully deleted
I20240430 20:11:23.156507 18981 ts_tablet_manager.cc:1909] T ef7a73d6307c4b099e967e7e70bf4f06 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:23.156319 18149 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet e5d1707ba0b34cc4b1cda39e36ccbd2f (table default.Table_7fec253c8f4a4fb1a66ec3af88a66ba6 [id=a5d580cffa2447d78fc9f2c8e31264f1]). Expected version 3 got 0
I20240430 20:11:23.156966 18985 tablet_bootstrap.cc:492] T 44699c84ebac49ef8f3765d4069e71ca P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:23.157124 18985 ts_tablet_manager.cc:1367] T 44699c84ebac49ef8f3765d4069e71ca P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.004s
I20240430 20:11:23.157886 18142 catalog_manager.cc:4830] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): tablet ef7a73d6307c4b099e967e7e70bf4f06 (table default.Table_50a4a93d5e7b4ec5b84f5c758cde8e9c [id=820d7f76444f42178fdb54864a998903]) successfully deleted
I20240430 20:11:23.153429 18987 tablet_replica.cc:330] T ef7a73d6307c4b099e967e7e70bf4f06 P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:23.158334 18152 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: Sending DeleteTablet for 3 replicas of tablet 235ea182e02740f68f30c3c34f77012a
I20240430 20:11:23.158295 18985 raft_consensus.cc:348] T 44699c84ebac49ef8f3765d4069e71ca P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:23.158541 18985 raft_consensus.cc:374] T 44699c84ebac49ef8f3765d4069e71ca P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:23.158581 18987 raft_consensus.cc:2229] T ef7a73d6307c4b099e967e7e70bf4f06 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:23.158689 18985 raft_consensus.cc:729] T 44699c84ebac49ef8f3765d4069e71ca P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:23.158934 18987 raft_consensus.cc:2258] T ef7a73d6307c4b099e967e7e70bf4f06 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:23.158972 18985 consensus_queue.cc:260] T 44699c84ebac49ef8f3765d4069e71ca P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:23.159727 18739 tablet_service.cc:1514] Processing DeleteTablet for tablet 05b5a0b272ca41509c3291336079ef81 with delete_type TABLET_DATA_DELETED (replaced by tablet da1c17aa2b4d4cc4a50ff958b11ac6d4) from {username='slave'} at 127.0.0.1:59818
I20240430 20:11:23.160389 18147 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet e5d1707ba0b34cc4b1cda39e36ccbd2f (table default.Table_7fec253c8f4a4fb1a66ec3af88a66ba6 [id=a5d580cffa2447d78fc9f2c8e31264f1]). Expected version 3 got 0
I20240430 20:11:23.160527 18739 tablet_service.cc:1514] Processing DeleteTablet for tablet 0810eb04f1dd4d4bae41845453a623ab with delete_type TABLET_DATA_DELETED (replaced by tablet 727326b84b41443894245c8a1d0fed20) from {username='slave'} at 127.0.0.1:59818
I20240430 20:11:23.160580 18147 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 269144de63db417a9c9423a913d4536a (table default.Table_a235489d8a2248c9b257cf1bd72de770 [id=b93708ff15ef4c8299967c8c91045c8f]). Expected version 1 got 0
I20240430 20:11:23.160779 18985 ts_tablet_manager.cc:1398] T 44699c84ebac49ef8f3765d4069e71ca P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240430 20:11:23.160809 18147 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet f6e23412ad5b4773bb62590e452abc7a (table default.Table_d4b5e1a26a9449cdbe70f185b1097321 [id=ee9addb7eb9f4430a57fd9ee4bfaae7b]). Expected version 1 got 0
I20240430 20:11:23.161317 18981 tablet_replica.cc:330] T ca95311f67c040f28b45356fcb284d82 P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:23.161520 18739 tablet_service.cc:1514] Processing DeleteTablet for tablet e49de8fd5081451693734b6d91522298 with delete_type TABLET_DATA_DELETED (replaced by tablet e5d1707ba0b34cc4b1cda39e36ccbd2f) from {username='slave'} at 127.0.0.1:59818
I20240430 20:11:23.161567 18981 raft_consensus.cc:2229] T ca95311f67c040f28b45356fcb284d82 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:23.161742 18981 raft_consensus.cc:2258] T ca95311f67c040f28b45356fcb284d82 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:23.162086 18985 tablet_bootstrap.cc:492] T a9e8a1052a2a42bf8dbb117c7a7b9c18 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:23.162582 18981 ts_tablet_manager.cc:1875] T ca95311f67c040f28b45356fcb284d82 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:23.163671 18150 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54524:
name: "default.Table_fcab4e9721f7470abc1df37d83ea299f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:23.164261 18150 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fcab4e9721f7470abc1df37d83ea299f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240430 20:11:23.164379 18142 catalog_manager.cc:4810] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): delete failed for tablet a9e8a1052a2a42bf8dbb117c7a7b9c18 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a9e8a1052a2a42bf8dbb117c7a7b9c18 already in progress: creating tablet
W20240430 20:11:23.164644 18142 catalog_manager.cc:4810] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): delete failed for tablet 3525f5a5051d4fb28cd9c95ca6ed6c48 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3525f5a5051d4fb28cd9c95ca6ed6c48 already in progress: creating tablet
I20240430 20:11:23.164832 18459 tablet_service.cc:1514] Processing DeleteTablet for tablet e49de8fd5081451693734b6d91522298 with delete_type TABLET_DATA_DELETED (replaced by tablet e5d1707ba0b34cc4b1cda39e36ccbd2f) from {username='slave'} at 127.0.0.1:41678
I20240430 20:11:23.165135 18462 tablet_service.cc:1514] Processing DeleteTablet for tablet 235ea182e02740f68f30c3c34f77012a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:22 UTC) from {username='slave'} at 127.0.0.1:41678
I20240430 20:11:23.165454 18152 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_d4b5e1a26a9449cdbe70f185b1097321" } modify_external_catalogs: true
I20240430 20:11:23.165637 18152 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_d4b5e1a26a9449cdbe70f185b1097321" } modify_external_catalogs: true
I20240430 20:11:23.165725 18987 ts_tablet_manager.cc:1875] T ef7a73d6307c4b099e967e7e70bf4f06 P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:23.165777 18985 tablet_bootstrap.cc:654] T a9e8a1052a2a42bf8dbb117c7a7b9c18 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:23.165845 18155 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54524:
name: "default.Table_773acff5477a41ce8504fb7a866dbccc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:23.166375 18155 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_773acff5477a41ce8504fb7a866dbccc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:23.167735 18601 tablet_service.cc:1514] Processing DeleteTablet for tablet 235ea182e02740f68f30c3c34f77012a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:22 UTC) from {username='slave'} at 127.0.0.1:40202
I20240430 20:11:23.168491 18739 tablet_service.cc:1514] Processing DeleteTablet for tablet 235ea182e02740f68f30c3c34f77012a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:22 UTC) from {username='slave'} at 127.0.0.1:59818
I20240430 20:11:23.169231 18147 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54524:
name: "default.Table_359e5a94ea604fc287e536a75406c664"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240430 20:11:23.169489 18988 tablet_bootstrap.cc:492] T 44699c84ebac49ef8f3765d4069e71ca P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:23.169634 18988 ts_tablet_manager.cc:1367] T 44699c84ebac49ef8f3765d4069e71ca P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.153s user 0.020s sys 0.004s
W20240430 20:11:23.169708 18147 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_359e5a94ea604fc287e536a75406c664 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:23.170480 18988 raft_consensus.cc:348] T 44699c84ebac49ef8f3765d4069e71ca P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:23.170764 18988 raft_consensus.cc:374] T 44699c84ebac49ef8f3765d4069e71ca P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:23.170886 18988 raft_consensus.cc:729] T 44699c84ebac49ef8f3765d4069e71ca P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:23.171159 18988 consensus_queue.cc:260] T 44699c84ebac49ef8f3765d4069e71ca P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:23.171679 18988 ts_tablet_manager.cc:1398] T 44699c84ebac49ef8f3765d4069e71ca P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240430 20:11:23.171998 18988 tablet_bootstrap.cc:492] T 27b21459592649dca1089a7a29021206 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:23.172008 18602 tablet_service.cc:1514] Processing DeleteTablet for tablet 05b5a0b272ca41509c3291336079ef81 with delete_type TABLET_DATA_DELETED (replaced by tablet da1c17aa2b4d4cc4a50ff958b11ac6d4) from {username='slave'} at 127.0.0.1:40202
I20240430 20:11:23.172173 18601 tablet_service.cc:1514] Processing DeleteTablet for tablet a9e8a1052a2a42bf8dbb117c7a7b9c18 with delete_type TABLET_DATA_DELETED (replaced by tablet 26a015250eeb4750a42df93840e8d47f) from {username='slave'} at 127.0.0.1:40202
I20240430 20:11:23.172742 18603 tablet_service.cc:1514] Processing DeleteTablet for tablet 0810eb04f1dd4d4bae41845453a623ab with delete_type TABLET_DATA_DELETED (replaced by tablet 727326b84b41443894245c8a1d0fed20) from {username='slave'} at 127.0.0.1:40202
I20240430 20:11:23.172902 18985 tablet_bootstrap.cc:492] T a9e8a1052a2a42bf8dbb117c7a7b9c18 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:23.173054 18985 ts_tablet_manager.cc:1367] T a9e8a1052a2a42bf8dbb117c7a7b9c18 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20240430 20:11:23.173177 18601 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b6be4a072094c02b14bddfea18c8600 with delete_type TABLET_DATA_DELETED (replaced by tablet 3525f5a5051d4fb28cd9c95ca6ed6c48) from {username='slave'} at 127.0.0.1:40202
I20240430 20:11:23.173242 18146 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_5feaab34a5cd439abdc705b85c22313b" } modify_external_catalogs: true
I20240430 20:11:23.173465 18988 tablet_bootstrap.cc:654] T 27b21459592649dca1089a7a29021206 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:23.173445 18146 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_5feaab34a5cd439abdc705b85c22313b" } modify_external_catalogs: true
I20240430 20:11:23.173749 18985 raft_consensus.cc:348] T a9e8a1052a2a42bf8dbb117c7a7b9c18 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:23.173954 18985 raft_consensus.cc:374] T a9e8a1052a2a42bf8dbb117c7a7b9c18 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:23.174018 18985 raft_consensus.cc:729] T a9e8a1052a2a42bf8dbb117c7a7b9c18 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
W20240430 20:11:23.174669 18139 catalog_manager.cc:4810] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): delete failed for tablet 3525f5a5051d4fb28cd9c95ca6ed6c48 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3525f5a5051d4fb28cd9c95ca6ed6c48 already in progress: creating tablet
W20240430 20:11:23.175148 18139 catalog_manager.cc:4810] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): delete failed for tablet ca95311f67c040f28b45356fcb284d82 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ca95311f67c040f28b45356fcb284d82 already in progress: creating tablet
W20240430 20:11:23.175375 18139 catalog_manager.cc:4810] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): delete failed for tablet 235ea182e02740f68f30c3c34f77012a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 235ea182e02740f68f30c3c34f77012a already in progress: creating tablet
W20240430 20:11:23.175595 18139 catalog_manager.cc:4797] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): delete failed for tablet 05b5a0b272ca41509c3291336079ef81 because the tablet was not found. No further retry: Not found: Tablet not found: 05b5a0b272ca41509c3291336079ef81
I20240430 20:11:23.175679 18459 tablet_service.cc:1514] Processing DeleteTablet for tablet ef7a73d6307c4b099e967e7e70bf4f06 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:22 UTC) from {username='slave'} at 127.0.0.1:41678
I20240430 20:11:23.175912 18462 tablet_service.cc:1514] Processing DeleteTablet for tablet 0810eb04f1dd4d4bae41845453a623ab with delete_type TABLET_DATA_DELETED (replaced by tablet 727326b84b41443894245c8a1d0fed20) from {username='slave'} at 127.0.0.1:41678
I20240430 20:11:23.175959 18602 tablet_service.cc:1514] Processing DeleteTablet for tablet ef7a73d6307c4b099e967e7e70bf4f06 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:22 UTC) from {username='slave'} at 127.0.0.1:40202
W20240430 20:11:23.176429 18139 catalog_manager.cc:4797] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): delete failed for tablet a9e8a1052a2a42bf8dbb117c7a7b9c18 because the tablet was not found. No further retry: Not found: Tablet not found: a9e8a1052a2a42bf8dbb117c7a7b9c18
W20240430 20:11:23.176641 18139 catalog_manager.cc:4797] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): delete failed for tablet 0810eb04f1dd4d4bae41845453a623ab because the tablet was not found. No further retry: Not found: Tablet not found: 0810eb04f1dd4d4bae41845453a623ab
W20240430 20:11:23.176863 18139 catalog_manager.cc:4797] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): delete failed for tablet 2b6be4a072094c02b14bddfea18c8600 because the tablet was not found. No further retry: Not found: Tablet not found: 2b6be4a072094c02b14bddfea18c8600
W20240430 20:11:23.179793 18139 catalog_manager.cc:4797] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): delete failed for tablet ef7a73d6307c4b099e967e7e70bf4f06 because the tablet was not found. No further retry: Not found: Tablet not found: ef7a73d6307c4b099e967e7e70bf4f06
I20240430 20:11:23.174233 18985 consensus_queue.cc:260] T a9e8a1052a2a42bf8dbb117c7a7b9c18 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:23.181126 18983 tablet_bootstrap.cc:492] T ca95311f67c040f28b45356fcb284d82 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:23.181264 18983 ts_tablet_manager.cc:1367] T ca95311f67c040f28b45356fcb284d82 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.173s user 0.025s sys 0.026s
I20240430 20:11:23.182058 18983 raft_consensus.cc:348] T ca95311f67c040f28b45356fcb284d82 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:23.182271 18983 raft_consensus.cc:374] T ca95311f67c040f28b45356fcb284d82 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:23.182353 18983 raft_consensus.cc:729] T ca95311f67c040f28b45356fcb284d82 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:23.182574 18983 consensus_queue.cc:260] T ca95311f67c040f28b45356fcb284d82 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:23.183714 18985 ts_tablet_manager.cc:1398] T a9e8a1052a2a42bf8dbb117c7a7b9c18 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240430 20:11:23.185480 18985 tablet_bootstrap.cc:492] T e5d1707ba0b34cc4b1cda39e36ccbd2f P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:23.185487 18983 ts_tablet_manager.cc:1398] T ca95311f67c040f28b45356fcb284d82 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240430 20:11:23.185817 18983 tablet_bootstrap.cc:492] T 44699c84ebac49ef8f3765d4069e71ca P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:23.187641 18983 tablet_bootstrap.cc:654] T 44699c84ebac49ef8f3765d4069e71ca P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:23.188824 18987 ts_tablet_manager.cc:1888] T ef7a73d6307c4b099e967e7e70bf4f06 P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:23.188980 18987 log.cc:1196] T ef7a73d6307c4b099e967e7e70bf4f06 P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/ef7a73d6307c4b099e967e7e70bf4f06
I20240430 20:11:23.189380 18987 ts_tablet_manager.cc:1909] T ef7a73d6307c4b099e967e7e70bf4f06 P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:23.190318 18987 tablet_replica.cc:330] T a9e8a1052a2a42bf8dbb117c7a7b9c18 P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:23.190518 18987 raft_consensus.cc:2229] T a9e8a1052a2a42bf8dbb117c7a7b9c18 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:23.190572 18141 catalog_manager.cc:4830] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): tablet ef7a73d6307c4b099e967e7e70bf4f06 (table default.Table_50a4a93d5e7b4ec5b84f5c758cde8e9c [id=820d7f76444f42178fdb54864a998903]) successfully deleted
W20240430 20:11:23.192157 18156 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bf925791ddcf4fa7b5a1645c63f0d696 consensus state has no leader
I20240430 20:11:23.194583 18985 tablet_bootstrap.cc:654] T e5d1707ba0b34cc4b1cda39e36ccbd2f P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:23.196844 18981 ts_tablet_manager.cc:1888] T ca95311f67c040f28b45356fcb284d82 P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:23.196995 18981 log.cc:1196] T ca95311f67c040f28b45356fcb284d82 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/ca95311f67c040f28b45356fcb284d82
I20240430 20:11:23.197309 18981 ts_tablet_manager.cc:1909] T ca95311f67c040f28b45356fcb284d82 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:23.197912 18983 tablet_bootstrap.cc:492] T 44699c84ebac49ef8f3765d4069e71ca P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:23.198082 18983 ts_tablet_manager.cc:1367] T 44699c84ebac49ef8f3765d4069e71ca P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.000s
I20240430 20:11:23.198318 18142 catalog_manager.cc:4830] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): tablet ca95311f67c040f28b45356fcb284d82 (table default.Table_d82824c897a94dd0ba45955f9473b626 [id=d794640d21a843a185038673f7eb2b00]) successfully deleted
I20240430 20:11:23.194319 18988 tablet_bootstrap.cc:492] T 27b21459592649dca1089a7a29021206 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:23.198873 18988 ts_tablet_manager.cc:1367] T 27b21459592649dca1089a7a29021206 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.027s user 0.008s sys 0.000s
W20240430 20:11:23.198889 18142 catalog_manager.cc:4797] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): delete failed for tablet e49de8fd5081451693734b6d91522298 because the tablet was not found. No further retry: Not found: Tablet not found: e49de8fd5081451693734b6d91522298
I20240430 20:11:23.198793 18983 raft_consensus.cc:348] T 44699c84ebac49ef8f3765d4069e71ca P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:23.199002 18983 raft_consensus.cc:374] T 44699c84ebac49ef8f3765d4069e71ca P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:23.199060 18983 raft_consensus.cc:729] T 44699c84ebac49ef8f3765d4069e71ca P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
W20240430 20:11:23.199126 18142 catalog_manager.cc:4810] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): delete failed for tablet 235ea182e02740f68f30c3c34f77012a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 235ea182e02740f68f30c3c34f77012a already in progress: creating tablet
W20240430 20:11:23.199323 18142 catalog_manager.cc:4797] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): delete failed for tablet ef7a73d6307c4b099e967e7e70bf4f06 because the tablet was not found. No further retry: Not found: Tablet not found: ef7a73d6307c4b099e967e7e70bf4f06
I20240430 20:11:23.199283 18983 consensus_queue.cc:260] T 44699c84ebac49ef8f3765d4069e71ca P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
W20240430 20:11:23.199568 18142 catalog_manager.cc:4797] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): delete failed for tablet 0810eb04f1dd4d4bae41845453a623ab because the tablet was not found. No further retry: Not found: Tablet not found: 0810eb04f1dd4d4bae41845453a623ab
I20240430 20:11:23.199862 18983 ts_tablet_manager.cc:1398] T 44699c84ebac49ef8f3765d4069e71ca P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240430 20:11:23.200134 18983 tablet_bootstrap.cc:492] T 235ea182e02740f68f30c3c34f77012a P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:23.201633 18983 tablet_bootstrap.cc:654] T 235ea182e02740f68f30c3c34f77012a P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:23.201586 18988 raft_consensus.cc:348] T 27b21459592649dca1089a7a29021206 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:23.201812 18988 raft_consensus.cc:374] T 27b21459592649dca1089a7a29021206 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:23.201884 18988 raft_consensus.cc:729] T 27b21459592649dca1089a7a29021206 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:23.202147 18988 consensus_queue.cc:260] T 27b21459592649dca1089a7a29021206 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:23.202765 18988 ts_tablet_manager.cc:1398] T 27b21459592649dca1089a7a29021206 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240430 20:11:23.203040 18988 tablet_bootstrap.cc:492] T 17843ea77664447eaaa19666e0445ada P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:23.204841 18988 tablet_bootstrap.cc:654] T 17843ea77664447eaaa19666e0445ada P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:23.205758 18987 raft_consensus.cc:2258] T a9e8a1052a2a42bf8dbb117c7a7b9c18 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:23.206883 18987 ts_tablet_manager.cc:1875] T a9e8a1052a2a42bf8dbb117c7a7b9c18 P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:23.213982 18987 ts_tablet_manager.cc:1888] T a9e8a1052a2a42bf8dbb117c7a7b9c18 P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:23.214125 18987 log.cc:1196] T a9e8a1052a2a42bf8dbb117c7a7b9c18 P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/a9e8a1052a2a42bf8dbb117c7a7b9c18
I20240430 20:11:23.216462 18987 ts_tablet_manager.cc:1909] T a9e8a1052a2a42bf8dbb117c7a7b9c18 P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:23.219327 18141 catalog_manager.cc:4830] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): tablet a9e8a1052a2a42bf8dbb117c7a7b9c18 (table default.Table_2e9c7c683df84a41942d31600a265641 [id=aca6fb02776a40bbac312191adbdc136]) successfully deleted
W20240430 20:11:23.219733 18141 catalog_manager.cc:4810] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): delete failed for tablet 3525f5a5051d4fb28cd9c95ca6ed6c48 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3525f5a5051d4fb28cd9c95ca6ed6c48 already in progress: creating tablet
I20240430 20:11:23.220683 18987 tablet_replica.cc:330] T ca95311f67c040f28b45356fcb284d82 P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:23.220865 18987 raft_consensus.cc:2229] T ca95311f67c040f28b45356fcb284d82 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:23.221072 18987 raft_consensus.cc:2258] T ca95311f67c040f28b45356fcb284d82 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:23.221640 18985 tablet_bootstrap.cc:492] T e5d1707ba0b34cc4b1cda39e36ccbd2f P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:23.221798 18985 ts_tablet_manager.cc:1367] T e5d1707ba0b34cc4b1cda39e36ccbd2f P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.036s user 0.003s sys 0.006s
I20240430 20:11:23.221926 18151 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_b78b1b2c95694d1faee32321f845047b" } modify_external_catalogs: true
I20240430 20:11:23.222123 18151 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_b78b1b2c95694d1faee32321f845047b" } modify_external_catalogs: true
I20240430 20:11:23.222427 18987 ts_tablet_manager.cc:1875] T ca95311f67c040f28b45356fcb284d82 P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:23.223058 18985 raft_consensus.cc:348] T e5d1707ba0b34cc4b1cda39e36ccbd2f P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:23.223261 18985 raft_consensus.cc:374] T e5d1707ba0b34cc4b1cda39e36ccbd2f P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:23.223330 18985 raft_consensus.cc:729] T e5d1707ba0b34cc4b1cda39e36ccbd2f P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:23.223541 18985 consensus_queue.cc:260] T e5d1707ba0b34cc4b1cda39e36ccbd2f P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:23.224061 18985 ts_tablet_manager.cc:1398] T e5d1707ba0b34cc4b1cda39e36ccbd2f P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240430 20:11:23.226776 18983 tablet_bootstrap.cc:492] T 235ea182e02740f68f30c3c34f77012a P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:23.226920 18983 ts_tablet_manager.cc:1367] T 235ea182e02740f68f30c3c34f77012a P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.027s user 0.006s sys 0.004s
I20240430 20:11:23.230650 18154 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54524:
name: "default.Table_c21d1f6f0bfb4f9ea6978a36db7a3701"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:23.231186 18154 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c21d1f6f0bfb4f9ea6978a36db7a3701 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:23.233282 18988 tablet_bootstrap.cc:492] T 17843ea77664447eaaa19666e0445ada P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:23.233457 18988 ts_tablet_manager.cc:1367] T 17843ea77664447eaaa19666e0445ada P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.030s user 0.003s sys 0.009s
I20240430 20:11:23.234275 18988 raft_consensus.cc:348] T 17843ea77664447eaaa19666e0445ada P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:23.234474 18988 raft_consensus.cc:374] T 17843ea77664447eaaa19666e0445ada P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:23.234745 18988 raft_consensus.cc:729] T 17843ea77664447eaaa19666e0445ada P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:23.235018 18988 consensus_queue.cc:260] T 17843ea77664447eaaa19666e0445ada P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:23.235005 18983 raft_consensus.cc:348] T 235ea182e02740f68f30c3c34f77012a P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:23.235229 18983 raft_consensus.cc:374] T 235ea182e02740f68f30c3c34f77012a P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:23.235337 18983 raft_consensus.cc:729] T 235ea182e02740f68f30c3c34f77012a P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:23.236347 18988 ts_tablet_manager.cc:1398] T 17843ea77664447eaaa19666e0445ada P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240430 20:11:23.236495 18152 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: Sending DeleteTablet for 3 replicas of tablet f6e23412ad5b4773bb62590e452abc7a
I20240430 20:11:23.236640 18988 tablet_bootstrap.cc:492] T e5d1707ba0b34cc4b1cda39e36ccbd2f P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:23.237198 18152 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54524:
name: "default.Table_5ec7f4e4018b417f923420df89e9e25f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:23.237715 18152 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5ec7f4e4018b417f923420df89e9e25f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:23.237835 18462 tablet_service.cc:1514] Processing DeleteTablet for tablet f6e23412ad5b4773bb62590e452abc7a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:23 UTC) from {username='slave'} at 127.0.0.1:41678
I20240430 20:11:23.238617 18742 tablet_service.cc:1514] Processing DeleteTablet for tablet f6e23412ad5b4773bb62590e452abc7a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:23 UTC) from {username='slave'} at 127.0.0.1:59818
W20240430 20:11:23.238924 18142 catalog_manager.cc:4810] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): delete failed for tablet f6e23412ad5b4773bb62590e452abc7a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f6e23412ad5b4773bb62590e452abc7a already in progress: creating tablet
I20240430 20:11:23.239229 18988 tablet_bootstrap.cc:654] T e5d1707ba0b34cc4b1cda39e36ccbd2f P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:23.239167 18155 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54524:
name: "default.Table_52f64b8de0684649aa824da65de3416f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240430 20:11:23.239509 18934 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
W20240430 20:11:23.239658 18155 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_52f64b8de0684649aa824da65de3416f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:23.240046 18983 consensus_queue.cc:260] T 235ea182e02740f68f30c3c34f77012a P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:23.241442 18602 tablet_service.cc:1514] Processing DeleteTablet for tablet f6e23412ad5b4773bb62590e452abc7a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:23 UTC) from {username='slave'} at 127.0.0.1:40202
I20240430 20:11:23.242141 18983 ts_tablet_manager.cc:1398] T 235ea182e02740f68f30c3c34f77012a P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20240430 20:11:23.242456 18983 tablet_bootstrap.cc:492] T 269144de63db417a9c9423a913d4536a P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
W20240430 20:11:23.242499 18139 catalog_manager.cc:4810] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): delete failed for tablet f6e23412ad5b4773bb62590e452abc7a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f6e23412ad5b4773bb62590e452abc7a already in progress: creating tablet
I20240430 20:11:23.243162 18987 ts_tablet_manager.cc:1888] T ca95311f67c040f28b45356fcb284d82 P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:23.243284 18987 log.cc:1196] T ca95311f67c040f28b45356fcb284d82 P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/ca95311f67c040f28b45356fcb284d82
I20240430 20:11:23.243304 18931 raft_consensus.cc:482] T 235ea182e02740f68f30c3c34f77012a P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:23.243505 18931 raft_consensus.cc:504] T 235ea182e02740f68f30c3c34f77012a P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:23.243657 18987 ts_tablet_manager.cc:1909] T ca95311f67c040f28b45356fcb284d82 P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:23.244184 18931 leader_election.cc:288] T 235ea182e02740f68f30c3c34f77012a P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:23.245076 18141 catalog_manager.cc:4830] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): tablet ca95311f67c040f28b45356fcb284d82 (table default.Table_d82824c897a94dd0ba45955f9473b626 [id=d794640d21a843a185038673f7eb2b00]) successfully deleted
I20240430 20:11:23.245118 18763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "235ea182e02740f68f30c3c34f77012a" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
W20240430 20:11:23.245674 18141 catalog_manager.cc:4797] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): delete failed for tablet 05b5a0b272ca41509c3291336079ef81 because the tablet was not found. No further retry: Not found: Tablet not found: 05b5a0b272ca41509c3291336079ef81
I20240430 20:11:23.245759 18482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "235ea182e02740f68f30c3c34f77012a" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
W20240430 20:11:23.246430 18558 leader_election.cc:341] T 235ea182e02740f68f30c3c34f77012a P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
W20240430 20:11:23.246865 18557 leader_election.cc:341] T 235ea182e02740f68f30c3c34f77012a P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:23.247001 18557 leader_election.cc:302] T 235ea182e02740f68f30c3c34f77012a P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4
W20240430 20:11:23.247169 18141 catalog_manager.cc:4797] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): delete failed for tablet 0810eb04f1dd4d4bae41845453a623ab because the tablet was not found. No further retry: Not found: Tablet not found: 0810eb04f1dd4d4bae41845453a623ab
W20240430 20:11:23.247376 18141 catalog_manager.cc:4797] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): delete failed for tablet e49de8fd5081451693734b6d91522298 because the tablet was not found. No further retry: Not found: Tablet not found: e49de8fd5081451693734b6d91522298
I20240430 20:11:23.247540 18931 raft_consensus.cc:2726] T 235ea182e02740f68f30c3c34f77012a P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240430 20:11:23.247597 18141 catalog_manager.cc:4810] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): delete failed for tablet 235ea182e02740f68f30c3c34f77012a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 235ea182e02740f68f30c3c34f77012a already in progress: creating tablet
W20240430 20:11:23.247843 18141 catalog_manager.cc:4810] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): delete failed for tablet f6e23412ad5b4773bb62590e452abc7a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f6e23412ad5b4773bb62590e452abc7a already in progress: creating tablet
I20240430 20:11:23.248787 18983 tablet_bootstrap.cc:654] T 269144de63db417a9c9423a913d4536a P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:23.249711 18151 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: Sending DeleteTablet for 3 replicas of tablet f79497bf22ed417d988530f6e69cc07f
I20240430 20:11:23.250083 18147 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_25ea07a841144dab969f3baf7ee7c0dc" } modify_external_catalogs: true
I20240430 20:11:23.250253 18147 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_25ea07a841144dab969f3baf7ee7c0dc" } modify_external_catalogs: true
I20240430 20:11:23.250331 18151 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_c6fcbdec7e67466da00ab71d5404fe4e" } new_table_name: "default.Table_545cc69db7d64874861a9fa404789c0b" modify_external_catalogs: true
I20240430 20:11:23.250582 18934 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240430 20:11:23.250926 18934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240430 20:11:23.251178 18154 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_4fd51e528ae6409e8987a3be835b54cc" } new_table_name: "default.Table_c0737f9f801a45eea21f6204a6232be1" modify_external_catalogs: true
I20240430 20:11:23.253594 18988 tablet_bootstrap.cc:492] T e5d1707ba0b34cc4b1cda39e36ccbd2f P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:23.253767 18988 ts_tablet_manager.cc:1367] T e5d1707ba0b34cc4b1cda39e36ccbd2f P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.003s
I20240430 20:11:23.254321 18462 tablet_service.cc:1514] Processing DeleteTablet for tablet f79497bf22ed417d988530f6e69cc07f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:23 UTC) from {username='slave'} at 127.0.0.1:41678
I20240430 20:11:23.254528 18988 raft_consensus.cc:348] T e5d1707ba0b34cc4b1cda39e36ccbd2f P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:23.254858 18988 raft_consensus.cc:374] T e5d1707ba0b34cc4b1cda39e36ccbd2f P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:23.254971 18988 raft_consensus.cc:729] T e5d1707ba0b34cc4b1cda39e36ccbd2f P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:23.250331 18150 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_a235489d8a2248c9b257cf1bd72de770" } new_table_name: "default.Table_156650e4b399419c97dcf5160b761ee1" modify_external_catalogs: true
I20240430 20:11:23.255267 18602 tablet_service.cc:1514] Processing DeleteTablet for tablet f79497bf22ed417d988530f6e69cc07f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:23 UTC) from {username='slave'} at 127.0.0.1:40202
I20240430 20:11:23.255281 18988 consensus_queue.cc:260] T e5d1707ba0b34cc4b1cda39e36ccbd2f P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
W20240430 20:11:23.256040 18139 catalog_manager.cc:4810] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): delete failed for tablet f79497bf22ed417d988530f6e69cc07f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f79497bf22ed417d988530f6e69cc07f already in progress: creating tablet
I20240430 20:11:23.256450 18983 tablet_bootstrap.cc:492] T 269144de63db417a9c9423a913d4536a P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
W20240430 20:11:23.256472 18142 catalog_manager.cc:4810] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): delete failed for tablet f79497bf22ed417d988530f6e69cc07f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f79497bf22ed417d988530f6e69cc07f already in progress: creating tablet
I20240430 20:11:23.256613 18983 ts_tablet_manager.cc:1367] T 269144de63db417a9c9423a913d4536a P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.000s
I20240430 20:11:23.257027 18742 tablet_service.cc:1514] Processing DeleteTablet for tablet f79497bf22ed417d988530f6e69cc07f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:23 UTC) from {username='slave'} at 127.0.0.1:59818
I20240430 20:11:23.257311 18983 raft_consensus.cc:348] T 269144de63db417a9c9423a913d4536a P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:23.257515 18983 raft_consensus.cc:374] T 269144de63db417a9c9423a913d4536a P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:23.257583 18983 raft_consensus.cc:729] T 269144de63db417a9c9423a913d4536a P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:23.257849 18983 consensus_queue.cc:260] T 269144de63db417a9c9423a913d4536a P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:23.258250 18985 tablet_bootstrap.cc:492] T f6e23412ad5b4773bb62590e452abc7a P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
W20240430 20:11:23.258229 18141 catalog_manager.cc:4810] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): delete failed for tablet f79497bf22ed417d988530f6e69cc07f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f79497bf22ed417d988530f6e69cc07f already in progress: creating tablet
I20240430 20:11:23.260407 18983 ts_tablet_manager.cc:1398] T 269144de63db417a9c9423a913d4536a P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240430 20:11:23.261497 18924 raft_consensus.cc:482] T e5d1707ba0b34cc4b1cda39e36ccbd2f P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:23.261651 18924 raft_consensus.cc:504] T e5d1707ba0b34cc4b1cda39e36ccbd2f P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:23.262348 18924 leader_election.cc:288] T e5d1707ba0b34cc4b1cda39e36ccbd2f P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:23.263017 18482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5d1707ba0b34cc4b1cda39e36ccbd2f" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:23.263247 18482 raft_consensus.cc:2454] T e5d1707ba0b34cc4b1cda39e36ccbd2f P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:23.263667 18699 leader_election.cc:302] T e5d1707ba0b34cc4b1cda39e36ccbd2f P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters:
I20240430 20:11:23.263895 18985 tablet_bootstrap.cc:654] T f6e23412ad5b4773bb62590e452abc7a P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:23.262545 18983 tablet_bootstrap.cc:492] T 3525f5a5051d4fb28cd9c95ca6ed6c48 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:23.263952 18924 raft_consensus.cc:2781] T e5d1707ba0b34cc4b1cda39e36ccbd2f P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:23.264072 18924 raft_consensus.cc:482] T e5d1707ba0b34cc4b1cda39e36ccbd2f P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:23.264173 18924 raft_consensus.cc:3037] T e5d1707ba0b34cc4b1cda39e36ccbd2f P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:23.265599 18622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5d1707ba0b34cc4b1cda39e36ccbd2f" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:23.266022 18983 tablet_bootstrap.cc:654] T 3525f5a5051d4fb28cd9c95ca6ed6c48 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
W20240430 20:11:23.266393 18696 leader_election.cc:341] T e5d1707ba0b34cc4b1cda39e36ccbd2f P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:23.268257 18924 raft_consensus.cc:504] T e5d1707ba0b34cc4b1cda39e36ccbd2f P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:23.269528 18985 tablet_bootstrap.cc:492] T f6e23412ad5b4773bb62590e452abc7a P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:23.269459 18482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5d1707ba0b34cc4b1cda39e36ccbd2f" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:23.269681 18482 raft_consensus.cc:3037] T e5d1707ba0b34cc4b1cda39e36ccbd2f P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:23.269712 18985 ts_tablet_manager.cc:1367] T f6e23412ad5b4773bb62590e452abc7a P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.000s
I20240430 20:11:23.269662 18622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5d1707ba0b34cc4b1cda39e36ccbd2f" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:23.270036 18155 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_7fec253c8f4a4fb1a66ec3af88a66ba6" } new_table_name: "default.Table_0ab5b2f09fa7468bb89ab703f6d3d25c" modify_external_catalogs: true
W20240430 20:11:23.270251 18696 leader_election.cc:341] T e5d1707ba0b34cc4b1cda39e36ccbd2f P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:23.270517 18924 leader_election.cc:288] T e5d1707ba0b34cc4b1cda39e36ccbd2f P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Requested vote from peers 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:23.263940 18988 ts_tablet_manager.cc:1398] T e5d1707ba0b34cc4b1cda39e36ccbd2f P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.010s user 0.003s sys 0.000s
W20240430 20:11:23.271452 18151 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4d3117a9a5f243b59a69d960c0a1eee4 consensus state has no leader
W20240430 20:11:23.271858 18154 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 17843ea77664447eaaa19666e0445ada consensus state has no leader
I20240430 20:11:23.271081 18985 raft_consensus.cc:348] T f6e23412ad5b4773bb62590e452abc7a P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:23.272486 18482 raft_consensus.cc:2454] T e5d1707ba0b34cc4b1cda39e36ccbd2f P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:23.272537 18985 raft_consensus.cc:374] T f6e23412ad5b4773bb62590e452abc7a P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:23.272624 18985 raft_consensus.cc:729] T f6e23412ad5b4773bb62590e452abc7a P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:23.272868 18985 consensus_queue.cc:260] T f6e23412ad5b4773bb62590e452abc7a P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:23.273082 18699 leader_election.cc:302] T e5d1707ba0b34cc4b1cda39e36ccbd2f P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters: 5e1c6397f09747f091534faf93542710
I20240430 20:11:23.273528 18924 raft_consensus.cc:2781] T e5d1707ba0b34cc4b1cda39e36ccbd2f P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:23.273882 18988 tablet_bootstrap.cc:492] T f6e23412ad5b4773bb62590e452abc7a P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:23.275245 18985 ts_tablet_manager.cc:1398] T f6e23412ad5b4773bb62590e452abc7a P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240430 20:11:23.276343 18988 tablet_bootstrap.cc:654] T f6e23412ad5b4773bb62590e452abc7a P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:23.277729 18985 tablet_bootstrap.cc:492] T 3525f5a5051d4fb28cd9c95ca6ed6c48 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:23.278637 18924 raft_consensus.cc:686] T e5d1707ba0b34cc4b1cda39e36ccbd2f P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Becoming Leader. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Running, Role: LEADER
I20240430 20:11:23.279107 18924 consensus_queue.cc:237] T e5d1707ba0b34cc4b1cda39e36ccbd2f P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:23.282131 18985 tablet_bootstrap.cc:654] T 3525f5a5051d4fb28cd9c95ca6ed6c48 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
W20240430 20:11:23.287328 18150 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 269144de63db417a9c9423a913d4536a consensus state has no leader
I20240430 20:11:23.288156 18988 tablet_bootstrap.cc:492] T f6e23412ad5b4773bb62590e452abc7a P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:23.288327 18988 ts_tablet_manager.cc:1367] T f6e23412ad5b4773bb62590e452abc7a P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.000s
I20240430 20:11:23.289547 18988 raft_consensus.cc:348] T f6e23412ad5b4773bb62590e452abc7a P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:23.289764 18988 raft_consensus.cc:374] T f6e23412ad5b4773bb62590e452abc7a P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:23.289867 18988 raft_consensus.cc:729] T f6e23412ad5b4773bb62590e452abc7a P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:23.290237 18985 tablet_bootstrap.cc:492] T 3525f5a5051d4fb28cd9c95ca6ed6c48 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:23.290145 18988 consensus_queue.cc:260] T f6e23412ad5b4773bb62590e452abc7a P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:23.290385 18985 ts_tablet_manager.cc:1367] T 3525f5a5051d4fb28cd9c95ca6ed6c48 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.002s
I20240430 20:11:23.292213 18462 tablet_service.cc:1467] Processing CreateTablet for tablet e6c0c808bfc94be3837ab835292b3c1b (DEFAULT_TABLE table=default.Table_8502447753b04761815bf9602fd28e37 [id=6ca223ca6354449c959d85ed2ac6a9c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:23.292732 18462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6c0c808bfc94be3837ab835292b3c1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:23.293587 18988 ts_tablet_manager.cc:1398] T f6e23412ad5b4773bb62590e452abc7a P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.005s user 0.002s sys 0.001s
I20240430 20:11:23.294776 18988 tablet_bootstrap.cc:492] T 3525f5a5051d4fb28cd9c95ca6ed6c48 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:23.295534 18602 tablet_service.cc:1467] Processing CreateTablet for tablet e6c0c808bfc94be3837ab835292b3c1b (DEFAULT_TABLE table=default.Table_8502447753b04761815bf9602fd28e37 [id=6ca223ca6354449c959d85ed2ac6a9c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:23.296082 18602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6c0c808bfc94be3837ab835292b3c1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:23.296492 18988 tablet_bootstrap.cc:654] T 3525f5a5051d4fb28cd9c95ca6ed6c48 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:23.297343 18924 raft_consensus.cc:482] T f6e23412ad5b4773bb62590e452abc7a P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:23.297554 18924 raft_consensus.cc:504] T f6e23412ad5b4773bb62590e452abc7a P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:23.298255 18924 leader_election.cc:288] T f6e23412ad5b4773bb62590e452abc7a P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:23.299029 18601 tablet_service.cc:1467] Processing CreateTablet for tablet 9750608faeab4be38ddce4e2e4d46964 (DEFAULT_TABLE table=default.Table_c06b54647f004ff7a7f7beafee702b4c [id=a6a27dcbd0e648a8bf29a8cd5b3a2a11]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:23.299207 18459 tablet_service.cc:1467] Processing CreateTablet for tablet 9750608faeab4be38ddce4e2e4d46964 (DEFAULT_TABLE table=default.Table_c06b54647f004ff7a7f7beafee702b4c [id=a6a27dcbd0e648a8bf29a8cd5b3a2a11]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:23.299330 18482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6e23412ad5b4773bb62590e452abc7a" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:23.299562 18601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9750608faeab4be38ddce4e2e4d46964. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:23.299557 18482 raft_consensus.cc:2454] T f6e23412ad5b4773bb62590e452abc7a P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:23.299625 18459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9750608faeab4be38ddce4e2e4d46964. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:23.300035 18622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6e23412ad5b4773bb62590e452abc7a" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:23.300628 18699 leader_election.cc:302] T f6e23412ad5b4773bb62590e452abc7a P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters:
W20240430 20:11:23.301105 18696 leader_election.cc:341] T f6e23412ad5b4773bb62590e452abc7a P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:23.301478 18924 raft_consensus.cc:2781] T f6e23412ad5b4773bb62590e452abc7a P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:23.301571 18924 raft_consensus.cc:482] T f6e23412ad5b4773bb62590e452abc7a P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:23.301674 18924 raft_consensus.cc:3037] T f6e23412ad5b4773bb62590e452abc7a P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:23.302466 18463 tablet_service.cc:1467] Processing CreateTablet for tablet 26a015250eeb4750a42df93840e8d47f (DEFAULT_TABLE table=default.Table_2e9c7c683df84a41942d31600a265641 [id=aca6fb02776a40bbac312191adbdc136]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:23.302857 18463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26a015250eeb4750a42df93840e8d47f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:23.302980 18603 tablet_service.cc:1467] Processing CreateTablet for tablet 26a015250eeb4750a42df93840e8d47f (DEFAULT_TABLE table=default.Table_2e9c7c683df84a41942d31600a265641 [id=aca6fb02776a40bbac312191adbdc136]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:23.303483 18603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26a015250eeb4750a42df93840e8d47f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:23.304409 18924 raft_consensus.cc:504] T f6e23412ad5b4773bb62590e452abc7a P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:23.305469 18482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6e23412ad5b4773bb62590e452abc7a" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:23.305671 18482 raft_consensus.cc:3037] T f6e23412ad5b4773bb62590e452abc7a P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:23.309092 18742 tablet_service.cc:1467] Processing CreateTablet for tablet e6c0c808bfc94be3837ab835292b3c1b (DEFAULT_TABLE table=default.Table_8502447753b04761815bf9602fd28e37 [id=6ca223ca6354449c959d85ed2ac6a9c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:23.309381 18740 tablet_service.cc:1467] Processing CreateTablet for tablet 26a015250eeb4750a42df93840e8d47f (DEFAULT_TABLE table=default.Table_2e9c7c683df84a41942d31600a265641 [id=aca6fb02776a40bbac312191adbdc136]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:23.309648 18742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6c0c808bfc94be3837ab835292b3c1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:23.309778 18622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6e23412ad5b4773bb62590e452abc7a" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:23.309978 18482 raft_consensus.cc:2454] T f6e23412ad5b4773bb62590e452abc7a P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:23.310407 18699 leader_election.cc:302] T f6e23412ad5b4773bb62590e452abc7a P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters:
I20240430 20:11:23.310608 18924 leader_election.cc:288] T f6e23412ad5b4773bb62590e452abc7a P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Requested vote from peers 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
W20240430 20:11:23.310684 18696 leader_election.cc:341] T f6e23412ad5b4773bb62590e452abc7a P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:23.310788 18924 raft_consensus.cc:2781] T f6e23412ad5b4773bb62590e452abc7a P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:23.310915 18924 raft_consensus.cc:686] T f6e23412ad5b4773bb62590e452abc7a P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Becoming Leader. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Running, Role: LEADER
I20240430 20:11:23.311163 18924 consensus_queue.cc:237] T f6e23412ad5b4773bb62590e452abc7a P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:23.311538 18740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26a015250eeb4750a42df93840e8d47f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:23.314991 18739 tablet_service.cc:1467] Processing CreateTablet for tablet 9750608faeab4be38ddce4e2e4d46964 (DEFAULT_TABLE table=default.Table_c06b54647f004ff7a7f7beafee702b4c [id=a6a27dcbd0e648a8bf29a8cd5b3a2a11]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:23.315493 18739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9750608faeab4be38ddce4e2e4d46964. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:23.317404 18985 raft_consensus.cc:348] T 3525f5a5051d4fb28cd9c95ca6ed6c48 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:23.317605 18985 raft_consensus.cc:374] T 3525f5a5051d4fb28cd9c95ca6ed6c48 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:23.317685 18985 raft_consensus.cc:729] T 3525f5a5051d4fb28cd9c95ca6ed6c48 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:23.317929 18985 consensus_queue.cc:260] T 3525f5a5051d4fb28cd9c95ca6ed6c48 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:23.321352 18985 ts_tablet_manager.cc:1398] T 3525f5a5051d4fb28cd9c95ca6ed6c48 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.031s user 0.002s sys 0.000s
I20240430 20:11:23.322779 18985 tablet_bootstrap.cc:492] T 269144de63db417a9c9423a913d4536a P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:23.324630 18985 tablet_bootstrap.cc:654] T 269144de63db417a9c9423a913d4536a P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:23.327379 18985 tablet_bootstrap.cc:492] T 269144de63db417a9c9423a913d4536a P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:23.327567 18985 ts_tablet_manager.cc:1367] T 269144de63db417a9c9423a913d4536a P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240430 20:11:23.329994 18985 raft_consensus.cc:348] T 269144de63db417a9c9423a913d4536a P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:23.330242 18985 raft_consensus.cc:374] T 269144de63db417a9c9423a913d4536a P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:23.330320 18985 raft_consensus.cc:729] T 269144de63db417a9c9423a913d4536a P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:23.330565 18985 consensus_queue.cc:260] T 269144de63db417a9c9423a913d4536a P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:23.333237 18985 ts_tablet_manager.cc:1398] T 269144de63db417a9c9423a913d4536a P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240430 20:11:23.333257 18599 tablet_service.cc:1514] Processing DeleteTablet for tablet ca95311f67c040f28b45356fcb284d82 with delete_type TABLET_DATA_DELETED (replaced by tablet 5809e7c9c2204ba293426c24129fb16e) from {username='slave'} at 127.0.0.1:40202
I20240430 20:11:23.334777 18980 tablet_replica.cc:330] T ca95311f67c040f28b45356fcb284d82 P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:23.334884 18150 master_service.cc:946] ReplaceTablet: received request to replace tablet 6f729a6e8dc14de9a2057cfe09ba9257 from {username='slave'} at 127.0.0.1:54524
I20240430 20:11:23.334981 18980 raft_consensus.cc:2229] T ca95311f67c040f28b45356fcb284d82 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:23.335209 18980 raft_consensus.cc:2258] T ca95311f67c040f28b45356fcb284d82 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:23.336760 18985 tablet_bootstrap.cc:492] T 17843ea77664447eaaa19666e0445ada P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
W20240430 20:11:23.337106 18155 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e5d1707ba0b34cc4b1cda39e36ccbd2f consensus state has no leader
I20240430 20:11:23.337129 18980 ts_tablet_manager.cc:1875] T ca95311f67c040f28b45356fcb284d82 P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:23.338323 18985 tablet_bootstrap.cc:654] T 17843ea77664447eaaa19666e0445ada P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:23.339620 18152 master_service.cc:946] ReplaceTablet: received request to replace tablet ca95311f67c040f28b45356fcb284d82 from {username='slave'} at 127.0.0.1:54524
I20240430 20:11:23.340507 18155 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54524:
name: "default.Table_5b83502eb2f24b9389c3d08006449aab"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:23.341243 18155 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5b83502eb2f24b9389c3d08006449aab in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:23.348085 18980 ts_tablet_manager.cc:1888] T ca95311f67c040f28b45356fcb284d82 P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:23.349063 18980 log.cc:1196] T ca95311f67c040f28b45356fcb284d82 P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/ca95311f67c040f28b45356fcb284d82
I20240430 20:11:23.349622 18980 ts_tablet_manager.cc:1909] T ca95311f67c040f28b45356fcb284d82 P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:23.352816 18924 raft_consensus.cc:482] T 27b21459592649dca1089a7a29021206 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:23.353130 19015 raft_consensus.cc:482] T 44699c84ebac49ef8f3765d4069e71ca P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:23.355249 19015 raft_consensus.cc:504] T 44699c84ebac49ef8f3765d4069e71ca P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:23.356359 19015 leader_election.cc:288] T 44699c84ebac49ef8f3765d4069e71ca P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:23.356405 18139 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet ca95311f67c040f28b45356fcb284d82 (table default.Table_d82824c897a94dd0ba45955f9473b626 [id=d794640d21a843a185038673f7eb2b00]) successfully deleted
I20240430 20:11:23.357087 18152 master_service.cc:946] ReplaceTablet: received request to replace tablet 269144de63db417a9c9423a913d4536a from {username='slave'} at 127.0.0.1:54524
I20240430 20:11:23.357589 18482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44699c84ebac49ef8f3765d4069e71ca" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:23.357872 18482 raft_consensus.cc:2454] T 44699c84ebac49ef8f3765d4069e71ca P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:23.358338 18622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44699c84ebac49ef8f3765d4069e71ca" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:23.358654 18622 raft_consensus.cc:2454] T 44699c84ebac49ef8f3765d4069e71ca P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:23.359350 18696 leader_election.cc:302] T 44699c84ebac49ef8f3765d4069e71ca P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:23.360543 19015 raft_consensus.cc:2781] T 44699c84ebac49ef8f3765d4069e71ca P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:23.360694 19015 raft_consensus.cc:482] T 44699c84ebac49ef8f3765d4069e71ca P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:23.360800 19015 raft_consensus.cc:3037] T 44699c84ebac49ef8f3765d4069e71ca P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:23.365851 18924 raft_consensus.cc:504] T 27b21459592649dca1089a7a29021206 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:23.366896 18155 master_service.cc:946] ReplaceTablet: received request to replace tablet 235ea182e02740f68f30c3c34f77012a from {username='slave'} at 127.0.0.1:54524
I20240430 20:11:23.367671 18482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27b21459592649dca1089a7a29021206" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:23.367921 18482 raft_consensus.cc:2454] T 27b21459592649dca1089a7a29021206 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:23.368356 18985 tablet_bootstrap.cc:492] T 17843ea77664447eaaa19666e0445ada P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:23.368383 18699 leader_election.cc:302] T 27b21459592649dca1089a7a29021206 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters:
I20240430 20:11:23.368511 18985 ts_tablet_manager.cc:1367] T 17843ea77664447eaaa19666e0445ada P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.032s user 0.016s sys 0.014s
I20240430 20:11:23.364778 19015 raft_consensus.cc:504] T 44699c84ebac49ef8f3765d4069e71ca P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:23.369365 18985 raft_consensus.cc:348] T 17843ea77664447eaaa19666e0445ada P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:23.369572 18985 raft_consensus.cc:374] T 17843ea77664447eaaa19666e0445ada P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:23.369663 18985 raft_consensus.cc:729] T 17843ea77664447eaaa19666e0445ada P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:23.369948 18622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27b21459592649dca1089a7a29021206" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:23.369935 18985 consensus_queue.cc:260] T 17843ea77664447eaaa19666e0445ada P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:23.370162 18622 raft_consensus.cc:2454] T 27b21459592649dca1089a7a29021206 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:23.371374 18154 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_32647411b85344b490966d64f135631b" } modify_external_catalogs: true
I20240430 20:11:23.371593 18154 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_32647411b85344b490966d64f135631b" } modify_external_catalogs: true
I20240430 20:11:23.371531 18482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44699c84ebac49ef8f3765d4069e71ca" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:23.371757 18482 raft_consensus.cc:3037] T 44699c84ebac49ef8f3765d4069e71ca P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
W20240430 20:11:23.372125 18139 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.177.60:37225: connect: Connection refused (error 111) [suppressed 11 similar messages]
I20240430 20:11:23.372834 18985 ts_tablet_manager.cc:1398] T 17843ea77664447eaaa19666e0445ada P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20240430 20:11:23.374835 19030 raft_consensus.cc:2781] T 27b21459592649dca1089a7a29021206 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:23.374577 18482 raft_consensus.cc:2454] T 44699c84ebac49ef8f3765d4069e71ca P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:23.375401 18985 tablet_bootstrap.cc:492] T 235ea182e02740f68f30c3c34f77012a P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:23.377460 18985 tablet_bootstrap.cc:654] T 235ea182e02740f68f30c3c34f77012a P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:23.378561 18151 master_service.cc:946] ReplaceTablet: received request to replace tablet 4d3117a9a5f243b59a69d960c0a1eee4 from {username='slave'} at 127.0.0.1:54524
I20240430 20:11:23.379143 18983 tablet_bootstrap.cc:492] T 3525f5a5051d4fb28cd9c95ca6ed6c48 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:23.379202 18699 leader_election.cc:302] T 44699c84ebac49ef8f3765d4069e71ca P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters:
I20240430 20:11:23.379297 18983 ts_tablet_manager.cc:1367] T 3525f5a5051d4fb28cd9c95ca6ed6c48 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.117s user 0.001s sys 0.003s
I20240430 20:11:23.380137 18983 raft_consensus.cc:348] T 3525f5a5051d4fb28cd9c95ca6ed6c48 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:23.380415 18983 raft_consensus.cc:374] T 3525f5a5051d4fb28cd9c95ca6ed6c48 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:23.380508 18983 raft_consensus.cc:729] T 3525f5a5051d4fb28cd9c95ca6ed6c48 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:23.380774 18983 consensus_queue.cc:260] T 3525f5a5051d4fb28cd9c95ca6ed6c48 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:23.381059 18924 leader_election.cc:288] T 27b21459592649dca1089a7a29021206 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:23.381328 18983 ts_tablet_manager.cc:1398] T 3525f5a5051d4fb28cd9c95ca6ed6c48 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240430 20:11:23.382824 18924 raft_consensus.cc:2781] T 44699c84ebac49ef8f3765d4069e71ca P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:23.383769 18622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44699c84ebac49ef8f3765d4069e71ca" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:23.383953 18622 raft_consensus.cc:3037] T 44699c84ebac49ef8f3765d4069e71ca P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:23.387347 18983 tablet_bootstrap.cc:492] T f6e23412ad5b4773bb62590e452abc7a P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:23.386576 18622 raft_consensus.cc:2454] T 44699c84ebac49ef8f3765d4069e71ca P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:23.388991 18983 tablet_bootstrap.cc:654] T f6e23412ad5b4773bb62590e452abc7a P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:23.389446 18924 raft_consensus.cc:686] T 44699c84ebac49ef8f3765d4069e71ca P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Becoming Leader. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Running, Role: LEADER
I20240430 20:11:23.390098 18924 consensus_queue.cc:237] T 44699c84ebac49ef8f3765d4069e71ca P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:23.389636 19015 leader_election.cc:288] T 44699c84ebac49ef8f3765d4069e71ca P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Requested vote from peers 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:23.398321 18988 tablet_bootstrap.cc:492] T 3525f5a5051d4fb28cd9c95ca6ed6c48 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:23.398578 18988 ts_tablet_manager.cc:1367] T 3525f5a5051d4fb28cd9c95ca6ed6c48 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.104s user 0.004s sys 0.004s
I20240430 20:11:23.399447 18988 raft_consensus.cc:348] T 3525f5a5051d4fb28cd9c95ca6ed6c48 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:23.399793 18988 raft_consensus.cc:374] T 3525f5a5051d4fb28cd9c95ca6ed6c48 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:23.399925 18988 raft_consensus.cc:729] T 3525f5a5051d4fb28cd9c95ca6ed6c48 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:23.375067 19030 raft_consensus.cc:482] T 27b21459592649dca1089a7a29021206 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:23.400228 19030 raft_consensus.cc:3037] T 27b21459592649dca1089a7a29021206 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:23.401631 18988 consensus_queue.cc:260] T 3525f5a5051d4fb28cd9c95ca6ed6c48 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:23.402228 18988 ts_tablet_manager.cc:1398] T 3525f5a5051d4fb28cd9c95ca6ed6c48 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240430 20:11:23.402257 18156 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_773acff5477a41ce8504fb7a866dbccc" } new_table_name: "default.Table_3c5df662747b48bf8beb6ebe4d2ea618" modify_external_catalogs: true
I20240430 20:11:23.403820 18148 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 269144de63db417a9c9423a913d4536a (table default.Table_156650e4b399419c97dcf5160b761ee1 [id=b93708ff15ef4c8299967c8c91045c8f]). Expected version 2 got 0
I20240430 20:11:23.404728 18155 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54524:
name: "default.Table_ffddf7fa447948b58de34494af4ed0fd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:23.405269 18155 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ffddf7fa447948b58de34494af4ed0fd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:23.409293 18602 tablet_service.cc:1514] Processing DeleteTablet for tablet ca95311f67c040f28b45356fcb284d82 with delete_type TABLET_DATA_DELETED (replaced by tablet 5809e7c9c2204ba293426c24129fb16e) from {username='slave'} at 127.0.0.1:40202
W20240430 20:11:23.410290 18139 catalog_manager.cc:4797] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): delete failed for tablet ca95311f67c040f28b45356fcb284d82 because the tablet was not found. No further retry: Not found: Tablet not found: ca95311f67c040f28b45356fcb284d82
I20240430 20:11:23.412104 18988 tablet_bootstrap.cc:492] T 269144de63db417a9c9423a913d4536a P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:23.413188 18147 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: Sending DeleteTablet for 3 replicas of tablet 2067cb2cf76d495c92c1f25c176eb6dc
I20240430 20:11:23.414322 18742 tablet_service.cc:1514] Processing DeleteTablet for tablet 2067cb2cf76d495c92c1f25c176eb6dc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:23 UTC) from {username='slave'} at 127.0.0.1:59818
I20240430 20:11:23.414332 18462 tablet_service.cc:1514] Processing DeleteTablet for tablet 2067cb2cf76d495c92c1f25c176eb6dc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:23 UTC) from {username='slave'} at 127.0.0.1:41678
I20240430 20:11:23.414767 18602 tablet_service.cc:1514] Processing DeleteTablet for tablet 2067cb2cf76d495c92c1f25c176eb6dc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:23 UTC) from {username='slave'} at 127.0.0.1:40202
W20240430 20:11:23.415163 18142 catalog_manager.cc:4810] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): delete failed for tablet 2067cb2cf76d495c92c1f25c176eb6dc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2067cb2cf76d495c92c1f25c176eb6dc already in progress: creating tablet
W20240430 20:11:23.415585 18139 catalog_manager.cc:4810] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): delete failed for tablet 2067cb2cf76d495c92c1f25c176eb6dc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2067cb2cf76d495c92c1f25c176eb6dc already in progress: creating tablet
W20240430 20:11:23.416468 18141 catalog_manager.cc:4810] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): delete failed for tablet 2067cb2cf76d495c92c1f25c176eb6dc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2067cb2cf76d495c92c1f25c176eb6dc already in progress: creating tablet
I20240430 20:11:23.417702 18602 tablet_service.cc:1514] Processing DeleteTablet for tablet 3525f5a5051d4fb28cd9c95ca6ed6c48 with delete_type TABLET_DATA_DELETED (replaced by tablet 9750608faeab4be38ddce4e2e4d46964) from {username='slave'} at 127.0.0.1:40202
I20240430 20:11:23.417973 18980 tablet_replica.cc:330] T 3525f5a5051d4fb28cd9c95ca6ed6c48 P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:23.418180 18980 raft_consensus.cc:2229] T 3525f5a5051d4fb28cd9c95ca6ed6c48 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:23.418378 18980 raft_consensus.cc:2258] T 3525f5a5051d4fb28cd9c95ca6ed6c48 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:23.419131 18980 ts_tablet_manager.cc:1875] T 3525f5a5051d4fb28cd9c95ca6ed6c48 P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:23.420023 18601 tablet_service.cc:1514] Processing DeleteTablet for tablet 235ea182e02740f68f30c3c34f77012a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:22 UTC) from {username='slave'} at 127.0.0.1:40202
W20240430 20:11:23.422842 18156 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0d6a4a63c004480b82d84381867bf8a4 consensus state has no leader
I20240430 20:11:23.425474 18156 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54524:
name: "default.Table_0af8d509e51a47c49580fa1d6cbee72f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:23.426188 18156 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0af8d509e51a47c49580fa1d6cbee72f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:23.429183 18601 tablet_service.cc:1467] Processing CreateTablet for tablet c3f8f9ada98b40bc8359a00e193c3e5f (DEFAULT_TABLE table=default.Table_c21d1f6f0bfb4f9ea6978a36db7a3701 [id=70551d80c9d8445fa158eee362758714]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:23.429631 18601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3f8f9ada98b40bc8359a00e193c3e5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:23.429637 18152 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: Sending DeleteTablet for 3 replicas of tablet 269144de63db417a9c9423a913d4536a
I20240430 20:11:23.430115 18152 catalog_manager.cc:6554] ReplaceTablet: tablet 269144de63db417a9c9423a913d4536a deleted and replaced by tablet 27f94f2d8f50495dababe417a9f82982
I20240430 20:11:23.431456 18149 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet e5d1707ba0b34cc4b1cda39e36ccbd2f (table default.Table_0ab5b2f09fa7468bb89ab703f6d3d25c [id=a5d580cffa2447d78fc9f2c8e31264f1]). Expected version 4 got 0
I20240430 20:11:23.431730 18149 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 17843ea77664447eaaa19666e0445ada (table default.Table_c0737f9f801a45eea21f6204a6232be1 [id=c0d8dc87d1f9426a85f6176f161595a1]). Expected version 2 got 0
I20240430 20:11:23.432165 18462 tablet_service.cc:1514] Processing DeleteTablet for tablet 269144de63db417a9c9423a913d4536a with delete_type TABLET_DATA_DELETED (replaced by tablet 27f94f2d8f50495dababe417a9f82982) from {username='slave'} at 127.0.0.1:41678
I20240430 20:11:23.432265 18152 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54524:
name: "default.Table_11aeb11694264720a7e795ac7a114ead"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240430 20:11:23.432502 18981 tablet_replica.cc:330] T 269144de63db417a9c9423a913d4536a P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:23.432737 18981 raft_consensus.cc:2229] T 269144de63db417a9c9423a913d4536a P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Raft consensus shutting down.
W20240430 20:11:23.432811 18152 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_11aeb11694264720a7e795ac7a114ead in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:23.432977 18981 raft_consensus.cc:2258] T 269144de63db417a9c9423a913d4536a P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:23.434147 18981 ts_tablet_manager.cc:1875] T 269144de63db417a9c9423a913d4536a P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:23.434475 18602 tablet_service.cc:1514] Processing DeleteTablet for tablet 269144de63db417a9c9423a913d4536a with delete_type TABLET_DATA_DELETED (replaced by tablet 27f94f2d8f50495dababe417a9f82982) from {username='slave'} at 127.0.0.1:40202
I20240430 20:11:23.434979 18983 tablet_bootstrap.cc:492] T f6e23412ad5b4773bb62590e452abc7a P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:23.435612 18742 tablet_service.cc:1514] Processing DeleteTablet for tablet 269144de63db417a9c9423a913d4536a with delete_type TABLET_DATA_DELETED (replaced by tablet 27f94f2d8f50495dababe417a9f82982) from {username='slave'} at 127.0.0.1:59818
W20240430 20:11:23.437194 18141 catalog_manager.cc:4810] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): delete failed for tablet 269144de63db417a9c9423a913d4536a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 269144de63db417a9c9423a913d4536a already in progress: creating tablet
I20240430 20:11:23.439339 18459 tablet_service.cc:1514] Processing DeleteTablet for tablet a9e8a1052a2a42bf8dbb117c7a7b9c18 with delete_type TABLET_DATA_DELETED (replaced by tablet 26a015250eeb4750a42df93840e8d47f) from {username='slave'} at 127.0.0.1:41678
I20240430 20:11:23.439611 18463 tablet_service.cc:1514] Processing DeleteTablet for tablet f6e23412ad5b4773bb62590e452abc7a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:23 UTC) from {username='slave'} at 127.0.0.1:41678
I20240430 20:11:23.440100 18457 tablet_service.cc:1467] Processing CreateTablet for tablet c3f8f9ada98b40bc8359a00e193c3e5f (DEFAULT_TABLE table=default.Table_c21d1f6f0bfb4f9ea6978a36db7a3701 [id=70551d80c9d8445fa158eee362758714]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:23.440670 18462 tablet_service.cc:1514] Processing DeleteTablet for tablet 3525f5a5051d4fb28cd9c95ca6ed6c48 with delete_type TABLET_DATA_DELETED (replaced by tablet 9750608faeab4be38ddce4e2e4d46964) from {username='slave'} at 127.0.0.1:41678
I20240430 20:11:23.441711 18983 ts_tablet_manager.cc:1367] T f6e23412ad5b4773bb62590e452abc7a P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.054s user 0.000s sys 0.007s
I20240430 20:11:23.442235 18149 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_52f64b8de0684649aa824da65de3416f" } modify_external_catalogs: true
I20240430 20:11:23.442453 18149 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_52f64b8de0684649aa824da65de3416f" } modify_external_catalogs: true
I20240430 20:11:23.443718 18602 tablet_service.cc:1467] Processing CreateTablet for tablet 81328e904ca94e2eb5ec94dd3ffb3c90 (DEFAULT_TABLE table=default.Table_fcab4e9721f7470abc1df37d83ea299f [id=e197b6008d9a40269e645c15412b21b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:23.444655 18151 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: Sending DeleteTablet for 3 replicas of tablet 4d3117a9a5f243b59a69d960c0a1eee4
I20240430 20:11:23.445164 18151 catalog_manager.cc:6554] ReplaceTablet: tablet 4d3117a9a5f243b59a69d960c0a1eee4 deleted and replaced by tablet e0d4cebfbfca45c08d1e7500228835b4
I20240430 20:11:23.445724 18739 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d3117a9a5f243b59a69d960c0a1eee4 with delete_type TABLET_DATA_DELETED (replaced by tablet e0d4cebfbfca45c08d1e7500228835b4) from {username='slave'} at 127.0.0.1:59818
I20240430 20:11:23.446147 18463 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d3117a9a5f243b59a69d960c0a1eee4 with delete_type TABLET_DATA_DELETED (replaced by tablet e0d4cebfbfca45c08d1e7500228835b4) from {username='slave'} at 127.0.0.1:41678
W20240430 20:11:23.446467 18141 catalog_manager.cc:4810] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): delete failed for tablet 4d3117a9a5f243b59a69d960c0a1eee4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4d3117a9a5f243b59a69d960c0a1eee4 already in progress: creating tablet
I20240430 20:11:23.447368 18739 tablet_service.cc:1467] Processing CreateTablet for tablet 81328e904ca94e2eb5ec94dd3ffb3c90 (DEFAULT_TABLE table=default.Table_fcab4e9721f7470abc1df37d83ea299f [id=e197b6008d9a40269e645c15412b21b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:23.447522 18463 tablet_service.cc:1467] Processing CreateTablet for tablet f15f5e1f3afd4224bbf39567cae74617 (DEFAULT_TABLE table=default.Table_6d2fc9759c4741f780a8096a79261d75 [id=d7e2c4300ff94fa2acc65d6d6c932c65]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:23.447806 18739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81328e904ca94e2eb5ec94dd3ffb3c90. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:23.448261 18983 raft_consensus.cc:348] T f6e23412ad5b4773bb62590e452abc7a P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:23.448655 18983 raft_consensus.cc:374] T f6e23412ad5b4773bb62590e452abc7a P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:23.448702 18462 tablet_service.cc:1467] Processing CreateTablet for tablet 81328e904ca94e2eb5ec94dd3ffb3c90 (DEFAULT_TABLE table=default.Table_fcab4e9721f7470abc1df37d83ea299f [id=e197b6008d9a40269e645c15412b21b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:23.448923 18983 raft_consensus.cc:729] T f6e23412ad5b4773bb62590e452abc7a P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:23.450150 18740 tablet_service.cc:1467] Processing CreateTablet for tablet f15f5e1f3afd4224bbf39567cae74617 (DEFAULT_TABLE table=default.Table_6d2fc9759c4741f780a8096a79261d75 [id=d7e2c4300ff94fa2acc65d6d6c932c65]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:23.450562 18740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f15f5e1f3afd4224bbf39567cae74617. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:23.451448 18459 tablet_service.cc:1467] Processing CreateTablet for tablet 0d6a4a63c004480b82d84381867bf8a4 (DEFAULT_TABLE table=default.Table_3c5df662747b48bf8beb6ebe4d2ea618 [id=663c8efbd7ac4a148d5f7626f1522033]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:23.451678 18599 tablet_service.cc:1467] Processing CreateTablet for tablet f15f5e1f3afd4224bbf39567cae74617 (DEFAULT_TABLE table=default.Table_6d2fc9759c4741f780a8096a79261d75 [id=d7e2c4300ff94fa2acc65d6d6c932c65]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:23.451835 18603 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d3117a9a5f243b59a69d960c0a1eee4 with delete_type TABLET_DATA_DELETED (replaced by tablet e0d4cebfbfca45c08d1e7500228835b4) from {username='slave'} at 127.0.0.1:40202
I20240430 20:11:23.450148 18983 consensus_queue.cc:260] T f6e23412ad5b4773bb62590e452abc7a P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:23.453811 18154 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: Sending DeleteTablet for 3 replicas of tablet b0e6cc37c6b1488fa88359d16b12a576
I20240430 20:11:23.455092 18146 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: Sending DeleteTablet for 3 replicas of tablet 6f729a6e8dc14de9a2057cfe09ba9257
I20240430 20:11:23.456624 18150 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54524:
name: "default.Table_a9b96b394f274b2bae5cfe2f0786ec92"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240430 20:11:23.457135 18743 tablet_service.cc:1514] Processing DeleteTablet for tablet b0e6cc37c6b1488fa88359d16b12a576 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:23 UTC) from {username='slave'} at 127.0.0.1:59818
W20240430 20:11:23.457195 18150 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a9b96b394f274b2bae5cfe2f0786ec92 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:23.457383 18741 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f729a6e8dc14de9a2057cfe09ba9257 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:23 UTC) from {username='slave'} at 127.0.0.1:59818
I20240430 20:11:23.458540 18600 tablet_service.cc:1514] Processing DeleteTablet for tablet f6e23412ad5b4773bb62590e452abc7a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:23 UTC) from {username='slave'} at 127.0.0.1:40202
I20240430 20:11:23.458622 18146 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_c21d1f6f0bfb4f9ea6978a36db7a3701" } modify_external_catalogs: true
I20240430 20:11:23.458783 18146 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_c21d1f6f0bfb4f9ea6978a36db7a3701" } modify_external_catalogs: true
I20240430 20:11:23.458811 18598 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f729a6e8dc14de9a2057cfe09ba9257 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:23 UTC) from {username='slave'} at 127.0.0.1:40202
W20240430 20:11:23.459151 18141 catalog_manager.cc:4797] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): delete failed for tablet b0e6cc37c6b1488fa88359d16b12a576 because the tablet was not found. No further retry: Not found: Tablet not found: b0e6cc37c6b1488fa88359d16b12a576
W20240430 20:11:23.459444 18141 catalog_manager.cc:4810] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): delete failed for tablet 6f729a6e8dc14de9a2057cfe09ba9257 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6f729a6e8dc14de9a2057cfe09ba9257 already in progress: creating tablet
I20240430 20:11:23.460413 18148 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54524:
name: "default.Table_80b9e5521fff4a119f3ce4977b3e44eb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:23.460933 18148 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_80b9e5521fff4a119f3ce4977b3e44eb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:23.461508 18461 tablet_service.cc:1514] Processing DeleteTablet for tablet b0e6cc37c6b1488fa88359d16b12a576 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:23 UTC) from {username='slave'} at 127.0.0.1:41678
I20240430 20:11:23.461737 18458 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f729a6e8dc14de9a2057cfe09ba9257 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:23 UTC) from {username='slave'} at 127.0.0.1:41678
I20240430 20:11:23.462445 18151 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54524:
name: "default.Table_f196eb2d3b014b878b9640fbc44cd952"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240430 20:11:23.462529 18156 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54524:
name: "default.Table_73903d0d2a4746f38fdb73932bde6597"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:23.462961 18151 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f196eb2d3b014b878b9640fbc44cd952 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240430 20:11:23.463016 18156 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_73903d0d2a4746f38fdb73932bde6597 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:23.463100 18457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3f8f9ada98b40bc8359a00e193c3e5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:23.463514 18459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d6a4a63c004480b82d84381867bf8a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:23.464213 18603 tablet_service.cc:1514] Processing DeleteTablet for tablet b0e6cc37c6b1488fa88359d16b12a576 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:23 UTC) from {username='slave'} at 127.0.0.1:40202
I20240430 20:11:23.468628 18742 tablet_service.cc:1467] Processing CreateTablet for tablet c3f8f9ada98b40bc8359a00e193c3e5f (DEFAULT_TABLE table=default.Table_c21d1f6f0bfb4f9ea6978a36db7a3701 [id=70551d80c9d8445fa158eee362758714]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:23.469107 18742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3f8f9ada98b40bc8359a00e193c3e5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:23.469818 18463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f15f5e1f3afd4224bbf39567cae74617. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:23.470822 18462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81328e904ca94e2eb5ec94dd3ffb3c90. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:23.475288 18599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f15f5e1f3afd4224bbf39567cae74617. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:23.475903 18602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81328e904ca94e2eb5ec94dd3ffb3c90. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:23.483120 18980 ts_tablet_manager.cc:1888] T 3525f5a5051d4fb28cd9c95ca6ed6c48 P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:23.483631 18155 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54524:
name: "default.Table_b29130df231a4316bcb8f9c2be002c92"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240430 20:11:23.484025 18154 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_545cc69db7d64874861a9fa404789c0b" } new_table_name: "default.Table_75dd71ce34a14788878b70c8354caba6" modify_external_catalogs: true
W20240430 20:11:23.484252 18155 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b29130df231a4316bcb8f9c2be002c92 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:23.485388 18152 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_c0737f9f801a45eea21f6204a6232be1" } new_table_name: "default.Table_16b082d140234e1c9a1e70654839b4cd" modify_external_catalogs: true
I20240430 20:11:23.487155 18149 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: Sending DeleteTablet for 3 replicas of tablet fe01b95104174279a7d64c0d4c0af982
I20240430 20:11:23.488613 18739 tablet_service.cc:1514] Processing DeleteTablet for tablet fe01b95104174279a7d64c0d4c0af982 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:23 UTC) from {username='slave'} at 127.0.0.1:59818
I20240430 20:11:23.488703 18458 tablet_service.cc:1514] Processing DeleteTablet for tablet fe01b95104174279a7d64c0d4c0af982 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:23 UTC) from {username='slave'} at 127.0.0.1:41678
W20240430 20:11:23.489310 18141 catalog_manager.cc:4797] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): delete failed for tablet fe01b95104174279a7d64c0d4c0af982 because the tablet was not found. No further retry: Not found: Tablet not found: fe01b95104174279a7d64c0d4c0af982
I20240430 20:11:23.489909 18980 log.cc:1196] T 3525f5a5051d4fb28cd9c95ca6ed6c48 P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/3525f5a5051d4fb28cd9c95ca6ed6c48
I20240430 20:11:23.490331 18980 ts_tablet_manager.cc:1909] T 3525f5a5051d4fb28cd9c95ca6ed6c48 P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:23.492020 18983 ts_tablet_manager.cc:1398] T f6e23412ad5b4773bb62590e452abc7a P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.050s user 0.002s sys 0.000s
I20240430 20:11:23.492293 18940 raft_consensus.cc:482] T 17843ea77664447eaaa19666e0445ada P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:23.492453 18940 raft_consensus.cc:504] T 17843ea77664447eaaa19666e0445ada P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:23.492612 18139 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet 3525f5a5051d4fb28cd9c95ca6ed6c48 (table default.Table_c06b54647f004ff7a7f7beafee702b4c [id=a6a27dcbd0e648a8bf29a8cd5b3a2a11]) successfully deleted
I20240430 20:11:23.492996 18980 tablet_replica.cc:330] T 235ea182e02740f68f30c3c34f77012a P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:23.493237 18980 raft_consensus.cc:2229] T 235ea182e02740f68f30c3c34f77012a P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:23.493265 18940 leader_election.cc:288] T 17843ea77664447eaaa19666e0445ada P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:23.493739 18739 tablet_service.cc:1467] Processing CreateTablet for tablet 0d6a4a63c004480b82d84381867bf8a4 (DEFAULT_TABLE table=default.Table_3c5df662747b48bf8beb6ebe4d2ea618 [id=663c8efbd7ac4a148d5f7626f1522033]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:23.494993 18763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17843ea77664447eaaa19666e0445ada" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:23.495223 18763 raft_consensus.cc:2454] T 17843ea77664447eaaa19666e0445ada P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 0.
I20240430 20:11:23.497593 18147 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54524:
name: "default.Table_c2b7ceaf18b24ff3b42e4fa585fcdb10"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240430 20:11:23.498085 18739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d6a4a63c004480b82d84381867bf8a4. 1 dirs total, 0 dirs full, 0 dirs failed
W20240430 20:11:23.498162 18147 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c2b7ceaf18b24ff3b42e4fa585fcdb10 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:23.498796 18417 leader_election.cc:302] T 17843ea77664447eaaa19666e0445ada P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters:
I20240430 20:11:23.500414 18940 raft_consensus.cc:2781] T 17843ea77664447eaaa19666e0445ada P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:23.500558 18940 raft_consensus.cc:482] T 17843ea77664447eaaa19666e0445ada P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:23.500672 18940 raft_consensus.cc:3037] T 17843ea77664447eaaa19666e0445ada P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:23.500978 18741 tablet_service.cc:1467] Processing CreateTablet for tablet 59548d560b4e4f7ca034d730aca79364 (DEFAULT_TABLE table=default.Table_359e5a94ea604fc287e536a75406c664 [id=f31f8f1f8e164a9d946c46164ba12f44]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:23.501400 18741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59548d560b4e4f7ca034d730aca79364. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:23.503814 18988 tablet_bootstrap.cc:654] T 269144de63db417a9c9423a913d4536a P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:23.504343 18940 raft_consensus.cc:504] T 17843ea77664447eaaa19666e0445ada P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:23.506189 18940 leader_election.cc:288] T 17843ea77664447eaaa19666e0445ada P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Requested vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:23.506865 18763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17843ea77664447eaaa19666e0445ada" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:23.507107 18763 raft_consensus.cc:3037] T 17843ea77664447eaaa19666e0445ada P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:23.509558 18763 raft_consensus.cc:2454] T 17843ea77664447eaaa19666e0445ada P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 1.
I20240430 20:11:23.511198 18417 leader_election.cc:302] T 17843ea77664447eaaa19666e0445ada P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters:
I20240430 20:11:23.487548 19030 raft_consensus.cc:504] T 27b21459592649dca1089a7a29021206 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:23.512878 18459 tablet_service.cc:1467] Processing CreateTablet for tablet 59548d560b4e4f7ca034d730aca79364 (DEFAULT_TABLE table=default.Table_359e5a94ea604fc287e536a75406c664 [id=f31f8f1f8e164a9d946c46164ba12f44]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:23.513767 19030 leader_election.cc:288] T 27b21459592649dca1089a7a29021206 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Requested vote from peers 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:23.514101 18980 raft_consensus.cc:2258] T 235ea182e02740f68f30c3c34f77012a P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:23.493844 18622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17843ea77664447eaaa19666e0445ada" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:23.515223 18621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27b21459592649dca1089a7a29021206" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:23.515452 18621 raft_consensus.cc:3037] T 27b21459592649dca1089a7a29021206 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
W20240430 20:11:23.515553 18415 leader_election.cc:341] T 17843ea77664447eaaa19666e0445ada P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:23.489728 18599 tablet_service.cc:1514] Processing DeleteTablet for tablet fe01b95104174279a7d64c0d4c0af982 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:23 UTC) from {username='slave'} at 127.0.0.1:40202
I20240430 20:11:23.498934 18602 tablet_service.cc:1467] Processing CreateTablet for tablet 0d6a4a63c004480b82d84381867bf8a4 (DEFAULT_TABLE table=default.Table_3c5df662747b48bf8beb6ebe4d2ea618 [id=663c8efbd7ac4a148d5f7626f1522033]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:23.517009 18602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d6a4a63c004480b82d84381867bf8a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:23.518138 18146 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: Sending DeleteTablet for 3 replicas of tablet c3f8f9ada98b40bc8359a00e193c3e5f
I20240430 20:11:23.518646 18482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27b21459592649dca1089a7a29021206" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:23.518898 18482 raft_consensus.cc:3037] T 27b21459592649dca1089a7a29021206 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:23.519938 18459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59548d560b4e4f7ca034d730aca79364. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:23.521291 18599 tablet_service.cc:1514] Processing DeleteTablet for tablet c3f8f9ada98b40bc8359a00e193c3e5f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:23 UTC) from {username='slave'} at 127.0.0.1:40202
I20240430 20:11:23.521637 18741 tablet_service.cc:1514] Processing DeleteTablet for tablet c3f8f9ada98b40bc8359a00e193c3e5f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:23 UTC) from {username='slave'} at 127.0.0.1:59818
I20240430 20:11:23.521723 18482 raft_consensus.cc:2454] T 27b21459592649dca1089a7a29021206 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:23.522218 18699 leader_election.cc:302] T 27b21459592649dca1089a7a29021206 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters:
W20240430 20:11:23.522382 18141 catalog_manager.cc:4810] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): delete failed for tablet c3f8f9ada98b40bc8359a00e193c3e5f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c3f8f9ada98b40bc8359a00e193c3e5f already in progress: creating tablet
I20240430 20:11:23.524410 18150 catalog_manager.cc:5484] T 44699c84ebac49ef8f3765d4069e71ca P 3c51c33d5616482cafb68e26688820f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c51c33d5616482cafb68e26688820f4 (127.17.177.3). New cstate: current_term: 1 leader_uuid: "3c51c33d5616482cafb68e26688820f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } health_report { overall_health: HEALTHY } } }
I20240430 20:11:23.525698 18150 catalog_manager.cc:5484] T e5d1707ba0b34cc4b1cda39e36ccbd2f P 3c51c33d5616482cafb68e26688820f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c51c33d5616482cafb68e26688820f4 (127.17.177.3). New cstate: current_term: 1 leader_uuid: "3c51c33d5616482cafb68e26688820f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } health_report { overall_health: HEALTHY } } }
I20240430 20:11:23.525959 18940 raft_consensus.cc:2781] T 17843ea77664447eaaa19666e0445ada P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:23.526516 18988 tablet_bootstrap.cc:492] T 269144de63db417a9c9423a913d4536a P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:23.526652 18621 raft_consensus.cc:2454] T 27b21459592649dca1089a7a29021206 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:23.526715 18988 ts_tablet_manager.cc:1367] T 269144de63db417a9c9423a913d4536a P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.115s user 0.004s sys 0.000s
I20240430 20:11:23.528072 18150 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet e5d1707ba0b34cc4b1cda39e36ccbd2f (table default.Table_0ab5b2f09fa7468bb89ab703f6d3d25c [id=a5d580cffa2447d78fc9f2c8e31264f1]). Expected version 4 got 0
I20240430 20:11:23.528239 18988 raft_consensus.cc:348] T 269144de63db417a9c9423a913d4536a P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:23.528470 18988 raft_consensus.cc:374] T 269144de63db417a9c9423a913d4536a P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:23.528553 18988 raft_consensus.cc:729] T 269144de63db417a9c9423a913d4536a P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:23.528402 18150 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 17843ea77664447eaaa19666e0445ada (table default.Table_c0737f9f801a45eea21f6204a6232be1 [id=c0d8dc87d1f9426a85f6176f161595a1]). Expected version 2 got 0
I20240430 20:11:23.528957 18988 consensus_queue.cc:260] T 269144de63db417a9c9423a913d4536a P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:23.526758 18146 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_359e5a94ea604fc287e536a75406c664" } new_table_name: "default.Table_6d2f70d1980647448e749e1779834218" modify_external_catalogs: true
I20240430 20:11:23.529325 18924 raft_consensus.cc:2781] T 27b21459592649dca1089a7a29021206 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:23.529469 18924 raft_consensus.cc:686] T 27b21459592649dca1089a7a29021206 P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Becoming Leader. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Running, Role: LEADER
I20240430 20:11:23.529714 18924 consensus_queue.cc:237] T 27b21459592649dca1089a7a29021206 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:23.530161 18981 ts_tablet_manager.cc:1888] T 269144de63db417a9c9423a913d4536a P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:23.530280 18981 log.cc:1196] T 269144de63db417a9c9423a913d4536a P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/269144de63db417a9c9423a913d4536a
I20240430 20:11:23.530659 18981 ts_tablet_manager.cc:1909] T 269144de63db417a9c9423a913d4536a P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:23.531520 18981 tablet_replica.cc:330] T a9e8a1052a2a42bf8dbb117c7a7b9c18 P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:23.531713 18981 raft_consensus.cc:2229] T a9e8a1052a2a42bf8dbb117c7a7b9c18 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:23.531844 18981 raft_consensus.cc:2258] T a9e8a1052a2a42bf8dbb117c7a7b9c18 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:23.534616 18142 catalog_manager.cc:4830] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): tablet 269144de63db417a9c9423a913d4536a (table default.Table_156650e4b399419c97dcf5160b761ee1 [id=b93708ff15ef4c8299967c8c91045c8f]) successfully deleted
I20240430 20:11:23.535009 18463 tablet_service.cc:1514] Processing DeleteTablet for tablet c3f8f9ada98b40bc8359a00e193c3e5f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:23 UTC) from {username='slave'} at 127.0.0.1:41678
I20240430 20:11:23.535401 18940 raft_consensus.cc:686] T 17843ea77664447eaaa19666e0445ada P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Becoming Leader. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Running, Role: LEADER
I20240430 20:11:23.535980 18980 ts_tablet_manager.cc:1875] T 235ea182e02740f68f30c3c34f77012a P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:23.536358 18940 consensus_queue.cc:237] T 17843ea77664447eaaa19666e0445ada P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:23.537092 18988 ts_tablet_manager.cc:1398] T 269144de63db417a9c9423a913d4536a P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240430 20:11:23.537431 18988 tablet_bootstrap.cc:492] T 235ea182e02740f68f30c3c34f77012a P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:23.499738 18983 tablet_bootstrap.cc:492] T 17843ea77664447eaaa19666e0445ada P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:23.505772 18623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17843ea77664447eaaa19666e0445ada" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
W20240430 20:11:23.539017 18415 leader_election.cc:341] T 17843ea77664447eaaa19666e0445ada P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:23.539109 18988 tablet_bootstrap.cc:654] T 235ea182e02740f68f30c3c34f77012a P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:23.541102 18599 tablet_service.cc:1467] Processing CreateTablet for tablet 59548d560b4e4f7ca034d730aca79364 (DEFAULT_TABLE table=default.Table_359e5a94ea604fc287e536a75406c664 [id=f31f8f1f8e164a9d946c46164ba12f44]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:23.541509 18599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59548d560b4e4f7ca034d730aca79364. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:23.542635 18983 tablet_bootstrap.cc:654] T 17843ea77664447eaaa19666e0445ada P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
W20240430 20:11:23.545950 18154 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e0d4cebfbfca45c08d1e7500228835b4 has no consensus state
I20240430 20:11:23.548132 18154 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_5ec7f4e4018b417f923420df89e9e25f" } new_table_name: "default.Table_b4c8addb5cfe442d8ec19324de4bb523" modify_external_catalogs: true
I20240430 20:11:23.551244 18457 tablet_service.cc:1467] Processing CreateTablet for tablet b0e6cc37c6b1488fa88359d16b12a576 (DEFAULT_TABLE table=default.Table_32647411b85344b490966d64f135631b [id=8cd243a627f84916902191f774d1e8cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:23.551378 18742 tablet_service.cc:1467] Processing CreateTablet for tablet b0e6cc37c6b1488fa88359d16b12a576 (DEFAULT_TABLE table=default.Table_32647411b85344b490966d64f135631b [id=8cd243a627f84916902191f774d1e8cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:23.551764 18457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0e6cc37c6b1488fa88359d16b12a576. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:23.551890 18742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0e6cc37c6b1488fa88359d16b12a576. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:23.555001 18981 ts_tablet_manager.cc:1875] T a9e8a1052a2a42bf8dbb117c7a7b9c18 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:23.558467 18601 tablet_service.cc:1467] Processing CreateTablet for tablet b0e6cc37c6b1488fa88359d16b12a576 (DEFAULT_TABLE table=default.Table_32647411b85344b490966d64f135631b [id=8cd243a627f84916902191f774d1e8cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:23.560261 18463 tablet_service.cc:1467] Processing CreateTablet for tablet a01ee128ba8a4c9fb8e125ba97239d4c (DEFAULT_TABLE table=default.Table_23b82186d14a44eeb81cad606fb8e9cc [id=172b1ea31b8142108ae466a018ecab47]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:23.560704 18463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a01ee128ba8a4c9fb8e125ba97239d4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:23.560374 18601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0e6cc37c6b1488fa88359d16b12a576. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:23.563177 18739 tablet_service.cc:1467] Processing CreateTablet for tablet a01ee128ba8a4c9fb8e125ba97239d4c (DEFAULT_TABLE table=default.Table_23b82186d14a44eeb81cad606fb8e9cc [id=172b1ea31b8142108ae466a018ecab47]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:23.563758 18985 tablet_bootstrap.cc:492] T 235ea182e02740f68f30c3c34f77012a P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:23.563946 18985 ts_tablet_manager.cc:1367] T 235ea182e02740f68f30c3c34f77012a P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.189s user 0.009s sys 0.017s
I20240430 20:11:23.564709 18155 master_service.cc:946] ReplaceTablet: received request to replace tablet 6f729a6e8dc14de9a2057cfe09ba9257 from {username='slave'} at 127.0.0.1:54524
I20240430 20:11:23.565104 18599 tablet_service.cc:1467] Processing CreateTablet for tablet a01ee128ba8a4c9fb8e125ba97239d4c (DEFAULT_TABLE table=default.Table_23b82186d14a44eeb81cad606fb8e9cc [id=172b1ea31b8142108ae466a018ecab47]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:23.565575 18599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a01ee128ba8a4c9fb8e125ba97239d4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:23.565948 18985 raft_consensus.cc:348] T 235ea182e02740f68f30c3c34f77012a P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:23.563789 18739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a01ee128ba8a4c9fb8e125ba97239d4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:23.566645 18985 raft_consensus.cc:374] T 235ea182e02740f68f30c3c34f77012a P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:23.566783 18985 raft_consensus.cc:729] T 235ea182e02740f68f30c3c34f77012a P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:23.567101 18985 consensus_queue.cc:260] T 235ea182e02740f68f30c3c34f77012a P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:23.567685 18985 ts_tablet_manager.cc:1398] T 235ea182e02740f68f30c3c34f77012a P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240430 20:11:23.568339 18462 tablet_service.cc:1467] Processing CreateTablet for tablet 5809e7c9c2204ba293426c24129fb16e (DEFAULT_TABLE table=default.Table_d82824c897a94dd0ba45955f9473b626 [id=d794640d21a843a185038673f7eb2b00]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:23.568861 18462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5809e7c9c2204ba293426c24129fb16e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:23.572813 18156 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54524:
name: "default.Table_2d41f75890eb40199519c2937c89e658"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:23.573550 18156 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2d41f75890eb40199519c2937c89e658 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:23.570333 18931 raft_consensus.cc:482] T f6e23412ad5b4773bb62590e452abc7a P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:23.575919 18931 raft_consensus.cc:504] T f6e23412ad5b4773bb62590e452abc7a P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:23.576637 18931 leader_election.cc:288] T f6e23412ad5b4773bb62590e452abc7a P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:23.571971 18603 tablet_service.cc:1467] Processing CreateTablet for tablet 5809e7c9c2204ba293426c24129fb16e (DEFAULT_TABLE table=default.Table_d82824c897a94dd0ba45955f9473b626 [id=d794640d21a843a185038673f7eb2b00]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:23.577154 18603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5809e7c9c2204ba293426c24129fb16e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:23.581537 18985 tablet_bootstrap.cc:492] T 4d3117a9a5f243b59a69d960c0a1eee4 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:23.582372 18482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6e23412ad5b4773bb62590e452abc7a" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:23.582845 18763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6e23412ad5b4773bb62590e452abc7a" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:23.583583 18482 raft_consensus.cc:2379] T f6e23412ad5b4773bb62590e452abc7a P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e1c6397f09747f091534faf93542710 in current term 1: Already voted for candidate 3c51c33d5616482cafb68e26688820f4 in this term.
I20240430 20:11:23.584136 18558 leader_election.cc:302] T f6e23412ad5b4773bb62590e452abc7a P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:23.585685 18931 raft_consensus.cc:3037] T f6e23412ad5b4773bb62590e452abc7a P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:23.587667 18741 tablet_service.cc:1467] Processing CreateTablet for tablet 5809e7c9c2204ba293426c24129fb16e (DEFAULT_TABLE table=default.Table_d82824c897a94dd0ba45955f9473b626 [id=d794640d21a843a185038673f7eb2b00]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:23.588111 18741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5809e7c9c2204ba293426c24129fb16e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:23.567660 18983 tablet_bootstrap.cc:492] T 17843ea77664447eaaa19666e0445ada P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:23.590395 18983 ts_tablet_manager.cc:1367] T 17843ea77664447eaaa19666e0445ada P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.091s user 0.004s sys 0.000s
I20240430 20:11:23.591135 18983 raft_consensus.cc:348] T 17843ea77664447eaaa19666e0445ada P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:23.591343 18983 raft_consensus.cc:374] T 17843ea77664447eaaa19666e0445ada P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:23.591435 18983 raft_consensus.cc:729] T 17843ea77664447eaaa19666e0445ada P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:23.593149 18983 consensus_queue.cc:260] T 17843ea77664447eaaa19666e0445ada P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:23.594859 18463 tablet_service.cc:1467] Processing CreateTablet for tablet fe01b95104174279a7d64c0d4c0af982 (DEFAULT_TABLE table=default.Table_52f64b8de0684649aa824da65de3416f [id=41ea2be4643c4e75a2c20b7f9f0a825e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:23.595333 18463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe01b95104174279a7d64c0d4c0af982. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:23.596758 18931 raft_consensus.cc:2726] T f6e23412ad5b4773bb62590e452abc7a P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:23.597582 18980 ts_tablet_manager.cc:1888] T 235ea182e02740f68f30c3c34f77012a P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:23.597748 18980 log.cc:1196] T 235ea182e02740f68f30c3c34f77012a P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/235ea182e02740f68f30c3c34f77012a
I20240430 20:11:23.598268 18980 ts_tablet_manager.cc:1909] T 235ea182e02740f68f30c3c34f77012a P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:23.599393 18985 tablet_bootstrap.cc:654] T 4d3117a9a5f243b59a69d960c0a1eee4 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:23.603945 18988 tablet_bootstrap.cc:492] T 235ea182e02740f68f30c3c34f77012a P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:23.604113 18940 raft_consensus.cc:482] T 235ea182e02740f68f30c3c34f77012a P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:23.604185 18988 ts_tablet_manager.cc:1367] T 235ea182e02740f68f30c3c34f77012a P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.067s user 0.003s sys 0.000s
I20240430 20:11:23.604274 18940 raft_consensus.cc:504] T 235ea182e02740f68f30c3c34f77012a P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:23.605175 18988 raft_consensus.cc:348] T 235ea182e02740f68f30c3c34f77012a P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:23.605459 18988 raft_consensus.cc:374] T 235ea182e02740f68f30c3c34f77012a P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:23.605595 18988 raft_consensus.cc:729] T 235ea182e02740f68f30c3c34f77012a P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
W20240430 20:11:23.605764 18146 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 59548d560b4e4f7ca034d730aca79364 consensus state has no leader
I20240430 20:11:23.605897 18988 consensus_queue.cc:260] T 235ea182e02740f68f30c3c34f77012a P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
W20240430 20:11:23.606089 18415 leader_election.cc:341] T 235ea182e02740f68f30c3c34f77012a P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Not found: Tablet not found: 235ea182e02740f68f30c3c34f77012a
I20240430 20:11:23.607414 18763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "235ea182e02740f68f30c3c34f77012a" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:23.607616 18763 raft_consensus.cc:2454] T 235ea182e02740f68f30c3c34f77012a P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 0.
I20240430 20:11:23.608182 18417 leader_election.cc:302] T 235ea182e02740f68f30c3c34f77012a P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters: 5e1c6397f09747f091534faf93542710
I20240430 20:11:23.608480 19037 raft_consensus.cc:2781] T 235ea182e02740f68f30c3c34f77012a P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:23.608563 19037 raft_consensus.cc:482] T 235ea182e02740f68f30c3c34f77012a P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:23.608666 19037 raft_consensus.cc:3037] T 235ea182e02740f68f30c3c34f77012a P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:23.611920 19037 raft_consensus.cc:504] T 235ea182e02740f68f30c3c34f77012a P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:23.612603 18146 master_service.cc:946] ReplaceTablet: received request to replace tablet f79497bf22ed417d988530f6e69cc07f from {username='slave'} at 127.0.0.1:54524
I20240430 20:11:23.613659 18763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "235ea182e02740f68f30c3c34f77012a" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:23.613875 18763 raft_consensus.cc:3037] T 235ea182e02740f68f30c3c34f77012a P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:23.614243 18743 tablet_service.cc:1467] Processing CreateTablet for tablet fe01b95104174279a7d64c0d4c0af982 (DEFAULT_TABLE table=default.Table_52f64b8de0684649aa824da65de3416f [id=41ea2be4643c4e75a2c20b7f9f0a825e]), partition=RANGE (key) PARTITION UNBOUNDED
W20240430 20:11:23.614506 18415 leader_election.cc:341] T 235ea182e02740f68f30c3c34f77012a P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Not found: Tablet not found: 235ea182e02740f68f30c3c34f77012a
I20240430 20:11:23.614706 18743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe01b95104174279a7d64c0d4c0af982. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:23.614802 19037 leader_election.cc:288] T 235ea182e02740f68f30c3c34f77012a P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Requested vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:23.614989 18940 leader_election.cc:288] T 235ea182e02740f68f30c3c34f77012a P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:23.617517 18981 ts_tablet_manager.cc:1888] T a9e8a1052a2a42bf8dbb117c7a7b9c18 P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:23.619261 18981 log.cc:1196] T a9e8a1052a2a42bf8dbb117c7a7b9c18 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/a9e8a1052a2a42bf8dbb117c7a7b9c18
I20240430 20:11:23.619578 18151 catalog_manager.cc:5484] T 17843ea77664447eaaa19666e0445ada P 2efb542aa55e439b99ccb98d1c5c884a reported cstate change: term changed from 0 to 1, leader changed from <none> to 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1). New cstate: current_term: 1 leader_uuid: "2efb542aa55e439b99ccb98d1c5c884a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } health_report { overall_health: UNKNOWN } } }
I20240430 20:11:23.619830 18981 ts_tablet_manager.cc:1909] T a9e8a1052a2a42bf8dbb117c7a7b9c18 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:23.620255 18151 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 17843ea77664447eaaa19666e0445ada (table default.Table_c0737f9f801a45eea21f6204a6232be1 [id=c0d8dc87d1f9426a85f6176f161595a1]). Expected version 2 got 0
I20240430 20:11:23.621778 18763 raft_consensus.cc:2454] T 235ea182e02740f68f30c3c34f77012a P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 1.
I20240430 20:11:23.622282 18417 leader_election.cc:302] T 235ea182e02740f68f30c3c34f77012a P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters: 5e1c6397f09747f091534faf93542710
I20240430 20:11:23.622771 18940 raft_consensus.cc:2781] T 235ea182e02740f68f30c3c34f77012a P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:23.622918 18940 raft_consensus.cc:686] T 235ea182e02740f68f30c3c34f77012a P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Becoming Leader. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Running, Role: LEADER
I20240430 20:11:23.623201 18940 consensus_queue.cc:237] T 235ea182e02740f68f30c3c34f77012a P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:23.623850 18142 catalog_manager.cc:4830] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): tablet a9e8a1052a2a42bf8dbb117c7a7b9c18 (table default.Table_2e9c7c683df84a41942d31600a265641 [id=aca6fb02776a40bbac312191adbdc136]) successfully deleted
I20240430 20:11:23.625783 18146 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_75dd71ce34a14788878b70c8354caba6" } modify_external_catalogs: true
I20240430 20:11:23.626034 18146 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_75dd71ce34a14788878b70c8354caba6" } modify_external_catalogs: true
I20240430 20:11:23.628165 18155 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_6d2f70d1980647448e749e1779834218" } new_table_name: "default.Table_f6d2f14f89184a32a523563bdc6ba478" modify_external_catalogs: true
I20240430 20:11:23.628784 18981 tablet_replica.cc:330] T f6e23412ad5b4773bb62590e452abc7a P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:23.628978 18981 raft_consensus.cc:2229] T f6e23412ad5b4773bb62590e452abc7a P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:23.629151 18981 raft_consensus.cc:2258] T f6e23412ad5b4773bb62590e452abc7a P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:23.631119 18981 ts_tablet_manager.cc:1875] T f6e23412ad5b4773bb62590e452abc7a P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:23.633790 18924 raft_consensus.cc:482] T 269144de63db417a9c9423a913d4536a P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:23.634022 18924 raft_consensus.cc:504] T 269144de63db417a9c9423a913d4536a P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
W20240430 20:11:23.635444 18699 leader_election.cc:341] T 269144de63db417a9c9423a913d4536a P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Not found: Tablet not found: 269144de63db417a9c9423a913d4536a
I20240430 20:11:23.635452 18623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "269144de63db417a9c9423a913d4536a" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:23.635669 18623 raft_consensus.cc:2454] T 269144de63db417a9c9423a913d4536a P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:23.636227 18696 leader_election.cc:302] T 269144de63db417a9c9423a913d4536a P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a
I20240430 20:11:23.636535 19030 raft_consensus.cc:2781] T 269144de63db417a9c9423a913d4536a P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20240430 20:11:23.636595 18154 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 609a1a7d22e84b69bd0a53577b62b69b has no consensus state
I20240430 20:11:23.636612 18924 leader_election.cc:288] T 269144de63db417a9c9423a913d4536a P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:23.636668 19030 raft_consensus.cc:482] T 269144de63db417a9c9423a913d4536a P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:23.636768 19030 raft_consensus.cc:3037] T 269144de63db417a9c9423a913d4536a P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:23.637441 18154 master_service.cc:946] ReplaceTablet: received request to replace tablet 9750608faeab4be38ddce4e2e4d46964 from {username='slave'} at 127.0.0.1:54524
I20240430 20:11:23.640736 19030 raft_consensus.cc:504] T 269144de63db417a9c9423a913d4536a P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:23.641777 18983 ts_tablet_manager.cc:1398] T 17843ea77664447eaaa19666e0445ada P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.051s user 0.005s sys 0.004s
I20240430 20:11:23.642158 19030 leader_election.cc:288] T 269144de63db417a9c9423a913d4536a P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Requested vote from peers 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
W20240430 20:11:23.642450 18699 leader_election.cc:341] T 269144de63db417a9c9423a913d4536a P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Not found: Tablet not found: 269144de63db417a9c9423a913d4536a
I20240430 20:11:23.642679 18623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "269144de63db417a9c9423a913d4536a" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:23.642899 18623 raft_consensus.cc:3037] T 269144de63db417a9c9423a913d4536a P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:23.644364 18983 tablet_bootstrap.cc:492] T e5d1707ba0b34cc4b1cda39e36ccbd2f P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:23.645644 18623 raft_consensus.cc:2454] T 269144de63db417a9c9423a913d4536a P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:23.647524 18696 leader_election.cc:302] T 269144de63db417a9c9423a913d4536a P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a
I20240430 20:11:23.647861 19030 raft_consensus.cc:2781] T 269144de63db417a9c9423a913d4536a P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:23.648028 19030 raft_consensus.cc:686] T 269144de63db417a9c9423a913d4536a P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Becoming Leader. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Running, Role: LEADER
I20240430 20:11:23.648486 18983 tablet_bootstrap.cc:654] T e5d1707ba0b34cc4b1cda39e36ccbd2f P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:23.648362 19030 consensus_queue.cc:237] T 269144de63db417a9c9423a913d4536a P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:23.651083 18148 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54524:
name: "default.Table_3acf40c6d58741d4a8e94ca96d642249"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:23.651842 18148 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3acf40c6d58741d4a8e94ca96d642249 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:23.651937 18156 master_service.cc:946] ReplaceTablet: received request to replace tablet 0d6a4a63c004480b82d84381867bf8a4 from {username='slave'} at 127.0.0.1:54524
I20240430 20:11:23.653591 18981 ts_tablet_manager.cc:1888] T f6e23412ad5b4773bb62590e452abc7a P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:23.654366 18981 log.cc:1196] T f6e23412ad5b4773bb62590e452abc7a P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/f6e23412ad5b4773bb62590e452abc7a
I20240430 20:11:23.659541 18139 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet 235ea182e02740f68f30c3c34f77012a (table default.Table_33059b681e564888992fa98876243716 [id=2eae27505e2b4218b933f03c397a8495]) successfully deleted
I20240430 20:11:23.660046 18599 tablet_service.cc:1467] Processing CreateTablet for tablet fe01b95104174279a7d64c0d4c0af982 (DEFAULT_TABLE table=default.Table_52f64b8de0684649aa824da65de3416f [id=41ea2be4643c4e75a2c20b7f9f0a825e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:23.662546 18149 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_917365c2d10941558081a9898db3d7ea" } modify_external_catalogs: true
I20240430 20:11:23.662801 18149 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_917365c2d10941558081a9898db3d7ea" } modify_external_catalogs: true
I20240430 20:11:23.665086 18599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe01b95104174279a7d64c0d4c0af982. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:23.665441 18147 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 17843ea77664447eaaa19666e0445ada (table default.Table_c0737f9f801a45eea21f6204a6232be1 [id=c0d8dc87d1f9426a85f6176f161595a1]). Expected version 2 got 0
I20240430 20:11:23.667930 18151 master_service.cc:946] ReplaceTablet: received request to replace tablet 27b21459592649dca1089a7a29021206 from {username='slave'} at 127.0.0.1:54524
I20240430 20:11:23.671458 18988 ts_tablet_manager.cc:1398] T 235ea182e02740f68f30c3c34f77012a P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.067s user 0.002s sys 0.000s
I20240430 20:11:23.672829 18743 tablet_service.cc:1514] Processing DeleteTablet for tablet 3525f5a5051d4fb28cd9c95ca6ed6c48 with delete_type TABLET_DATA_DELETED (replaced by tablet 9750608faeab4be38ddce4e2e4d46964) from {username='slave'} at 127.0.0.1:59818
I20240430 20:11:23.672930 18740 tablet_service.cc:1514] Processing DeleteTablet for tablet f6e23412ad5b4773bb62590e452abc7a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:23 UTC) from {username='slave'} at 127.0.0.1:59818
I20240430 20:11:23.673259 18987 tablet_replica.cc:330] T 3525f5a5051d4fb28cd9c95ca6ed6c48 P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:23.673475 18987 raft_consensus.cc:2229] T 3525f5a5051d4fb28cd9c95ca6ed6c48 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:23.673673 18987 raft_consensus.cc:2258] T 3525f5a5051d4fb28cd9c95ca6ed6c48 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:23.674261 18981 ts_tablet_manager.cc:1909] T f6e23412ad5b4773bb62590e452abc7a P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:23.675698 18987 ts_tablet_manager.cc:1875] T 3525f5a5051d4fb28cd9c95ca6ed6c48 P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:23.677222 18152 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54524:
name: "default.Table_7bfb47ba65114665943b7edccabd5a8b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240430 20:11:23.677665 18142 catalog_manager.cc:4830] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): tablet f6e23412ad5b4773bb62590e452abc7a (table default.Table_d4b5e1a26a9449cdbe70f185b1097321 [id=ee9addb7eb9f4430a57fd9ee4bfaae7b]) successfully deleted
W20240430 20:11:23.677969 18152 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7bfb47ba65114665943b7edccabd5a8b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:23.678561 18981 tablet_replica.cc:330] T 3525f5a5051d4fb28cd9c95ca6ed6c48 P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:23.678781 18981 raft_consensus.cc:2229] T 3525f5a5051d4fb28cd9c95ca6ed6c48 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:23.678949 18981 raft_consensus.cc:2258] T 3525f5a5051d4fb28cd9c95ca6ed6c48 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:23.679323 18763 raft_consensus.cc:1261] T 17843ea77664447eaaa19666e0445ada P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Refusing update from remote peer 2efb542aa55e439b99ccb98d1c5c884a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240430 20:11:23.680294 18940 consensus_queue.cc:1035] T 17843ea77664447eaaa19666e0445ada P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:23.680601 18981 ts_tablet_manager.cc:1875] T 3525f5a5051d4fb28cd9c95ca6ed6c48 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:23.681489 18147 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_0af8d509e51a47c49580fa1d6cbee72f" } modify_external_catalogs: true
I20240430 20:11:23.681668 18147 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_0af8d509e51a47c49580fa1d6cbee72f" } modify_external_catalogs: true
I20240430 20:11:23.683132 18623 raft_consensus.cc:3037] T 17843ea77664447eaaa19666e0445ada P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:23.685479 18150 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 17843ea77664447eaaa19666e0445ada (table default.Table_16b082d140234e1c9a1e70654839b4cd [id=c0d8dc87d1f9426a85f6176f161595a1]). Expected version 3 got 0
I20240430 20:11:23.687991 18623 raft_consensus.cc:1261] T 17843ea77664447eaaa19666e0445ada P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Refusing update from remote peer 2efb542aa55e439b99ccb98d1c5c884a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240430 20:11:23.689163 18940 consensus_queue.cc:1035] T 17843ea77664447eaaa19666e0445ada P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:23.689047 18150 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54524:
name: "default.Table_ddbb3fa4bcd7461cba6de52ebacdb12d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240430 20:11:23.689721 18457 tablet_service.cc:1514] Processing DeleteTablet for tablet 235ea182e02740f68f30c3c34f77012a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:22 UTC) from {username='slave'} at 127.0.0.1:41678
W20240430 20:11:23.689814 18150 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ddbb3fa4bcd7461cba6de52ebacdb12d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:23.692708 18980 tablet_replica.cc:330] T 269144de63db417a9c9423a913d4536a P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:23.693953 18985 tablet_bootstrap.cc:492] T 4d3117a9a5f243b59a69d960c0a1eee4 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:23.694229 18985 ts_tablet_manager.cc:1367] T 4d3117a9a5f243b59a69d960c0a1eee4 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.126s user 0.003s sys 0.002s
I20240430 20:11:23.695363 18985 raft_consensus.cc:348] T 4d3117a9a5f243b59a69d960c0a1eee4 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:23.695742 18985 raft_consensus.cc:374] T 4d3117a9a5f243b59a69d960c0a1eee4 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:23.695914 18985 raft_consensus.cc:729] T 4d3117a9a5f243b59a69d960c0a1eee4 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:23.696363 18985 consensus_queue.cc:260] T 4d3117a9a5f243b59a69d960c0a1eee4 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:23.697252 18985 ts_tablet_manager.cc:1398] T 4d3117a9a5f243b59a69d960c0a1eee4 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240430 20:11:23.697597 18980 raft_consensus.cc:2229] T 269144de63db417a9c9423a913d4536a P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:23.705958 18146 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet e0d4cebfbfca45c08d1e7500228835b4
I20240430 20:11:23.706502 18987 ts_tablet_manager.cc:1888] T 3525f5a5051d4fb28cd9c95ca6ed6c48 P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:23.711625 18146 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_5b83502eb2f24b9389c3d08006449aab" } modify_external_catalogs: true
I20240430 20:11:23.712144 18146 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_5b83502eb2f24b9389c3d08006449aab" } modify_external_catalogs: true
I20240430 20:11:23.715037 18987 log.cc:1196] T 3525f5a5051d4fb28cd9c95ca6ed6c48 P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/3525f5a5051d4fb28cd9c95ca6ed6c48
I20240430 20:11:23.715790 18987 ts_tablet_manager.cc:1909] T 3525f5a5051d4fb28cd9c95ca6ed6c48 P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:23.715850 18980 raft_consensus.cc:2258] T 269144de63db417a9c9423a913d4536a P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:23.713016 18981 ts_tablet_manager.cc:1888] T 3525f5a5051d4fb28cd9c95ca6ed6c48 P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:23.716324 18981 log.cc:1196] T 3525f5a5051d4fb28cd9c95ca6ed6c48 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/3525f5a5051d4fb28cd9c95ca6ed6c48
I20240430 20:11:23.716936 18981 ts_tablet_manager.cc:1909] T 3525f5a5051d4fb28cd9c95ca6ed6c48 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:23.718295 18985 tablet_bootstrap.cc:492] T f79497bf22ed417d988530f6e69cc07f P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:23.720261 18980 ts_tablet_manager.cc:1875] T 269144de63db417a9c9423a913d4536a P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:23.724846 18142 catalog_manager.cc:4830] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): tablet 3525f5a5051d4fb28cd9c95ca6ed6c48 (table default.Table_c06b54647f004ff7a7f7beafee702b4c [id=a6a27dcbd0e648a8bf29a8cd5b3a2a11]) successfully deleted
I20240430 20:11:23.726786 18981 tablet_replica.cc:330] T 4d3117a9a5f243b59a69d960c0a1eee4 P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:23.729606 18981 raft_consensus.cc:2229] T 4d3117a9a5f243b59a69d960c0a1eee4 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Raft consensus shutting down.
W20240430 20:11:23.730793 18155 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 59548d560b4e4f7ca034d730aca79364 consensus state has no leader
I20240430 20:11:23.731053 18482 raft_consensus.cc:1261] T e5d1707ba0b34cc4b1cda39e36ccbd2f P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Refusing update from remote peer 3c51c33d5616482cafb68e26688820f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240430 20:11:23.732178 19030 consensus_queue.cc:1035] T e5d1707ba0b34cc4b1cda39e36ccbd2f P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240430 20:11:23.738381 18696 consensus_peers.cc:485] T e5d1707ba0b34cc4b1cda39e36ccbd2f P 3c51c33d5616482cafb68e26688820f4 -> Peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Couldn't send request to peer 5e1c6397f09747f091534faf93542710. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240430 20:11:23.738284 18981 raft_consensus.cc:2258] T 4d3117a9a5f243b59a69d960c0a1eee4 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:23.739590 18141 catalog_manager.cc:4830] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): tablet 3525f5a5051d4fb28cd9c95ca6ed6c48 (table default.Table_c06b54647f004ff7a7f7beafee702b4c [id=a6a27dcbd0e648a8bf29a8cd5b3a2a11]) successfully deleted
I20240430 20:11:23.742770 18988 tablet_bootstrap.cc:492] T 6f729a6e8dc14de9a2057cfe09ba9257 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:23.744237 18987 tablet_replica.cc:330] T f6e23412ad5b4773bb62590e452abc7a P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:23.744526 18987 raft_consensus.cc:2229] T f6e23412ad5b4773bb62590e452abc7a P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Raft consensus shutting down.
I20240430 20:11:23.745026 18987 pending_rounds.cc:62] T f6e23412ad5b4773bb62590e452abc7a P 3c51c33d5616482cafb68e26688820f4: Trying to abort 1 pending ops.
I20240430 20:11:23.745244 18987 pending_rounds.cc:68] T f6e23412ad5b4773bb62590e452abc7a P 3c51c33d5616482cafb68e26688820f4: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7022624290071240704 op_type: NO_OP noop_request { }
I20240430 20:11:23.745544 18987 raft_consensus.cc:2866] T f6e23412ad5b4773bb62590e452abc7a P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240430 20:11:23.745743 18987 raft_consensus.cc:2258] T f6e23412ad5b4773bb62590e452abc7a P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:23.745833 18985 tablet_bootstrap.cc:654] T f79497bf22ed417d988530f6e69cc07f P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:23.747455 18987 ts_tablet_manager.cc:1875] T f6e23412ad5b4773bb62590e452abc7a P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:23.749235 19060 mvcc.cc:205] Tried to move back new op lower bound from 7022624291753701376 to 7022624289914105856. Current Snapshot: MvccSnapshot[applied={T|T < 7022624291753701376}]
I20240430 20:11:23.750718 19053 mvcc.cc:205] Tried to move back new op lower bound from 7022624291753701376 to 7022624289914105856. Current Snapshot: MvccSnapshot[applied={T|T < 7022624291753701376}]
I20240430 20:11:23.752223 18154 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: Sending DeleteTablet for 3 replicas of tablet 9750608faeab4be38ddce4e2e4d46964
I20240430 20:11:23.752892 18154 catalog_manager.cc:6554] ReplaceTablet: tablet 9750608faeab4be38ddce4e2e4d46964 deleted and replaced by tablet 0fb0c0ca09154f7a809dd523a859929a
I20240430 20:11:23.757819 18154 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet e5d1707ba0b34cc4b1cda39e36ccbd2f (table default.Table_0ab5b2f09fa7468bb89ab703f6d3d25c [id=a5d580cffa2447d78fc9f2c8e31264f1]). Expected version 4 got 0
I20240430 20:11:23.758968 18154 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54524:
name: "default.Table_3a865e57c98d4c02bfb7910ab5f0be50"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:23.759717 18154 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3a865e57c98d4c02bfb7910ab5f0be50 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:23.761940 18462 tablet_service.cc:1514] Processing DeleteTablet for tablet 9750608faeab4be38ddce4e2e4d46964 with delete_type TABLET_DATA_DELETED (replaced by tablet 0fb0c0ca09154f7a809dd523a859929a) from {username='slave'} at 127.0.0.1:41678
I20240430 20:11:23.763763 18742 tablet_service.cc:1514] Processing DeleteTablet for tablet 9750608faeab4be38ddce4e2e4d46964 with delete_type TABLET_DATA_DELETED (replaced by tablet 0fb0c0ca09154f7a809dd523a859929a) from {username='slave'} at 127.0.0.1:59818
I20240430 20:11:23.768793 18981 ts_tablet_manager.cc:1875] T 4d3117a9a5f243b59a69d960c0a1eee4 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:23.769881 18988 tablet_bootstrap.cc:654] T 6f729a6e8dc14de9a2057cfe09ba9257 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
W20240430 20:11:23.773397 18140 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet bf925791ddcf4fa7b5a1645c63f0d696 (table default.Table_917365c2d10941558081a9898db3d7ea [id=b536a31acaea48d78885b0e585d00832], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet bf925791ddcf4fa7b5a1645c63f0d696 consensus state has no leader
I20240430 20:11:23.776465 18156 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: Sending DeleteTablet for 3 replicas of tablet 0d6a4a63c004480b82d84381867bf8a4
I20240430 20:11:23.777067 18156 catalog_manager.cc:6554] ReplaceTablet: tablet 0d6a4a63c004480b82d84381867bf8a4 deleted and replaced by tablet c9ff972ade694938888514979e1638aa
I20240430 20:11:23.781744 18599 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d6a4a63c004480b82d84381867bf8a4 with delete_type TABLET_DATA_DELETED (replaced by tablet c9ff972ade694938888514979e1638aa) from {username='slave'} at 127.0.0.1:40202
I20240430 20:11:23.787639 18462 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d6a4a63c004480b82d84381867bf8a4 with delete_type TABLET_DATA_DELETED (replaced by tablet c9ff972ade694938888514979e1638aa) from {username='slave'} at 127.0.0.1:41678
I20240430 20:11:23.789218 19056 mvcc.cc:205] Tried to move back new op lower bound from 7022624291524702208 to 7022624290972053504. Current Snapshot: MvccSnapshot[applied={T|T < 7022624291524702208}]
I20240430 20:11:23.789654 18601 tablet_service.cc:1514] Processing DeleteTablet for tablet 9750608faeab4be38ddce4e2e4d46964 with delete_type TABLET_DATA_DELETED (replaced by tablet 0fb0c0ca09154f7a809dd523a859929a) from {username='slave'} at 127.0.0.1:40202
I20240430 20:11:23.774786 19064 tablet.cc:1722] T e5d1707ba0b34cc4b1cda39e36ccbd2f P 3c51c33d5616482cafb68e26688820f4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20240430 20:11:23.781111 19069 tablet.cc:1722] T 17843ea77664447eaaa19666e0445ada P 5e1c6397f09747f091534faf93542710: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240430 20:11:23.788275 18742 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d6a4a63c004480b82d84381867bf8a4 with delete_type TABLET_DATA_DELETED (replaced by tablet c9ff972ade694938888514979e1638aa) from {username='slave'} at 127.0.0.1:59818
I20240430 20:11:23.792858 19065 tablet.cc:1722] T e5d1707ba0b34cc4b1cda39e36ccbd2f P 2efb542aa55e439b99ccb98d1c5c884a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20240430 20:11:23.798259 18149 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: Sending DeleteTablet for 3 replicas of tablet bf925791ddcf4fa7b5a1645c63f0d696
I20240430 20:11:23.799393 18742 tablet_service.cc:1514] Processing DeleteTablet for tablet bf925791ddcf4fa7b5a1645c63f0d696 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:23 UTC) from {username='slave'} at 127.0.0.1:59818
I20240430 20:11:23.799367 18149 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_c06b54647f004ff7a7f7beafee702b4c" } modify_external_catalogs: true
I20240430 20:11:23.799643 18149 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_c06b54647f004ff7a7f7beafee702b4c" } modify_external_catalogs: true
I20240430 20:11:23.800143 18462 tablet_service.cc:1514] Processing DeleteTablet for tablet bf925791ddcf4fa7b5a1645c63f0d696 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:23 UTC) from {username='slave'} at 127.0.0.1:41678
I20240430 20:11:23.800537 18601 tablet_service.cc:1514] Processing DeleteTablet for tablet bf925791ddcf4fa7b5a1645c63f0d696 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:23 UTC) from {username='slave'} at 127.0.0.1:40202
I20240430 20:11:23.800680 19067 tablet.cc:1722] T 17843ea77664447eaaa19666e0445ada P 2efb542aa55e439b99ccb98d1c5c884a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240430 20:11:23.802071 18148 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet e5d1707ba0b34cc4b1cda39e36ccbd2f (table default.Table_0ab5b2f09fa7468bb89ab703f6d3d25c [id=a5d580cffa2447d78fc9f2c8e31264f1]). Expected version 4 got 0
I20240430 20:11:23.803542 18148 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54524:
name: "default.Table_7199d7c36d3f45f79cf9e4e2b4145592"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:23.804852 18148 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7199d7c36d3f45f79cf9e4e2b4145592 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:23.819350 18980 ts_tablet_manager.cc:1888] T 269144de63db417a9c9423a913d4536a P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:23.819981 18980 log.cc:1196] T 269144de63db417a9c9423a913d4536a P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/269144de63db417a9c9423a913d4536a
I20240430 20:11:23.821388 18980 ts_tablet_manager.cc:1909] T 269144de63db417a9c9423a913d4536a P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:23.825091 18985 tablet_bootstrap.cc:492] T f79497bf22ed417d988530f6e69cc07f P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:23.825920 18985 ts_tablet_manager.cc:1367] T f79497bf22ed417d988530f6e69cc07f P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.108s user 0.005s sys 0.000s
I20240430 20:11:23.826131 18139 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet 269144de63db417a9c9423a913d4536a (table default.Table_156650e4b399419c97dcf5160b761ee1 [id=b93708ff15ef4c8299967c8c91045c8f]) successfully deleted
I20240430 20:11:23.827073 18985 raft_consensus.cc:348] T f79497bf22ed417d988530f6e69cc07f P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:23.827412 18985 raft_consensus.cc:374] T f79497bf22ed417d988530f6e69cc07f P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:23.827513 18985 raft_consensus.cc:729] T f79497bf22ed417d988530f6e69cc07f P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:23.827808 18985 consensus_queue.cc:260] T f79497bf22ed417d988530f6e69cc07f P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:23.828143 19068 tablet.cc:1722] T 17843ea77664447eaaa19666e0445ada P 3c51c33d5616482cafb68e26688820f4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240430 20:11:23.829155 18988 tablet_bootstrap.cc:492] T 6f729a6e8dc14de9a2057cfe09ba9257 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:23.829818 18988 ts_tablet_manager.cc:1367] T 6f729a6e8dc14de9a2057cfe09ba9257 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.092s user 0.004s sys 0.000s
I20240430 20:11:23.831521 18980 tablet_replica.cc:330] T f6e23412ad5b4773bb62590e452abc7a P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:23.832232 18980 raft_consensus.cc:2229] T f6e23412ad5b4773bb62590e452abc7a P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:23.832626 18980 raft_consensus.cc:2258] T f6e23412ad5b4773bb62590e452abc7a P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:23.832543 18988 raft_consensus.cc:348] T 6f729a6e8dc14de9a2057cfe09ba9257 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:23.832778 18988 raft_consensus.cc:374] T 6f729a6e8dc14de9a2057cfe09ba9257 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:23.832875 18988 raft_consensus.cc:729] T 6f729a6e8dc14de9a2057cfe09ba9257 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:23.833736 18983 tablet_bootstrap.cc:492] T e5d1707ba0b34cc4b1cda39e36ccbd2f P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:23.833920 18983 ts_tablet_manager.cc:1367] T e5d1707ba0b34cc4b1cda39e36ccbd2f P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.190s user 0.011s sys 0.015s
I20240430 20:11:23.834434 18980 ts_tablet_manager.cc:1875] T f6e23412ad5b4773bb62590e452abc7a P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:23.834717 18983 raft_consensus.cc:348] T e5d1707ba0b34cc4b1cda39e36ccbd2f P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:23.834981 18983 raft_consensus.cc:374] T e5d1707ba0b34cc4b1cda39e36ccbd2f P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:23.835067 18983 raft_consensus.cc:729] T e5d1707ba0b34cc4b1cda39e36ccbd2f P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:23.835335 18983 consensus_queue.cc:260] T e5d1707ba0b34cc4b1cda39e36ccbd2f P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:23.835997 18983 ts_tablet_manager.cc:1398] T e5d1707ba0b34cc4b1cda39e36ccbd2f P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240430 20:11:23.836481 18151 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: Sending DeleteTablet for 3 replicas of tablet 27b21459592649dca1089a7a29021206
I20240430 20:11:23.837150 18151 catalog_manager.cc:6554] ReplaceTablet: tablet 27b21459592649dca1089a7a29021206 deleted and replaced by tablet cace1398492f4d968dcb6899385d527c
I20240430 20:11:23.837162 18983 tablet_bootstrap.cc:492] T 4d3117a9a5f243b59a69d960c0a1eee4 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:23.839833 18601 tablet_service.cc:1514] Processing DeleteTablet for tablet 27b21459592649dca1089a7a29021206 with delete_type TABLET_DATA_DELETED (replaced by tablet cace1398492f4d968dcb6899385d527c) from {username='slave'} at 127.0.0.1:40202
I20240430 20:11:23.841017 18983 tablet_bootstrap.cc:654] T 4d3117a9a5f243b59a69d960c0a1eee4 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:23.842162 18742 tablet_service.cc:1514] Processing DeleteTablet for tablet 27b21459592649dca1089a7a29021206 with delete_type TABLET_DATA_DELETED (replaced by tablet cace1398492f4d968dcb6899385d527c) from {username='slave'} at 127.0.0.1:59818
I20240430 20:11:23.843429 18151 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54524:
name: "default.Table_67a6f326000b49c98d52cfcdf5dea396"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:23.844233 18151 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_67a6f326000b49c98d52cfcdf5dea396 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:23.849009 18987 ts_tablet_manager.cc:1888] T f6e23412ad5b4773bb62590e452abc7a P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:23.853546 18980 ts_tablet_manager.cc:1888] T f6e23412ad5b4773bb62590e452abc7a P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:23.853967 18980 log.cc:1196] T f6e23412ad5b4773bb62590e452abc7a P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/f6e23412ad5b4773bb62590e452abc7a
I20240430 20:11:23.854753 18980 ts_tablet_manager.cc:1909] T f6e23412ad5b4773bb62590e452abc7a P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:23.856130 18147 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet f22d19bea9014700a58aa7ba14943c3d
I20240430 20:11:23.856240 18463 tablet_service.cc:1514] Processing DeleteTablet for tablet 27b21459592649dca1089a7a29021206 with delete_type TABLET_DATA_DELETED (replaced by tablet cace1398492f4d968dcb6899385d527c) from {username='slave'} at 127.0.0.1:41678
I20240430 20:11:23.857643 18139 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet f6e23412ad5b4773bb62590e452abc7a (table default.Table_d4b5e1a26a9449cdbe70f185b1097321 [id=ee9addb7eb9f4430a57fd9ee4bfaae7b]) successfully deleted
W20240430 20:11:23.858647 18139 catalog_manager.cc:4810] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): delete failed for tablet 6f729a6e8dc14de9a2057cfe09ba9257 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6f729a6e8dc14de9a2057cfe09ba9257 already in progress: creating tablet
W20240430 20:11:23.859490 18139 catalog_manager.cc:4810] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): delete failed for tablet b0e6cc37c6b1488fa88359d16b12a576 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b0e6cc37c6b1488fa88359d16b12a576 already in progress: creating tablet
W20240430 20:11:23.860363 18139 catalog_manager.cc:4810] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): delete failed for tablet fe01b95104174279a7d64c0d4c0af982 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fe01b95104174279a7d64c0d4c0af982 already in progress: creating tablet
I20240430 20:11:23.860239 18147 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54524:
name: "default.Table_14776edb17d64a83be63345e68995156"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:23.860982 18147 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_14776edb17d64a83be63345e68995156 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240430 20:11:23.861217 18139 catalog_manager.cc:4810] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): delete failed for tablet c3f8f9ada98b40bc8359a00e193c3e5f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c3f8f9ada98b40bc8359a00e193c3e5f already in progress: creating tablet
W20240430 20:11:23.861950 18139 catalog_manager.cc:4810] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): delete failed for tablet 0d6a4a63c004480b82d84381867bf8a4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0d6a4a63c004480b82d84381867bf8a4 already in progress: creating tablet
I20240430 20:11:23.862342 18983 tablet_bootstrap.cc:492] T 4d3117a9a5f243b59a69d960c0a1eee4 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:23.862520 18983 ts_tablet_manager.cc:1367] T 4d3117a9a5f243b59a69d960c0a1eee4 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.026s user 0.003s sys 0.003s
W20240430 20:11:23.862677 18139 catalog_manager.cc:4810] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): delete failed for tablet 9750608faeab4be38ddce4e2e4d46964 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9750608faeab4be38ddce4e2e4d46964 already in progress: creating tablet
W20240430 20:11:23.863523 18139 catalog_manager.cc:4810] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): delete failed for tablet bf925791ddcf4fa7b5a1645c63f0d696 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bf925791ddcf4fa7b5a1645c63f0d696 already in progress: creating tablet
I20240430 20:11:23.863718 18983 raft_consensus.cc:348] T 4d3117a9a5f243b59a69d960c0a1eee4 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:23.863935 18980 tablet_replica.cc:330] T 27b21459592649dca1089a7a29021206 P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:23.864449 18980 raft_consensus.cc:2229] T 27b21459592649dca1089a7a29021206 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:23.864756 18980 raft_consensus.cc:2258] T 27b21459592649dca1089a7a29021206 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:23.865782 18980 ts_tablet_manager.cc:1875] T 27b21459592649dca1089a7a29021206 P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:23.864037 18983 raft_consensus.cc:374] T 4d3117a9a5f243b59a69d960c0a1eee4 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:23.866564 18983 raft_consensus.cc:729] T 4d3117a9a5f243b59a69d960c0a1eee4 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:23.872269 18987 log.cc:1196] T f6e23412ad5b4773bb62590e452abc7a P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/f6e23412ad5b4773bb62590e452abc7a
I20240430 20:11:23.872891 18987 ts_tablet_manager.cc:1909] T f6e23412ad5b4773bb62590e452abc7a P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:23.874136 18141 catalog_manager.cc:4830] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): tablet f6e23412ad5b4773bb62590e452abc7a (table default.Table_d4b5e1a26a9449cdbe70f185b1097321 [id=ee9addb7eb9f4430a57fd9ee4bfaae7b]) successfully deleted
I20240430 20:11:23.874583 18987 tablet_replica.cc:330] T 27b21459592649dca1089a7a29021206 P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
W20240430 20:11:23.874794 18141 catalog_manager.cc:4810] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): delete failed for tablet 9750608faeab4be38ddce4e2e4d46964 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9750608faeab4be38ddce4e2e4d46964 already in progress: creating tablet
I20240430 20:11:23.874837 18987 raft_consensus.cc:2229] T 27b21459592649dca1089a7a29021206 P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Raft consensus shutting down.
I20240430 20:11:23.875098 18987 pending_rounds.cc:62] T 27b21459592649dca1089a7a29021206 P 3c51c33d5616482cafb68e26688820f4: Trying to abort 1 pending ops.
W20240430 20:11:23.875094 18141 catalog_manager.cc:4810] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): delete failed for tablet 0d6a4a63c004480b82d84381867bf8a4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0d6a4a63c004480b82d84381867bf8a4 already in progress: creating tablet
I20240430 20:11:23.875155 18987 pending_rounds.cc:68] T 27b21459592649dca1089a7a29021206 P 3c51c33d5616482cafb68e26688820f4: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7022624290939555840 op_type: NO_OP noop_request { }
I20240430 20:11:23.875293 18987 raft_consensus.cc:2866] T 27b21459592649dca1089a7a29021206 P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
W20240430 20:11:23.875313 18141 catalog_manager.cc:4810] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): delete failed for tablet bf925791ddcf4fa7b5a1645c63f0d696 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bf925791ddcf4fa7b5a1645c63f0d696 already in progress: creating tablet
I20240430 20:11:23.875403 18987 raft_consensus.cc:2258] T 27b21459592649dca1089a7a29021206 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:23.876242 18987 ts_tablet_manager.cc:1875] T 27b21459592649dca1089a7a29021206 P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:23.878122 19067 tablet.cc:1717] T 17843ea77664447eaaa19666e0445ada P 2efb542aa55e439b99ccb98d1c5c884a: Skipping requested alter to schema version 3, tablet already version 3
I20240430 20:11:23.878942 18150 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_6d2fc9759c4741f780a8096a79261d75" } new_table_name: "default.Table_873c21f6dffd4d47a742b3699ab85268" modify_external_catalogs: true
I20240430 20:11:23.881213 18985 ts_tablet_manager.cc:1398] T f79497bf22ed417d988530f6e69cc07f P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.055s user 0.003s sys 0.000s
I20240430 20:11:23.883262 18987 ts_tablet_manager.cc:1888] T 27b21459592649dca1089a7a29021206 P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:23.883544 18987 log.cc:1196] T 27b21459592649dca1089a7a29021206 P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/27b21459592649dca1089a7a29021206
I20240430 20:11:23.884053 18987 ts_tablet_manager.cc:1909] T 27b21459592649dca1089a7a29021206 P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:23.884233 19068 tablet.cc:1717] T 17843ea77664447eaaa19666e0445ada P 3c51c33d5616482cafb68e26688820f4: Skipping requested alter to schema version 3, tablet already version 3
I20240430 20:11:23.885185 18141 catalog_manager.cc:4830] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): tablet 27b21459592649dca1089a7a29021206 (table default.Table_a8d5ac73c6d047698d87b3d1ba34e66f [id=9c80193cde2346c982cf84ae9612e2f7]) successfully deleted
I20240430 20:11:23.854163 18988 consensus_queue.cc:260] T 6f729a6e8dc14de9a2057cfe09ba9257 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:23.885555 18985 tablet_bootstrap.cc:492] T bf925791ddcf4fa7b5a1645c63f0d696 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:23.885854 18988 ts_tablet_manager.cc:1398] T 6f729a6e8dc14de9a2057cfe09ba9257 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.054s user 0.002s sys 0.000s
I20240430 20:11:23.886159 18988 tablet_bootstrap.cc:492] T bf925791ddcf4fa7b5a1645c63f0d696 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:23.887414 18985 tablet_bootstrap.cc:654] T bf925791ddcf4fa7b5a1645c63f0d696 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:23.887743 18988 tablet_bootstrap.cc:654] T bf925791ddcf4fa7b5a1645c63f0d696 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:23.891691 18981 ts_tablet_manager.cc:1888] T 4d3117a9a5f243b59a69d960c0a1eee4 P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:23.891849 18981 log.cc:1196] T 4d3117a9a5f243b59a69d960c0a1eee4 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/4d3117a9a5f243b59a69d960c0a1eee4
I20240430 20:11:23.892269 18981 ts_tablet_manager.cc:1909] T 4d3117a9a5f243b59a69d960c0a1eee4 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
W20240430 20:11:23.893620 18142 catalog_manager.cc:4810] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): delete failed for tablet b0e6cc37c6b1488fa88359d16b12a576 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b0e6cc37c6b1488fa88359d16b12a576 already in progress: creating tablet
I20240430 20:11:23.893865 18981 tablet_replica.cc:330] T 235ea182e02740f68f30c3c34f77012a P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
W20240430 20:11:23.894039 18142 catalog_manager.cc:4810] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): delete failed for tablet 6f729a6e8dc14de9a2057cfe09ba9257 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6f729a6e8dc14de9a2057cfe09ba9257 already in progress: creating tablet
I20240430 20:11:23.894088 18981 raft_consensus.cc:2229] T 235ea182e02740f68f30c3c34f77012a P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Raft consensus shutting down.
I20240430 20:11:23.894343 18981 pending_rounds.cc:62] T 235ea182e02740f68f30c3c34f77012a P 2efb542aa55e439b99ccb98d1c5c884a: Trying to abort 1 pending ops.
W20240430 20:11:23.894404 18142 catalog_manager.cc:4810] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): delete failed for tablet fe01b95104174279a7d64c0d4c0af982 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fe01b95104174279a7d64c0d4c0af982 already in progress: creating tablet
W20240430 20:11:23.895012 18142 catalog_manager.cc:4810] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): delete failed for tablet c3f8f9ada98b40bc8359a00e193c3e5f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c3f8f9ada98b40bc8359a00e193c3e5f already in progress: creating tablet
I20240430 20:11:23.898363 19069 tablet.cc:1717] T 17843ea77664447eaaa19666e0445ada P 5e1c6397f09747f091534faf93542710: Skipping requested alter to schema version 3, tablet already version 3
I20240430 20:11:23.898408 18983 consensus_queue.cc:260] T 4d3117a9a5f243b59a69d960c0a1eee4 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:23.899063 18983 ts_tablet_manager.cc:1398] T 4d3117a9a5f243b59a69d960c0a1eee4 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.036s user 0.002s sys 0.000s
I20240430 20:11:23.899315 18985 tablet_bootstrap.cc:492] T bf925791ddcf4fa7b5a1645c63f0d696 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:23.899405 18983 tablet_bootstrap.cc:492] T f79497bf22ed417d988530f6e69cc07f P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:23.899593 18985 ts_tablet_manager.cc:1367] T bf925791ddcf4fa7b5a1645c63f0d696 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.003s
I20240430 20:11:23.900653 18985 raft_consensus.cc:348] T bf925791ddcf4fa7b5a1645c63f0d696 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:23.901072 18985 raft_consensus.cc:374] T bf925791ddcf4fa7b5a1645c63f0d696 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:23.901222 18985 raft_consensus.cc:729] T bf925791ddcf4fa7b5a1645c63f0d696 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:23.901576 18985 consensus_queue.cc:260] T bf925791ddcf4fa7b5a1645c63f0d696 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:23.903896 18983 tablet_bootstrap.cc:654] T f79497bf22ed417d988530f6e69cc07f P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:23.903854 18985 ts_tablet_manager.cc:1398] T bf925791ddcf4fa7b5a1645c63f0d696 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240430 20:11:23.905483 18985 tablet_bootstrap.cc:492] T 2067cb2cf76d495c92c1f25c176eb6dc P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:23.905838 18980 ts_tablet_manager.cc:1888] T 27b21459592649dca1089a7a29021206 P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:23.905993 18980 log.cc:1196] T 27b21459592649dca1089a7a29021206 P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/27b21459592649dca1089a7a29021206
I20240430 20:11:23.906407 18980 ts_tablet_manager.cc:1909] T 27b21459592649dca1089a7a29021206 P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:23.907492 18139 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet 27b21459592649dca1089a7a29021206 (table default.Table_a8d5ac73c6d047698d87b3d1ba34e66f [id=9c80193cde2346c982cf84ae9612e2f7]) successfully deleted
I20240430 20:11:23.909268 18985 tablet_bootstrap.cc:654] T 2067cb2cf76d495c92c1f25c176eb6dc P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:23.910348 18988 tablet_bootstrap.cc:492] T bf925791ddcf4fa7b5a1645c63f0d696 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:23.910619 18988 ts_tablet_manager.cc:1367] T bf925791ddcf4fa7b5a1645c63f0d696 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.000s
I20240430 20:11:23.911535 18988 raft_consensus.cc:348] T bf925791ddcf4fa7b5a1645c63f0d696 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:23.911855 18988 raft_consensus.cc:374] T bf925791ddcf4fa7b5a1645c63f0d696 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:23.912051 18988 raft_consensus.cc:729] T bf925791ddcf4fa7b5a1645c63f0d696 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:23.912370 18988 consensus_queue.cc:260] T bf925791ddcf4fa7b5a1645c63f0d696 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:23.912938 18983 tablet_bootstrap.cc:492] T f79497bf22ed417d988530f6e69cc07f P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:23.912963 18988 ts_tablet_manager.cc:1398] T bf925791ddcf4fa7b5a1645c63f0d696 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240430 20:11:23.913182 18983 ts_tablet_manager.cc:1367] T f79497bf22ed417d988530f6e69cc07f P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.003s
I20240430 20:11:23.913337 18988 tablet_bootstrap.cc:492] T f79497bf22ed417d988530f6e69cc07f P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:23.913447 18146 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet d4aba87631de460d9ccbe18fe6354655
I20240430 20:11:23.914048 18983 raft_consensus.cc:348] T f79497bf22ed417d988530f6e69cc07f P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:23.914307 18983 raft_consensus.cc:374] T f79497bf22ed417d988530f6e69cc07f P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:23.914399 18983 raft_consensus.cc:729] T f79497bf22ed417d988530f6e69cc07f P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:23.914710 18983 consensus_queue.cc:260] T f79497bf22ed417d988530f6e69cc07f P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:23.916679 18988 tablet_bootstrap.cc:654] T f79497bf22ed417d988530f6e69cc07f P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:23.923681 18983 ts_tablet_manager.cc:1398] T f79497bf22ed417d988530f6e69cc07f P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240430 20:11:23.927428 18983 tablet_bootstrap.cc:492] T bf925791ddcf4fa7b5a1645c63f0d696 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:23.929716 18983 tablet_bootstrap.cc:654] T bf925791ddcf4fa7b5a1645c63f0d696 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:23.931456 18154 master_service.cc:946] ReplaceTablet: received request to replace tablet e5d1707ba0b34cc4b1cda39e36ccbd2f from {username='slave'} at 127.0.0.1:54524
I20240430 20:11:23.932973 18149 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet 0fb0c0ca09154f7a809dd523a859929a
I20240430 20:11:23.933423 18149 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54524:
name: "default.Table_40d60ca37bac4d53a9297289f18c1719"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:23.934188 18149 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_40d60ca37bac4d53a9297289f18c1719 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:23.934253 18983 tablet_bootstrap.cc:492] T bf925791ddcf4fa7b5a1645c63f0d696 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:23.934408 18983 ts_tablet_manager.cc:1367] T bf925791ddcf4fa7b5a1645c63f0d696 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20240430 20:11:23.935858 18985 tablet_bootstrap.cc:492] T 2067cb2cf76d495c92c1f25c176eb6dc P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:23.936098 18985 ts_tablet_manager.cc:1367] T 2067cb2cf76d495c92c1f25c176eb6dc P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.031s user 0.002s sys 0.005s
I20240430 20:11:23.935978 18983 raft_consensus.cc:348] T bf925791ddcf4fa7b5a1645c63f0d696 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:23.936184 18983 raft_consensus.cc:374] T bf925791ddcf4fa7b5a1645c63f0d696 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:23.936282 18983 raft_consensus.cc:729] T bf925791ddcf4fa7b5a1645c63f0d696 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:23.936556 18983 consensus_queue.cc:260] T bf925791ddcf4fa7b5a1645c63f0d696 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:23.937502 18155 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_11aeb11694264720a7e795ac7a114ead" } new_table_name: "default.Table_b927029492584700848792c3500ac17e" modify_external_catalogs: true
I20240430 20:11:23.938419 18983 ts_tablet_manager.cc:1398] T bf925791ddcf4fa7b5a1645c63f0d696 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240430 20:11:23.938731 18983 tablet_bootstrap.cc:492] T 6f729a6e8dc14de9a2057cfe09ba9257 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:23.938694 19042 raft_consensus.cc:482] T e5d1707ba0b34cc4b1cda39e36ccbd2f P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:23.938831 19042 raft_consensus.cc:504] T e5d1707ba0b34cc4b1cda39e36ccbd2f P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:23.939705 19042 leader_election.cc:288] T e5d1707ba0b34cc4b1cda39e36ccbd2f P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:23.939976 18146 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54524:
name: "default.Table_0c6652c070534fbbaab6c2b8d175f139"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:23.940706 18146 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0c6652c070534fbbaab6c2b8d175f139 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:23.940732 18742 tablet_service.cc:1514] Processing DeleteTablet for tablet 3525f5a5051d4fb28cd9c95ca6ed6c48 with delete_type TABLET_DATA_DELETED (replaced by tablet 9750608faeab4be38ddce4e2e4d46964) from {username='slave'} at 127.0.0.1:59818
I20240430 20:11:23.940971 18739 tablet_service.cc:1514] Processing DeleteTablet for tablet 27b21459592649dca1089a7a29021206 with delete_type TABLET_DATA_DELETED (replaced by tablet cace1398492f4d968dcb6899385d527c) from {username='slave'} at 127.0.0.1:59818
I20240430 20:11:23.941113 18741 tablet_service.cc:1514] Processing DeleteTablet for tablet 235ea182e02740f68f30c3c34f77012a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:22 UTC) from {username='slave'} at 127.0.0.1:59818
I20240430 20:11:23.941280 18743 tablet_service.cc:1514] Processing DeleteTablet for tablet 269144de63db417a9c9423a913d4536a with delete_type TABLET_DATA_DELETED (replaced by tablet 27f94f2d8f50495dababe417a9f82982) from {username='slave'} at 127.0.0.1:59818
I20240430 20:11:23.941665 18762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5d1707ba0b34cc4b1cda39e36ccbd2f" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:23.942497 18482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5d1707ba0b34cc4b1cda39e36ccbd2f" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:23.944921 18558 leader_election.cc:302] T e5d1707ba0b34cc4b1cda39e36ccbd2f P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:23.949381 19042 raft_consensus.cc:2726] T e5d1707ba0b34cc4b1cda39e36ccbd2f P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240430 20:11:23.950249 18141 catalog_manager.cc:4797] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): delete failed for tablet 3525f5a5051d4fb28cd9c95ca6ed6c48 because the tablet was not found. No further retry: Not found: Tablet not found: 3525f5a5051d4fb28cd9c95ca6ed6c48
I20240430 20:11:23.950381 18987 tablet_replica.cc:330] T 235ea182e02740f68f30c3c34f77012a P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:23.950574 18987 raft_consensus.cc:2229] T 235ea182e02740f68f30c3c34f77012a P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:23.950769 18987 raft_consensus.cc:2258] T 235ea182e02740f68f30c3c34f77012a P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:23.951387 18987 ts_tablet_manager.cc:1875] T 235ea182e02740f68f30c3c34f77012a P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240430 20:11:23.951370 18141 catalog_manager.cc:4797] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): delete failed for tablet 27b21459592649dca1089a7a29021206 because the tablet was not found. No further retry: Not found: Tablet not found: 27b21459592649dca1089a7a29021206
I20240430 20:11:23.954623 18985 raft_consensus.cc:348] T 2067cb2cf76d495c92c1f25c176eb6dc P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:23.954875 18985 raft_consensus.cc:374] T 2067cb2cf76d495c92c1f25c176eb6dc P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:23.954955 18985 raft_consensus.cc:729] T 2067cb2cf76d495c92c1f25c176eb6dc P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:23.955219 18985 consensus_queue.cc:260] T 2067cb2cf76d495c92c1f25c176eb6dc P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:23.956434 18985 ts_tablet_manager.cc:1398] T 2067cb2cf76d495c92c1f25c176eb6dc P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.020s user 0.000s sys 0.002s
I20240430 20:11:23.956779 18985 tablet_bootstrap.cc:492] T 6f729a6e8dc14de9a2057cfe09ba9257 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:23.958375 18985 tablet_bootstrap.cc:654] T 6f729a6e8dc14de9a2057cfe09ba9257 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:23.894495 18981 pending_rounds.cc:68] T 235ea182e02740f68f30c3c34f77012a P 2efb542aa55e439b99ccb98d1c5c884a: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7022624291329019904 op_type: NO_OP noop_request { }
I20240430 20:11:23.966975 18983 tablet_bootstrap.cc:654] T 6f729a6e8dc14de9a2057cfe09ba9257 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:23.966914 18981 raft_consensus.cc:2866] T 235ea182e02740f68f30c3c34f77012a P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240430 20:11:23.968935 18151 master_service.cc:946] ReplaceTablet: received request to replace tablet 6f729a6e8dc14de9a2057cfe09ba9257 from {username='slave'} at 127.0.0.1:54524
I20240430 20:11:23.969686 18151 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_b4c8addb5cfe442d8ec19324de4bb523" } new_table_name: "default.Table_062e0e190e17433395f8f47234987d92" modify_external_catalogs: true
I20240430 20:11:23.974161 18987 ts_tablet_manager.cc:1888] T 235ea182e02740f68f30c3c34f77012a P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:23.975261 18147 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54524:
name: "default.Table_99fded9dcd7a4147a4373fc15d9d2677"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240430 20:11:23.976130 18983 tablet_bootstrap.cc:492] T 6f729a6e8dc14de9a2057cfe09ba9257 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:23.976315 18983 ts_tablet_manager.cc:1367] T 6f729a6e8dc14de9a2057cfe09ba9257 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.038s user 0.003s sys 0.000s
W20240430 20:11:23.976769 18147 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_99fded9dcd7a4147a4373fc15d9d2677 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:23.976974 18981 raft_consensus.cc:2258] T 235ea182e02740f68f30c3c34f77012a P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:23.977106 18983 raft_consensus.cc:348] T 6f729a6e8dc14de9a2057cfe09ba9257 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:23.977314 18983 raft_consensus.cc:374] T 6f729a6e8dc14de9a2057cfe09ba9257 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:23.977389 18983 raft_consensus.cc:729] T 6f729a6e8dc14de9a2057cfe09ba9257 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:23.977676 18983 consensus_queue.cc:260] T 6f729a6e8dc14de9a2057cfe09ba9257 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:23.980722 18985 tablet_bootstrap.cc:492] T 6f729a6e8dc14de9a2057cfe09ba9257 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:23.981565 18981 ts_tablet_manager.cc:1875] T 235ea182e02740f68f30c3c34f77012a P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240430 20:11:23.982569 18150 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f15f5e1f3afd4224bbf39567cae74617 consensus state has no leader
I20240430 20:11:23.981022 18985 ts_tablet_manager.cc:1367] T 6f729a6e8dc14de9a2057cfe09ba9257 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.024s user 0.001s sys 0.005s
I20240430 20:11:23.984372 18983 ts_tablet_manager.cc:1398] T 6f729a6e8dc14de9a2057cfe09ba9257 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240430 20:11:23.984710 18983 tablet_bootstrap.cc:492] T 2067cb2cf76d495c92c1f25c176eb6dc P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:23.985347 18150 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_a9b96b394f274b2bae5cfe2f0786ec92" } new_table_name: "default.Table_a4d0abf694944132bd0029c817df66fc" modify_external_catalogs: true
I20240430 20:11:23.988258 18983 tablet_bootstrap.cc:654] T 2067cb2cf76d495c92c1f25c176eb6dc P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:23.995695 18156 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 17843ea77664447eaaa19666e0445ada (table default.Table_16b082d140234e1c9a1e70654839b4cd [id=c0d8dc87d1f9426a85f6176f161595a1]). Expected version 3 got 0
I20240430 20:11:23.997649 18981 ts_tablet_manager.cc:1888] T 235ea182e02740f68f30c3c34f77012a P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:23.998780 18981 log.cc:1196] T 235ea182e02740f68f30c3c34f77012a P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/235ea182e02740f68f30c3c34f77012a
I20240430 20:11:23.999454 18981 ts_tablet_manager.cc:1909] T 235ea182e02740f68f30c3c34f77012a P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:23.999490 18743 tablet_service.cc:1467] Processing CreateTablet for tablet 3f49eda7aeca42308d01b83a19d7d037 (DEFAULT_TABLE table=default.Table_2d41f75890eb40199519c2937c89e658 [id=ff86c0357c164dd0991098c416087773]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:24.000056 18743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f49eda7aeca42308d01b83a19d7d037. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:24.001375 18981 tablet_replica.cc:330] T bf925791ddcf4fa7b5a1645c63f0d696 P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:24.002131 18987 log.cc:1196] T 235ea182e02740f68f30c3c34f77012a P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/235ea182e02740f68f30c3c34f77012a
I20240430 20:11:24.002950 18981 raft_consensus.cc:2229] T bf925791ddcf4fa7b5a1645c63f0d696 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:24.003350 18987 ts_tablet_manager.cc:1909] T 235ea182e02740f68f30c3c34f77012a P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:24.003424 18601 tablet_service.cc:1467] Processing CreateTablet for tablet 3f49eda7aeca42308d01b83a19d7d037 (DEFAULT_TABLE table=default.Table_2d41f75890eb40199519c2937c89e658 [id=ff86c0357c164dd0991098c416087773]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:24.003571 18981 raft_consensus.cc:2258] T bf925791ddcf4fa7b5a1645c63f0d696 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:24.003947 18601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f49eda7aeca42308d01b83a19d7d037. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:24.004452 18985 raft_consensus.cc:348] T 6f729a6e8dc14de9a2057cfe09ba9257 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:24.004693 18985 raft_consensus.cc:374] T 6f729a6e8dc14de9a2057cfe09ba9257 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:24.004999 18987 tablet_replica.cc:330] T 269144de63db417a9c9423a913d4536a P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:24.005298 18987 raft_consensus.cc:2229] T 269144de63db417a9c9423a913d4536a P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Raft consensus shutting down.
I20240430 20:11:24.005571 18987 pending_rounds.cc:62] T 269144de63db417a9c9423a913d4536a P 3c51c33d5616482cafb68e26688820f4: Trying to abort 1 pending ops.
I20240430 20:11:24.005704 18985 raft_consensus.cc:729] T 6f729a6e8dc14de9a2057cfe09ba9257 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:24.005676 18987 pending_rounds.cc:68] T 269144de63db417a9c9423a913d4536a P 3c51c33d5616482cafb68e26688820f4: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7022624291425566720 op_type: NO_OP noop_request { }
I20240430 20:11:24.005805 18987 raft_consensus.cc:2866] T 269144de63db417a9c9423a913d4536a P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240430 20:11:24.005937 18987 raft_consensus.cc:2258] T 269144de63db417a9c9423a913d4536a P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:24.006156 18985 consensus_queue.cc:260] T 6f729a6e8dc14de9a2057cfe09ba9257 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:24.007174 18985 ts_tablet_manager.cc:1398] T 6f729a6e8dc14de9a2057cfe09ba9257 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.024s user 0.001s sys 0.001s
I20240430 20:11:24.007544 18985 tablet_bootstrap.cc:492] T 9750608faeab4be38ddce4e2e4d46964 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:24.008288 18463 tablet_service.cc:1467] Processing CreateTablet for tablet 3f49eda7aeca42308d01b83a19d7d037 (DEFAULT_TABLE table=default.Table_2d41f75890eb40199519c2937c89e658 [id=ff86c0357c164dd0991098c416087773]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:24.008738 18463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f49eda7aeca42308d01b83a19d7d037. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:24.009409 18985 tablet_bootstrap.cc:654] T 9750608faeab4be38ddce4e2e4d46964 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:24.010927 18981 ts_tablet_manager.cc:1875] T bf925791ddcf4fa7b5a1645c63f0d696 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:24.011066 19079 raft_consensus.cc:482] T 2067cb2cf76d495c92c1f25c176eb6dc P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:24.011253 19079 raft_consensus.cc:504] T 2067cb2cf76d495c92c1f25c176eb6dc P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:24.012177 19079 leader_election.cc:288] T 2067cb2cf76d495c92c1f25c176eb6dc P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:24.012679 18621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2067cb2cf76d495c92c1f25c176eb6dc" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:24.012814 18141 catalog_manager.cc:4830] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): tablet 235ea182e02740f68f30c3c34f77012a (table default.Table_33059b681e564888992fa98876243716 [id=2eae27505e2b4218b933f03c397a8495]) successfully deleted
I20240430 20:11:24.013506 18142 catalog_manager.cc:4830] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): tablet 235ea182e02740f68f30c3c34f77012a (table default.Table_33059b681e564888992fa98876243716 [id=2eae27505e2b4218b933f03c397a8495]) successfully deleted
W20240430 20:11:24.013847 18142 catalog_manager.cc:4810] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): delete failed for tablet 0d6a4a63c004480b82d84381867bf8a4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0d6a4a63c004480b82d84381867bf8a4 already in progress: creating tablet
W20240430 20:11:24.014322 18415 leader_election.cc:341] T 2067cb2cf76d495c92c1f25c176eb6dc P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:24.018641 18762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2067cb2cf76d495c92c1f25c176eb6dc" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:24.019920 18987 ts_tablet_manager.cc:1875] T 269144de63db417a9c9423a913d4536a P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240430 20:11:24.019943 18417 leader_election.cc:341] T 2067cb2cf76d495c92c1f25c176eb6dc P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:24.020118 18417 leader_election.cc:302] T 2067cb2cf76d495c92c1f25c176eb6dc P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a; no voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710
I20240430 20:11:24.020725 19079 raft_consensus.cc:2726] T 2067cb2cf76d495c92c1f25c176eb6dc P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:24.030016 18599 tablet_service.cc:1467] Processing CreateTablet for tablet 81787eccca6c42c1b421377a3bf9ebe6 (DEFAULT_TABLE table=default.Table_c2b7ceaf18b24ff3b42e4fa585fcdb10 [id=7f602584cef9486eb7d9b4389be3f43e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:24.030512 18599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81787eccca6c42c1b421377a3bf9ebe6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:24.030494 18741 tablet_service.cc:1467] Processing CreateTablet for tablet 81787eccca6c42c1b421377a3bf9ebe6 (DEFAULT_TABLE table=default.Table_c2b7ceaf18b24ff3b42e4fa585fcdb10 [id=7f602584cef9486eb7d9b4389be3f43e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:24.030882 18741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81787eccca6c42c1b421377a3bf9ebe6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:24.032893 18981 ts_tablet_manager.cc:1888] T bf925791ddcf4fa7b5a1645c63f0d696 P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:24.033593 18981 log.cc:1196] T bf925791ddcf4fa7b5a1645c63f0d696 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/bf925791ddcf4fa7b5a1645c63f0d696
I20240430 20:11:24.033951 18981 ts_tablet_manager.cc:1909] T bf925791ddcf4fa7b5a1645c63f0d696 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:24.034943 18603 tablet_service.cc:1467] Processing CreateTablet for tablet bd7392f3dfd643cdb1634ee10ad41651 (DEFAULT_TABLE table=default.Table_b29130df231a4316bcb8f9c2be002c92 [id=5e77a9e6adfa4bf0b1c1f81be22e6499]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:24.035382 18603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd7392f3dfd643cdb1634ee10ad41651. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:24.035494 18985 tablet_bootstrap.cc:492] T 9750608faeab4be38ddce4e2e4d46964 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:24.035652 18985 ts_tablet_manager.cc:1367] T 9750608faeab4be38ddce4e2e4d46964 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.028s user 0.001s sys 0.005s
I20240430 20:11:24.036656 18987 ts_tablet_manager.cc:1888] T 269144de63db417a9c9423a913d4536a P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:24.036803 18987 log.cc:1196] T 269144de63db417a9c9423a913d4536a P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/269144de63db417a9c9423a913d4536a
I20240430 20:11:24.036479 18985 raft_consensus.cc:348] T 9750608faeab4be38ddce4e2e4d46964 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:24.037127 18985 raft_consensus.cc:374] T 9750608faeab4be38ddce4e2e4d46964 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:24.037204 18985 raft_consensus.cc:729] T 9750608faeab4be38ddce4e2e4d46964 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:24.037290 18987 ts_tablet_manager.cc:1909] T 269144de63db417a9c9423a913d4536a P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:24.038143 18985 consensus_queue.cc:260] T 9750608faeab4be38ddce4e2e4d46964 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:24.038391 18141 catalog_manager.cc:4830] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): tablet 269144de63db417a9c9423a913d4536a (table default.Table_156650e4b399419c97dcf5160b761ee1 [id=b93708ff15ef4c8299967c8c91045c8f]) successfully deleted
I20240430 20:11:24.039258 18457 tablet_service.cc:1467] Processing CreateTablet for tablet bd7392f3dfd643cdb1634ee10ad41651 (DEFAULT_TABLE table=default.Table_b29130df231a4316bcb8f9c2be002c92 [id=5e77a9e6adfa4bf0b1c1f81be22e6499]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:24.039669 18457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd7392f3dfd643cdb1634ee10ad41651. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:24.043350 18462 tablet_service.cc:1467] Processing CreateTablet for tablet 81787eccca6c42c1b421377a3bf9ebe6 (DEFAULT_TABLE table=default.Table_c2b7ceaf18b24ff3b42e4fa585fcdb10 [id=7f602584cef9486eb7d9b4389be3f43e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:24.043741 18462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81787eccca6c42c1b421377a3bf9ebe6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:24.045111 18985 ts_tablet_manager.cc:1398] T 9750608faeab4be38ddce4e2e4d46964 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20240430 20:11:24.045481 18985 tablet_bootstrap.cc:492] T 26a015250eeb4750a42df93840e8d47f P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:24.050134 18142 catalog_manager.cc:4830] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): tablet bf925791ddcf4fa7b5a1645c63f0d696 (table default.Table_917365c2d10941558081a9898db3d7ea [id=b536a31acaea48d78885b0e585d00832]) successfully deleted
I20240430 20:11:24.052327 18985 tablet_bootstrap.cc:654] T 26a015250eeb4750a42df93840e8d47f P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:24.053311 18981 tablet_replica.cc:330] T 27b21459592649dca1089a7a29021206 P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:24.053519 18981 raft_consensus.cc:2229] T 27b21459592649dca1089a7a29021206 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:24.053720 18981 raft_consensus.cc:2258] T 27b21459592649dca1089a7a29021206 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:24.054231 18981 ts_tablet_manager.cc:1875] T 27b21459592649dca1089a7a29021206 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:24.062875 18739 tablet_service.cc:1467] Processing CreateTablet for tablet bd7392f3dfd643cdb1634ee10ad41651 (DEFAULT_TABLE table=default.Table_b29130df231a4316bcb8f9c2be002c92 [id=5e77a9e6adfa4bf0b1c1f81be22e6499]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:24.063328 18739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd7392f3dfd643cdb1634ee10ad41651. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:24.067775 19032 raft_consensus.cc:482] T bf925791ddcf4fa7b5a1645c63f0d696 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:24.067919 19032 raft_consensus.cc:504] T bf925791ddcf4fa7b5a1645c63f0d696 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:24.068682 19032 leader_election.cc:288] T bf925791ddcf4fa7b5a1645c63f0d696 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:24.069402 18621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf925791ddcf4fa7b5a1645c63f0d696" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:24.069609 18621 raft_consensus.cc:2454] T bf925791ddcf4fa7b5a1645c63f0d696 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:24.070062 18696 leader_election.cc:302] T bf925791ddcf4fa7b5a1645c63f0d696 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
W20240430 20:11:24.071341 18699 leader_election.cc:341] T bf925791ddcf4fa7b5a1645c63f0d696 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Not found: Tablet not found: bf925791ddcf4fa7b5a1645c63f0d696
I20240430 20:11:24.071712 19032 raft_consensus.cc:2781] T bf925791ddcf4fa7b5a1645c63f0d696 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:24.071813 19032 raft_consensus.cc:482] T bf925791ddcf4fa7b5a1645c63f0d696 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:24.071884 19032 raft_consensus.cc:3037] T bf925791ddcf4fa7b5a1645c63f0d696 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:24.078521 19032 raft_consensus.cc:504] T bf925791ddcf4fa7b5a1645c63f0d696 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
W20240430 20:11:24.080228 18699 leader_election.cc:341] T bf925791ddcf4fa7b5a1645c63f0d696 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Not found: Tablet not found: bf925791ddcf4fa7b5a1645c63f0d696
I20240430 20:11:24.081219 18981 ts_tablet_manager.cc:1888] T 27b21459592649dca1089a7a29021206 P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:24.081177 18621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf925791ddcf4fa7b5a1645c63f0d696" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:24.081351 18981 log.cc:1196] T 27b21459592649dca1089a7a29021206 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/27b21459592649dca1089a7a29021206
I20240430 20:11:24.081426 18621 raft_consensus.cc:3037] T bf925791ddcf4fa7b5a1645c63f0d696 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:24.081727 18981 ts_tablet_manager.cc:1909] T 27b21459592649dca1089a7a29021206 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:24.082693 18142 catalog_manager.cc:4830] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): tablet 27b21459592649dca1089a7a29021206 (table default.Table_a8d5ac73c6d047698d87b3d1ba34e66f [id=9c80193cde2346c982cf84ae9612e2f7]) successfully deleted
I20240430 20:11:24.085218 18621 raft_consensus.cc:2454] T bf925791ddcf4fa7b5a1645c63f0d696 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:24.085906 18696 leader_election.cc:302] T bf925791ddcf4fa7b5a1645c63f0d696 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a
I20240430 20:11:24.086210 18924 raft_consensus.cc:2781] T bf925791ddcf4fa7b5a1645c63f0d696 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:24.086324 18924 raft_consensus.cc:686] T bf925791ddcf4fa7b5a1645c63f0d696 P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Becoming Leader. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Running, Role: LEADER
I20240430 20:11:24.086616 18924 consensus_queue.cc:237] T bf925791ddcf4fa7b5a1645c63f0d696 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:24.093312 18599 tablet_service.cc:1467] Processing CreateTablet for tablet ffe059eeac91476aaeef8c56d2f49ee8 (DEFAULT_TABLE table=default.Table_73903d0d2a4746f38fdb73932bde6597 [id=6b56400601e6453495da25f608db780f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:24.093793 18599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffe059eeac91476aaeef8c56d2f49ee8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:24.094485 18459 tablet_service.cc:1467] Processing CreateTablet for tablet ffe059eeac91476aaeef8c56d2f49ee8 (DEFAULT_TABLE table=default.Table_73903d0d2a4746f38fdb73932bde6597 [id=6b56400601e6453495da25f608db780f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:24.094873 18459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffe059eeac91476aaeef8c56d2f49ee8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:24.099772 18742 tablet_service.cc:1467] Processing CreateTablet for tablet ffe059eeac91476aaeef8c56d2f49ee8 (DEFAULT_TABLE table=default.Table_73903d0d2a4746f38fdb73932bde6597 [id=6b56400601e6453495da25f608db780f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:24.100173 18742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffe059eeac91476aaeef8c56d2f49ee8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:24.100344 18599 tablet_service.cc:1467] Processing CreateTablet for tablet b4bc111e183f40a0acc9dd377195f6d9 (DEFAULT_TABLE table=default.Table_80b9e5521fff4a119f3ce4977b3e44eb [id=d5cea90cbc2c48df9d8b80da1eccc949]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:24.100718 18599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4bc111e183f40a0acc9dd377195f6d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:24.101219 18458 tablet_service.cc:1467] Processing CreateTablet for tablet b4bc111e183f40a0acc9dd377195f6d9 (DEFAULT_TABLE table=default.Table_80b9e5521fff4a119f3ce4977b3e44eb [id=d5cea90cbc2c48df9d8b80da1eccc949]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:24.101622 18458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4bc111e183f40a0acc9dd377195f6d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:24.117447 19032 leader_election.cc:288] T bf925791ddcf4fa7b5a1645c63f0d696 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Requested vote from peers 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:24.125624 18739 tablet_service.cc:1467] Processing CreateTablet for tablet b4bc111e183f40a0acc9dd377195f6d9 (DEFAULT_TABLE table=default.Table_80b9e5521fff4a119f3ce4977b3e44eb [id=d5cea90cbc2c48df9d8b80da1eccc949]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:24.126425 18739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4bc111e183f40a0acc9dd377195f6d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:24.127789 18599 tablet_service.cc:1467] Processing CreateTablet for tablet 0879b02a0bcc4f619fb34cff7b9d7c4c (DEFAULT_TABLE table=default.Table_f196eb2d3b014b878b9640fbc44cd952 [id=6d6fa10e4bcc437aaee661cd8abb441f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:24.128513 18599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0879b02a0bcc4f619fb34cff7b9d7c4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:24.129328 18601 tablet_service.cc:1467] Processing CreateTablet for tablet a0e8e07d363f48c698c7e2b17edc1545 (DEFAULT_TABLE table=default.Table_a9b96b394f274b2bae5cfe2f0786ec92 [id=50771665d40d466f92f3a1babd60e39b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:24.130306 18156 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_f6d2f14f89184a32a523563bdc6ba478" } modify_external_catalogs: true
I20240430 20:11:24.130537 18156 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_f6d2f14f89184a32a523563bdc6ba478" } modify_external_catalogs: true
I20240430 20:11:24.134317 18743 tablet_service.cc:1467] Processing CreateTablet for tablet a0e8e07d363f48c698c7e2b17edc1545 (DEFAULT_TABLE table=default.Table_a9b96b394f274b2bae5cfe2f0786ec92 [id=50771665d40d466f92f3a1babd60e39b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:24.134930 18743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0e8e07d363f48c698c7e2b17edc1545. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:24.136859 18457 tablet_service.cc:1467] Processing CreateTablet for tablet 0879b02a0bcc4f619fb34cff7b9d7c4c (DEFAULT_TABLE table=default.Table_f196eb2d3b014b878b9640fbc44cd952 [id=6d6fa10e4bcc437aaee661cd8abb441f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:24.137362 18457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0879b02a0bcc4f619fb34cff7b9d7c4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:24.139464 18601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0e8e07d363f48c698c7e2b17edc1545. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:24.140791 18741 tablet_service.cc:1467] Processing CreateTablet for tablet 0879b02a0bcc4f619fb34cff7b9d7c4c (DEFAULT_TABLE table=default.Table_f196eb2d3b014b878b9640fbc44cd952 [id=6d6fa10e4bcc437aaee661cd8abb441f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:24.141320 18741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0879b02a0bcc4f619fb34cff7b9d7c4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:24.142122 18603 tablet_service.cc:1467] Processing CreateTablet for tablet 187a27a6449342f99de3c5a2f93d49fc (DEFAULT_TABLE table=default.Table_11aeb11694264720a7e795ac7a114ead [id=f61d04c63e844a7b99d503770e5fcd87]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:24.142550 18603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 187a27a6449342f99de3c5a2f93d49fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:24.143944 18463 tablet_service.cc:1467] Processing CreateTablet for tablet a0e8e07d363f48c698c7e2b17edc1545 (DEFAULT_TABLE table=default.Table_a9b96b394f274b2bae5cfe2f0786ec92 [id=50771665d40d466f92f3a1babd60e39b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:24.144325 18463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0e8e07d363f48c698c7e2b17edc1545. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:24.148927 18742 tablet_service.cc:1467] Processing CreateTablet for tablet 187a27a6449342f99de3c5a2f93d49fc (DEFAULT_TABLE table=default.Table_11aeb11694264720a7e795ac7a114ead [id=f61d04c63e844a7b99d503770e5fcd87]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:24.149339 18742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 187a27a6449342f99de3c5a2f93d49fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:24.151968 18462 tablet_service.cc:1467] Processing CreateTablet for tablet 187a27a6449342f99de3c5a2f93d49fc (DEFAULT_TABLE table=default.Table_11aeb11694264720a7e795ac7a114ead [id=f61d04c63e844a7b99d503770e5fcd87]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:24.152437 18462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 187a27a6449342f99de3c5a2f93d49fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:24.157042 18603 tablet_service.cc:1467] Processing CreateTablet for tablet 680553d4d0bc4fcb839235c1832757e1 (DEFAULT_TABLE table=default.Table_ffddf7fa447948b58de34494af4ed0fd [id=eecb2f8eeef547c5ab1abe6a2430e592]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:24.157537 18603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 680553d4d0bc4fcb839235c1832757e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:24.157617 18154 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: Sending DeleteTablet for 3 replicas of tablet e5d1707ba0b34cc4b1cda39e36ccbd2f
I20240430 20:11:24.158133 18154 catalog_manager.cc:6554] ReplaceTablet: tablet e5d1707ba0b34cc4b1cda39e36ccbd2f deleted and replaced by tablet fae9a35a2ac844b4bc2ebdf214aa7171
I20240430 20:11:24.160442 18602 tablet_service.cc:1514] Processing DeleteTablet for tablet e5d1707ba0b34cc4b1cda39e36ccbd2f with delete_type TABLET_DATA_DELETED (replaced by tablet fae9a35a2ac844b4bc2ebdf214aa7171) from {username='slave'} at 127.0.0.1:40202
I20240430 20:11:24.161410 18459 tablet_service.cc:1514] Processing DeleteTablet for tablet e5d1707ba0b34cc4b1cda39e36ccbd2f with delete_type TABLET_DATA_DELETED (replaced by tablet fae9a35a2ac844b4bc2ebdf214aa7171) from {username='slave'} at 127.0.0.1:41678
I20240430 20:11:24.161867 18981 tablet_replica.cc:330] T e5d1707ba0b34cc4b1cda39e36ccbd2f P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:24.161991 18980 tablet_replica.cc:330] T e5d1707ba0b34cc4b1cda39e36ccbd2f P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:24.162194 18980 raft_consensus.cc:2229] T e5d1707ba0b34cc4b1cda39e36ccbd2f P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:24.162209 18981 raft_consensus.cc:2229] T e5d1707ba0b34cc4b1cda39e36ccbd2f P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:24.162377 18980 raft_consensus.cc:2258] T e5d1707ba0b34cc4b1cda39e36ccbd2f P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:24.162469 18981 raft_consensus.cc:2258] T e5d1707ba0b34cc4b1cda39e36ccbd2f P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:24.163391 18980 ts_tablet_manager.cc:1875] T e5d1707ba0b34cc4b1cda39e36ccbd2f P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:24.163592 18981 ts_tablet_manager.cc:1875] T e5d1707ba0b34cc4b1cda39e36ccbd2f P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:24.167433 18742 tablet_service.cc:1514] Processing DeleteTablet for tablet e5d1707ba0b34cc4b1cda39e36ccbd2f with delete_type TABLET_DATA_DELETED (replaced by tablet fae9a35a2ac844b4bc2ebdf214aa7171) from {username='slave'} at 127.0.0.1:59818
I20240430 20:11:24.167637 18988 tablet_bootstrap.cc:492] T f79497bf22ed417d988530f6e69cc07f P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:24.167730 18987 tablet_replica.cc:330] T e5d1707ba0b34cc4b1cda39e36ccbd2f P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:24.167822 18988 ts_tablet_manager.cc:1367] T f79497bf22ed417d988530f6e69cc07f P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.255s user 0.043s sys 0.056s
I20240430 20:11:24.167948 18987 raft_consensus.cc:2229] T e5d1707ba0b34cc4b1cda39e36ccbd2f P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Raft consensus shutting down.
I20240430 20:11:24.168357 18987 raft_consensus.cc:2258] T e5d1707ba0b34cc4b1cda39e36ccbd2f P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:24.169193 18987 ts_tablet_manager.cc:1875] T e5d1707ba0b34cc4b1cda39e36ccbd2f P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:24.170943 18981 ts_tablet_manager.cc:1888] T e5d1707ba0b34cc4b1cda39e36ccbd2f P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240430 20:11:24.171082 18981 log.cc:1196] T e5d1707ba0b34cc4b1cda39e36ccbd2f P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/e5d1707ba0b34cc4b1cda39e36ccbd2f
I20240430 20:11:24.171458 18981 ts_tablet_manager.cc:1909] T e5d1707ba0b34cc4b1cda39e36ccbd2f P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:24.173195 18985 tablet_bootstrap.cc:492] T 26a015250eeb4750a42df93840e8d47f P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:24.173368 18985 ts_tablet_manager.cc:1367] T 26a015250eeb4750a42df93840e8d47f P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.128s user 0.009s sys 0.027s
I20240430 20:11:24.174360 18985 raft_consensus.cc:348] T 26a015250eeb4750a42df93840e8d47f P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:24.174661 18985 raft_consensus.cc:374] T 26a015250eeb4750a42df93840e8d47f P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:24.174758 18985 raft_consensus.cc:729] T 26a015250eeb4750a42df93840e8d47f P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:24.175017 18985 consensus_queue.cc:260] T 26a015250eeb4750a42df93840e8d47f P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:24.175534 18985 ts_tablet_manager.cc:1398] T 26a015250eeb4750a42df93840e8d47f P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240430 20:11:24.176527 18142 catalog_manager.cc:4830] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): tablet e5d1707ba0b34cc4b1cda39e36ccbd2f (table default.Table_0ab5b2f09fa7468bb89ab703f6d3d25c [id=a5d580cffa2447d78fc9f2c8e31264f1]) successfully deleted
I20240430 20:11:24.177961 18985 tablet_bootstrap.cc:492] T e6c0c808bfc94be3837ab835292b3c1b P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:24.179747 18985 tablet_bootstrap.cc:654] T e6c0c808bfc94be3837ab835292b3c1b P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:24.180824 18987 ts_tablet_manager.cc:1888] T e5d1707ba0b34cc4b1cda39e36ccbd2f P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240430 20:11:24.181061 18987 log.cc:1196] T e5d1707ba0b34cc4b1cda39e36ccbd2f P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/e5d1707ba0b34cc4b1cda39e36ccbd2f
I20240430 20:11:24.181526 18987 ts_tablet_manager.cc:1909] T e5d1707ba0b34cc4b1cda39e36ccbd2f P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:24.183682 18141 catalog_manager.cc:4830] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): tablet e5d1707ba0b34cc4b1cda39e36ccbd2f (table default.Table_0ab5b2f09fa7468bb89ab703f6d3d25c [id=a5d580cffa2447d78fc9f2c8e31264f1]) successfully deleted
I20240430 20:11:24.182284 18988 raft_consensus.cc:348] T f79497bf22ed417d988530f6e69cc07f P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:24.183835 18988 raft_consensus.cc:374] T f79497bf22ed417d988530f6e69cc07f P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:24.184854 18988 raft_consensus.cc:729] T f79497bf22ed417d988530f6e69cc07f P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:24.185159 18988 consensus_queue.cc:260] T f79497bf22ed417d988530f6e69cc07f P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:24.185825 18988 ts_tablet_manager.cc:1398] T f79497bf22ed417d988530f6e69cc07f P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.018s user 0.002s sys 0.001s
W20240430 20:11:24.187872 18155 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 187a27a6449342f99de3c5a2f93d49fc consensus state has no leader
I20240430 20:11:24.188129 18155 master_service.cc:946] ReplaceTablet: received request to replace tablet e5d1707ba0b34cc4b1cda39e36ccbd2f from {username='slave'} at 127.0.0.1:54524
I20240430 20:11:24.189816 18146 master_service.cc:946] ReplaceTablet: received request to replace tablet f15f5e1f3afd4224bbf39567cae74617 from {username='slave'} at 127.0.0.1:54524
I20240430 20:11:24.190397 18985 tablet_bootstrap.cc:492] T e6c0c808bfc94be3837ab835292b3c1b P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:24.190663 18985 ts_tablet_manager.cc:1367] T e6c0c808bfc94be3837ab835292b3c1b P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.003s
I20240430 20:11:24.191623 18988 tablet_bootstrap.cc:492] T 2067cb2cf76d495c92c1f25c176eb6dc P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:24.191684 18985 raft_consensus.cc:348] T e6c0c808bfc94be3837ab835292b3c1b P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:24.192029 18985 raft_consensus.cc:374] T e6c0c808bfc94be3837ab835292b3c1b P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:24.192131 18985 raft_consensus.cc:729] T e6c0c808bfc94be3837ab835292b3c1b P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:24.192483 18985 consensus_queue.cc:260] T e6c0c808bfc94be3837ab835292b3c1b P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:24.194482 18985 ts_tablet_manager.cc:1398] T e6c0c808bfc94be3837ab835292b3c1b P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240430 20:11:24.195632 18988 tablet_bootstrap.cc:654] T 2067cb2cf76d495c92c1f25c176eb6dc P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:24.195962 18985 tablet_bootstrap.cc:492] T 0d6a4a63c004480b82d84381867bf8a4 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:24.197556 18983 tablet_bootstrap.cc:492] T 2067cb2cf76d495c92c1f25c176eb6dc P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:24.197738 18983 ts_tablet_manager.cc:1367] T 2067cb2cf76d495c92c1f25c176eb6dc P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.213s user 0.012s sys 0.032s
I20240430 20:11:24.197999 18985 tablet_bootstrap.cc:654] T 0d6a4a63c004480b82d84381867bf8a4 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:24.198752 18983 raft_consensus.cc:348] T 2067cb2cf76d495c92c1f25c176eb6dc P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:24.199010 18983 raft_consensus.cc:374] T 2067cb2cf76d495c92c1f25c176eb6dc P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:24.199112 18983 raft_consensus.cc:729] T 2067cb2cf76d495c92c1f25c176eb6dc P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:24.199486 18983 consensus_queue.cc:260] T 2067cb2cf76d495c92c1f25c176eb6dc P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:24.202850 18980 ts_tablet_manager.cc:1888] T e5d1707ba0b34cc4b1cda39e36ccbd2f P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:24.203200 18980 log.cc:1196] T e5d1707ba0b34cc4b1cda39e36ccbd2f P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/e5d1707ba0b34cc4b1cda39e36ccbd2f
I20240430 20:11:24.204409 18980 ts_tablet_manager.cc:1909] T e5d1707ba0b34cc4b1cda39e36ccbd2f P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:24.205971 18983 ts_tablet_manager.cc:1398] T 2067cb2cf76d495c92c1f25c176eb6dc P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.008s user 0.003s sys 0.000s
I20240430 20:11:24.206287 18983 tablet_bootstrap.cc:492] T 26a015250eeb4750a42df93840e8d47f P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:24.206260 18139 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet e5d1707ba0b34cc4b1cda39e36ccbd2f (table default.Table_0ab5b2f09fa7468bb89ab703f6d3d25c [id=a5d580cffa2447d78fc9f2c8e31264f1]) successfully deleted
I20240430 20:11:24.209540 18462 tablet_service.cc:1467] Processing CreateTablet for tablet 680553d4d0bc4fcb839235c1832757e1 (DEFAULT_TABLE table=default.Table_ffddf7fa447948b58de34494af4ed0fd [id=eecb2f8eeef547c5ab1abe6a2430e592]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:24.210212 18462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 680553d4d0bc4fcb839235c1832757e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:24.212291 18743 tablet_service.cc:1467] Processing CreateTablet for tablet 680553d4d0bc4fcb839235c1832757e1 (DEFAULT_TABLE table=default.Table_ffddf7fa447948b58de34494af4ed0fd [id=eecb2f8eeef547c5ab1abe6a2430e592]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:24.213284 18743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 680553d4d0bc4fcb839235c1832757e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:24.213678 18603 tablet_service.cc:1467] Processing CreateTablet for tablet 609a1a7d22e84b69bd0a53577b62b69b (DEFAULT_TABLE table=default.Table_b4c8addb5cfe442d8ec19324de4bb523 [id=c092ea63bd4742bd9876634a1dd2e2b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:24.214367 18459 tablet_service.cc:1467] Processing CreateTablet for tablet 609a1a7d22e84b69bd0a53577b62b69b (DEFAULT_TABLE table=default.Table_b4c8addb5cfe442d8ec19324de4bb523 [id=c092ea63bd4742bd9876634a1dd2e2b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:24.214854 18459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 609a1a7d22e84b69bd0a53577b62b69b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:24.216398 18603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 609a1a7d22e84b69bd0a53577b62b69b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:24.216378 18741 tablet_service.cc:1467] Processing CreateTablet for tablet 609a1a7d22e84b69bd0a53577b62b69b (DEFAULT_TABLE table=default.Table_b4c8addb5cfe442d8ec19324de4bb523 [id=c092ea63bd4742bd9876634a1dd2e2b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:24.216754 18741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 609a1a7d22e84b69bd0a53577b62b69b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:24.219297 18458 tablet_service.cc:1467] Processing CreateTablet for tablet 27f94f2d8f50495dababe417a9f82982 (DEFAULT_TABLE table=default.Table_156650e4b399419c97dcf5160b761ee1 [id=b93708ff15ef4c8299967c8c91045c8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:24.220310 18983 tablet_bootstrap.cc:654] T 26a015250eeb4750a42df93840e8d47f P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:24.221031 18985 tablet_bootstrap.cc:492] T 0d6a4a63c004480b82d84381867bf8a4 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:24.221196 18985 ts_tablet_manager.cc:1367] T 0d6a4a63c004480b82d84381867bf8a4 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.025s user 0.007s sys 0.000s
I20240430 20:11:24.222266 18602 tablet_service.cc:1467] Processing CreateTablet for tablet 27f94f2d8f50495dababe417a9f82982 (DEFAULT_TABLE table=default.Table_156650e4b399419c97dcf5160b761ee1 [id=b93708ff15ef4c8299967c8c91045c8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:24.222666 18458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27f94f2d8f50495dababe417a9f82982. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:24.222733 18602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27f94f2d8f50495dababe417a9f82982. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:24.223438 18149 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_ffddf7fa447948b58de34494af4ed0fd" } new_table_name: "default.Table_49a012a7c4c8457fadb0a8f0fc9654c6" modify_external_catalogs: true
I20240430 20:11:24.226362 18155 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54524:
name: "default.Table_9d39d94262a7481aad7df7868a2aff26"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:24.227010 18155 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9d39d94262a7481aad7df7868a2aff26 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:24.227345 18985 raft_consensus.cc:348] T 0d6a4a63c004480b82d84381867bf8a4 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:24.227710 18985 raft_consensus.cc:374] T 0d6a4a63c004480b82d84381867bf8a4 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:24.227831 18985 raft_consensus.cc:729] T 0d6a4a63c004480b82d84381867bf8a4 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:24.229715 18985 consensus_queue.cc:260] T 0d6a4a63c004480b82d84381867bf8a4 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:24.231824 18985 ts_tablet_manager.cc:1398] T 0d6a4a63c004480b82d84381867bf8a4 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.011s user 0.003s sys 0.000s
I20240430 20:11:24.231979 18743 tablet_service.cc:1467] Processing CreateTablet for tablet 27f94f2d8f50495dababe417a9f82982 (DEFAULT_TABLE table=default.Table_156650e4b399419c97dcf5160b761ee1 [id=b93708ff15ef4c8299967c8c91045c8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:24.232430 18743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27f94f2d8f50495dababe417a9f82982. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:24.233076 18988 tablet_bootstrap.cc:492] T 2067cb2cf76d495c92c1f25c176eb6dc P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:24.233331 18988 ts_tablet_manager.cc:1367] T 2067cb2cf76d495c92c1f25c176eb6dc P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.042s user 0.002s sys 0.001s
W20240430 20:11:24.233654 18151 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 609a1a7d22e84b69bd0a53577b62b69b consensus state has no leader
I20240430 20:11:24.233886 18985 tablet_bootstrap.cc:492] T f15f5e1f3afd4224bbf39567cae74617 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:24.234061 18151 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_73903d0d2a4746f38fdb73932bde6597" } modify_external_catalogs: true
I20240430 20:11:24.234253 18151 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_73903d0d2a4746f38fdb73932bde6597" } modify_external_catalogs: true
I20240430 20:11:24.234629 18988 raft_consensus.cc:348] T 2067cb2cf76d495c92c1f25c176eb6dc P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:24.234947 18988 raft_consensus.cc:374] T 2067cb2cf76d495c92c1f25c176eb6dc P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:24.235111 18988 raft_consensus.cc:729] T 2067cb2cf76d495c92c1f25c176eb6dc P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:24.235522 18988 consensus_queue.cc:260] T 2067cb2cf76d495c92c1f25c176eb6dc P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:24.236709 18988 ts_tablet_manager.cc:1398] T 2067cb2cf76d495c92c1f25c176eb6dc P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240430 20:11:24.239171 18985 tablet_bootstrap.cc:654] T f15f5e1f3afd4224bbf39567cae74617 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:24.241590 18988 tablet_bootstrap.cc:492] T 4d3117a9a5f243b59a69d960c0a1eee4 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:24.244611 18988 tablet_bootstrap.cc:654] T 4d3117a9a5f243b59a69d960c0a1eee4 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:24.245733 19079 raft_consensus.cc:482] T 0d6a4a63c004480b82d84381867bf8a4 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:24.245958 19079 raft_consensus.cc:504] T 0d6a4a63c004480b82d84381867bf8a4 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:24.247474 19079 leader_election.cc:288] T 0d6a4a63c004480b82d84381867bf8a4 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:24.248170 18621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d6a4a63c004480b82d84381867bf8a4" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:24.248275 18762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d6a4a63c004480b82d84381867bf8a4" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
W20240430 20:11:24.248777 18415 leader_election.cc:341] T 0d6a4a63c004480b82d84381867bf8a4 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Illegal state: must be running to vote when last-logged opid is not known
W20240430 20:11:24.249194 18417 leader_election.cc:341] T 0d6a4a63c004480b82d84381867bf8a4 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:24.249354 18417 leader_election.cc:302] T 0d6a4a63c004480b82d84381867bf8a4 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a; no voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710
I20240430 20:11:24.249639 19079 raft_consensus.cc:2726] T 0d6a4a63c004480b82d84381867bf8a4 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:24.250137 18983 tablet_bootstrap.cc:492] T 26a015250eeb4750a42df93840e8d47f P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:24.250334 18983 ts_tablet_manager.cc:1367] T 26a015250eeb4750a42df93840e8d47f P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.044s user 0.002s sys 0.001s
I20240430 20:11:24.251324 18983 raft_consensus.cc:348] T 26a015250eeb4750a42df93840e8d47f P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:24.251574 18983 raft_consensus.cc:374] T 26a015250eeb4750a42df93840e8d47f P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:24.251665 18983 raft_consensus.cc:729] T 26a015250eeb4750a42df93840e8d47f P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:24.255800 18985 tablet_bootstrap.cc:492] T f15f5e1f3afd4224bbf39567cae74617 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:24.256016 18985 ts_tablet_manager.cc:1367] T f15f5e1f3afd4224bbf39567cae74617 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.022s user 0.001s sys 0.004s
I20240430 20:11:24.256232 18983 consensus_queue.cc:260] T 26a015250eeb4750a42df93840e8d47f P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:24.256877 19032 consensus_queue.cc:1035] T 44699c84ebac49ef8f3765d4069e71ca P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:24.257180 18985 raft_consensus.cc:348] T f15f5e1f3afd4224bbf39567cae74617 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:24.257391 18985 raft_consensus.cc:374] T f15f5e1f3afd4224bbf39567cae74617 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:24.257477 18985 raft_consensus.cc:729] T f15f5e1f3afd4224bbf39567cae74617 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:24.257869 18985 consensus_queue.cc:260] T f15f5e1f3afd4224bbf39567cae74617 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:24.259264 18983 ts_tablet_manager.cc:1398] T 26a015250eeb4750a42df93840e8d47f P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.009s user 0.003s sys 0.000s
I20240430 20:11:24.259609 18983 tablet_bootstrap.cc:492] T 9750608faeab4be38ddce4e2e4d46964 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:24.260360 18985 ts_tablet_manager.cc:1398] T f15f5e1f3afd4224bbf39567cae74617 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20240430 20:11:24.262269 18985 tablet_bootstrap.cc:492] T c3f8f9ada98b40bc8359a00e193c3e5f P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:24.264497 18985 tablet_bootstrap.cc:654] T c3f8f9ada98b40bc8359a00e193c3e5f P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:24.265225 18983 tablet_bootstrap.cc:654] T 9750608faeab4be38ddce4e2e4d46964 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:24.269047 19030 consensus_queue.cc:1035] T 44699c84ebac49ef8f3765d4069e71ca P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:24.271278 18988 tablet_bootstrap.cc:492] T 4d3117a9a5f243b59a69d960c0a1eee4 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:24.271739 18988 ts_tablet_manager.cc:1367] T 4d3117a9a5f243b59a69d960c0a1eee4 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.035s user 0.003s sys 0.001s
I20240430 20:11:24.273231 19079 raft_consensus.cc:482] T 26a015250eeb4750a42df93840e8d47f P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:24.273394 19079 raft_consensus.cc:504] T 26a015250eeb4750a42df93840e8d47f P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:24.274061 19079 leader_election.cc:288] T 26a015250eeb4750a42df93840e8d47f P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:24.275372 18985 tablet_bootstrap.cc:492] T c3f8f9ada98b40bc8359a00e193c3e5f P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:24.275552 18985 ts_tablet_manager.cc:1367] T c3f8f9ada98b40bc8359a00e193c3e5f P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.000s
I20240430 20:11:24.275718 18621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26a015250eeb4750a42df93840e8d47f" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:24.275946 18621 raft_consensus.cc:2454] T 26a015250eeb4750a42df93840e8d47f P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 0.
I20240430 20:11:24.276552 18415 leader_election.cc:302] T 26a015250eeb4750a42df93840e8d47f P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:24.276624 18762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26a015250eeb4750a42df93840e8d47f" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:24.277098 19079 raft_consensus.cc:2781] T 26a015250eeb4750a42df93840e8d47f P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:24.277252 19079 raft_consensus.cc:482] T 26a015250eeb4750a42df93840e8d47f P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:24.277369 19079 raft_consensus.cc:3037] T 26a015250eeb4750a42df93840e8d47f P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
W20240430 20:11:24.277333 18417 leader_election.cc:341] T 26a015250eeb4750a42df93840e8d47f P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:24.272598 18988 raft_consensus.cc:348] T 4d3117a9a5f243b59a69d960c0a1eee4 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:24.277801 18988 raft_consensus.cc:374] T 4d3117a9a5f243b59a69d960c0a1eee4 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:24.277940 18988 raft_consensus.cc:729] T 4d3117a9a5f243b59a69d960c0a1eee4 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:24.278266 18988 consensus_queue.cc:260] T 4d3117a9a5f243b59a69d960c0a1eee4 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:24.280380 18988 ts_tablet_manager.cc:1398] T 4d3117a9a5f243b59a69d960c0a1eee4 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240430 20:11:24.280827 18988 tablet_bootstrap.cc:492] T 26a015250eeb4750a42df93840e8d47f P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:24.282686 19079 raft_consensus.cc:504] T 26a015250eeb4750a42df93840e8d47f P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:24.284193 19042 raft_consensus.cc:482] T 6f729a6e8dc14de9a2057cfe09ba9257 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:24.284292 18621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26a015250eeb4750a42df93840e8d47f" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:24.284387 19042 raft_consensus.cc:504] T 6f729a6e8dc14de9a2057cfe09ba9257 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:24.284547 18621 raft_consensus.cc:3037] T 26a015250eeb4750a42df93840e8d47f P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:24.285099 19042 leader_election.cc:288] T 6f729a6e8dc14de9a2057cfe09ba9257 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:24.285661 18762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f729a6e8dc14de9a2057cfe09ba9257" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:24.285885 18762 raft_consensus.cc:2454] T 6f729a6e8dc14de9a2057cfe09ba9257 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 0.
I20240430 20:11:24.286304 18557 leader_election.cc:302] T 6f729a6e8dc14de9a2057cfe09ba9257 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:24.286943 19042 raft_consensus.cc:2781] T 6f729a6e8dc14de9a2057cfe09ba9257 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:24.287067 19042 raft_consensus.cc:482] T 6f729a6e8dc14de9a2057cfe09ba9257 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:24.287155 19042 raft_consensus.cc:3037] T 6f729a6e8dc14de9a2057cfe09ba9257 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
W20240430 20:11:24.287664 18150 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a0e8e07d363f48c698c7e2b17edc1545 consensus state has no leader
I20240430 20:11:24.287940 18150 master_service.cc:946] ReplaceTablet: received request to replace tablet 2067cb2cf76d495c92c1f25c176eb6dc from {username='slave'} at 127.0.0.1:54524
I20240430 20:11:24.288345 18156 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: Sending DeleteTablet for 3 replicas of tablet 59548d560b4e4f7ca034d730aca79364
I20240430 20:11:24.288390 18621 raft_consensus.cc:2454] T 26a015250eeb4750a42df93840e8d47f P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 1.
I20240430 20:11:24.289130 18415 leader_election.cc:302] T 26a015250eeb4750a42df93840e8d47f P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:24.289490 19072 raft_consensus.cc:2781] T 26a015250eeb4750a42df93840e8d47f P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:24.289667 19079 leader_election.cc:288] T 26a015250eeb4750a42df93840e8d47f P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Requested vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:24.289664 19072 raft_consensus.cc:686] T 26a015250eeb4750a42df93840e8d47f P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Becoming Leader. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Running, Role: LEADER
I20240430 20:11:24.289914 18985 raft_consensus.cc:348] T c3f8f9ada98b40bc8359a00e193c3e5f P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:24.290179 18985 raft_consensus.cc:374] T c3f8f9ada98b40bc8359a00e193c3e5f P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:24.290283 18985 raft_consensus.cc:729] T c3f8f9ada98b40bc8359a00e193c3e5f P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:24.290359 19072 consensus_queue.cc:237] T 26a015250eeb4750a42df93840e8d47f P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:24.290522 18762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26a015250eeb4750a42df93840e8d47f" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:24.290450 19042 raft_consensus.cc:504] T 6f729a6e8dc14de9a2057cfe09ba9257 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:24.290630 18985 consensus_queue.cc:260] T c3f8f9ada98b40bc8359a00e193c3e5f P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
W20240430 20:11:24.291098 18417 leader_election.cc:341] T 26a015250eeb4750a42df93840e8d47f P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:24.291239 18985 ts_tablet_manager.cc:1398] T c3f8f9ada98b40bc8359a00e193c3e5f P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.016s user 0.003s sys 0.000s
I20240430 20:11:24.291535 18985 tablet_bootstrap.cc:492] T 81328e904ca94e2eb5ec94dd3ffb3c90 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:24.291766 18483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f729a6e8dc14de9a2057cfe09ba9257" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:24.292042 18483 raft_consensus.cc:2454] T 6f729a6e8dc14de9a2057cfe09ba9257 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 0.
I20240430 20:11:24.292340 18762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f729a6e8dc14de9a2057cfe09ba9257" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:24.292559 18762 raft_consensus.cc:3037] T 6f729a6e8dc14de9a2057cfe09ba9257 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:24.292445 18481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f729a6e8dc14de9a2057cfe09ba9257" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:24.292745 18481 raft_consensus.cc:3037] T 6f729a6e8dc14de9a2057cfe09ba9257 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:24.294164 18985 tablet_bootstrap.cc:654] T 81328e904ca94e2eb5ec94dd3ffb3c90 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:24.295001 19042 leader_election.cc:288] T 6f729a6e8dc14de9a2057cfe09ba9257 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Requested vote from peers 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:24.295107 18743 tablet_service.cc:1514] Processing DeleteTablet for tablet 59548d560b4e4f7ca034d730aca79364 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:24 UTC) from {username='slave'} at 127.0.0.1:59818
I20240430 20:11:24.295253 18603 tablet_service.cc:1514] Processing DeleteTablet for tablet 59548d560b4e4f7ca034d730aca79364 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:24 UTC) from {username='slave'} at 127.0.0.1:40202
I20240430 20:11:24.295699 18152 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: default.Table_16b082d140234e1c9a1e70654839b4cd [id=c0d8dc87d1f9426a85f6176f161595a1] alter complete (version 3)
I20240430 20:11:24.295904 18462 tablet_service.cc:1514] Processing DeleteTablet for tablet 59548d560b4e4f7ca034d730aca79364 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:24 UTC) from {username='slave'} at 127.0.0.1:41678
W20240430 20:11:24.296579 18142 catalog_manager.cc:4810] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): delete failed for tablet 59548d560b4e4f7ca034d730aca79364 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 59548d560b4e4f7ca034d730aca79364 already in progress: creating tablet
I20240430 20:11:24.297940 18983 tablet_bootstrap.cc:492] T 9750608faeab4be38ddce4e2e4d46964 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:24.298110 18983 ts_tablet_manager.cc:1367] T 9750608faeab4be38ddce4e2e4d46964 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.039s user 0.002s sys 0.001s
I20240430 20:11:24.298959 18983 raft_consensus.cc:348] T 9750608faeab4be38ddce4e2e4d46964 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:24.299150 18983 raft_consensus.cc:374] T 9750608faeab4be38ddce4e2e4d46964 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:24.299235 18983 raft_consensus.cc:729] T 9750608faeab4be38ddce4e2e4d46964 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:24.299505 18983 consensus_queue.cc:260] T 9750608faeab4be38ddce4e2e4d46964 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
W20240430 20:11:24.300213 18141 catalog_manager.cc:4810] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): delete failed for tablet 59548d560b4e4f7ca034d730aca79364 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 59548d560b4e4f7ca034d730aca79364 already in progress: creating tablet
I20240430 20:11:24.302184 18481 raft_consensus.cc:2454] T 6f729a6e8dc14de9a2057cfe09ba9257 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
I20240430 20:11:24.302335 18602 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f729a6e8dc14de9a2057cfe09ba9257 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:23 UTC) from {username='slave'} at 127.0.0.1:40202
I20240430 20:11:24.302567 18598 tablet_service.cc:1514] Processing DeleteTablet for tablet f79497bf22ed417d988530f6e69cc07f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:23 UTC) from {username='slave'} at 127.0.0.1:40202
I20240430 20:11:24.302609 18600 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d3117a9a5f243b59a69d960c0a1eee4 with delete_type TABLET_DATA_DELETED (replaced by tablet e0d4cebfbfca45c08d1e7500228835b4) from {username='slave'} at 127.0.0.1:40202
I20240430 20:11:24.302956 18980 tablet_replica.cc:330] T 6f729a6e8dc14de9a2057cfe09ba9257 P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:24.302789 18558 leader_election.cc:302] T 6f729a6e8dc14de9a2057cfe09ba9257 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:24.303167 18980 raft_consensus.cc:2229] T 6f729a6e8dc14de9a2057cfe09ba9257 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:24.303330 18980 raft_consensus.cc:2258] T 6f729a6e8dc14de9a2057cfe09ba9257 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240430 20:11:24.303417 18139 catalog_manager.cc:4810] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): delete failed for tablet 59548d560b4e4f7ca034d730aca79364 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 59548d560b4e4f7ca034d730aca79364 already in progress: creating tablet
I20240430 20:11:24.304306 18980 ts_tablet_manager.cc:1875] T 6f729a6e8dc14de9a2057cfe09ba9257 P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:24.304355 18603 tablet_service.cc:1514] Processing DeleteTablet for tablet e5d1707ba0b34cc4b1cda39e36ccbd2f with delete_type TABLET_DATA_DELETED (replaced by tablet fae9a35a2ac844b4bc2ebdf214aa7171) from {username='slave'} at 127.0.0.1:40202
I20240430 20:11:24.311411 19030 raft_consensus.cc:482] T 2067cb2cf76d495c92c1f25c176eb6dc P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:24.313177 18743 tablet_service.cc:1514] Processing DeleteTablet for tablet e5d1707ba0b34cc4b1cda39e36ccbd2f with delete_type TABLET_DATA_DELETED (replaced by tablet fae9a35a2ac844b4bc2ebdf214aa7171) from {username='slave'} at 127.0.0.1:59818
I20240430 20:11:24.313360 18988 tablet_bootstrap.cc:654] T 26a015250eeb4750a42df93840e8d47f P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:24.313531 18741 tablet_service.cc:1514] Processing DeleteTablet for tablet bf925791ddcf4fa7b5a1645c63f0d696 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:23 UTC) from {username='slave'} at 127.0.0.1:59818
I20240430 20:11:24.314116 18987 tablet_replica.cc:330] T bf925791ddcf4fa7b5a1645c63f0d696 P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:24.314373 18987 raft_consensus.cc:2229] T bf925791ddcf4fa7b5a1645c63f0d696 P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Raft consensus shutting down.
I20240430 20:11:24.313120 19030 raft_consensus.cc:504] T 2067cb2cf76d495c92c1f25c176eb6dc P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:24.314575 18983 ts_tablet_manager.cc:1398] T 9750608faeab4be38ddce4e2e4d46964 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20240430 20:11:24.315151 19030 leader_election.cc:288] T 2067cb2cf76d495c92c1f25c176eb6dc P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:24.316936 18980 ts_tablet_manager.cc:1888] T 6f729a6e8dc14de9a2057cfe09ba9257 P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:24.317080 18980 log.cc:1196] T 6f729a6e8dc14de9a2057cfe09ba9257 P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/6f729a6e8dc14de9a2057cfe09ba9257
I20240430 20:11:24.317567 18987 pending_rounds.cc:62] T bf925791ddcf4fa7b5a1645c63f0d696 P 3c51c33d5616482cafb68e26688820f4: Trying to abort 1 pending ops.
I20240430 20:11:24.317649 18987 pending_rounds.cc:68] T bf925791ddcf4fa7b5a1645c63f0d696 P 3c51c33d5616482cafb68e26688820f4: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7022624293221056512 op_type: NO_OP noop_request { }
I20240430 20:11:24.317827 18987 raft_consensus.cc:2866] T bf925791ddcf4fa7b5a1645c63f0d696 P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240430 20:11:24.317946 18987 raft_consensus.cc:2258] T bf925791ddcf4fa7b5a1645c63f0d696 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:24.317766 18481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2067cb2cf76d495c92c1f25c176eb6dc" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:24.318152 18481 raft_consensus.cc:2454] T 2067cb2cf76d495c92c1f25c176eb6dc P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:24.318873 18699 leader_election.cc:302] T 2067cb2cf76d495c92c1f25c176eb6dc P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters:
I20240430 20:11:24.319237 19030 raft_consensus.cc:2781] T 2067cb2cf76d495c92c1f25c176eb6dc P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:24.319339 19030 raft_consensus.cc:482] T 2067cb2cf76d495c92c1f25c176eb6dc P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:24.319453 19030 raft_consensus.cc:3037] T 2067cb2cf76d495c92c1f25c176eb6dc P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:24.321111 18983 tablet_bootstrap.cc:492] T e6c0c808bfc94be3837ab835292b3c1b P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:24.323382 18987 ts_tablet_manager.cc:1875] T bf925791ddcf4fa7b5a1645c63f0d696 P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:24.327780 18621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2067cb2cf76d495c92c1f25c176eb6dc" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:24.328111 18621 raft_consensus.cc:2454] T 2067cb2cf76d495c92c1f25c176eb6dc P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
W20240430 20:11:24.329339 18141 catalog_manager.cc:4797] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): delete failed for tablet e5d1707ba0b34cc4b1cda39e36ccbd2f because the tablet was not found. No further retry: Not found: Tablet not found: e5d1707ba0b34cc4b1cda39e36ccbd2f
I20240430 20:11:24.329984 18988 tablet_bootstrap.cc:492] T 26a015250eeb4750a42df93840e8d47f P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:24.330086 18739 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f729a6e8dc14de9a2057cfe09ba9257 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:23 UTC) from {username='slave'} at 127.0.0.1:59818
I20240430 20:11:24.330197 18988 ts_tablet_manager.cc:1367] T 26a015250eeb4750a42df93840e8d47f P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.049s user 0.003s sys 0.000s
I20240430 20:11:24.330549 18601 tablet_service.cc:1514] Processing DeleteTablet for tablet bf925791ddcf4fa7b5a1645c63f0d696 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:23 UTC) from {username='slave'} at 127.0.0.1:40202
I20240430 20:11:24.330902 18762 raft_consensus.cc:2454] T 6f729a6e8dc14de9a2057cfe09ba9257 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
I20240430 20:11:24.331833 19030 raft_consensus.cc:504] T 2067cb2cf76d495c92c1f25c176eb6dc P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:24.333088 18481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2067cb2cf76d495c92c1f25c176eb6dc" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:24.333390 18481 raft_consensus.cc:3037] T 2067cb2cf76d495c92c1f25c176eb6dc P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:24.337021 18481 raft_consensus.cc:2454] T 2067cb2cf76d495c92c1f25c176eb6dc P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:24.337775 18699 leader_election.cc:302] T 2067cb2cf76d495c92c1f25c176eb6dc P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters:
I20240430 20:11:24.338080 18987 ts_tablet_manager.cc:1888] T bf925791ddcf4fa7b5a1645c63f0d696 P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:24.338194 18924 raft_consensus.cc:2781] T 2067cb2cf76d495c92c1f25c176eb6dc P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:24.338294 18987 log.cc:1196] T bf925791ddcf4fa7b5a1645c63f0d696 P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/bf925791ddcf4fa7b5a1645c63f0d696
I20240430 20:11:24.338364 18924 raft_consensus.cc:686] T 2067cb2cf76d495c92c1f25c176eb6dc P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Becoming Leader. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Running, Role: LEADER
I20240430 20:11:24.338927 19030 leader_election.cc:288] T 2067cb2cf76d495c92c1f25c176eb6dc P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Requested vote from peers 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:24.339411 18924 consensus_queue.cc:237] T 2067cb2cf76d495c92c1f25c176eb6dc P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:24.340185 18621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2067cb2cf76d495c92c1f25c176eb6dc" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:24.340498 18621 raft_consensus.cc:3037] T 2067cb2cf76d495c92c1f25c176eb6dc P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:24.340701 19042 raft_consensus.cc:482] T f79497bf22ed417d988530f6e69cc07f P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:24.340852 19042 raft_consensus.cc:504] T f79497bf22ed417d988530f6e69cc07f P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:24.342283 19042 leader_election.cc:288] T f79497bf22ed417d988530f6e69cc07f P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:24.342604 18983 tablet_bootstrap.cc:654] T e6c0c808bfc94be3837ab835292b3c1b P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:24.343824 18762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f79497bf22ed417d988530f6e69cc07f" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:24.344986 18762 raft_consensus.cc:2454] T f79497bf22ed417d988530f6e69cc07f P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 0.
I20240430 20:11:24.344491 18481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f79497bf22ed417d988530f6e69cc07f" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:24.345357 18481 raft_consensus.cc:2454] T f79497bf22ed417d988530f6e69cc07f P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 0.
I20240430 20:11:24.345968 18557 leader_election.cc:302] T f79497bf22ed417d988530f6e69cc07f P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:24.346174 18980 ts_tablet_manager.cc:1909] T 6f729a6e8dc14de9a2057cfe09ba9257 P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:24.346318 19042 raft_consensus.cc:2781] T f79497bf22ed417d988530f6e69cc07f P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:24.346432 19042 raft_consensus.cc:482] T f79497bf22ed417d988530f6e69cc07f P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:24.346527 19042 raft_consensus.cc:3037] T f79497bf22ed417d988530f6e69cc07f P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:24.346577 18621 raft_consensus.cc:2454] T 2067cb2cf76d495c92c1f25c176eb6dc P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:24.347414 18139 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet 6f729a6e8dc14de9a2057cfe09ba9257 (table default.Table_5feaab34a5cd439abdc705b85c22313b [id=1cefefb9abd64daa914f73fb65c4b201]) successfully deleted
I20240430 20:11:24.349841 19042 raft_consensus.cc:504] T f79497bf22ed417d988530f6e69cc07f P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:24.350042 18980 tablet_replica.cc:330] T f79497bf22ed417d988530f6e69cc07f P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:24.350360 18988 raft_consensus.cc:348] T 26a015250eeb4750a42df93840e8d47f P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:24.350682 18988 raft_consensus.cc:374] T 26a015250eeb4750a42df93840e8d47f P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:24.350771 18988 raft_consensus.cc:729] T 26a015250eeb4750a42df93840e8d47f P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:24.351557 18980 raft_consensus.cc:2229] T f79497bf22ed417d988530f6e69cc07f P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:24.352161 18980 raft_consensus.cc:2258] T f79497bf22ed417d988530f6e69cc07f P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:24.352706 18481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f79497bf22ed417d988530f6e69cc07f" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:24.352988 18987 ts_tablet_manager.cc:1909] T bf925791ddcf4fa7b5a1645c63f0d696 P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:24.353070 19042 leader_election.cc:288] T f79497bf22ed417d988530f6e69cc07f P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Requested vote from peers 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:24.353291 18481 raft_consensus.cc:3037] T f79497bf22ed417d988530f6e69cc07f P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:24.354976 18987 tablet_replica.cc:330] T 6f729a6e8dc14de9a2057cfe09ba9257 P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:24.355229 18987 raft_consensus.cc:2229] T 6f729a6e8dc14de9a2057cfe09ba9257 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:24.355444 18987 raft_consensus.cc:2258] T 6f729a6e8dc14de9a2057cfe09ba9257 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:24.356025 18141 catalog_manager.cc:4830] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): tablet bf925791ddcf4fa7b5a1645c63f0d696 (table default.Table_917365c2d10941558081a9898db3d7ea [id=b536a31acaea48d78885b0e585d00832]) successfully deleted
I20240430 20:11:24.356110 18987 ts_tablet_manager.cc:1875] T 6f729a6e8dc14de9a2057cfe09ba9257 P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:24.356587 18762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f79497bf22ed417d988530f6e69cc07f" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:24.357582 18980 ts_tablet_manager.cc:1875] T f79497bf22ed417d988530f6e69cc07f P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:24.357611 18762 raft_consensus.cc:3037] T f79497bf22ed417d988530f6e69cc07f P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:24.360255 18481 raft_consensus.cc:2454] T f79497bf22ed417d988530f6e69cc07f P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
I20240430 20:11:24.361093 18558 leader_election.cc:302] T f79497bf22ed417d988530f6e69cc07f P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
W20240430 20:11:24.361310 18558 raft_consensus.cc:2670] T f79497bf22ed417d988530f6e69cc07f P 5e1c6397f09747f091534faf93542710: unable to run election callback: Service unavailable: Thread pool token was shut down
I20240430 20:11:24.361510 18762 raft_consensus.cc:2454] T f79497bf22ed417d988530f6e69cc07f P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
I20240430 20:11:24.352380 18988 consensus_queue.cc:260] T 26a015250eeb4750a42df93840e8d47f P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:24.364035 18988 ts_tablet_manager.cc:1398] T 26a015250eeb4750a42df93840e8d47f P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.033s user 0.000s sys 0.003s
I20240430 20:11:24.364470 18988 tablet_bootstrap.cc:492] T 9750608faeab4be38ddce4e2e4d46964 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:24.365038 18147 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54524:
name: "default.Table_36adeb442f1245aba00549559bb329f3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:24.365664 18147 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_36adeb442f1245aba00549559bb329f3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:24.367061 18148 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_23b82186d14a44eeb81cad606fb8e9cc" } modify_external_catalogs: true
I20240430 20:11:24.367237 18148 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_23b82186d14a44eeb81cad606fb8e9cc" } modify_external_catalogs: true
I20240430 20:11:24.368916 18988 tablet_bootstrap.cc:654] T 9750608faeab4be38ddce4e2e4d46964 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:24.374634 18156 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_7bfb47ba65114665943b7edccabd5a8b" } modify_external_catalogs: true
I20240430 20:11:24.374841 18985 tablet_bootstrap.cc:492] T 81328e904ca94e2eb5ec94dd3ffb3c90 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:24.374835 18156 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_7bfb47ba65114665943b7edccabd5a8b" } modify_external_catalogs: true
I20240430 20:11:24.374990 18985 ts_tablet_manager.cc:1367] T 81328e904ca94e2eb5ec94dd3ffb3c90 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.084s user 0.002s sys 0.006s
I20240430 20:11:24.375479 18154 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_b29130df231a4316bcb8f9c2be002c92" } modify_external_catalogs: true
I20240430 20:11:24.375636 18154 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_b29130df231a4316bcb8f9c2be002c92" } modify_external_catalogs: true
I20240430 20:11:24.375855 18985 raft_consensus.cc:348] T 81328e904ca94e2eb5ec94dd3ffb3c90 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:24.376055 18985 raft_consensus.cc:374] T 81328e904ca94e2eb5ec94dd3ffb3c90 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:24.376121 18985 raft_consensus.cc:729] T 81328e904ca94e2eb5ec94dd3ffb3c90 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:24.376020 18152 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54524:
name: "default.Table_085ba776e84d4d66bc187679878700cf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240430 20:11:24.376359 18985 consensus_queue.cc:260] T 81328e904ca94e2eb5ec94dd3ffb3c90 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
W20240430 20:11:24.376622 18152 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_085ba776e84d4d66bc187679878700cf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:24.378031 18985 ts_tablet_manager.cc:1398] T 81328e904ca94e2eb5ec94dd3ffb3c90 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240430 20:11:24.382396 18983 tablet_bootstrap.cc:492] T e6c0c808bfc94be3837ab835292b3c1b P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:24.382612 18983 ts_tablet_manager.cc:1367] T e6c0c808bfc94be3837ab835292b3c1b P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.062s user 0.005s sys 0.004s
I20240430 20:11:24.383380 18983 raft_consensus.cc:348] T e6c0c808bfc94be3837ab835292b3c1b P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:24.383637 18983 raft_consensus.cc:374] T e6c0c808bfc94be3837ab835292b3c1b P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:24.383718 18983 raft_consensus.cc:729] T e6c0c808bfc94be3837ab835292b3c1b P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:24.383960 18983 consensus_queue.cc:260] T e6c0c808bfc94be3837ab835292b3c1b P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:24.384877 18987 ts_tablet_manager.cc:1888] T 6f729a6e8dc14de9a2057cfe09ba9257 P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:24.385057 18987 log.cc:1196] T 6f729a6e8dc14de9a2057cfe09ba9257 P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/6f729a6e8dc14de9a2057cfe09ba9257
I20240430 20:11:24.385437 18987 ts_tablet_manager.cc:1909] T 6f729a6e8dc14de9a2057cfe09ba9257 P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
W20240430 20:11:24.386904 18149 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 680553d4d0bc4fcb839235c1832757e1 consensus state has no leader
I20240430 20:11:24.385133 18983 ts_tablet_manager.cc:1398] T e6c0c808bfc94be3837ab835292b3c1b P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240430 20:11:24.388017 18141 catalog_manager.cc:4830] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): tablet 6f729a6e8dc14de9a2057cfe09ba9257 (table default.Table_5feaab34a5cd439abdc705b85c22313b [id=1cefefb9abd64daa914f73fb65c4b201]) successfully deleted
I20240430 20:11:24.389683 18146 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: Sending DeleteTablet for 3 replicas of tablet f15f5e1f3afd4224bbf39567cae74617
I20240430 20:11:24.390210 18146 catalog_manager.cc:6554] ReplaceTablet: tablet f15f5e1f3afd4224bbf39567cae74617 deleted and replaced by tablet 6e826459c3f4437da81c9e6e09f0c22f
I20240430 20:11:24.391265 18739 tablet_service.cc:1514] Processing DeleteTablet for tablet f15f5e1f3afd4224bbf39567cae74617 with delete_type TABLET_DATA_DELETED (replaced by tablet 6e826459c3f4437da81c9e6e09f0c22f) from {username='slave'} at 127.0.0.1:59818
I20240430 20:11:24.391350 18983 tablet_bootstrap.cc:492] T c3f8f9ada98b40bc8359a00e193c3e5f P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:24.392117 18601 tablet_service.cc:1514] Processing DeleteTablet for tablet f15f5e1f3afd4224bbf39567cae74617 with delete_type TABLET_DATA_DELETED (replaced by tablet 6e826459c3f4437da81c9e6e09f0c22f) from {username='slave'} at 127.0.0.1:40202
W20240430 20:11:24.392225 18141 catalog_manager.cc:4810] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): delete failed for tablet f15f5e1f3afd4224bbf39567cae74617 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f15f5e1f3afd4224bbf39567cae74617 already in progress: creating tablet
I20240430 20:11:24.393381 18983 tablet_bootstrap.cc:654] T c3f8f9ada98b40bc8359a00e193c3e5f P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:24.393424 18463 tablet_service.cc:1514] Processing DeleteTablet for tablet f15f5e1f3afd4224bbf39567cae74617 with delete_type TABLET_DATA_DELETED (replaced by tablet 6e826459c3f4437da81c9e6e09f0c22f) from {username='slave'} at 127.0.0.1:41678
I20240430 20:11:24.393791 18981 tablet_replica.cc:330] T f15f5e1f3afd4224bbf39567cae74617 P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:24.393996 18981 raft_consensus.cc:2229] T f15f5e1f3afd4224bbf39567cae74617 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:24.394209 18981 raft_consensus.cc:2258] T f15f5e1f3afd4224bbf39567cae74617 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:24.394229 18924 raft_consensus.cc:482] T 4d3117a9a5f243b59a69d960c0a1eee4 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:24.394431 18924 raft_consensus.cc:504] T 4d3117a9a5f243b59a69d960c0a1eee4 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:24.395211 18924 leader_election.cc:288] T 4d3117a9a5f243b59a69d960c0a1eee4 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:24.395294 18981 ts_tablet_manager.cc:1875] T f15f5e1f3afd4224bbf39567cae74617 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:24.396317 18621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3117a9a5f243b59a69d960c0a1eee4" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:24.396543 18621 raft_consensus.cc:2454] T 4d3117a9a5f243b59a69d960c0a1eee4 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:24.396566 18985 tablet_bootstrap.cc:492] T 59548d560b4e4f7ca034d730aca79364 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:24.397063 18696 leader_election.cc:302] T 4d3117a9a5f243b59a69d960c0a1eee4 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:24.397099 18151 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: Sending DeleteTablet for 3 replicas of tablet ffe059eeac91476aaeef8c56d2f49ee8
I20240430 20:11:24.397435 18924 raft_consensus.cc:2781] T 4d3117a9a5f243b59a69d960c0a1eee4 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:24.397564 18924 raft_consensus.cc:482] T 4d3117a9a5f243b59a69d960c0a1eee4 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:24.397642 18924 raft_consensus.cc:3037] T 4d3117a9a5f243b59a69d960c0a1eee4 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:24.398744 18739 tablet_service.cc:1514] Processing DeleteTablet for tablet ffe059eeac91476aaeef8c56d2f49ee8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:24 UTC) from {username='slave'} at 127.0.0.1:59818
I20240430 20:11:24.399246 18601 tablet_service.cc:1514] Processing DeleteTablet for tablet ffe059eeac91476aaeef8c56d2f49ee8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:24 UTC) from {username='slave'} at 127.0.0.1:40202
I20240430 20:11:24.400385 18924 raft_consensus.cc:504] T 4d3117a9a5f243b59a69d960c0a1eee4 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:24.400791 18980 ts_tablet_manager.cc:1888] T f79497bf22ed417d988530f6e69cc07f P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:24.400916 18980 log.cc:1196] T f79497bf22ed417d988530f6e69cc07f P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/f79497bf22ed417d988530f6e69cc07f
I20240430 20:11:24.401284 18980 ts_tablet_manager.cc:1909] T f79497bf22ed417d988530f6e69cc07f P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:24.402050 18463 tablet_service.cc:1514] Processing DeleteTablet for tablet ffe059eeac91476aaeef8c56d2f49ee8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:24 UTC) from {username='slave'} at 127.0.0.1:41678
W20240430 20:11:24.402177 18699 leader_election.cc:341] T 4d3117a9a5f243b59a69d960c0a1eee4 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Not found: Tablet not found: 4d3117a9a5f243b59a69d960c0a1eee4
I20240430 20:11:24.402288 18981 ts_tablet_manager.cc:1888] T f15f5e1f3afd4224bbf39567cae74617 P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:24.402324 18139 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet f79497bf22ed417d988530f6e69cc07f (table default.Table_b78b1b2c95694d1faee32321f845047b [id=2a3033894afd47ed8d907628a2e9712d]) successfully deleted
I20240430 20:11:24.402406 18981 log.cc:1196] T f15f5e1f3afd4224bbf39567cae74617 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/f15f5e1f3afd4224bbf39567cae74617
I20240430 20:11:24.402498 18985 tablet_bootstrap.cc:654] T 59548d560b4e4f7ca034d730aca79364 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:24.402819 18981 ts_tablet_manager.cc:1909] T f15f5e1f3afd4224bbf39567cae74617 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:24.402743 18621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3117a9a5f243b59a69d960c0a1eee4" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:24.402946 18621 raft_consensus.cc:3037] T 4d3117a9a5f243b59a69d960c0a1eee4 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:24.403415 18142 catalog_manager.cc:4830] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): tablet f15f5e1f3afd4224bbf39567cae74617 (table default.Table_873c21f6dffd4d47a742b3699ab85268 [id=d7e2c4300ff94fa2acc65d6d6c932c65]) successfully deleted
W20240430 20:11:24.403980 18142 catalog_manager.cc:4810] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): delete failed for tablet ffe059eeac91476aaeef8c56d2f49ee8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ffe059eeac91476aaeef8c56d2f49ee8 already in progress: creating tablet
I20240430 20:11:24.405637 18983 tablet_bootstrap.cc:492] T c3f8f9ada98b40bc8359a00e193c3e5f P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:24.405786 18983 ts_tablet_manager.cc:1367] T c3f8f9ada98b40bc8359a00e193c3e5f P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.003s
I20240430 20:11:24.406566 18621 raft_consensus.cc:2454] T 4d3117a9a5f243b59a69d960c0a1eee4 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:24.406890 18980 tablet_replica.cc:330] T 4d3117a9a5f243b59a69d960c0a1eee4 P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:24.407132 18980 raft_consensus.cc:2229] T 4d3117a9a5f243b59a69d960c0a1eee4 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:24.407285 18980 raft_consensus.cc:2258] T 4d3117a9a5f243b59a69d960c0a1eee4 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:24.407487 18696 leader_election.cc:302] T 4d3117a9a5f243b59a69d960c0a1eee4 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a
I20240430 20:11:24.407801 18980 ts_tablet_manager.cc:1875] T 4d3117a9a5f243b59a69d960c0a1eee4 P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:24.408110 19030 raft_consensus.cc:2781] T 4d3117a9a5f243b59a69d960c0a1eee4 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:24.408318 19030 raft_consensus.cc:686] T 4d3117a9a5f243b59a69d960c0a1eee4 P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Becoming Leader. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Running, Role: LEADER
I20240430 20:11:24.408664 19030 consensus_queue.cc:237] T 4d3117a9a5f243b59a69d960c0a1eee4 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:24.410807 18924 leader_election.cc:288] T 4d3117a9a5f243b59a69d960c0a1eee4 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Requested vote from peers 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:24.411893 18146 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54524:
name: "default.Table_ea83912975774b6089fd85c252f45ea5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:24.412637 18146 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ea83912975774b6089fd85c252f45ea5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:24.414019 18151 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54524:
name: "default.Table_1bb7ee3d454b4c1eac6646e9b3eae7e1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240430 20:11:24.406711 18983 raft_consensus.cc:348] T c3f8f9ada98b40bc8359a00e193c3e5f P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:24.414435 18983 raft_consensus.cc:374] T c3f8f9ada98b40bc8359a00e193c3e5f P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20240430 20:11:24.414536 18151 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1bb7ee3d454b4c1eac6646e9b3eae7e1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:24.414562 18983 raft_consensus.cc:729] T c3f8f9ada98b40bc8359a00e193c3e5f P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
W20240430 20:11:24.415899 18699 leader_election.cc:341] T 4d3117a9a5f243b59a69d960c0a1eee4 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Not found: Tablet not found: 4d3117a9a5f243b59a69d960c0a1eee4
I20240430 20:11:24.416708 18980 ts_tablet_manager.cc:1888] T 4d3117a9a5f243b59a69d960c0a1eee4 P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:24.416620 18983 consensus_queue.cc:260] T c3f8f9ada98b40bc8359a00e193c3e5f P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:24.417431 18980 log.cc:1196] T 4d3117a9a5f243b59a69d960c0a1eee4 P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/4d3117a9a5f243b59a69d960c0a1eee4
I20240430 20:11:24.418411 18155 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54524:
name: "default.Table_fe1773cf77654e2a86e95444437b7201"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240430 20:11:24.419071 18983 ts_tablet_manager.cc:1398] T c3f8f9ada98b40bc8359a00e193c3e5f P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.013s user 0.003s sys 0.000s
W20240430 20:11:24.419085 18155 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fe1773cf77654e2a86e95444437b7201 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:24.419446 18983 tablet_bootstrap.cc:492] T f15f5e1f3afd4224bbf39567cae74617 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
W20240430 20:11:24.422809 18141 catalog_manager.cc:4810] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): delete failed for tablet ffe059eeac91476aaeef8c56d2f49ee8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ffe059eeac91476aaeef8c56d2f49ee8 already in progress: creating tablet
I20240430 20:11:24.428227 18980 ts_tablet_manager.cc:1909] T 4d3117a9a5f243b59a69d960c0a1eee4 P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:24.428228 18983 tablet_bootstrap.cc:654] T f15f5e1f3afd4224bbf39567cae74617 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:24.432689 18980 tablet_replica.cc:330] T bf925791ddcf4fa7b5a1645c63f0d696 P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:24.434808 18980 raft_consensus.cc:2229] T bf925791ddcf4fa7b5a1645c63f0d696 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:24.435014 18980 raft_consensus.cc:2258] T bf925791ddcf4fa7b5a1645c63f0d696 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:24.435554 18980 ts_tablet_manager.cc:1875] T bf925791ddcf4fa7b5a1645c63f0d696 P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:24.449364 18147 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54524:
name: "default.Table_eea9acffa45b4f3ab6d4524dc460853c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:24.450101 18147 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_eea9acffa45b4f3ab6d4524dc460853c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:24.458468 18152 master_service.cc:946] ReplaceTablet: received request to replace tablet b4bc111e183f40a0acc9dd377195f6d9 from {username='slave'} at 127.0.0.1:54524
I20240430 20:11:24.458643 18980 ts_tablet_manager.cc:1888] T bf925791ddcf4fa7b5a1645c63f0d696 P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:24.458770 18980 log.cc:1196] T bf925791ddcf4fa7b5a1645c63f0d696 P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/bf925791ddcf4fa7b5a1645c63f0d696
I20240430 20:11:24.459129 18980 ts_tablet_manager.cc:1909] T bf925791ddcf4fa7b5a1645c63f0d696 P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:24.463299 18934 rpc_server.cc:307] RPC server started. Bound to: 127.17.177.60:37225
I20240430 20:11:24.463229 19139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.177.60:37225 every 8 connection(s)
I20240430 20:11:24.464102 19042 raft_consensus.cc:482] T e6c0c808bfc94be3837ab835292b3c1b P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:24.464251 19042 raft_consensus.cc:504] T e6c0c808bfc94be3837ab835292b3c1b P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:24.465508 18934 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/master-2/data/info.pb
I20240430 20:11:24.465433 18481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6c0c808bfc94be3837ab835292b3c1b" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:24.465692 18481 raft_consensus.cc:2454] T e6c0c808bfc94be3837ab835292b3c1b P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 0.
I20240430 20:11:24.465855 18739 tablet_service.cc:1467] Processing CreateTablet for tablet 15674aeb56b74210a4a2809608fd4457 (DEFAULT_TABLE table=default.Table_40d60ca37bac4d53a9297289f18c1719 [id=efc0011648254789a4f3b41a1fffdf4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:24.466213 18741 tablet_service.cc:1467] Processing CreateTablet for tablet fd47db5ca8b24c2b827df0a64e195866 (DEFAULT_TABLE table=default.Table_14776edb17d64a83be63345e68995156 [id=6544b0a3871d41c9bf3e524f669aa0fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:24.466399 18739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15674aeb56b74210a4a2809608fd4457. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:24.466697 18741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd47db5ca8b24c2b827df0a64e195866. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:24.467628 18743 tablet_service.cc:1467] Processing CreateTablet for tablet 372b135dd01f44f98cd713ca8857ba61 (DEFAULT_TABLE table=default.Table_67a6f326000b49c98d52cfcdf5dea396 [id=7f3a15a442984827a7521d5d1c98dc3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:24.468044 18743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 372b135dd01f44f98cd713ca8857ba61. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:24.470960 18558 leader_election.cc:302] T e6c0c808bfc94be3837ab835292b3c1b P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:24.470930 18742 tablet_service.cc:1467] Processing CreateTablet for tablet 13c1107ea4d04eb78a74d3f658c72fb3 (DEFAULT_TABLE table=default.Table_ddbb3fa4bcd7461cba6de52ebacdb12d [id=77aff20b7b63407c91f98a01a6f46e4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:24.467623 18740 tablet_service.cc:1467] Processing CreateTablet for tablet 0fe1247349344863a6273a769ce8ddf6 (DEFAULT_TABLE table=default.Table_7bfb47ba65114665943b7edccabd5a8b [id=c487c255c03741199e57b71670bc84e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:24.473397 18150 master_service.cc:946] ReplaceTablet: received request to replace tablet 44699c84ebac49ef8f3765d4069e71ca from {username='slave'} at 127.0.0.1:54524
I20240430 20:11:24.473910 18116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNG3HqZ/build/asan/bin/kudu as pid 18934
I20240430 20:11:24.476119 19042 leader_election.cc:288] T e6c0c808bfc94be3837ab835292b3c1b P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:24.476342 19042 raft_consensus.cc:2781] T e6c0c808bfc94be3837ab835292b3c1b P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:24.476475 19042 raft_consensus.cc:482] T e6c0c808bfc94be3837ab835292b3c1b P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:24.476586 19042 raft_consensus.cc:3037] T e6c0c808bfc94be3837ab835292b3c1b P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:24.482167 18456 tablet_service.cc:1467] Processing CreateTablet for tablet bbe9a2429a434f3f95d6e7f314fcfe4c (DEFAULT_TABLE table=default.Table_3acf40c6d58741d4a8e94ca96d642249 [id=71226c1fc71242a6aad13625c9ca3c23]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:24.482676 18456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbe9a2429a434f3f95d6e7f314fcfe4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:24.482841 18458 tablet_service.cc:1467] Processing CreateTablet for tablet 13c1107ea4d04eb78a74d3f658c72fb3 (DEFAULT_TABLE table=default.Table_ddbb3fa4bcd7461cba6de52ebacdb12d [id=77aff20b7b63407c91f98a01a6f46e4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:24.483196 18458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13c1107ea4d04eb78a74d3f658c72fb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:24.485891 19042 raft_consensus.cc:504] T e6c0c808bfc94be3837ab835292b3c1b P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:24.486485 18459 tablet_service.cc:1467] Processing CreateTablet for tablet 0fe1247349344863a6273a769ce8ddf6 (DEFAULT_TABLE table=default.Table_7bfb47ba65114665943b7edccabd5a8b [id=c487c255c03741199e57b71670bc84e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:24.486869 18459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fe1247349344863a6273a769ce8ddf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:24.490257 18461 tablet_service.cc:1467] Processing CreateTablet for tablet f5236574672b477a9026e91adcaf70ec (DEFAULT_TABLE table=default.Table_0c6652c070534fbbaab6c2b8d175f139 [id=00dd4c11a1cc4018bb900da95768bb88]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:24.490964 18455 tablet_service.cc:1467] Processing CreateTablet for tablet fae9a35a2ac844b4bc2ebdf214aa7171 (DEFAULT_TABLE table=default.Table_0ab5b2f09fa7468bb89ab703f6d3d25c [id=a5d580cffa2447d78fc9f2c8e31264f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:24.491035 19042 leader_election.cc:288] T e6c0c808bfc94be3837ab835292b3c1b P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Requested vote from peers 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:24.491642 18454 tablet_service.cc:1467] Processing CreateTablet for tablet 4e97f7315f8b4ac29db16e00fc35bf99 (DEFAULT_TABLE table=default.Table_3a865e57c98d4c02bfb7910ab5f0be50 [id=6747400b7ddf4d809488469d1d66ee1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:24.492352 18453 tablet_service.cc:1467] Processing CreateTablet for tablet cace1398492f4d968dcb6899385d527c (DEFAULT_TABLE table=default.Table_a8d5ac73c6d047698d87b3d1ba34e66f [id=9c80193cde2346c982cf84ae9612e2f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:24.493147 18452 tablet_service.cc:1467] Processing CreateTablet for tablet c9ff972ade694938888514979e1638aa (DEFAULT_TABLE table=default.Table_3c5df662747b48bf8beb6ebe4d2ea618 [id=663c8efbd7ac4a148d5f7626f1522033]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:24.493897 18451 tablet_service.cc:1467] Processing CreateTablet for tablet 7bde1d6934784c9aa5911181e8df9eeb (DEFAULT_TABLE table=default.Table_7199d7c36d3f45f79cf9e4e2b4145592 [id=f6aef8f6a9164dd78f49e03f723b45a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:24.482492 18462 tablet_service.cc:1467] Processing CreateTablet for tablet fd47db5ca8b24c2b827df0a64e195866 (DEFAULT_TABLE table=default.Table_14776edb17d64a83be63345e68995156 [id=6544b0a3871d41c9bf3e524f669aa0fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:24.494799 18461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5236574672b477a9026e91adcaf70ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:24.496254 18462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd47db5ca8b24c2b827df0a64e195866. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:24.498823 18455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fae9a35a2ac844b4bc2ebdf214aa7171. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:24.504166 18454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e97f7315f8b4ac29db16e00fc35bf99. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:24.506831 18453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cace1398492f4d968dcb6899385d527c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:24.509824 18451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bde1d6934784c9aa5911181e8df9eeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:24.490970 18481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6c0c808bfc94be3837ab835292b3c1b" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:24.514999 18481 raft_consensus.cc:3037] T e6c0c808bfc94be3837ab835292b3c1b P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:24.518391 18481 raft_consensus.cc:2454] T e6c0c808bfc94be3837ab835292b3c1b P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
I20240430 20:11:24.518769 18452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9ff972ade694938888514979e1638aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:24.519804 19140 sys_catalog.cc:263] Verifying existing consensus state
I20240430 20:11:24.530201 19140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: Bootstrap starting.
I20240430 20:11:24.535401 18740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fe1247349344863a6273a769ce8ddf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:24.551148 18742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13c1107ea4d04eb78a74d3f658c72fb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:24.562963 18762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6c0c808bfc94be3837ab835292b3c1b" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:24.563477 18763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6c0c808bfc94be3837ab835292b3c1b" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
W20240430 20:11:24.564405 18557 leader_election.cc:341] T e6c0c808bfc94be3837ab835292b3c1b P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
W20240430 20:11:24.566717 18557 leader_election.cc:341] T e6c0c808bfc94be3837ab835292b3c1b P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:24.567553 18744 tablet_service.cc:1467] Processing CreateTablet for tablet bbe9a2429a434f3f95d6e7f314fcfe4c (DEFAULT_TABLE table=default.Table_3acf40c6d58741d4a8e94ca96d642249 [id=71226c1fc71242a6aad13625c9ca3c23]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:24.572260 18737 tablet_service.cc:1467] Processing CreateTablet for tablet 4e97f7315f8b4ac29db16e00fc35bf99 (DEFAULT_TABLE table=default.Table_3a865e57c98d4c02bfb7910ab5f0be50 [id=6747400b7ddf4d809488469d1d66ee1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:24.572772 18736 tablet_service.cc:1467] Processing CreateTablet for tablet cace1398492f4d968dcb6899385d527c (DEFAULT_TABLE table=default.Table_a8d5ac73c6d047698d87b3d1ba34e66f [id=9c80193cde2346c982cf84ae9612e2f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:24.573336 18744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbe9a2429a434f3f95d6e7f314fcfe4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:24.577153 19140 log.cc:824] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: Log is configured to *not* fsync() on all Append() calls
I20240430 20:11:24.578830 18736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cace1398492f4d968dcb6899385d527c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:24.581369 18737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e97f7315f8b4ac29db16e00fc35bf99. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:24.572119 18741 tablet_service.cc:1467] Processing CreateTablet for tablet f5236574672b477a9026e91adcaf70ec (DEFAULT_TABLE table=default.Table_0c6652c070534fbbaab6c2b8d175f139 [id=00dd4c11a1cc4018bb900da95768bb88]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:24.585489 18738 tablet_service.cc:1467] Processing CreateTablet for tablet fae9a35a2ac844b4bc2ebdf214aa7171 (DEFAULT_TABLE table=default.Table_0ab5b2f09fa7468bb89ab703f6d3d25c [id=a5d580cffa2447d78fc9f2c8e31264f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:24.589184 18741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5236574672b477a9026e91adcaf70ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:24.594403 18558 leader_election.cc:302] T e6c0c808bfc94be3837ab835292b3c1b P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters: 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:24.594787 19042 raft_consensus.cc:2781] T e6c0c808bfc94be3837ab835292b3c1b P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:24.594970 19042 raft_consensus.cc:686] T e6c0c808bfc94be3837ab835292b3c1b P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Becoming Leader. State: Replica: 5e1c6397f09747f091534faf93542710, State: Running, Role: LEADER
I20240430 20:11:24.595357 19042 consensus_queue.cc:237] T e6c0c808bfc94be3837ab835292b3c1b P 5e1c6397f09747f091534faf93542710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:24.597119 18733 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d3117a9a5f243b59a69d960c0a1eee4 with delete_type TABLET_DATA_DELETED (replaced by tablet e0d4cebfbfca45c08d1e7500228835b4) from {username='slave'} at 127.0.0.1:59818
I20240430 20:11:24.597865 18987 tablet_replica.cc:330] T 4d3117a9a5f243b59a69d960c0a1eee4 P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:24.587033 18734 tablet_service.cc:1467] Processing CreateTablet for tablet 7bde1d6934784c9aa5911181e8df9eeb (DEFAULT_TABLE table=default.Table_7199d7c36d3f45f79cf9e4e2b4145592 [id=f6aef8f6a9164dd78f49e03f723b45a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:24.598802 18463 tablet_service.cc:1467] Processing CreateTablet for tablet 15674aeb56b74210a4a2809608fd4457 (DEFAULT_TABLE table=default.Table_40d60ca37bac4d53a9297289f18c1719 [id=efc0011648254789a4f3b41a1fffdf4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:24.599242 18463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15674aeb56b74210a4a2809608fd4457. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:24.602504 18732 tablet_service.cc:1514] Processing DeleteTablet for tablet f79497bf22ed417d988530f6e69cc07f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:23 UTC) from {username='slave'} at 127.0.0.1:59818
I20240430 20:11:24.603098 18731 tablet_service.cc:1514] Processing DeleteTablet for tablet 2067cb2cf76d495c92c1f25c176eb6dc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:23 UTC) from {username='slave'} at 127.0.0.1:59818
I20240430 20:11:24.599764 18735 tablet_service.cc:1467] Processing CreateTablet for tablet c9ff972ade694938888514979e1638aa (DEFAULT_TABLE table=default.Table_3c5df662747b48bf8beb6ebe4d2ea618 [id=663c8efbd7ac4a148d5f7626f1522033]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:24.609534 18457 tablet_service.cc:1467] Processing CreateTablet for tablet 372b135dd01f44f98cd713ca8857ba61 (DEFAULT_TABLE table=default.Table_67a6f326000b49c98d52cfcdf5dea396 [id=7f3a15a442984827a7521d5d1c98dc3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:24.609789 18730 tablet_service.cc:1514] Processing DeleteTablet for tablet bf925791ddcf4fa7b5a1645c63f0d696 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:23 UTC) from {username='slave'} at 127.0.0.1:59818
I20240430 20:11:24.609926 18457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 372b135dd01f44f98cd713ca8857ba61. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:24.610013 18738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fae9a35a2ac844b4bc2ebdf214aa7171. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:24.612162 18669 heartbeater.cc:347] Connected to a master server at 127.17.177.60:37225
I20240430 20:11:24.612358 18669 heartbeater.cc:503] Master 127.17.177.60:37225 was elected leader, sending a full tablet report...
I20240430 20:11:24.613096 18735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9ff972ade694938888514979e1638aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:24.613340 18734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bde1d6934784c9aa5911181e8df9eeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:24.657296 18146 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_ddbb3fa4bcd7461cba6de52ebacdb12d" } new_table_name: "default.Table_52dcbd45fadf4a4d85c4533f1d69f976" modify_external_catalogs: true
I20240430 20:11:24.663201 18149 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_b927029492584700848792c3500ac17e" } new_table_name: "default.Table_173d3907261b4390aa5647f01794d901" modify_external_catalogs: true
I20240430 20:11:24.695552 18155 master_service.cc:946] ReplaceTablet: received request to replace tablet 0d6a4a63c004480b82d84381867bf8a4 from {username='slave'} at 127.0.0.1:54524
I20240430 20:11:24.696889 18151 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54524:
name: "default.Table_40857db02a904104a4850e51d5996da2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:24.697715 18151 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_40857db02a904104a4850e51d5996da2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:24.699318 18147 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54524:
name: "default.Table_4e15b6827cbe4bd3a7c8264b73f6e999"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240430 20:11:24.696170 18155 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_67a6f326000b49c98d52cfcdf5dea396" } new_table_name: "default.Table_31f8c79337974bd9957c65f54d3c74e5" modify_external_catalogs: true
W20240430 20:11:24.700136 18147 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4e15b6827cbe4bd3a7c8264b73f6e999 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:24.703055 19072 raft_consensus.cc:482] T 81328e904ca94e2eb5ec94dd3ffb3c90 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:24.703246 19072 raft_consensus.cc:504] T 81328e904ca94e2eb5ec94dd3ffb3c90 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:24.703989 19072 leader_election.cc:288] T 81328e904ca94e2eb5ec94dd3ffb3c90 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:24.704075 18148 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: Sending DeleteTablet for 3 replicas of tablet a01ee128ba8a4c9fb8e125ba97239d4c
I20240430 20:11:24.704699 18621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81328e904ca94e2eb5ec94dd3ffb3c90" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:24.705529 18742 tablet_service.cc:1514] Processing DeleteTablet for tablet a01ee128ba8a4c9fb8e125ba97239d4c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:24 UTC) from {username='slave'} at 127.0.0.1:59818
W20240430 20:11:24.705941 18415 leader_election.cc:341] T 81328e904ca94e2eb5ec94dd3ffb3c90 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:24.706712 18148 catalog_manager.cc:5484] T e6c0c808bfc94be3837ab835292b3c1b P 5e1c6397f09747f091534faf93542710 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e1c6397f09747f091534faf93542710 (127.17.177.2). New cstate: current_term: 1 leader_uuid: "5e1c6397f09747f091534faf93542710" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } health_report { overall_health: UNKNOWN } } }
I20240430 20:11:24.706951 18763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81328e904ca94e2eb5ec94dd3ffb3c90" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
W20240430 20:11:24.707523 18417 leader_election.cc:341] T 81328e904ca94e2eb5ec94dd3ffb3c90 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:24.707638 18417 leader_election.cc:302] T 81328e904ca94e2eb5ec94dd3ffb3c90 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a; no voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710
I20240430 20:11:24.708092 19072 raft_consensus.cc:2726] T 81328e904ca94e2eb5ec94dd3ffb3c90 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:24.708408 18463 tablet_service.cc:1514] Processing DeleteTablet for tablet a01ee128ba8a4c9fb8e125ba97239d4c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:24 UTC) from {username='slave'} at 127.0.0.1:41678
I20240430 20:11:24.708899 18154 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: Sending DeleteTablet for 3 replicas of tablet bd7392f3dfd643cdb1634ee10ad41651
I20240430 20:11:24.710389 18742 tablet_service.cc:1514] Processing DeleteTablet for tablet bd7392f3dfd643cdb1634ee10ad41651 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:24 UTC) from {username='slave'} at 127.0.0.1:59818
I20240430 20:11:24.710711 18463 tablet_service.cc:1514] Processing DeleteTablet for tablet bd7392f3dfd643cdb1634ee10ad41651 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:24 UTC) from {username='slave'} at 127.0.0.1:41678
W20240430 20:11:24.711055 18142 catalog_manager.cc:4810] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): delete failed for tablet a01ee128ba8a4c9fb8e125ba97239d4c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a01ee128ba8a4c9fb8e125ba97239d4c already in progress: creating tablet
W20240430 20:11:24.712759 18142 catalog_manager.cc:4810] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): delete failed for tablet bd7392f3dfd643cdb1634ee10ad41651 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bd7392f3dfd643cdb1634ee10ad41651 already in progress: creating tablet
I20240430 20:11:24.713837 18154 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54524:
name: "default.Table_9854f29c4fdd4c4bb14859cbc9c48fb3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:24.714531 18154 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9854f29c4fdd4c4bb14859cbc9c48fb3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:24.717136 18156 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: Sending DeleteTablet for 3 replicas of tablet 0fe1247349344863a6273a769ce8ddf6
I20240430 20:11:24.718654 18742 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fe1247349344863a6273a769ce8ddf6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:24 UTC) from {username='slave'} at 127.0.0.1:59818
I20240430 20:11:24.720052 18463 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fe1247349344863a6273a769ce8ddf6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:24 UTC) from {username='slave'} at 127.0.0.1:41678
I20240430 20:11:24.721678 18156 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_99fded9dcd7a4147a4373fc15d9d2677" } modify_external_catalogs: true
I20240430 20:11:24.722030 18156 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_99fded9dcd7a4147a4373fc15d9d2677" } modify_external_catalogs: true
W20240430 20:11:24.723462 18142 catalog_manager.cc:4810] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): delete failed for tablet 0fe1247349344863a6273a769ce8ddf6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0fe1247349344863a6273a769ce8ddf6 already in progress: creating tablet
I20240430 20:11:24.725368 19140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: Bootstrap replayed 1/1 log segments. Stats: ops{read=35 overwritten=0 applied=30 ignored=0} inserts{seen=27 ignored=0} mutations{seen=20 ignored=0} orphaned_commits=0. Pending: 5 replicates
I20240430 20:11:24.726336 19140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: Bootstrap complete.
I20240430 20:11:24.728071 18152 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: Sending DeleteTablet for 3 replicas of tablet b4bc111e183f40a0acc9dd377195f6d9
I20240430 20:11:24.728727 18152 catalog_manager.cc:6554] ReplaceTablet: tablet b4bc111e183f40a0acc9dd377195f6d9 deleted and replaced by tablet 0b4d9e860d7c45fd91d7a566605c9283
I20240430 20:11:24.729318 18742 tablet_service.cc:1514] Processing DeleteTablet for tablet b4bc111e183f40a0acc9dd377195f6d9 with delete_type TABLET_DATA_DELETED (replaced by tablet 0b4d9e860d7c45fd91d7a566605c9283) from {username='slave'} at 127.0.0.1:59818
I20240430 20:11:24.730302 18987 raft_consensus.cc:2229] T 4d3117a9a5f243b59a69d960c0a1eee4 P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Raft consensus shutting down.
W20240430 20:11:24.731199 18146 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 13c1107ea4d04eb78a74d3f658c72fb3 consensus state has no leader
I20240430 20:11:24.731380 18987 pending_rounds.cc:62] T 4d3117a9a5f243b59a69d960c0a1eee4 P 3c51c33d5616482cafb68e26688820f4: Trying to abort 1 pending ops.
I20240430 20:11:24.731518 18987 pending_rounds.cc:68] T 4d3117a9a5f243b59a69d960c0a1eee4 P 3c51c33d5616482cafb68e26688820f4: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7022624294540988416 op_type: NO_OP noop_request { }
I20240430 20:11:24.731702 18987 raft_consensus.cc:2866] T 4d3117a9a5f243b59a69d960c0a1eee4 P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240430 20:11:24.731850 18987 raft_consensus.cc:2258] T 4d3117a9a5f243b59a69d960c0a1eee4 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:24.732697 18463 tablet_service.cc:1514] Processing DeleteTablet for tablet b4bc111e183f40a0acc9dd377195f6d9 with delete_type TABLET_DATA_DELETED (replaced by tablet 0b4d9e860d7c45fd91d7a566605c9283) from {username='slave'} at 127.0.0.1:41678
W20240430 20:11:24.733430 18142 catalog_manager.cc:4810] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): delete failed for tablet b4bc111e183f40a0acc9dd377195f6d9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b4bc111e183f40a0acc9dd377195f6d9 already in progress: creating tablet
I20240430 20:11:24.735415 18601 tablet_service.cc:1467] Processing CreateTablet for tablet 15674aeb56b74210a4a2809608fd4457 (DEFAULT_TABLE table=default.Table_40d60ca37bac4d53a9297289f18c1719 [id=efc0011648254789a4f3b41a1fffdf4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:24.735949 18601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15674aeb56b74210a4a2809608fd4457. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:24.742659 19140 raft_consensus.cc:348] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33 [term 3 FOLLOWER]: Replica starting. Triggering 5 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b3f98793354c7aaa5c106e8ec31ed8" member_type: VOTER last_known_addr { host: "127.17.177.62" port: 36441 } } peers { permanent_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e" member_type: VOTER last_known_addr { host: "127.17.177.61" port: 34777 } } peers { permanent_uuid: "b632d6ef4c104a058a6db208024cab33" member_type: VOTER last_known_addr { host: "127.17.177.60" port: 37225 } }
I20240430 20:11:24.745270 18599 tablet_service.cc:1467] Processing CreateTablet for tablet fd47db5ca8b24c2b827df0a64e195866 (DEFAULT_TABLE table=default.Table_14776edb17d64a83be63345e68995156 [id=6544b0a3871d41c9bf3e524f669aa0fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:24.745450 18603 tablet_service.cc:1467] Processing CreateTablet for tablet 372b135dd01f44f98cd713ca8857ba61 (DEFAULT_TABLE table=default.Table_67a6f326000b49c98d52cfcdf5dea396 [id=7f3a15a442984827a7521d5d1c98dc3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:24.745704 18599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd47db5ca8b24c2b827df0a64e195866. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:24.745920 18603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 372b135dd01f44f98cd713ca8857ba61. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:24.753485 18600 tablet_service.cc:1467] Processing CreateTablet for tablet bbe9a2429a434f3f95d6e7f314fcfe4c (DEFAULT_TABLE table=default.Table_3acf40c6d58741d4a8e94ca96d642249 [id=71226c1fc71242a6aad13625c9ca3c23]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:24.753983 18600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbe9a2429a434f3f95d6e7f314fcfe4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:24.754124 18602 tablet_service.cc:1514] Processing DeleteTablet for tablet 2067cb2cf76d495c92c1f25c176eb6dc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:23 UTC) from {username='slave'} at 127.0.0.1:40202
I20240430 20:11:24.754175 18596 tablet_service.cc:1467] Processing CreateTablet for tablet 4e97f7315f8b4ac29db16e00fc35bf99 (DEFAULT_TABLE table=default.Table_3a865e57c98d4c02bfb7910ab5f0be50 [id=6747400b7ddf4d809488469d1d66ee1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:24.754377 18980 tablet_replica.cc:330] T 2067cb2cf76d495c92c1f25c176eb6dc P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:24.754630 18980 raft_consensus.cc:2229] T 2067cb2cf76d495c92c1f25c176eb6dc P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:24.754820 18980 raft_consensus.cc:2258] T 2067cb2cf76d495c92c1f25c176eb6dc P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:24.755108 18593 tablet_service.cc:1467] Processing CreateTablet for tablet c9ff972ade694938888514979e1638aa (DEFAULT_TABLE table=default.Table_3c5df662747b48bf8beb6ebe4d2ea618 [id=663c8efbd7ac4a148d5f7626f1522033]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:24.755463 18980 ts_tablet_manager.cc:1875] T 2067cb2cf76d495c92c1f25c176eb6dc P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:24.755790 18593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9ff972ade694938888514979e1638aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:24.758867 18598 tablet_service.cc:1467] Processing CreateTablet for tablet fae9a35a2ac844b4bc2ebdf214aa7171 (DEFAULT_TABLE table=default.Table_0ab5b2f09fa7468bb89ab703f6d3d25c [id=a5d580cffa2447d78fc9f2c8e31264f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:24.759207 18598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fae9a35a2ac844b4bc2ebdf214aa7171. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:24.761816 18596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e97f7315f8b4ac29db16e00fc35bf99. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:24.762807 18597 tablet_service.cc:1514] Processing DeleteTablet for tablet 9750608faeab4be38ddce4e2e4d46964 with delete_type TABLET_DATA_DELETED (replaced by tablet 0fb0c0ca09154f7a809dd523a859929a) from {username='slave'} at 127.0.0.1:40202
I20240430 20:11:24.763430 18595 tablet_service.cc:1467] Processing CreateTablet for tablet cace1398492f4d968dcb6899385d527c (DEFAULT_TABLE table=default.Table_a8d5ac73c6d047698d87b3d1ba34e66f [id=9c80193cde2346c982cf84ae9612e2f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:24.764063 18595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cace1398492f4d968dcb6899385d527c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:24.767506 18592 tablet_service.cc:1467] Processing CreateTablet for tablet 7bde1d6934784c9aa5911181e8df9eeb (DEFAULT_TABLE table=default.Table_7199d7c36d3f45f79cf9e4e2b4145592 [id=f6aef8f6a9164dd78f49e03f723b45a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:24.768066 18592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bde1d6934784c9aa5911181e8df9eeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:24.759410 18599 tablet_service.cc:1467] Processing CreateTablet for tablet f5236574672b477a9026e91adcaf70ec (DEFAULT_TABLE table=default.Table_0c6652c070534fbbaab6c2b8d175f139 [id=00dd4c11a1cc4018bb900da95768bb88]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:24.773267 18152 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_3a865e57c98d4c02bfb7910ab5f0be50" } new_table_name: "default.Table_af1c8032c64245218a81d5bda4c68302" modify_external_catalogs: true
I20240430 20:11:24.779840 18980 ts_tablet_manager.cc:1888] T 2067cb2cf76d495c92c1f25c176eb6dc P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:24.779891 18599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5236574672b477a9026e91adcaf70ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:24.780649 18980 log.cc:1196] T 2067cb2cf76d495c92c1f25c176eb6dc P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/2067cb2cf76d495c92c1f25c176eb6dc
I20240430 20:11:24.779951 18603 tablet_service.cc:1467] Processing CreateTablet for tablet 0fe1247349344863a6273a769ce8ddf6 (DEFAULT_TABLE table=default.Table_7bfb47ba65114665943b7edccabd5a8b [id=c487c255c03741199e57b71670bc84e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:24.782814 18603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fe1247349344863a6273a769ce8ddf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:24.783303 18593 tablet_service.cc:1514] Processing DeleteTablet for tablet c3f8f9ada98b40bc8359a00e193c3e5f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:23 UTC) from {username='slave'} at 127.0.0.1:40202
I20240430 20:11:24.783798 18601 tablet_service.cc:1467] Processing CreateTablet for tablet 13c1107ea4d04eb78a74d3f658c72fb3 (DEFAULT_TABLE table=default.Table_ddbb3fa4bcd7461cba6de52ebacdb12d [id=77aff20b7b63407c91f98a01a6f46e4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:24.784057 18597 tablet_service.cc:1514] Processing DeleteTablet for tablet a01ee128ba8a4c9fb8e125ba97239d4c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:24 UTC) from {username='slave'} at 127.0.0.1:40202
I20240430 20:11:24.784768 18602 tablet_service.cc:1514] Processing DeleteTablet for tablet bd7392f3dfd643cdb1634ee10ad41651 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:24 UTC) from {username='slave'} at 127.0.0.1:40202
I20240430 20:11:24.785080 18980 ts_tablet_manager.cc:1909] T 2067cb2cf76d495c92c1f25c176eb6dc P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:24.786836 18980 tablet_replica.cc:330] T 9750608faeab4be38ddce4e2e4d46964 P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:24.787077 18602 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fe1247349344863a6273a769ce8ddf6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:24 UTC) from {username='slave'} at 127.0.0.1:40202
I20240430 20:11:24.788465 18980 raft_consensus.cc:2229] T 9750608faeab4be38ddce4e2e4d46964 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:24.788470 18597 tablet_service.cc:1514] Processing DeleteTablet for tablet b4bc111e183f40a0acc9dd377195f6d9 with delete_type TABLET_DATA_DELETED (replaced by tablet 0b4d9e860d7c45fd91d7a566605c9283) from {username='slave'} at 127.0.0.1:40202
I20240430 20:11:24.789978 18601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13c1107ea4d04eb78a74d3f658c72fb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:24.790122 18150 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: Sending DeleteTablet for 3 replicas of tablet 44699c84ebac49ef8f3765d4069e71ca
I20240430 20:11:24.791393 18150 catalog_manager.cc:6554] ReplaceTablet: tablet 44699c84ebac49ef8f3765d4069e71ca deleted and replaced by tablet 34ab40d8d91e401db299ddee7175ba28
I20240430 20:11:24.791738 18150 master_service.cc:946] ReplaceTablet: received request to replace tablet 26a015250eeb4750a42df93840e8d47f from {username='slave'} at 127.0.0.1:54524
I20240430 20:11:24.791744 18742 tablet_service.cc:1514] Processing DeleteTablet for tablet 44699c84ebac49ef8f3765d4069e71ca with delete_type TABLET_DATA_DELETED (replaced by tablet 34ab40d8d91e401db299ddee7175ba28) from {username='slave'} at 127.0.0.1:59818
W20240430 20:11:24.792608 18141 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 13c1107ea4d04eb78a74d3f658c72fb3 (table default.Table_52dcbd45fadf4a4d85c4533f1d69f976 [id=77aff20b7b63407c91f98a01a6f46e4c], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 13c1107ea4d04eb78a74d3f658c72fb3 consensus state has no leader
W20240430 20:11:24.793584 18149 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 187a27a6449342f99de3c5a2f93d49fc consensus state has no leader
I20240430 20:11:24.794216 18980 raft_consensus.cc:2258] T 9750608faeab4be38ddce4e2e4d46964 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:24.794963 18980 ts_tablet_manager.cc:1875] T 9750608faeab4be38ddce4e2e4d46964 P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:24.805258 18463 tablet_service.cc:1514] Processing DeleteTablet for tablet 44699c84ebac49ef8f3765d4069e71ca with delete_type TABLET_DATA_DELETED (replaced by tablet 34ab40d8d91e401db299ddee7175ba28) from {username='slave'} at 127.0.0.1:41678
I20240430 20:11:24.805624 18981 tablet_replica.cc:330] T 44699c84ebac49ef8f3765d4069e71ca P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:24.813527 19072 raft_consensus.cc:482] T 9750608faeab4be38ddce4e2e4d46964 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:24.813750 19072 raft_consensus.cc:504] T 9750608faeab4be38ddce4e2e4d46964 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:24.814471 19072 leader_election.cc:288] T 9750608faeab4be38ddce4e2e4d46964 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:24.815735 18763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9750608faeab4be38ddce4e2e4d46964" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
W20240430 20:11:24.816444 18417 leader_election.cc:341] T 9750608faeab4be38ddce4e2e4d46964 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:24.818626 18621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9750608faeab4be38ddce4e2e4d46964" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
W20240430 20:11:24.821627 18415 leader_election.cc:341] T 9750608faeab4be38ddce4e2e4d46964 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Illegal state: Tablet not RUNNING: STOPPED
I20240430 20:11:24.821790 18415 leader_election.cc:302] T 9750608faeab4be38ddce4e2e4d46964 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a; no voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710
I20240430 20:11:24.822371 19072 raft_consensus.cc:2726] T 9750608faeab4be38ddce4e2e4d46964 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240430 20:11:24.827877 19127 consensus_queue.cc:1159] T 4d3117a9a5f243b59a69d960c0a1eee4 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "3c51c33d5616482cafb68e26688820f4" status { last_received { term: 1 index: 1 } last_committed_idx: 0 last_received_current_leader { term: 1 index: 1 } }
I20240430 20:11:24.828850 18987 ts_tablet_manager.cc:1875] T 4d3117a9a5f243b59a69d960c0a1eee4 P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:24.836603 18980 ts_tablet_manager.cc:1888] T 9750608faeab4be38ddce4e2e4d46964 P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:24.836761 18980 log.cc:1196] T 9750608faeab4be38ddce4e2e4d46964 P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/9750608faeab4be38ddce4e2e4d46964
I20240430 20:11:24.837077 18980 ts_tablet_manager.cc:1909] T 9750608faeab4be38ddce4e2e4d46964 P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:24.837837 18980 tablet_replica.cc:330] T c3f8f9ada98b40bc8359a00e193c3e5f P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:24.838045 18980 raft_consensus.cc:2229] T c3f8f9ada98b40bc8359a00e193c3e5f P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:24.838196 18980 raft_consensus.cc:2258] T c3f8f9ada98b40bc8359a00e193c3e5f P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:24.839057 18980 ts_tablet_manager.cc:1875] T c3f8f9ada98b40bc8359a00e193c3e5f P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:24.839447 18987 ts_tablet_manager.cc:1888] T 4d3117a9a5f243b59a69d960c0a1eee4 P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:24.839591 18987 log.cc:1196] T 4d3117a9a5f243b59a69d960c0a1eee4 P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/4d3117a9a5f243b59a69d960c0a1eee4
I20240430 20:11:24.840049 18987 ts_tablet_manager.cc:1909] T 4d3117a9a5f243b59a69d960c0a1eee4 P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:24.841284 18141 catalog_manager.cc:4830] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): tablet 4d3117a9a5f243b59a69d960c0a1eee4 (table default.Table_75dd71ce34a14788878b70c8354caba6 [id=ff76c635446a4b4583759831970f784e]) successfully deleted
I20240430 20:11:24.841704 18987 tablet_replica.cc:330] T 2067cb2cf76d495c92c1f25c176eb6dc P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:24.841912 18987 raft_consensus.cc:2229] T 2067cb2cf76d495c92c1f25c176eb6dc P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Raft consensus shutting down.
I20240430 20:11:24.842350 18987 pending_rounds.cc:62] T 2067cb2cf76d495c92c1f25c176eb6dc P 3c51c33d5616482cafb68e26688820f4: Trying to abort 1 pending ops.
I20240430 20:11:24.842444 18987 pending_rounds.cc:68] T 2067cb2cf76d495c92c1f25c176eb6dc P 3c51c33d5616482cafb68e26688820f4: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7022624294257532928 op_type: NO_OP noop_request { }
I20240430 20:11:24.842626 18987 raft_consensus.cc:2866] T 2067cb2cf76d495c92c1f25c176eb6dc P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240430 20:11:24.842756 18987 raft_consensus.cc:2258] T 2067cb2cf76d495c92c1f25c176eb6dc P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:24.847172 18980 ts_tablet_manager.cc:1888] T c3f8f9ada98b40bc8359a00e193c3e5f P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:24.847954 18987 ts_tablet_manager.cc:1875] T 2067cb2cf76d495c92c1f25c176eb6dc P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:24.855573 18987 ts_tablet_manager.cc:1888] T 2067cb2cf76d495c92c1f25c176eb6dc P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:24.855898 18980 log.cc:1196] T c3f8f9ada98b40bc8359a00e193c3e5f P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/c3f8f9ada98b40bc8359a00e193c3e5f
I20240430 20:11:24.856475 18980 ts_tablet_manager.cc:1909] T c3f8f9ada98b40bc8359a00e193c3e5f P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:24.859246 18987 log.cc:1196] T 2067cb2cf76d495c92c1f25c176eb6dc P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/2067cb2cf76d495c92c1f25c176eb6dc
I20240430 20:11:24.859812 18987 ts_tablet_manager.cc:1909] T 2067cb2cf76d495c92c1f25c176eb6dc P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:24.861100 18987 tablet_replica.cc:330] T f79497bf22ed417d988530f6e69cc07f P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:24.861075 18141 catalog_manager.cc:4830] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): tablet 2067cb2cf76d495c92c1f25c176eb6dc (table default.Table_25ea07a841144dab969f3baf7ee7c0dc [id=b4bd1ac2bb0b4989af99b2f4936ce927]) successfully deleted
W20240430 20:11:24.860828 18482 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.590s user 0.002s sys 0.013s
W20240430 20:11:24.861425 18482 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.591s user 0.002s sys 0.013s
I20240430 20:11:24.861403 18987 raft_consensus.cc:2229] T f79497bf22ed417d988530f6e69cc07f P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:24.861675 18987 raft_consensus.cc:2258] T f79497bf22ed417d988530f6e69cc07f P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:24.862506 18987 ts_tablet_manager.cc:1875] T f79497bf22ed417d988530f6e69cc07f P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:24.862752 19072 raft_consensus.cc:482] T c3f8f9ada98b40bc8359a00e193c3e5f P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:24.862926 19072 raft_consensus.cc:504] T c3f8f9ada98b40bc8359a00e193c3e5f P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:24.863641 19072 leader_election.cc:288] T c3f8f9ada98b40bc8359a00e193c3e5f P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
W20240430 20:11:24.864714 18415 leader_election.cc:341] T c3f8f9ada98b40bc8359a00e193c3e5f P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Not found: Tablet not found: c3f8f9ada98b40bc8359a00e193c3e5f
I20240430 20:11:24.865455 18763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3f8f9ada98b40bc8359a00e193c3e5f" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
W20240430 20:11:24.866364 18139 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.177.60:37225, user_credentials={real_user=slave}} blocked reactor thread for 94225.2us
I20240430 20:11:24.866988 18139 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet 4d3117a9a5f243b59a69d960c0a1eee4 (table default.Table_75dd71ce34a14788878b70c8354caba6 [id=ff76c635446a4b4583759831970f784e]) successfully deleted
W20240430 20:11:24.868530 18139 catalog_manager.cc:4797] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): delete failed for tablet e5d1707ba0b34cc4b1cda39e36ccbd2f because the tablet was not found. No further retry: Not found: Tablet not found: e5d1707ba0b34cc4b1cda39e36ccbd2f
I20240430 20:11:24.868856 18139 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet bf925791ddcf4fa7b5a1645c63f0d696 (table default.Table_917365c2d10941558081a9898db3d7ea [id=b536a31acaea48d78885b0e585d00832]) successfully deleted
W20240430 20:11:24.869220 18139 catalog_manager.cc:4810] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): delete failed for tablet f15f5e1f3afd4224bbf39567cae74617 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f15f5e1f3afd4224bbf39567cae74617 already in progress: creating tablet
W20240430 20:11:24.869485 18139 catalog_manager.cc:4810] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): delete failed for tablet ffe059eeac91476aaeef8c56d2f49ee8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ffe059eeac91476aaeef8c56d2f49ee8 already in progress: creating tablet
I20240430 20:11:24.870273 18987 ts_tablet_manager.cc:1888] T f79497bf22ed417d988530f6e69cc07f P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:24.870273 18139 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet 2067cb2cf76d495c92c1f25c176eb6dc (table default.Table_25ea07a841144dab969f3baf7ee7c0dc [id=b4bd1ac2bb0b4989af99b2f4936ce927]) successfully deleted
W20240430 20:11:24.870489 18417 leader_election.cc:341] T c3f8f9ada98b40bc8359a00e193c3e5f P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:24.870711 18417 leader_election.cc:302] T c3f8f9ada98b40bc8359a00e193c3e5f P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a; no voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710
I20240430 20:11:24.870823 18987 log.cc:1196] T f79497bf22ed417d988530f6e69cc07f P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/f79497bf22ed417d988530f6e69cc07f
I20240430 20:11:24.871186 18987 ts_tablet_manager.cc:1909] T f79497bf22ed417d988530f6e69cc07f P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:24.871351 19072 raft_consensus.cc:2726] T c3f8f9ada98b40bc8359a00e193c3e5f P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:24.872260 18141 catalog_manager.cc:4830] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): tablet f79497bf22ed417d988530f6e69cc07f (table default.Table_b78b1b2c95694d1faee32321f845047b [id=2a3033894afd47ed8d907628a2e9712d]) successfully deleted
W20240430 20:11:24.872674 18141 catalog_manager.cc:4797] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): delete failed for tablet bf925791ddcf4fa7b5a1645c63f0d696 because the tablet was not found. No further retry: Not found: Tablet not found: bf925791ddcf4fa7b5a1645c63f0d696
I20240430 20:11:24.870501 18985 tablet_bootstrap.cc:492] T 59548d560b4e4f7ca034d730aca79364 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
W20240430 20:11:24.873041 18141 catalog_manager.cc:4810] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): delete failed for tablet a01ee128ba8a4c9fb8e125ba97239d4c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a01ee128ba8a4c9fb8e125ba97239d4c already in progress: creating tablet
W20240430 20:11:24.873461 18141 catalog_manager.cc:4810] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): delete failed for tablet bd7392f3dfd643cdb1634ee10ad41651 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bd7392f3dfd643cdb1634ee10ad41651 already in progress: creating tablet
I20240430 20:11:24.873275 18985 ts_tablet_manager.cc:1367] T 59548d560b4e4f7ca034d730aca79364 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.477s user 0.040s sys 0.084s
W20240430 20:11:24.873776 18141 catalog_manager.cc:4810] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): delete failed for tablet 0fe1247349344863a6273a769ce8ddf6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0fe1247349344863a6273a769ce8ddf6 already in progress: creating tablet
W20240430 20:11:24.874195 18141 catalog_manager.cc:4810] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): delete failed for tablet b4bc111e183f40a0acc9dd377195f6d9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b4bc111e183f40a0acc9dd377195f6d9 already in progress: creating tablet
I20240430 20:11:24.874568 18987 tablet_replica.cc:330] T 44699c84ebac49ef8f3765d4069e71ca P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:24.874822 18987 raft_consensus.cc:2229] T 44699c84ebac49ef8f3765d4069e71ca P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Raft consensus shutting down.
I20240430 20:11:24.875046 18985 raft_consensus.cc:348] T 59548d560b4e4f7ca034d730aca79364 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:24.875175 18987 raft_consensus.cc:2258] T 44699c84ebac49ef8f3765d4069e71ca P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:24.875267 18985 raft_consensus.cc:374] T 59548d560b4e4f7ca034d730aca79364 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:24.875357 18985 raft_consensus.cc:729] T 59548d560b4e4f7ca034d730aca79364 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:24.875628 18985 consensus_queue.cc:260] T 59548d560b4e4f7ca034d730aca79364 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:24.876487 18985 ts_tablet_manager.cc:1398] T 59548d560b4e4f7ca034d730aca79364 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240430 20:11:24.876778 18985 tablet_bootstrap.cc:492] T a01ee128ba8a4c9fb8e125ba97239d4c P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:24.877843 18987 ts_tablet_manager.cc:1875] T 44699c84ebac49ef8f3765d4069e71ca P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:24.878441 18985 tablet_bootstrap.cc:654] T a01ee128ba8a4c9fb8e125ba97239d4c P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:24.879413 18149 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_40d60ca37bac4d53a9297289f18c1719" } new_table_name: "default.Table_7974c4f995c047469ccc69cc0d0521f3" modify_external_catalogs: true
I20240430 20:11:24.884841 18987 ts_tablet_manager.cc:1888] T 44699c84ebac49ef8f3765d4069e71ca P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:24.885063 18987 log.cc:1196] T 44699c84ebac49ef8f3765d4069e71ca P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/44699c84ebac49ef8f3765d4069e71ca
I20240430 20:11:24.885946 18987 ts_tablet_manager.cc:1909] T 44699c84ebac49ef8f3765d4069e71ca P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:24.888465 18141 catalog_manager.cc:4830] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): tablet 44699c84ebac49ef8f3765d4069e71ca (table default.Table_0b4b99c1c03a4e779046063159ce6861 [id=43d7bdc84bab4c54bac58dd1dd0f74ac]) successfully deleted
I20240430 20:11:24.888926 18529 heartbeater.cc:347] Connected to a master server at 127.17.177.60:37225
I20240430 20:11:24.889128 18529 heartbeater.cc:503] Master 127.17.177.60:37225 was elected leader, sending a full tablet report...
I20240430 20:11:24.893200 18983 tablet_bootstrap.cc:492] T f15f5e1f3afd4224bbf39567cae74617 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:24.893497 18983 ts_tablet_manager.cc:1367] T f15f5e1f3afd4224bbf39567cae74617 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.474s user 0.033s sys 0.093s
I20240430 20:11:24.894634 18983 raft_consensus.cc:348] T f15f5e1f3afd4224bbf39567cae74617 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:24.895045 18983 raft_consensus.cc:374] T f15f5e1f3afd4224bbf39567cae74617 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:24.895187 18983 raft_consensus.cc:729] T f15f5e1f3afd4224bbf39567cae74617 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:24.895558 18983 consensus_queue.cc:260] T f15f5e1f3afd4224bbf39567cae74617 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:24.895699 18139 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet 9750608faeab4be38ddce4e2e4d46964 (table default.Table_c06b54647f004ff7a7f7beafee702b4c [id=a6a27dcbd0e648a8bf29a8cd5b3a2a11]) successfully deleted
I20240430 20:11:24.895989 18139 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet c3f8f9ada98b40bc8359a00e193c3e5f (table default.Table_c21d1f6f0bfb4f9ea6978a36db7a3701 [id=70551d80c9d8445fa158eee362758714]) successfully deleted
I20240430 20:11:24.896081 18669 heartbeater.cc:503] Master 127.17.177.60:37225 was elected leader, sending a full tablet report...
W20240430 20:11:24.896235 18139 catalog_manager.cc:4810] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): delete failed for tablet a01ee128ba8a4c9fb8e125ba97239d4c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a01ee128ba8a4c9fb8e125ba97239d4c already in progress: creating tablet
W20240430 20:11:24.896459 18139 catalog_manager.cc:4810] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): delete failed for tablet bd7392f3dfd643cdb1634ee10ad41651 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bd7392f3dfd643cdb1634ee10ad41651 already in progress: creating tablet
W20240430 20:11:24.896657 18139 catalog_manager.cc:4810] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): delete failed for tablet 0fe1247349344863a6273a769ce8ddf6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0fe1247349344863a6273a769ce8ddf6 already in progress: creating tablet
W20240430 20:11:24.896862 18139 catalog_manager.cc:4810] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): delete failed for tablet b4bc111e183f40a0acc9dd377195f6d9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b4bc111e183f40a0acc9dd377195f6d9 already in progress: creating tablet
I20240430 20:11:24.900370 18981 raft_consensus.cc:2229] T 44699c84ebac49ef8f3765d4069e71ca P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:24.900756 18601 tablet_service.cc:1514] Processing DeleteTablet for tablet 44699c84ebac49ef8f3765d4069e71ca with delete_type TABLET_DATA_DELETED (replaced by tablet 34ab40d8d91e401db299ddee7175ba28) from {username='slave'} at 127.0.0.1:40202
I20240430 20:11:24.900863 18981 raft_consensus.cc:2258] T 44699c84ebac49ef8f3765d4069e71ca P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:24.901096 18980 tablet_replica.cc:330] T 44699c84ebac49ef8f3765d4069e71ca P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:24.901342 18980 raft_consensus.cc:2229] T 44699c84ebac49ef8f3765d4069e71ca P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:24.901571 18980 raft_consensus.cc:2258] T 44699c84ebac49ef8f3765d4069e71ca P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240430 20:11:24.901909 18988 thread.cc:636] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.523s user 0.097s sys 0.149s
W20240430 20:11:24.902015 18988 thread.cc:602] log-alloc [worker] (thread pool) Time spent starting thread: real 0.523s user 0.097s sys 0.149s
I20240430 20:11:24.902756 18980 ts_tablet_manager.cc:1875] T 44699c84ebac49ef8f3765d4069e71ca P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:24.902984 18988 tablet_bootstrap.cc:492] T 9750608faeab4be38ddce4e2e4d46964 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:24.903129 18988 ts_tablet_manager.cc:1367] T 9750608faeab4be38ddce4e2e4d46964 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.539s user 0.098s sys 0.152s
I20240430 20:11:24.905347 18981 ts_tablet_manager.cc:1875] T 44699c84ebac49ef8f3765d4069e71ca P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:24.909000 18147 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_156650e4b399419c97dcf5160b761ee1" } modify_external_catalogs: true
I20240430 20:11:24.909233 18147 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_156650e4b399419c97dcf5160b761ee1" } modify_external_catalogs: true
I20240430 20:11:24.909835 18151 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54524:
name: "default.Table_3ad9d7307b80412f87283e58ffb6f290"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:24.910436 18151 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3ad9d7307b80412f87283e58ffb6f290 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:24.910542 18980 ts_tablet_manager.cc:1888] T 44699c84ebac49ef8f3765d4069e71ca P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:24.910534 18988 raft_consensus.cc:348] T 9750608faeab4be38ddce4e2e4d46964 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:24.910773 18988 raft_consensus.cc:374] T 9750608faeab4be38ddce4e2e4d46964 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:24.910971 18988 raft_consensus.cc:729] T 9750608faeab4be38ddce4e2e4d46964 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:24.910709 18980 log.cc:1196] T 44699c84ebac49ef8f3765d4069e71ca P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/44699c84ebac49ef8f3765d4069e71ca
I20240430 20:11:24.911968 18980 ts_tablet_manager.cc:1909] T 44699c84ebac49ef8f3765d4069e71ca P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:24.912194 18529 heartbeater.cc:503] Master 127.17.177.60:37225 was elected leader, sending a full tablet report...
I20240430 20:11:24.912583 18988 consensus_queue.cc:260] T 9750608faeab4be38ddce4e2e4d46964 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:24.913314 18988 ts_tablet_manager.cc:1398] T 9750608faeab4be38ddce4e2e4d46964 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240430 20:11:24.913625 18988 tablet_bootstrap.cc:492] T e6c0c808bfc94be3837ab835292b3c1b P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:24.915707 18983 ts_tablet_manager.cc:1398] T f15f5e1f3afd4224bbf39567cae74617 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.022s user 0.002s sys 0.000s
I20240430 20:11:24.916033 18669 heartbeater.cc:503] Master 127.17.177.60:37225 was elected leader, sending a full tablet report...
I20240430 20:11:24.921223 18983 tablet_bootstrap.cc:492] T 81328e904ca94e2eb5ec94dd3ffb3c90 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:24.921584 18981 ts_tablet_manager.cc:1888] T 44699c84ebac49ef8f3765d4069e71ca P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:24.921806 18981 log.cc:1196] T 44699c84ebac49ef8f3765d4069e71ca P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/44699c84ebac49ef8f3765d4069e71ca
I20240430 20:11:24.922315 18981 ts_tablet_manager.cc:1909] T 44699c84ebac49ef8f3765d4069e71ca P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:24.923058 18983 tablet_bootstrap.cc:654] T 81328e904ca94e2eb5ec94dd3ffb3c90 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:24.923441 18142 catalog_manager.cc:4830] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): tablet 44699c84ebac49ef8f3765d4069e71ca (table default.Table_0b4b99c1c03a4e779046063159ce6861 [id=43d7bdc84bab4c54bac58dd1dd0f74ac]) successfully deleted
I20240430 20:11:24.923816 18139 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet 44699c84ebac49ef8f3765d4069e71ca (table default.Table_0b4b99c1c03a4e779046063159ce6861 [id=43d7bdc84bab4c54bac58dd1dd0f74ac]) successfully deleted
I20240430 20:11:24.924028 18988 tablet_bootstrap.cc:654] T e6c0c808bfc94be3837ab835292b3c1b P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
W20240430 20:11:24.932586 18155 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 372b135dd01f44f98cd713ca8857ba61 consensus state has no leader
I20240430 20:11:24.933168 19140 raft_consensus.cc:729] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: b632d6ef4c104a058a6db208024cab33, State: Initialized, Role: FOLLOWER
I20240430 20:11:24.934450 19140 consensus_queue.cc:260] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 30, Last appended: 3.35, Last appended by leader: 35, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b3f98793354c7aaa5c106e8ec31ed8" member_type: VOTER last_known_addr { host: "127.17.177.62" port: 36441 } } peers { permanent_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e" member_type: VOTER last_known_addr { host: "127.17.177.61" port: 34777 } } peers { permanent_uuid: "b632d6ef4c104a058a6db208024cab33" member_type: VOTER last_known_addr { host: "127.17.177.60" port: 37225 } }
I20240430 20:11:24.937932 18983 tablet_bootstrap.cc:492] T 81328e904ca94e2eb5ec94dd3ffb3c90 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:24.938161 18983 ts_tablet_manager.cc:1367] T 81328e904ca94e2eb5ec94dd3ffb3c90 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.005s
I20240430 20:11:24.938329 18148 master_service.cc:946] ReplaceTablet: received request to replace tablet 2067cb2cf76d495c92c1f25c176eb6dc from {username='slave'} at 127.0.0.1:54524
I20240430 20:11:24.939069 18983 raft_consensus.cc:348] T 81328e904ca94e2eb5ec94dd3ffb3c90 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:24.939729 18983 raft_consensus.cc:374] T 81328e904ca94e2eb5ec94dd3ffb3c90 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:24.939893 18983 raft_consensus.cc:729] T 81328e904ca94e2eb5ec94dd3ffb3c90 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:24.940207 18983 consensus_queue.cc:260] T 81328e904ca94e2eb5ec94dd3ffb3c90 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:24.940694 18669 heartbeater.cc:503] Master 127.17.177.60:37225 was elected leader, sending a full tablet report...
I20240430 20:11:24.944859 18983 ts_tablet_manager.cc:1398] T 81328e904ca94e2eb5ec94dd3ffb3c90 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240430 20:11:24.944784 18154 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54524:
name: "default.Table_086ca54daa8e48cb915da0ff5c7272b7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240430 20:11:24.945273 18601 tablet_service.cc:1514] Processing DeleteTablet for tablet f15f5e1f3afd4224bbf39567cae74617 with delete_type TABLET_DATA_DELETED (replaced by tablet 6e826459c3f4437da81c9e6e09f0c22f) from {username='slave'} at 127.0.0.1:40202
W20240430 20:11:24.945470 18154 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_086ca54daa8e48cb915da0ff5c7272b7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:24.947494 18980 tablet_replica.cc:330] T f15f5e1f3afd4224bbf39567cae74617 P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:24.947695 18980 raft_consensus.cc:2229] T f15f5e1f3afd4224bbf39567cae74617 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:24.947863 18980 raft_consensus.cc:2258] T f15f5e1f3afd4224bbf39567cae74617 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:24.948238 18983 tablet_bootstrap.cc:492] T 59548d560b4e4f7ca034d730aca79364 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:24.948678 18980 ts_tablet_manager.cc:1875] T f15f5e1f3afd4224bbf39567cae74617 P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:24.949970 18983 tablet_bootstrap.cc:654] T 59548d560b4e4f7ca034d730aca79364 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:24.951089 19154 sys_catalog.cc:455] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b3f98793354c7aaa5c106e8ec31ed8" member_type: VOTER last_known_addr { host: "127.17.177.62" port: 36441 } } peers { permanent_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e" member_type: VOTER last_known_addr { host: "127.17.177.61" port: 34777 } } peers { permanent_uuid: "b632d6ef4c104a058a6db208024cab33" member_type: VOTER last_known_addr { host: "127.17.177.60" port: 37225 } } }
I20240430 20:11:24.951439 19154 sys_catalog.cc:458] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33 [sys.catalog]: This master's current role is: FOLLOWER
I20240430 20:11:24.951685 18669 heartbeater.cc:503] Master 127.17.177.60:37225 was elected leader, sending a full tablet report...
I20240430 20:11:24.954782 18988 tablet_bootstrap.cc:492] T e6c0c808bfc94be3837ab835292b3c1b P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:24.954977 18988 ts_tablet_manager.cc:1367] T e6c0c808bfc94be3837ab835292b3c1b P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.041s user 0.007s sys 0.001s
I20240430 20:11:24.955789 18988 raft_consensus.cc:348] T e6c0c808bfc94be3837ab835292b3c1b P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:24.956043 18988 raft_consensus.cc:374] T e6c0c808bfc94be3837ab835292b3c1b P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:24.956113 18988 raft_consensus.cc:729] T e6c0c808bfc94be3837ab835292b3c1b P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:24.956596 18988 consensus_queue.cc:260] T e6c0c808bfc94be3837ab835292b3c1b P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:24.957314 18988 ts_tablet_manager.cc:1398] T e6c0c808bfc94be3837ab835292b3c1b P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240430 20:11:24.959074 18988 tablet_bootstrap.cc:492] T f15f5e1f3afd4224bbf39567cae74617 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:24.961823 18988 tablet_bootstrap.cc:654] T f15f5e1f3afd4224bbf39567cae74617 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:24.961989 18980 ts_tablet_manager.cc:1888] T f15f5e1f3afd4224bbf39567cae74617 P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:24.962214 19077 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33 [term 3 FOLLOWER]: Advancing to term 4
I20240430 20:11:24.966768 18980 log.cc:1196] T f15f5e1f3afd4224bbf39567cae74617 P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/f15f5e1f3afd4224bbf39567cae74617
I20240430 20:11:24.967249 18980 ts_tablet_manager.cc:1909] T f15f5e1f3afd4224bbf39567cae74617 P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:24.967527 19140 sys_catalog.cc:564] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33 [sys.catalog]: configured and running, proceeding with master startup.
I20240430 20:11:24.968494 18156 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet fe4636497588415fbf7a626f79cebcc9
I20240430 20:11:24.973389 18139 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet f15f5e1f3afd4224bbf39567cae74617 (table default.Table_873c21f6dffd4d47a742b3699ab85268 [id=d7e2c4300ff94fa2acc65d6d6c932c65]) successfully deleted
I20240430 20:11:24.974025 18988 tablet_bootstrap.cc:492] T f15f5e1f3afd4224bbf39567cae74617 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:24.974201 18988 ts_tablet_manager.cc:1367] T f15f5e1f3afd4224bbf39567cae74617 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.003s
I20240430 20:11:24.974334 19077 pending_rounds.cc:76] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: Aborting all ops after (but not including) 34
I20240430 20:11:24.974478 19077 pending_rounds.cc:97] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: Aborting uncommitted WRITE_OP operation due to leader change: 3.35
I20240430 20:11:24.975083 18988 raft_consensus.cc:348] T f15f5e1f3afd4224bbf39567cae74617 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:24.975364 18988 raft_consensus.cc:374] T f15f5e1f3afd4224bbf39567cae74617 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:24.975494 18988 raft_consensus.cc:729] T f15f5e1f3afd4224bbf39567cae74617 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:24.975769 18988 consensus_queue.cc:260] T f15f5e1f3afd4224bbf39567cae74617 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:24.976577 18988 ts_tablet_manager.cc:1398] T f15f5e1f3afd4224bbf39567cae74617 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240430 20:11:24.976967 18988 tablet_bootstrap.cc:492] T 81328e904ca94e2eb5ec94dd3ffb3c90 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:24.978870 18810 heartbeater.cc:347] Connected to a master server at 127.17.177.60:37225
I20240430 20:11:24.979110 18810 heartbeater.cc:503] Master 127.17.177.60:37225 was elected leader, sending a full tablet report...
I20240430 20:11:24.980160 18988 tablet_bootstrap.cc:654] T 81328e904ca94e2eb5ec94dd3ffb3c90 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:24.981855 18983 tablet_bootstrap.cc:492] T 59548d560b4e4f7ca034d730aca79364 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:24.982098 18983 ts_tablet_manager.cc:1367] T 59548d560b4e4f7ca034d730aca79364 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.034s user 0.008s sys 0.002s
I20240430 20:11:24.982890 18983 raft_consensus.cc:348] T 59548d560b4e4f7ca034d730aca79364 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:24.983256 18983 raft_consensus.cc:374] T 59548d560b4e4f7ca034d730aca79364 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:24.983388 18983 raft_consensus.cc:729] T 59548d560b4e4f7ca034d730aca79364 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:24.983776 18983 consensus_queue.cc:260] T 59548d560b4e4f7ca034d730aca79364 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:24.984288 18669 heartbeater.cc:503] Master 127.17.177.60:37225 was elected leader, sending a full tablet report...
W20240430 20:11:24.986703 18152 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4e97f7315f8b4ac29db16e00fc35bf99 consensus state has no leader
W20240430 20:11:24.987517 18149 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 15674aeb56b74210a4a2809608fd4457 consensus state has no leader
I20240430 20:11:24.991230 18810 heartbeater.cc:503] Master 127.17.177.60:37225 was elected leader, sending a full tablet report...
I20240430 20:11:24.998973 18983 ts_tablet_manager.cc:1398] T 59548d560b4e4f7ca034d730aca79364 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.017s user 0.002s sys 0.000s
I20240430 20:11:25.000166 18669 heartbeater.cc:347] Connected to a master server at 127.17.177.60:37225
I20240430 20:11:25.000341 18669 heartbeater.cc:503] Master 127.17.177.60:37225 was elected leader, sending a full tablet report...
I20240430 20:11:25.001725 18983 tablet_bootstrap.cc:492] T 0d6a4a63c004480b82d84381867bf8a4 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:25.004032 18983 tablet_bootstrap.cc:654] T 0d6a4a63c004480b82d84381867bf8a4 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:25.004513 18985 tablet_bootstrap.cc:492] T a01ee128ba8a4c9fb8e125ba97239d4c P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:25.006624 18985 ts_tablet_manager.cc:1367] T a01ee128ba8a4c9fb8e125ba97239d4c P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.130s user 0.000s sys 0.004s
I20240430 20:11:25.009084 18985 raft_consensus.cc:348] T a01ee128ba8a4c9fb8e125ba97239d4c P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:25.009946 18985 raft_consensus.cc:374] T a01ee128ba8a4c9fb8e125ba97239d4c P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:25.010557 18985 raft_consensus.cc:729] T a01ee128ba8a4c9fb8e125ba97239d4c P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:25.011303 18985 consensus_queue.cc:260] T a01ee128ba8a4c9fb8e125ba97239d4c P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:25.016108 18529 heartbeater.cc:503] Master 127.17.177.60:37225 was elected leader, sending a full tablet report...
I20240430 20:11:25.019870 18988 tablet_bootstrap.cc:492] T 81328e904ca94e2eb5ec94dd3ffb3c90 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:25.020745 18988 ts_tablet_manager.cc:1367] T 81328e904ca94e2eb5ec94dd3ffb3c90 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.043s user 0.002s sys 0.001s
I20240430 20:11:25.021716 18988 raft_consensus.cc:348] T 81328e904ca94e2eb5ec94dd3ffb3c90 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:25.021934 18988 raft_consensus.cc:374] T 81328e904ca94e2eb5ec94dd3ffb3c90 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:25.021988 18988 raft_consensus.cc:729] T 81328e904ca94e2eb5ec94dd3ffb3c90 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:25.022264 18988 consensus_queue.cc:260] T 81328e904ca94e2eb5ec94dd3ffb3c90 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:25.024421 18983 tablet_bootstrap.cc:492] T 0d6a4a63c004480b82d84381867bf8a4 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:25.025085 18988 ts_tablet_manager.cc:1398] T 81328e904ca94e2eb5ec94dd3ffb3c90 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240430 20:11:25.025316 18983 ts_tablet_manager.cc:1367] T 0d6a4a63c004480b82d84381867bf8a4 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.024s user 0.004s sys 0.000s
I20240430 20:11:25.026571 18529 heartbeater.cc:503] Master 127.17.177.60:37225 was elected leader, sending a full tablet report...
I20240430 20:11:25.027817 18983 raft_consensus.cc:348] T 0d6a4a63c004480b82d84381867bf8a4 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:25.028391 18988 tablet_bootstrap.cc:492] T 59548d560b4e4f7ca034d730aca79364 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:25.031850 18983 raft_consensus.cc:374] T 0d6a4a63c004480b82d84381867bf8a4 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:25.035517 18810 heartbeater.cc:503] Master 127.17.177.60:37225 was elected leader, sending a full tablet report...
I20240430 20:11:25.038892 18985 ts_tablet_manager.cc:1398] T a01ee128ba8a4c9fb8e125ba97239d4c P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.032s user 0.001s sys 0.001s
I20240430 20:11:25.039381 18988 tablet_bootstrap.cc:654] T 59548d560b4e4f7ca034d730aca79364 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:25.039578 18983 raft_consensus.cc:729] T 0d6a4a63c004480b82d84381867bf8a4 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:25.039762 18529 heartbeater.cc:503] Master 127.17.177.60:37225 was elected leader, sending a full tablet report...
I20240430 20:11:25.039952 18983 consensus_queue.cc:260] T 0d6a4a63c004480b82d84381867bf8a4 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:25.040552 18669 heartbeater.cc:503] Master 127.17.177.60:37225 was elected leader, sending a full tablet report...
I20240430 20:11:25.046289 18983 ts_tablet_manager.cc:1398] T 0d6a4a63c004480b82d84381867bf8a4 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.019s user 0.002s sys 0.001s
I20240430 20:11:25.046603 18669 heartbeater.cc:503] Master 127.17.177.60:37225 was elected leader, sending a full tablet report...
I20240430 20:11:25.046967 18985 tablet_bootstrap.cc:492] T b0e6cc37c6b1488fa88359d16b12a576 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:25.048681 18983 tablet_bootstrap.cc:492] T a01ee128ba8a4c9fb8e125ba97239d4c P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:25.049224 18985 tablet_bootstrap.cc:654] T b0e6cc37c6b1488fa88359d16b12a576 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:25.052147 18983 tablet_bootstrap.cc:654] T a01ee128ba8a4c9fb8e125ba97239d4c P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:25.054689 18149 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54524:
name: "default.Table_ddd446dad29647018f226059e9933e56"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:25.055387 18149 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ddd446dad29647018f226059e9933e56 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:25.056104 18988 tablet_bootstrap.cc:492] T 59548d560b4e4f7ca034d730aca79364 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:25.056294 18988 ts_tablet_manager.cc:1367] T 59548d560b4e4f7ca034d730aca79364 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.028s user 0.001s sys 0.003s
I20240430 20:11:25.057099 18988 raft_consensus.cc:348] T 59548d560b4e4f7ca034d730aca79364 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:25.057312 18988 raft_consensus.cc:374] T 59548d560b4e4f7ca034d730aca79364 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:25.057384 18988 raft_consensus.cc:729] T 59548d560b4e4f7ca034d730aca79364 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:25.057633 18988 consensus_queue.cc:260] T 59548d560b4e4f7ca034d730aca79364 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:25.057598 18151 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54524:
name: "default.Table_28b63364f56b484eb48763e662dda489"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:25.058204 18151 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_28b63364f56b484eb48763e662dda489 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:25.058280 18988 ts_tablet_manager.cc:1398] T 59548d560b4e4f7ca034d730aca79364 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240430 20:11:25.058554 18988 tablet_bootstrap.cc:492] T c3f8f9ada98b40bc8359a00e193c3e5f P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:25.059827 18152 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54524:
name: "default.Table_6f2fd99337344da2ac5a0c83d9b56bb8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240430 20:11:25.060004 18988 tablet_bootstrap.cc:654] T c3f8f9ada98b40bc8359a00e193c3e5f P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
W20240430 20:11:25.060344 18152 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6f2fd99337344da2ac5a0c83d9b56bb8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:25.072232 18988 tablet_bootstrap.cc:492] T c3f8f9ada98b40bc8359a00e193c3e5f P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:25.072433 18988 ts_tablet_manager.cc:1367] T c3f8f9ada98b40bc8359a00e193c3e5f P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.003s
I20240430 20:11:25.073349 18988 raft_consensus.cc:348] T c3f8f9ada98b40bc8359a00e193c3e5f P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:25.073585 18988 raft_consensus.cc:374] T c3f8f9ada98b40bc8359a00e193c3e5f P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:25.073658 18988 raft_consensus.cc:729] T c3f8f9ada98b40bc8359a00e193c3e5f P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:25.074041 18988 consensus_queue.cc:260] T c3f8f9ada98b40bc8359a00e193c3e5f P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:25.074673 18988 ts_tablet_manager.cc:1398] T c3f8f9ada98b40bc8359a00e193c3e5f P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20240430 20:11:25.074983 18988 tablet_bootstrap.cc:492] T 0d6a4a63c004480b82d84381867bf8a4 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:25.076610 18988 tablet_bootstrap.cc:654] T 0d6a4a63c004480b82d84381867bf8a4 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:25.103399 18742 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d3117a9a5f243b59a69d960c0a1eee4 with delete_type TABLET_DATA_DELETED (replaced by tablet e0d4cebfbfca45c08d1e7500228835b4) from {username='slave'} at 127.0.0.1:59818
I20240430 20:11:25.103760 18738 tablet_service.cc:1514] Processing DeleteTablet for tablet 2067cb2cf76d495c92c1f25c176eb6dc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:23 UTC) from {username='slave'} at 127.0.0.1:59818
W20240430 20:11:25.105008 18141 catalog_manager.cc:4797] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): delete failed for tablet 4d3117a9a5f243b59a69d960c0a1eee4 because the tablet was not found. No further retry: Not found: Tablet not found: 4d3117a9a5f243b59a69d960c0a1eee4
W20240430 20:11:25.105300 18141 catalog_manager.cc:4797] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): delete failed for tablet 2067cb2cf76d495c92c1f25c176eb6dc because the tablet was not found. No further retry: Not found: Tablet not found: 2067cb2cf76d495c92c1f25c176eb6dc
I20240430 20:11:25.105429 19042 raft_consensus.cc:482] T 0d6a4a63c004480b82d84381867bf8a4 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:25.105696 19042 raft_consensus.cc:504] T 0d6a4a63c004480b82d84381867bf8a4 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:25.106637 19042 leader_election.cc:288] T 0d6a4a63c004480b82d84381867bf8a4 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:25.108207 18156 master_service.cc:946] ReplaceTablet: received request to replace tablet bd7392f3dfd643cdb1634ee10ad41651 from {username='slave'} at 127.0.0.1:54524
I20240430 20:11:25.108784 18482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d6a4a63c004480b82d84381867bf8a4" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:25.109113 18482 raft_consensus.cc:2454] T 0d6a4a63c004480b82d84381867bf8a4 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 0.
I20240430 20:11:25.109769 18558 leader_election.cc:302] T 0d6a4a63c004480b82d84381867bf8a4 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:25.110328 18763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d6a4a63c004480b82d84381867bf8a4" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
W20240430 20:11:25.111234 18557 leader_election.cc:341] T 0d6a4a63c004480b82d84381867bf8a4 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:25.113669 19042 raft_consensus.cc:2781] T 0d6a4a63c004480b82d84381867bf8a4 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:25.113804 19042 raft_consensus.cc:482] T 0d6a4a63c004480b82d84381867bf8a4 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:25.113926 19042 raft_consensus.cc:3037] T 0d6a4a63c004480b82d84381867bf8a4 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:25.116708 18601 tablet_service.cc:1467] Processing CreateTablet for tablet c5fd0730133c472299c93f4b0830e527 (DEFAULT_TABLE table=default.Table_ea83912975774b6089fd85c252f45ea5 [id=99cbf7b92c7943838fce9f3ba42babde]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:25.117285 18601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5fd0730133c472299c93f4b0830e527. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:25.131340 19042 raft_consensus.cc:504] T 0d6a4a63c004480b82d84381867bf8a4 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:25.136931 19072 consensus_queue.cc:1035] T 26a015250eeb4750a42df93840e8d47f P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:25.142532 18738 tablet_service.cc:1467] Processing CreateTablet for tablet c5fd0730133c472299c93f4b0830e527 (DEFAULT_TABLE table=default.Table_ea83912975774b6089fd85c252f45ea5 [id=99cbf7b92c7943838fce9f3ba42babde]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:25.143218 18738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5fd0730133c472299c93f4b0830e527. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:25.152264 18599 tablet_service.cc:1467] Processing CreateTablet for tablet 93d158de64994f3c9fa45a1250a4f533 (DEFAULT_TABLE table=default.Table_9d39d94262a7481aad7df7868a2aff26 [id=06b16d46a7484d0dbc7971f070d0f37e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:25.152818 18457 tablet_service.cc:1467] Processing CreateTablet for tablet c5fd0730133c472299c93f4b0830e527 (DEFAULT_TABLE table=default.Table_ea83912975774b6089fd85c252f45ea5 [id=99cbf7b92c7943838fce9f3ba42babde]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:25.153008 18599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93d158de64994f3c9fa45a1250a4f533. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:25.153231 18457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5fd0730133c472299c93f4b0830e527. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:25.160682 18482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d6a4a63c004480b82d84381867bf8a4" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:25.160989 18482 raft_consensus.cc:3037] T 0d6a4a63c004480b82d84381867bf8a4 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:25.164914 18482 raft_consensus.cc:2454] T 0d6a4a63c004480b82d84381867bf8a4 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
I20240430 20:11:25.165652 18558 leader_election.cc:302] T 0d6a4a63c004480b82d84381867bf8a4 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:25.175242 18742 tablet_service.cc:1467] Processing CreateTablet for tablet 93d158de64994f3c9fa45a1250a4f533 (DEFAULT_TABLE table=default.Table_9d39d94262a7481aad7df7868a2aff26 [id=06b16d46a7484d0dbc7971f070d0f37e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:25.175627 19030 raft_consensus.cc:482] T 26a015250eeb4750a42df93840e8d47f P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:25.175913 18742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93d158de64994f3c9fa45a1250a4f533. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:25.175827 19030 raft_consensus.cc:504] T 26a015250eeb4750a42df93840e8d47f P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:25.176733 19030 leader_election.cc:288] T 26a015250eeb4750a42df93840e8d47f P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:25.177714 18482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26a015250eeb4750a42df93840e8d47f" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:25.188246 18763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d6a4a63c004480b82d84381867bf8a4" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
W20240430 20:11:25.189101 18557 leader_election.cc:341] T 0d6a4a63c004480b82d84381867bf8a4 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:25.189687 19042 leader_election.cc:288] T 0d6a4a63c004480b82d84381867bf8a4 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Requested vote from peers 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:25.190171 18669 heartbeater.cc:503] Master 127.17.177.60:37225 was elected leader, sending a full tablet report...
I20240430 20:11:25.193915 19042 raft_consensus.cc:2781] T 0d6a4a63c004480b82d84381867bf8a4 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:25.194202 19042 raft_consensus.cc:686] T 0d6a4a63c004480b82d84381867bf8a4 P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Becoming Leader. State: Replica: 5e1c6397f09747f091534faf93542710, State: Running, Role: LEADER
I20240430 20:11:25.194653 19042 consensus_queue.cc:237] T 0d6a4a63c004480b82d84381867bf8a4 P 5e1c6397f09747f091534faf93542710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:25.199054 18457 tablet_service.cc:1467] Processing CreateTablet for tablet 93d158de64994f3c9fa45a1250a4f533 (DEFAULT_TABLE table=default.Table_9d39d94262a7481aad7df7868a2aff26 [id=06b16d46a7484d0dbc7971f070d0f37e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:25.199594 18457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93d158de64994f3c9fa45a1250a4f533. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:25.212667 19154 sys_catalog.cc:455] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88b3f98793354c7aaa5c106e8ec31ed8. Latest consensus state: current_term: 4 leader_uuid: "88b3f98793354c7aaa5c106e8ec31ed8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b3f98793354c7aaa5c106e8ec31ed8" member_type: VOTER last_known_addr { host: "127.17.177.62" port: 36441 } } peers { permanent_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e" member_type: VOTER last_known_addr { host: "127.17.177.61" port: 34777 } } peers { permanent_uuid: "b632d6ef4c104a058a6db208024cab33" member_type: VOTER last_known_addr { host: "127.17.177.60" port: 37225 } } }
I20240430 20:11:25.212975 19154 sys_catalog.cc:458] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33 [sys.catalog]: This master's current role is: FOLLOWER
I20240430 20:11:25.215111 18744 tablet_service.cc:1467] Processing CreateTablet for tablet a745334c12c545c1ba825c7552c60359 (DEFAULT_TABLE table=default.Table_36adeb442f1245aba00549559bb329f3 [id=4ab2e4b038744db8b2cc94c26dbcc35f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:25.218137 18463 tablet_service.cc:1467] Processing CreateTablet for tablet a745334c12c545c1ba825c7552c60359 (DEFAULT_TABLE table=default.Table_36adeb442f1245aba00549559bb329f3 [id=4ab2e4b038744db8b2cc94c26dbcc35f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:25.218667 18463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a745334c12c545c1ba825c7552c60359. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:25.212713 18595 tablet_service.cc:1467] Processing CreateTablet for tablet a745334c12c545c1ba825c7552c60359 (DEFAULT_TABLE table=default.Table_36adeb442f1245aba00549559bb329f3 [id=4ab2e4b038744db8b2cc94c26dbcc35f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:25.221577 18595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a745334c12c545c1ba825c7552c60359. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:25.222728 19179 sys_catalog.cc:455] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "88b3f98793354c7aaa5c106e8ec31ed8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b3f98793354c7aaa5c106e8ec31ed8" member_type: VOTER last_known_addr { host: "127.17.177.62" port: 36441 } } peers { permanent_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e" member_type: VOTER last_known_addr { host: "127.17.177.61" port: 34777 } } peers { permanent_uuid: "b632d6ef4c104a058a6db208024cab33" member_type: VOTER last_known_addr { host: "127.17.177.60" port: 37225 } } }
I20240430 20:11:25.223114 19179 sys_catalog.cc:458] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33 [sys.catalog]: This master's current role is: FOLLOWER
I20240430 20:11:25.229823 19182 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20240430 20:11:25.241565 18744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a745334c12c545c1ba825c7552c60359. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:25.256623 19072 raft_consensus.cc:482] T 59548d560b4e4f7ca034d730aca79364 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:25.256868 19072 raft_consensus.cc:504] T 59548d560b4e4f7ca034d730aca79364 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:25.257876 19072 leader_election.cc:288] T 59548d560b4e4f7ca034d730aca79364 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:25.258837 18763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59548d560b4e4f7ca034d730aca79364" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:25.259161 18763 raft_consensus.cc:2454] T 59548d560b4e4f7ca034d730aca79364 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 0.
I20240430 20:11:25.259824 18417 leader_election.cc:302] T 59548d560b4e4f7ca034d730aca79364 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters:
I20240430 20:11:25.260854 18623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59548d560b4e4f7ca034d730aca79364" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:25.261164 18623 raft_consensus.cc:2454] T 59548d560b4e4f7ca034d730aca79364 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 0.
I20240430 20:11:25.262194 19072 raft_consensus.cc:2781] T 59548d560b4e4f7ca034d730aca79364 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:25.262337 19072 raft_consensus.cc:482] T 59548d560b4e4f7ca034d730aca79364 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:25.262451 19072 raft_consensus.cc:3037] T 59548d560b4e4f7ca034d730aca79364 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:25.266475 19072 raft_consensus.cc:504] T 59548d560b4e4f7ca034d730aca79364 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:25.274387 18763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59548d560b4e4f7ca034d730aca79364" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:25.274793 18763 raft_consensus.cc:3037] T 59548d560b4e4f7ca034d730aca79364 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:25.279007 19030 raft_consensus.cc:482] T e6c0c808bfc94be3837ab835292b3c1b P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:25.279259 19030 raft_consensus.cc:504] T e6c0c808bfc94be3837ab835292b3c1b P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:25.280225 19030 leader_election.cc:288] T e6c0c808bfc94be3837ab835292b3c1b P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:25.281617 18482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6c0c808bfc94be3837ab835292b3c1b" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:25.282011 18482 raft_consensus.cc:2379] T e6c0c808bfc94be3837ab835292b3c1b P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c51c33d5616482cafb68e26688820f4 in current term 1: Already voted for candidate 5e1c6397f09747f091534faf93542710 in this term.
I20240430 20:11:25.282454 18763 raft_consensus.cc:2454] T 59548d560b4e4f7ca034d730aca79364 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 1.
I20240430 20:11:25.283432 18417 leader_election.cc:302] T 59548d560b4e4f7ca034d730aca79364 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters:
I20240430 20:11:25.284122 19185 raft_consensus.cc:2781] T 59548d560b4e4f7ca034d730aca79364 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:25.288347 18953 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "88b3f98793354c7aaa5c106e8ec31ed8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b3f98793354c7aaa5c106e8ec31ed8" member_type: VOTER last_known_addr { host: "127.17.177.62" port: 36441 } } peers { permanent_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e" member_type: VOTER last_known_addr { host: "127.17.177.61" port: 34777 } } peers { permanent_uuid: "b632d6ef4c104a058a6db208024cab33" member_type: VOTER last_known_addr { host: "127.17.177.60" port: 37225 } } }
I20240430 20:11:25.288511 18154 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54524:
name: "default.Table_172b295132814722b733e25f446730a7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240430 20:11:25.288708 18953 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8 [sys.catalog]: This master's current role is: LEADER
W20240430 20:11:25.289161 18154 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_172b295132814722b733e25f446730a7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:25.290099 18146 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54524:
name: "default.Table_f61a45518e134c4496e4ac81fbe40fce"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:25.290630 18146 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f61a45518e134c4496e4ac81fbe40fce in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:25.291852 18148 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_173d3907261b4390aa5647f01794d901" } modify_external_catalogs: true
I20240430 20:11:25.292100 18148 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_173d3907261b4390aa5647f01794d901" } modify_external_catalogs: true
I20240430 20:11:25.295352 18156 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_ea83912975774b6089fd85c252f45ea5" } modify_external_catalogs: true
I20240430 20:11:25.295723 18156 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_ea83912975774b6089fd85c252f45ea5" } modify_external_catalogs: true
I20240430 20:11:25.298633 18152 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_9d39d94262a7481aad7df7868a2aff26" } modify_external_catalogs: true
I20240430 20:11:25.298908 18152 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_9d39d94262a7481aad7df7868a2aff26" } modify_external_catalogs: true
I20240430 20:11:25.304859 18735 tablet_service.cc:1514] Processing DeleteTablet for tablet f15f5e1f3afd4224bbf39567cae74617 with delete_type TABLET_DATA_DELETED (replaced by tablet 6e826459c3f4437da81c9e6e09f0c22f) from {username='slave'} at 127.0.0.1:59818
I20240430 20:11:25.305169 18987 tablet_replica.cc:330] T f15f5e1f3afd4224bbf39567cae74617 P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:25.305378 18987 raft_consensus.cc:2229] T f15f5e1f3afd4224bbf39567cae74617 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:25.305567 18987 raft_consensus.cc:2258] T f15f5e1f3afd4224bbf39567cae74617 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:25.306545 18987 ts_tablet_manager.cc:1875] T f15f5e1f3afd4224bbf39567cae74617 P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:25.306623 18735 tablet_service.cc:1514] Processing DeleteTablet for tablet 9750608faeab4be38ddce4e2e4d46964 with delete_type TABLET_DATA_DELETED (replaced by tablet 0fb0c0ca09154f7a809dd523a859929a) from {username='slave'} at 127.0.0.1:59818
I20240430 20:11:25.307415 18735 tablet_service.cc:1514] Processing DeleteTablet for tablet c3f8f9ada98b40bc8359a00e193c3e5f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:23 UTC) from {username='slave'} at 127.0.0.1:59818
I20240430 20:11:25.308351 18735 tablet_service.cc:1514] Processing DeleteTablet for tablet 59548d560b4e4f7ca034d730aca79364 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:24 UTC) from {username='slave'} at 127.0.0.1:59818
I20240430 20:11:25.315093 18987 ts_tablet_manager.cc:1888] T f15f5e1f3afd4224bbf39567cae74617 P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:25.315373 18987 log.cc:1196] T f15f5e1f3afd4224bbf39567cae74617 P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/f15f5e1f3afd4224bbf39567cae74617
I20240430 20:11:25.315879 18987 ts_tablet_manager.cc:1909] T f15f5e1f3afd4224bbf39567cae74617 P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:25.317129 18987 tablet_replica.cc:330] T 9750608faeab4be38ddce4e2e4d46964 P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:25.317340 18141 catalog_manager.cc:4830] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): tablet f15f5e1f3afd4224bbf39567cae74617 (table default.Table_873c21f6dffd4d47a742b3699ab85268 [id=d7e2c4300ff94fa2acc65d6d6c932c65]) successfully deleted
I20240430 20:11:25.317477 18987 raft_consensus.cc:2229] T 9750608faeab4be38ddce4e2e4d46964 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:25.317692 18987 raft_consensus.cc:2258] T 9750608faeab4be38ddce4e2e4d46964 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:25.323195 18987 ts_tablet_manager.cc:1875] T 9750608faeab4be38ddce4e2e4d46964 P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:25.323376 18735 tablet_service.cc:1467] Processing CreateTablet for tablet 6e826459c3f4437da81c9e6e09f0c22f (DEFAULT_TABLE table=default.Table_873c21f6dffd4d47a742b3699ab85268 [id=d7e2c4300ff94fa2acc65d6d6c932c65]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:25.323858 18735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e826459c3f4437da81c9e6e09f0c22f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:25.330278 18150 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: Sending DeleteTablet for 3 replicas of tablet 26a015250eeb4750a42df93840e8d47f
I20240430 20:11:25.330966 18150 catalog_manager.cc:6554] ReplaceTablet: tablet 26a015250eeb4750a42df93840e8d47f deleted and replaced by tablet ec060e847f024f9fa369cff320e71558
I20240430 20:11:25.331213 18150 master_service.cc:946] ReplaceTablet: received request to replace tablet 609a1a7d22e84b69bd0a53577b62b69b from {username='slave'} at 127.0.0.1:54524
I20240430 20:11:25.337877 18155 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_a4d0abf694944132bd0029c817df66fc" } new_table_name: "default.Table_a058c17d17cb4a68a597fe273e1c1dfd" modify_external_catalogs: true
I20240430 20:11:25.341241 18737 tablet_service.cc:1514] Processing DeleteTablet for tablet 26a015250eeb4750a42df93840e8d47f with delete_type TABLET_DATA_DELETED (replaced by tablet ec060e847f024f9fa369cff320e71558) from {username='slave'} at 127.0.0.1:59818
I20240430 20:11:25.345695 18592 tablet_service.cc:1514] Processing DeleteTablet for tablet 59548d560b4e4f7ca034d730aca79364 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:24 UTC) from {username='slave'} at 127.0.0.1:40202
I20240430 20:11:25.346064 18980 tablet_replica.cc:330] T 59548d560b4e4f7ca034d730aca79364 P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:25.346279 18980 raft_consensus.cc:2229] T 59548d560b4e4f7ca034d730aca79364 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:25.346503 18980 raft_consensus.cc:2258] T 59548d560b4e4f7ca034d730aca79364 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:25.346820 18592 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d6a4a63c004480b82d84381867bf8a4 with delete_type TABLET_DATA_DELETED (replaced by tablet c9ff972ade694938888514979e1638aa) from {username='slave'} at 127.0.0.1:40202
I20240430 20:11:25.348181 18980 ts_tablet_manager.cc:1875] T 59548d560b4e4f7ca034d730aca79364 P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:25.354049 18149 master_service.cc:946] ReplaceTablet: received request to replace tablet b4bc111e183f40a0acc9dd377195f6d9 from {username='slave'} at 127.0.0.1:54524
I20240430 20:11:25.354208 18980 ts_tablet_manager.cc:1888] T 59548d560b4e4f7ca034d730aca79364 P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:25.354342 18980 log.cc:1196] T 59548d560b4e4f7ca034d730aca79364 P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/59548d560b4e4f7ca034d730aca79364
I20240430 20:11:25.354836 18980 ts_tablet_manager.cc:1909] T 59548d560b4e4f7ca034d730aca79364 P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:25.355180 18149 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54524:
name: "default.Table_5924e4520c6c49988ccef21ca04d25d4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240430 20:11:25.355528 18139 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet 59548d560b4e4f7ca034d730aca79364 (table default.Table_f6d2f14f89184a32a523563bdc6ba478 [id=f31f8f1f8e164a9d946c46164ba12f44]) successfully deleted
I20240430 20:11:25.355886 18980 tablet_replica.cc:330] T 0d6a4a63c004480b82d84381867bf8a4 P 5e1c6397f09747f091534faf93542710: stopping tablet replica
W20240430 20:11:25.356067 18149 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5924e4520c6c49988ccef21ca04d25d4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:25.357548 18599 tablet_service.cc:1514] Processing DeleteTablet for tablet 26a015250eeb4750a42df93840e8d47f with delete_type TABLET_DATA_DELETED (replaced by tablet ec060e847f024f9fa369cff320e71558) from {username='slave'} at 127.0.0.1:40202
W20240430 20:11:25.363513 18415 leader_election.cc:341] T 59548d560b4e4f7ca034d730aca79364 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Not found: Tablet not found: 59548d560b4e4f7ca034d730aca79364
I20240430 20:11:25.363843 19072 leader_election.cc:288] T 59548d560b4e4f7ca034d730aca79364 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:25.365267 18151 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_c2b7ceaf18b24ff3b42e4fa585fcdb10" } new_table_name: "default.Table_b74297eb78cc4912a25657bbf0180340" modify_external_catalogs: true
I20240430 20:11:25.368538 18987 ts_tablet_manager.cc:1888] T 9750608faeab4be38ddce4e2e4d46964 P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:25.374179 18595 tablet_service.cc:1467] Processing CreateTablet for tablet 6e826459c3f4437da81c9e6e09f0c22f (DEFAULT_TABLE table=default.Table_873c21f6dffd4d47a742b3699ab85268 [id=d7e2c4300ff94fa2acc65d6d6c932c65]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:25.374912 18595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e826459c3f4437da81c9e6e09f0c22f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:25.378876 18147 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: Sending DeleteTablet for 3 replicas of tablet 27f94f2d8f50495dababe417a9f82982
I20240430 20:11:25.379793 18147 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54524:
name: "default.Table_529678d6f1444c9294902598ac7c551e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:25.380479 18147 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_529678d6f1444c9294902598ac7c551e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:25.383649 18599 tablet_service.cc:1514] Processing DeleteTablet for tablet 27f94f2d8f50495dababe417a9f82982 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:24 UTC) from {username='slave'} at 127.0.0.1:40202
I20240430 20:11:25.375959 18735 tablet_service.cc:1467] Processing CreateTablet for tablet aec9c09cc0454acab6006bb3e10982b7 (DEFAULT_TABLE table=default.Table_085ba776e84d4d66bc187679878700cf [id=26d4860fb33047b7b812ebd4309f2e28]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:25.384455 18735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aec9c09cc0454acab6006bb3e10982b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:25.375206 18601 tablet_service.cc:1467] Processing CreateTablet for tablet aec9c09cc0454acab6006bb3e10982b7 (DEFAULT_TABLE table=default.Table_085ba776e84d4d66bc187679878700cf [id=26d4860fb33047b7b812ebd4309f2e28]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:25.385491 18601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aec9c09cc0454acab6006bb3e10982b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:25.390336 18738 tablet_service.cc:1514] Processing DeleteTablet for tablet 27f94f2d8f50495dababe417a9f82982 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:24 UTC) from {username='slave'} at 127.0.0.1:59818
I20240430 20:11:25.391005 18987 log.cc:1196] T 9750608faeab4be38ddce4e2e4d46964 P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/9750608faeab4be38ddce4e2e4d46964
I20240430 20:11:25.393493 18987 ts_tablet_manager.cc:1909] T 9750608faeab4be38ddce4e2e4d46964 P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:25.396265 18987 tablet_replica.cc:330] T c3f8f9ada98b40bc8359a00e193c3e5f P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:25.396389 18141 catalog_manager.cc:4830] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): tablet 9750608faeab4be38ddce4e2e4d46964 (table default.Table_c06b54647f004ff7a7f7beafee702b4c [id=a6a27dcbd0e648a8bf29a8cd5b3a2a11]) successfully deleted
I20240430 20:11:25.397168 18980 raft_consensus.cc:2229] T 0d6a4a63c004480b82d84381867bf8a4 P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Raft consensus shutting down.
I20240430 20:11:25.397755 18980 pending_rounds.cc:62] T 0d6a4a63c004480b82d84381867bf8a4 P 5e1c6397f09747f091534faf93542710: Trying to abort 1 pending ops.
I20240430 20:11:25.397948 18980 pending_rounds.cc:68] T 0d6a4a63c004480b82d84381867bf8a4 P 5e1c6397f09747f091534faf93542710: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7022624298127032320 op_type: NO_OP noop_request { }
I20240430 20:11:25.398204 18980 raft_consensus.cc:2866] T 0d6a4a63c004480b82d84381867bf8a4 P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240430 20:11:25.398411 18980 raft_consensus.cc:2258] T 0d6a4a63c004480b82d84381867bf8a4 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240430 20:11:25.398993 18558 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.177.1:41217, user_credentials={real_user=slave}} blocked reactor thread for 233384us
I20240430 20:11:25.400946 18623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26a015250eeb4750a42df93840e8d47f" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:25.401919 18623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6c0c808bfc94be3837ab835292b3c1b" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:25.402531 18696 leader_election.cc:302] T 26a015250eeb4750a42df93840e8d47f P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710
I20240430 20:11:25.406107 18696 leader_election.cc:302] T e6c0c808bfc94be3837ab835292b3c1b P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710
I20240430 20:11:25.406399 18463 tablet_service.cc:1514] Processing DeleteTablet for tablet 26a015250eeb4750a42df93840e8d47f with delete_type TABLET_DATA_DELETED (replaced by tablet ec060e847f024f9fa369cff320e71558) from {username='slave'} at 127.0.0.1:41678
I20240430 20:11:25.406786 18981 tablet_replica.cc:330] T 26a015250eeb4750a42df93840e8d47f P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:25.406931 18463 tablet_service.cc:1514] Processing DeleteTablet for tablet 44699c84ebac49ef8f3765d4069e71ca with delete_type TABLET_DATA_DELETED (replaced by tablet 34ab40d8d91e401db299ddee7175ba28) from {username='slave'} at 127.0.0.1:41678
I20240430 20:11:25.407028 18981 raft_consensus.cc:2229] T 26a015250eeb4750a42df93840e8d47f P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Raft consensus shutting down.
I20240430 20:11:25.407387 18981 pending_rounds.cc:62] T 26a015250eeb4750a42df93840e8d47f P 2efb542aa55e439b99ccb98d1c5c884a: Trying to abort 1 pending ops.
I20240430 20:11:25.407464 18463 tablet_service.cc:1514] Processing DeleteTablet for tablet 2067cb2cf76d495c92c1f25c176eb6dc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:23 UTC) from {username='slave'} at 127.0.0.1:41678
I20240430 20:11:25.407471 18981 pending_rounds.cc:68] T 26a015250eeb4750a42df93840e8d47f P 2efb542aa55e439b99ccb98d1c5c884a: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7022624294056210432 op_type: NO_OP noop_request { }
I20240430 20:11:25.407773 18981 raft_consensus.cc:2866] T 26a015250eeb4750a42df93840e8d47f P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240430 20:11:25.407873 18981 raft_consensus.cc:2258] T 26a015250eeb4750a42df93840e8d47f P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:25.403999 18763 raft_consensus.cc:3037] T 26a015250eeb4750a42df93840e8d47f P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:25.408514 18981 ts_tablet_manager.cc:1875] T 26a015250eeb4750a42df93840e8d47f P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:25.412709 18148 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: Sending DeleteTablet for 3 replicas of tablet 187a27a6449342f99de3c5a2f93d49fc
I20240430 20:11:25.413606 19030 raft_consensus.cc:2726] T 26a015250eeb4750a42df93840e8d47f P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:25.414027 19030 raft_consensus.cc:3037] T e6c0c808bfc94be3837ab835292b3c1b P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:25.415478 18156 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: Sending DeleteTablet for 3 replicas of tablet c5fd0730133c472299c93f4b0830e527
I20240430 20:11:25.416193 18592 tablet_service.cc:1514] Processing DeleteTablet for tablet c5fd0730133c472299c93f4b0830e527 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:25 UTC) from {username='slave'} at 127.0.0.1:40202
I20240430 20:11:25.416502 18599 tablet_service.cc:1514] Processing DeleteTablet for tablet 187a27a6449342f99de3c5a2f93d49fc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:25 UTC) from {username='slave'} at 127.0.0.1:40202
I20240430 20:11:25.417455 19030 raft_consensus.cc:2726] T e6c0c808bfc94be3837ab835292b3c1b P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:25.419104 18457 tablet_service.cc:1467] Processing CreateTablet for tablet 6e826459c3f4437da81c9e6e09f0c22f (DEFAULT_TABLE table=default.Table_873c21f6dffd4d47a742b3699ab85268 [id=d7e2c4300ff94fa2acc65d6d6c932c65]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:25.419852 18457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e826459c3f4437da81c9e6e09f0c22f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:25.417551 18981 ts_tablet_manager.cc:1888] T 26a015250eeb4750a42df93840e8d47f P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:25.423964 18981 log.cc:1196] T 26a015250eeb4750a42df93840e8d47f P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/26a015250eeb4750a42df93840e8d47f
I20240430 20:11:25.424690 18981 ts_tablet_manager.cc:1909] T 26a015250eeb4750a42df93840e8d47f P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:25.425737 18981 tablet_replica.cc:330] T 2067cb2cf76d495c92c1f25c176eb6dc P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:25.426071 18981 raft_consensus.cc:2229] T 2067cb2cf76d495c92c1f25c176eb6dc P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:25.426481 18981 raft_consensus.cc:2258] T 2067cb2cf76d495c92c1f25c176eb6dc P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:25.427268 18981 ts_tablet_manager.cc:1875] T 2067cb2cf76d495c92c1f25c176eb6dc P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:25.427596 18987 raft_consensus.cc:2229] T c3f8f9ada98b40bc8359a00e193c3e5f P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:25.429678 19042 consensus_queue.cc:1035] T e6c0c808bfc94be3837ab835292b3c1b P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:25.428412 18744 tablet_service.cc:1514] Processing DeleteTablet for tablet c5fd0730133c472299c93f4b0830e527 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:25 UTC) from {username='slave'} at 127.0.0.1:59818
I20240430 20:11:25.430661 18738 tablet_service.cc:1514] Processing DeleteTablet for tablet 187a27a6449342f99de3c5a2f93d49fc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:25 UTC) from {username='slave'} at 127.0.0.1:59818
I20240430 20:11:25.393661 18810 heartbeater.cc:347] Connected to a master server at 127.17.177.60:37225
I20240430 20:11:25.431073 18810 heartbeater.cc:503] Master 127.17.177.60:37225 was elected leader, sending a full tablet report...
I20240430 20:11:25.431289 18987 raft_consensus.cc:2258] T c3f8f9ada98b40bc8359a00e193c3e5f P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:25.432464 18980 ts_tablet_manager.cc:1875] T 0d6a4a63c004480b82d84381867bf8a4 P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:25.433053 18987 ts_tablet_manager.cc:1875] T c3f8f9ada98b40bc8359a00e193c3e5f P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:25.437678 18463 tablet_service.cc:1514] Processing DeleteTablet for tablet f15f5e1f3afd4224bbf39567cae74617 with delete_type TABLET_DATA_DELETED (replaced by tablet 6e826459c3f4437da81c9e6e09f0c22f) from {username='slave'} at 127.0.0.1:41678
I20240430 20:11:25.438624 18463 tablet_service.cc:1514] Processing DeleteTablet for tablet c3f8f9ada98b40bc8359a00e193c3e5f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:23 UTC) from {username='slave'} at 127.0.0.1:41678
I20240430 20:11:25.439148 18463 tablet_service.cc:1514] Processing DeleteTablet for tablet f79497bf22ed417d988530f6e69cc07f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:23 UTC) from {username='slave'} at 127.0.0.1:41678
I20240430 20:11:25.439643 18463 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d6a4a63c004480b82d84381867bf8a4 with delete_type TABLET_DATA_DELETED (replaced by tablet c9ff972ade694938888514979e1638aa) from {username='slave'} at 127.0.0.1:41678
I20240430 20:11:25.440153 18463 tablet_service.cc:1514] Processing DeleteTablet for tablet 9750608faeab4be38ddce4e2e4d46964 with delete_type TABLET_DATA_DELETED (replaced by tablet 0fb0c0ca09154f7a809dd523a859929a) from {username='slave'} at 127.0.0.1:41678
I20240430 20:11:25.440832 18463 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f729a6e8dc14de9a2057cfe09ba9257 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:23 UTC) from {username='slave'} at 127.0.0.1:41678
I20240430 20:11:25.442219 18463 tablet_service.cc:1467] Processing CreateTablet for tablet aec9c09cc0454acab6006bb3e10982b7 (DEFAULT_TABLE table=default.Table_085ba776e84d4d66bc187679878700cf [id=26d4860fb33047b7b812ebd4309f2e28]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:25.442672 18463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aec9c09cc0454acab6006bb3e10982b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:25.449352 18987 ts_tablet_manager.cc:1888] T c3f8f9ada98b40bc8359a00e193c3e5f P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:25.457361 18987 log.cc:1196] T c3f8f9ada98b40bc8359a00e193c3e5f P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/c3f8f9ada98b40bc8359a00e193c3e5f
I20240430 20:11:25.458341 18987 ts_tablet_manager.cc:1909] T c3f8f9ada98b40bc8359a00e193c3e5f P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:25.460422 18987 tablet_replica.cc:330] T 59548d560b4e4f7ca034d730aca79364 P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:25.460685 18141 catalog_manager.cc:4830] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): tablet c3f8f9ada98b40bc8359a00e193c3e5f (table default.Table_c21d1f6f0bfb4f9ea6978a36db7a3701 [id=70551d80c9d8445fa158eee362758714]) successfully deleted
I20240430 20:11:25.462954 18987 raft_consensus.cc:2229] T 59548d560b4e4f7ca034d730aca79364 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:25.464977 18987 raft_consensus.cc:2258] T 59548d560b4e4f7ca034d730aca79364 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:25.465494 18980 ts_tablet_manager.cc:1888] T 0d6a4a63c004480b82d84381867bf8a4 P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:25.465647 18980 log.cc:1196] T 0d6a4a63c004480b82d84381867bf8a4 P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/0d6a4a63c004480b82d84381867bf8a4
I20240430 20:11:25.466069 18980 ts_tablet_manager.cc:1909] T 0d6a4a63c004480b82d84381867bf8a4 P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:25.466992 18987 ts_tablet_manager.cc:1875] T 59548d560b4e4f7ca034d730aca79364 P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:25.467002 18139 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet 0d6a4a63c004480b82d84381867bf8a4 (table default.Table_3c5df662747b48bf8beb6ebe4d2ea618 [id=663c8efbd7ac4a148d5f7626f1522033]) successfully deleted
I20240430 20:11:25.469269 18463 tablet_service.cc:1514] Processing DeleteTablet for tablet 27f94f2d8f50495dababe417a9f82982 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:24 UTC) from {username='slave'} at 127.0.0.1:41678
I20240430 20:11:25.472831 18529 heartbeater.cc:347] Connected to a master server at 127.17.177.60:37225
I20240430 20:11:25.473120 18529 heartbeater.cc:503] Master 127.17.177.60:37225 was elected leader, sending a full tablet report...
I20240430 20:11:25.475032 18150 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: Sending DeleteTablet for 3 replicas of tablet 609a1a7d22e84b69bd0a53577b62b69b
I20240430 20:11:25.475543 18150 catalog_manager.cc:6554] ReplaceTablet: tablet 609a1a7d22e84b69bd0a53577b62b69b deleted and replaced by tablet d57e59f225114af1b1522be6cb805991
I20240430 20:11:25.476402 18152 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: Sending DeleteTablet for 3 replicas of tablet 93d158de64994f3c9fa45a1250a4f533
I20240430 20:11:25.476918 19051 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "5e1c6397f09747f091534faf93542710" instance_seqno: 1714507880456452) as {username='slave'} at 127.17.177.2:41749; Asking this server to re-register.
I20240430 20:11:25.478073 18592 tablet_service.cc:1514] Processing DeleteTablet for tablet 93d158de64994f3c9fa45a1250a4f533 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:25 UTC) from {username='slave'} at 127.0.0.1:40202
W20240430 20:11:25.478826 18155 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a0e8e07d363f48c698c7e2b17edc1545 consensus state has no leader
I20240430 20:11:25.480954 18669 heartbeater.cc:464] Registering TS with master...
I20240430 20:11:25.481536 18987 ts_tablet_manager.cc:1888] T 59548d560b4e4f7ca034d730aca79364 P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:25.482504 18154 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54524:
name: "default.Table_73ee808b1dda40dbafb28cf5669d53bc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240430 20:11:25.483150 18987 log.cc:1196] T 59548d560b4e4f7ca034d730aca79364 P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/59548d560b4e4f7ca034d730aca79364
W20240430 20:11:25.483144 18154 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_73ee808b1dda40dbafb28cf5669d53bc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:25.483541 18987 ts_tablet_manager.cc:1909] T 59548d560b4e4f7ca034d730aca79364 P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:25.484633 18156 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_086ca54daa8e48cb915da0ff5c7272b7" } new_table_name: "default.Table_88c44b20f2ce4eeca7ed9583a70c7c65" modify_external_catalogs: true
I20240430 20:11:25.484910 19051 ts_manager.cc:194] Registered new tserver with Master: 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:25.486116 18150 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54524:
name: "default.Table_2f1d187ff53442efbe9a5266b89f1267"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240430 20:11:25.486303 18599 tablet_service.cc:1514] Processing DeleteTablet for tablet 609a1a7d22e84b69bd0a53577b62b69b with delete_type TABLET_DATA_DELETED (replaced by tablet d57e59f225114af1b1522be6cb805991) from {username='slave'} at 127.0.0.1:40202
I20240430 20:11:25.486677 18987 tablet_replica.cc:330] T 26a015250eeb4750a42df93840e8d47f P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
W20240430 20:11:25.486737 18150 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2f1d187ff53442efbe9a5266b89f1267 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:25.486884 18987 raft_consensus.cc:2229] T 26a015250eeb4750a42df93840e8d47f P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:25.487056 18987 raft_consensus.cc:2258] T 26a015250eeb4750a42df93840e8d47f P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:25.486738 18148 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_9854f29c4fdd4c4bb14859cbc9c48fb3" } new_table_name: "default.Table_a67165ab67884e57955b604826a006d6" modify_external_catalogs: true
I20240430 20:11:25.487602 18987 ts_tablet_manager.cc:1875] T 26a015250eeb4750a42df93840e8d47f P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:25.488369 18147 rpcz_store.cc:275] Call kudu.master.MasterService.GetMasterRegistration from 127.0.0.1:59916 (request call id 2) took 1426 ms. Trace:
I20240430 20:11:25.488489 18147 rpcz_store.cc:276] 0430 20:11:24.061736 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:25.481786 (+1420050us) service_pool.cc:225] Handling call
0430 20:11:25.488167 (+ 6381us) spinlock_profiling.cc:243] Waited 6.03 ms on lock 0x616000021d54. stack: 00007f00b6ff3756 00007f00b6ff3510 00007f00b602648d 0000000000955804 00000000009557cd 00007f00c5b3215d 00007f00c5b32253 00007f00c2864d41 00007f00ca45034d 00007f00ca5e51bf 00007f00c13d7227 00007f00c13d6f55 00007f00bdcfb04e 00007f00bdcf9f2e 00007f00bdd00405 00007f00bdd00d2f
0430 20:11:25.488288 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":13859200}
I20240430 20:11:25.490095 18146 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_eea9acffa45b4f3ab6d4524dc460853c" } modify_external_catalogs: true
I20240430 20:11:25.490366 18146 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_eea9acffa45b4f3ab6d4524dc460853c" } modify_external_catalogs: true
I20240430 20:11:25.491034 18152 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_4e15b6827cbe4bd3a7c8264b73f6e999" } new_table_name: "default.Table_40e4655ab0b449cb8f9688a222e92813" modify_external_catalogs: true
I20240430 20:11:25.492224 18141 catalog_manager.cc:4830] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): tablet 59548d560b4e4f7ca034d730aca79364 (table default.Table_f6d2f14f89184a32a523563bdc6ba478 [id=f31f8f1f8e164a9d946c46164ba12f44]) successfully deleted
I20240430 20:11:25.492954 18744 tablet_service.cc:1514] Processing DeleteTablet for tablet 93d158de64994f3c9fa45a1250a4f533 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:25 UTC) from {username='slave'} at 127.0.0.1:59818
W20240430 20:11:25.494879 18696 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.177.2:46027, user_credentials={real_user=slave}} blocked reactor thread for 88809.1us
W20240430 20:11:25.500110 18151 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 81787eccca6c42c1b421377a3bf9ebe6 consensus state has no leader
I20240430 20:11:25.501046 18738 tablet_service.cc:1514] Processing DeleteTablet for tablet 609a1a7d22e84b69bd0a53577b62b69b with delete_type TABLET_DATA_DELETED (replaced by tablet d57e59f225114af1b1522be6cb805991) from {username='slave'} at 127.0.0.1:59818
I20240430 20:11:25.501498 18987 ts_tablet_manager.cc:1888] T 26a015250eeb4750a42df93840e8d47f P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:25.504447 19051 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "3c51c33d5616482cafb68e26688820f4" instance_seqno: 1714507880977799) as {username='slave'} at 127.17.177.3:42157; Asking this server to re-register.
I20240430 20:11:25.505520 18987 log.cc:1196] T 26a015250eeb4750a42df93840e8d47f P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/26a015250eeb4750a42df93840e8d47f
I20240430 20:11:25.506201 18987 ts_tablet_manager.cc:1909] T 26a015250eeb4750a42df93840e8d47f P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:25.506615 19042 consensus_queue.cc:1035] T e6c0c808bfc94be3837ab835292b3c1b P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:25.507642 18985 tablet_bootstrap.cc:492] T b0e6cc37c6b1488fa88359d16b12a576 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:25.508430 18985 ts_tablet_manager.cc:1367] T b0e6cc37c6b1488fa88359d16b12a576 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.461s user 0.100s sys 0.096s
I20240430 20:11:25.508786 18141 catalog_manager.cc:4830] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): tablet 26a015250eeb4750a42df93840e8d47f (table default.Table_2e9c7c683df84a41942d31600a265641 [id=aca6fb02776a40bbac312191adbdc136]) successfully deleted
I20240430 20:11:25.509464 18985 raft_consensus.cc:348] T b0e6cc37c6b1488fa88359d16b12a576 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:25.509733 18985 raft_consensus.cc:374] T b0e6cc37c6b1488fa88359d16b12a576 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:25.509826 18985 raft_consensus.cc:729] T b0e6cc37c6b1488fa88359d16b12a576 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
W20240430 20:11:25.509621 18141 catalog_manager.cc:4810] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): delete failed for tablet 27f94f2d8f50495dababe417a9f82982 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 27f94f2d8f50495dababe417a9f82982 already in progress: creating tablet
I20240430 20:11:25.510090 18985 consensus_queue.cc:260] T b0e6cc37c6b1488fa88359d16b12a576 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
W20240430 20:11:25.510984 18141 catalog_manager.cc:4810] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): delete failed for tablet c5fd0730133c472299c93f4b0830e527 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c5fd0730133c472299c93f4b0830e527 already in progress: creating tablet
W20240430 20:11:25.511834 18141 catalog_manager.cc:4810] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): delete failed for tablet 187a27a6449342f99de3c5a2f93d49fc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 187a27a6449342f99de3c5a2f93d49fc already in progress: creating tablet
W20240430 20:11:25.512715 18141 catalog_manager.cc:4810] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): delete failed for tablet 93d158de64994f3c9fa45a1250a4f533 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 93d158de64994f3c9fa45a1250a4f533 already in progress: creating tablet
W20240430 20:11:25.513512 18141 catalog_manager.cc:4810] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): delete failed for tablet 609a1a7d22e84b69bd0a53577b62b69b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 609a1a7d22e84b69bd0a53577b62b69b already in progress: creating tablet
W20240430 20:11:25.514564 18148 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a1aeee1b02214975a376cfd36957afe8 has no consensus state
W20240430 20:11:25.515156 18152 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet aacabead3c874e77a47d2ecf19dbfa82 has no consensus state
W20240430 20:11:25.517064 18156 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bd11c0335a58448da56533c3f2e785a9 has no consensus state
I20240430 20:11:25.521342 18156 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54524:
name: "default.Table_7a920899dad84902aed804e1990aec4e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:25.522281 18156 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7a920899dad84902aed804e1990aec4e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:25.530833 18152 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54524:
name: "default.Table_a7add9298f1a44c6a2b2a9d73405c834"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:25.531912 18152 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a7add9298f1a44c6a2b2a9d73405c834 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:25.535082 18148 master_service.cc:946] ReplaceTablet: received request to replace tablet fe01b95104174279a7d64c0d4c0af982 from {username='slave'} at 127.0.0.1:54524
I20240430 20:11:25.535454 18983 tablet_bootstrap.cc:492] T a01ee128ba8a4c9fb8e125ba97239d4c P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:25.535663 18983 ts_tablet_manager.cc:1367] T a01ee128ba8a4c9fb8e125ba97239d4c P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.487s user 0.021s sys 0.035s
I20240430 20:11:25.536505 18983 raft_consensus.cc:348] T a01ee128ba8a4c9fb8e125ba97239d4c P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:25.536732 18983 raft_consensus.cc:374] T a01ee128ba8a4c9fb8e125ba97239d4c P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:25.536801 18983 raft_consensus.cc:729] T a01ee128ba8a4c9fb8e125ba97239d4c P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:25.537096 18983 consensus_queue.cc:260] T a01ee128ba8a4c9fb8e125ba97239d4c P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:25.537885 18463 tablet_service.cc:1514] Processing DeleteTablet for tablet 187a27a6449342f99de3c5a2f93d49fc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:25 UTC) from {username='slave'} at 127.0.0.1:41678
I20240430 20:11:25.538103 18983 ts_tablet_manager.cc:1398] T a01ee128ba8a4c9fb8e125ba97239d4c P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240430 20:11:25.538445 19147 mvcc.cc:205] Tried to move back new op lower bound from 7022624294598426624 to 7022624284890099712. Current Snapshot: MvccSnapshot[applied={T|T < 7022624294598426624 or (T in {7022624294598426624})}]
I20240430 20:11:25.541744 18983 tablet_bootstrap.cc:492] T 5809e7c9c2204ba293426c24129fb16e P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:25.545094 18810 heartbeater.cc:464] Registering TS with master...
I20240430 20:11:25.546295 18983 tablet_bootstrap.cc:654] T 5809e7c9c2204ba293426c24129fb16e P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:25.548218 18455 tablet_service.cc:1514] Processing DeleteTablet for tablet 609a1a7d22e84b69bd0a53577b62b69b with delete_type TABLET_DATA_DELETED (replaced by tablet d57e59f225114af1b1522be6cb805991) from {username='slave'} at 127.0.0.1:41678
I20240430 20:11:25.548614 18463 tablet_service.cc:1514] Processing DeleteTablet for tablet c5fd0730133c472299c93f4b0830e527 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:25 UTC) from {username='slave'} at 127.0.0.1:41678
I20240430 20:11:25.551544 18463 tablet_service.cc:1514] Processing DeleteTablet for tablet 93d158de64994f3c9fa45a1250a4f533 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:25 UTC) from {username='slave'} at 127.0.0.1:41678
I20240430 20:11:25.560824 18595 tablet_service.cc:1514] Processing DeleteTablet for tablet a01ee128ba8a4c9fb8e125ba97239d4c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:24 UTC) from {username='slave'} at 127.0.0.1:40202
I20240430 20:11:25.561519 18985 ts_tablet_manager.cc:1398] T b0e6cc37c6b1488fa88359d16b12a576 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.053s user 0.002s sys 0.002s
I20240430 20:11:25.561889 18985 tablet_bootstrap.cc:492] T 5809e7c9c2204ba293426c24129fb16e P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:25.563822 18142 catalog_manager.cc:4830] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): tablet 26a015250eeb4750a42df93840e8d47f (table default.Table_2e9c7c683df84a41942d31600a265641 [id=aca6fb02776a40bbac312191adbdc136]) successfully deleted
W20240430 20:11:25.567580 18142 catalog_manager.cc:4797] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): delete failed for tablet 44699c84ebac49ef8f3765d4069e71ca because the tablet was not found. No further retry: Not found: Tablet not found: 44699c84ebac49ef8f3765d4069e71ca
I20240430 20:11:25.568148 19051 ts_manager.cc:194] Registered new tserver with Master: 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:25.568553 19185 raft_consensus.cc:686] T 59548d560b4e4f7ca034d730aca79364 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Becoming Leader. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Running, Role: LEADER
I20240430 20:11:25.568907 19185 consensus_queue.cc:237] T 59548d560b4e4f7ca034d730aca79364 P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:25.570569 18985 tablet_bootstrap.cc:654] T 5809e7c9c2204ba293426c24129fb16e P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:25.573563 18980 tablet_replica.cc:330] T 26a015250eeb4750a42df93840e8d47f P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:25.573925 18980 raft_consensus.cc:2229] T 26a015250eeb4750a42df93840e8d47f P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:25.574223 18980 pending_rounds.cc:62] T 26a015250eeb4750a42df93840e8d47f P 5e1c6397f09747f091534faf93542710: Trying to abort 1 pending ops.
I20240430 20:11:25.574374 18980 pending_rounds.cc:68] T 26a015250eeb4750a42df93840e8d47f P 5e1c6397f09747f091534faf93542710: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7022624294056210432 op_type: NO_OP noop_request { }
I20240430 20:11:25.574645 18980 raft_consensus.cc:2866] T 26a015250eeb4750a42df93840e8d47f P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20240430 20:11:25.574898 18980 raft_consensus.cc:2258] T 26a015250eeb4750a42df93840e8d47f P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:25.576146 18980 ts_tablet_manager.cc:1875] T 26a015250eeb4750a42df93840e8d47f P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240430 20:11:25.581032 18988 thread.cc:636] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.504s user 0.106s sys 0.183s
W20240430 20:11:25.581164 18988 thread.cc:602] log-alloc [worker] (thread pool) Time spent starting thread: real 0.504s user 0.106s sys 0.183s
I20240430 20:11:25.585047 18988 tablet_bootstrap.cc:492] T 0d6a4a63c004480b82d84381867bf8a4 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:25.585237 18988 ts_tablet_manager.cc:1367] T 0d6a4a63c004480b82d84381867bf8a4 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.510s user 0.106s sys 0.184s
I20240430 20:11:25.586267 18988 raft_consensus.cc:348] T 0d6a4a63c004480b82d84381867bf8a4 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:25.586494 18988 raft_consensus.cc:374] T 0d6a4a63c004480b82d84381867bf8a4 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:25.586609 18988 raft_consensus.cc:729] T 0d6a4a63c004480b82d84381867bf8a4 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:25.586931 18988 consensus_queue.cc:260] T 0d6a4a63c004480b82d84381867bf8a4 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:25.587586 18988 ts_tablet_manager.cc:1398] T 0d6a4a63c004480b82d84381867bf8a4 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240430 20:11:25.587900 18988 tablet_bootstrap.cc:492] T fe01b95104174279a7d64c0d4c0af982 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:25.590910 18988 tablet_bootstrap.cc:654] T fe01b95104174279a7d64c0d4c0af982 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:25.592844 18735 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d6a4a63c004480b82d84381867bf8a4 with delete_type TABLET_DATA_DELETED (replaced by tablet c9ff972ade694938888514979e1638aa) from {username='slave'} at 127.0.0.1:59818
I20240430 20:11:25.593117 18987 tablet_replica.cc:330] T 0d6a4a63c004480b82d84381867bf8a4 P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:25.593338 18987 raft_consensus.cc:2229] T 0d6a4a63c004480b82d84381867bf8a4 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:25.593533 18987 raft_consensus.cc:2258] T 0d6a4a63c004480b82d84381867bf8a4 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:25.597044 18146 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: Sending DeleteTablet for 3 replicas of tablet 1398d4d2305b48e08fdb12432bd6c4b0
I20240430 20:11:25.598122 18735 tablet_service.cc:1514] Processing DeleteTablet for tablet 1398d4d2305b48e08fdb12432bd6c4b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:25 UTC) from {username='slave'} at 127.0.0.1:59818
I20240430 20:11:25.600692 18463 tablet_service.cc:1514] Processing DeleteTablet for tablet 1398d4d2305b48e08fdb12432bd6c4b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:25 UTC) from {username='slave'} at 127.0.0.1:41678
I20240430 20:11:25.601644 18987 ts_tablet_manager.cc:1875] T 0d6a4a63c004480b82d84381867bf8a4 P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:25.612236 18987 ts_tablet_manager.cc:1888] T 0d6a4a63c004480b82d84381867bf8a4 P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:25.612514 18987 log.cc:1196] T 0d6a4a63c004480b82d84381867bf8a4 P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/0d6a4a63c004480b82d84381867bf8a4
I20240430 20:11:25.613276 18987 ts_tablet_manager.cc:1909] T 0d6a4a63c004480b82d84381867bf8a4 P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:25.614548 18141 catalog_manager.cc:4830] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): tablet 0d6a4a63c004480b82d84381867bf8a4 (table default.Table_3c5df662747b48bf8beb6ebe4d2ea618 [id=663c8efbd7ac4a148d5f7626f1522033]) successfully deleted
I20240430 20:11:25.615670 18735 tablet_service.cc:1467] Processing CreateTablet for tablet e2d90ff17e574f88b2061b1c117d4b7d (DEFAULT_TABLE table=default.Table_6f2fd99337344da2ac5a0c83d9b56bb8 [id=8f1ad1f8d0224b3daa5d84dc6916227b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:25.616111 18735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2d90ff17e574f88b2061b1c117d4b7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:25.616473 18146 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54524:
name: "default.Table_da5216d025844afda5a83ada555bc120"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:25.617210 18146 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_da5216d025844afda5a83ada555bc120 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240430 20:11:25.619061 18141 catalog_manager.cc:4797] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): delete failed for tablet 1398d4d2305b48e08fdb12432bd6c4b0 because the tablet was not found. No further retry: Not found: Tablet not found: 1398d4d2305b48e08fdb12432bd6c4b0
I20240430 20:11:25.627683 18457 tablet_service.cc:1467] Processing CreateTablet for tablet e2d90ff17e574f88b2061b1c117d4b7d (DEFAULT_TABLE table=default.Table_6f2fd99337344da2ac5a0c83d9b56bb8 [id=8f1ad1f8d0224b3daa5d84dc6916227b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:25.624033 18156 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54524:
name: "default.Table_cd719a43207f4bf983e1b8e6d236bb16"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240430 20:11:25.628265 18457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2d90ff17e574f88b2061b1c117d4b7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:25.630714 18981 ts_tablet_manager.cc:1888] T 2067cb2cf76d495c92c1f25c176eb6dc P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:25.630932 18981 log.cc:1196] T 2067cb2cf76d495c92c1f25c176eb6dc P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/2067cb2cf76d495c92c1f25c176eb6dc
I20240430 20:11:25.631356 18981 ts_tablet_manager.cc:1909] T 2067cb2cf76d495c92c1f25c176eb6dc P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:25.632251 18981 tablet_replica.cc:330] T c3f8f9ada98b40bc8359a00e193c3e5f P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:25.632509 18981 raft_consensus.cc:2229] T c3f8f9ada98b40bc8359a00e193c3e5f P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:25.632726 18981 raft_consensus.cc:2258] T c3f8f9ada98b40bc8359a00e193c3e5f P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:25.633309 18981 ts_tablet_manager.cc:1875] T c3f8f9ada98b40bc8359a00e193c3e5f P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:25.634873 18142 catalog_manager.cc:4830] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): tablet 2067cb2cf76d495c92c1f25c176eb6dc (table default.Table_25ea07a841144dab969f3baf7ee7c0dc [id=b4bd1ac2bb0b4989af99b2f4936ce927]) successfully deleted
W20240430 20:11:25.635232 18156 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cd719a43207f4bf983e1b8e6d236bb16 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:25.636343 18983 tablet_bootstrap.cc:492] T 5809e7c9c2204ba293426c24129fb16e P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:25.636509 18983 ts_tablet_manager.cc:1367] T 5809e7c9c2204ba293426c24129fb16e P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.095s user 0.000s sys 0.003s
I20240430 20:11:25.637688 18983 raft_consensus.cc:348] T 5809e7c9c2204ba293426c24129fb16e P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:25.637902 18983 raft_consensus.cc:374] T 5809e7c9c2204ba293426c24129fb16e P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:25.637992 18983 raft_consensus.cc:729] T 5809e7c9c2204ba293426c24129fb16e P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:25.638204 18983 consensus_queue.cc:260] T 5809e7c9c2204ba293426c24129fb16e P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:25.638767 18983 ts_tablet_manager.cc:1398] T 5809e7c9c2204ba293426c24129fb16e P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240430 20:11:25.639127 18983 tablet_bootstrap.cc:492] T fe01b95104174279a7d64c0d4c0af982 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
W20240430 20:11:25.635264 18142 catalog_manager.cc:4797] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): delete failed for tablet f15f5e1f3afd4224bbf39567cae74617 because the tablet was not found. No further retry: Not found: Tablet not found: f15f5e1f3afd4224bbf39567cae74617
I20240430 20:11:25.645709 18595 tablet_service.cc:1467] Processing CreateTablet for tablet e2d90ff17e574f88b2061b1c117d4b7d (DEFAULT_TABLE table=default.Table_6f2fd99337344da2ac5a0c83d9b56bb8 [id=8f1ad1f8d0224b3daa5d84dc6916227b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:25.647759 18595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2d90ff17e574f88b2061b1c117d4b7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:25.650693 18988 tablet_bootstrap.cc:492] T fe01b95104174279a7d64c0d4c0af982 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:25.650884 18988 ts_tablet_manager.cc:1367] T fe01b95104174279a7d64c0d4c0af982 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.063s user 0.026s sys 0.011s
I20240430 20:11:25.651665 18988 raft_consensus.cc:348] T fe01b95104174279a7d64c0d4c0af982 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:25.651906 18988 raft_consensus.cc:374] T fe01b95104174279a7d64c0d4c0af982 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:25.651990 18988 raft_consensus.cc:729] T fe01b95104174279a7d64c0d4c0af982 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:25.652256 18988 consensus_queue.cc:260] T fe01b95104174279a7d64c0d4c0af982 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:25.652781 18988 ts_tablet_manager.cc:1398] T fe01b95104174279a7d64c0d4c0af982 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240430 20:11:25.653165 18988 tablet_bootstrap.cc:492] T 5809e7c9c2204ba293426c24129fb16e P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:25.654372 18735 tablet_service.cc:1467] Processing CreateTablet for tablet bd11c0335a58448da56533c3f2e785a9 (DEFAULT_TABLE table=default.Table_88c44b20f2ce4eeca7ed9583a70c7c65 [id=73caec5fb65e4cb090e7ae8c391501fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:25.656647 18738 tablet_service.cc:1514] Processing DeleteTablet for tablet fe01b95104174279a7d64c0d4c0af982 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:23 UTC) from {username='slave'} at 127.0.0.1:59818
I20240430 20:11:25.659081 18985 tablet_bootstrap.cc:492] T 5809e7c9c2204ba293426c24129fb16e P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:25.659526 18735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd11c0335a58448da56533c3f2e785a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:25.661185 18738 tablet_service.cc:1467] Processing CreateTablet for tablet 1398d4d2305b48e08fdb12432bd6c4b0 (DEFAULT_TABLE table=default.Table_eea9acffa45b4f3ab6d4524dc460853c [id=52cd3a55893547cabf418f4d01b11d08]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:25.661644 18738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1398d4d2305b48e08fdb12432bd6c4b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:25.665231 18744 tablet_service.cc:1467] Processing CreateTablet for tablet a1aeee1b02214975a376cfd36957afe8 (DEFAULT_TABLE table=default.Table_a67165ab67884e57955b604826a006d6 [id=8b480956f4d146e08dcfed0d0a0e4e0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:25.665993 18742 tablet_service.cc:1467] Processing CreateTablet for tablet aacabead3c874e77a47d2ecf19dbfa82 (DEFAULT_TABLE table=default.Table_40e4655ab0b449cb8f9688a222e92813 [id=da9ad991953a40c4b77b31317597ad1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:25.666484 18987 tablet_replica.cc:330] T fe01b95104174279a7d64c0d4c0af982 P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:25.666476 18988 tablet_bootstrap.cc:654] T 5809e7c9c2204ba293426c24129fb16e P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:25.669190 18744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1aeee1b02214975a376cfd36957afe8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:25.669503 18455 tablet_service.cc:1467] Processing CreateTablet for tablet a1aeee1b02214975a376cfd36957afe8 (DEFAULT_TABLE table=default.Table_a67165ab67884e57955b604826a006d6 [id=8b480956f4d146e08dcfed0d0a0e4e0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:25.670138 18463 tablet_service.cc:1467] Processing CreateTablet for tablet bd11c0335a58448da56533c3f2e785a9 (DEFAULT_TABLE table=default.Table_88c44b20f2ce4eeca7ed9583a70c7c65 [id=73caec5fb65e4cb090e7ae8c391501fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:25.671645 18146 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_7974c4f995c047469ccc69cc0d0521f3" } modify_external_catalogs: true
I20240430 20:11:25.671880 18146 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_7974c4f995c047469ccc69cc0d0521f3" } modify_external_catalogs: true
I20240430 20:11:25.672803 19051 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" instance_seqno: 1714507879927603) as {username='slave'} at 127.17.177.1:50353; Asking this server to re-register.
I20240430 20:11:25.673849 18987 raft_consensus.cc:2229] T fe01b95104174279a7d64c0d4c0af982 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:25.673979 18452 tablet_service.cc:1467] Processing CreateTablet for tablet aacabead3c874e77a47d2ecf19dbfa82 (DEFAULT_TABLE table=default.Table_40e4655ab0b449cb8f9688a222e92813 [id=da9ad991953a40c4b77b31317597ad1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:25.674074 18987 raft_consensus.cc:2258] T fe01b95104174279a7d64c0d4c0af982 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:25.675644 18455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1aeee1b02214975a376cfd36957afe8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:25.676391 18987 ts_tablet_manager.cc:1875] T fe01b95104174279a7d64c0d4c0af982 P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:25.676606 18601 tablet_service.cc:1514] Processing DeleteTablet for tablet 1398d4d2305b48e08fdb12432bd6c4b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:25 UTC) from {username='slave'} at 127.0.0.1:40202
I20240430 20:11:25.676875 18463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd11c0335a58448da56533c3f2e785a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:25.677739 18461 tablet_service.cc:1467] Processing CreateTablet for tablet 1398d4d2305b48e08fdb12432bd6c4b0 (DEFAULT_TABLE table=default.Table_eea9acffa45b4f3ab6d4524dc460853c [id=52cd3a55893547cabf418f4d01b11d08]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:25.678050 18452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aacabead3c874e77a47d2ecf19dbfa82. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:25.678577 18461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1398d4d2305b48e08fdb12432bd6c4b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:25.678676 18742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aacabead3c874e77a47d2ecf19dbfa82. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:25.679872 18529 heartbeater.cc:464] Registering TS with master...
I20240430 20:11:25.685940 18987 ts_tablet_manager.cc:1888] T fe01b95104174279a7d64c0d4c0af982 P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:25.686205 18987 log.cc:1196] T fe01b95104174279a7d64c0d4c0af982 P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/fe01b95104174279a7d64c0d4c0af982
I20240430 20:11:25.686724 18987 ts_tablet_manager.cc:1909] T fe01b95104174279a7d64c0d4c0af982 P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:25.687986 18141 catalog_manager.cc:4830] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): tablet fe01b95104174279a7d64c0d4c0af982 (table default.Table_52f64b8de0684649aa824da65de3416f [id=41ea2be4643c4e75a2c20b7f9f0a825e]) successfully deleted
I20240430 20:11:25.688593 18146 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: Sending DeleteTablet for 3 replicas of tablet 15674aeb56b74210a4a2809608fd4457
I20240430 20:11:25.691493 18988 tablet_bootstrap.cc:492] T 5809e7c9c2204ba293426c24129fb16e P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:25.691653 18988 ts_tablet_manager.cc:1367] T 5809e7c9c2204ba293426c24129fb16e P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.039s user 0.002s sys 0.001s
I20240430 20:11:25.693076 18985 ts_tablet_manager.cc:1367] T 5809e7c9c2204ba293426c24129fb16e P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.097s user 0.001s sys 0.006s
I20240430 20:11:25.693444 18742 tablet_service.cc:1514] Processing DeleteTablet for tablet 15674aeb56b74210a4a2809608fd4457 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:25 UTC) from {username='slave'} at 127.0.0.1:59818
I20240430 20:11:25.693579 18988 raft_consensus.cc:348] T 5809e7c9c2204ba293426c24129fb16e P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:25.693807 18988 raft_consensus.cc:374] T 5809e7c9c2204ba293426c24129fb16e P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:25.694072 18599 tablet_service.cc:1467] Processing CreateTablet for tablet bd11c0335a58448da56533c3f2e785a9 (DEFAULT_TABLE table=default.Table_88c44b20f2ce4eeca7ed9583a70c7c65 [id=73caec5fb65e4cb090e7ae8c391501fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:25.694662 19051 ts_manager.cc:194] Registered new tserver with Master: 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:25.693943 18988 raft_consensus.cc:729] T 5809e7c9c2204ba293426c24129fb16e P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:25.695880 18988 consensus_queue.cc:260] T 5809e7c9c2204ba293426c24129fb16e P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
W20240430 20:11:25.696079 18141 catalog_manager.cc:4810] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): delete failed for tablet 15674aeb56b74210a4a2809608fd4457 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 15674aeb56b74210a4a2809608fd4457 already in progress: creating tablet
I20240430 20:11:25.696466 18988 ts_tablet_manager.cc:1398] T 5809e7c9c2204ba293426c24129fb16e P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240430 20:11:25.696449 18459 tablet_service.cc:1514] Processing DeleteTablet for tablet 26a015250eeb4750a42df93840e8d47f with delete_type TABLET_DATA_DELETED (replaced by tablet ec060e847f024f9fa369cff320e71558) from {username='slave'} at 127.0.0.1:41678
I20240430 20:11:25.697413 18454 tablet_service.cc:1514] Processing DeleteTablet for tablet 59548d560b4e4f7ca034d730aca79364 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:24 UTC) from {username='slave'} at 127.0.0.1:41678
I20240430 20:11:25.697714 18988 tablet_bootstrap.cc:492] T b0e6cc37c6b1488fa88359d16b12a576 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:25.697715 18983 tablet_bootstrap.cc:654] T fe01b95104174279a7d64c0d4c0af982 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:25.700106 18988 tablet_bootstrap.cc:654] T b0e6cc37c6b1488fa88359d16b12a576 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:25.703593 18742 tablet_service.cc:1467] Processing CreateTablet for tablet 02176eee8e044a7db8cac48439e9f594 (DEFAULT_TABLE table=default.Table_1bb7ee3d454b4c1eac6646e9b3eae7e1 [id=1fb10bddcfe94ec488ecc71e43ca339d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:25.704118 18742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02176eee8e044a7db8cac48439e9f594. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:25.706609 19191 raft_consensus.cc:482] T 81328e904ca94e2eb5ec94dd3ffb3c90 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:25.706743 18451 tablet_service.cc:1514] Processing DeleteTablet for tablet a01ee128ba8a4c9fb8e125ba97239d4c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:24 UTC) from {username='slave'} at 127.0.0.1:41678
I20240430 20:11:25.706766 19191 raft_consensus.cc:504] T 81328e904ca94e2eb5ec94dd3ffb3c90 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:25.707551 19191 leader_election.cc:288] T 81328e904ca94e2eb5ec94dd3ffb3c90 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:25.702879 18601 tablet_service.cc:1467] Processing CreateTablet for tablet a1aeee1b02214975a376cfd36957afe8 (DEFAULT_TABLE table=default.Table_a67165ab67884e57955b604826a006d6 [id=8b480956f4d146e08dcfed0d0a0e4e0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:25.709604 18621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81328e904ca94e2eb5ec94dd3ffb3c90" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:25.709991 18592 tablet_service.cc:1467] Processing CreateTablet for tablet aacabead3c874e77a47d2ecf19dbfa82 (DEFAULT_TABLE table=default.Table_40e4655ab0b449cb8f9688a222e92813 [id=da9ad991953a40c4b77b31317597ad1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:25.713382 18599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd11c0335a58448da56533c3f2e785a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:25.714352 18601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1aeee1b02214975a376cfd36957afe8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:25.714730 18603 tablet_service.cc:1467] Processing CreateTablet for tablet 1398d4d2305b48e08fdb12432bd6c4b0 (DEFAULT_TABLE table=default.Table_eea9acffa45b4f3ab6d4524dc460853c [id=52cd3a55893547cabf418f4d01b11d08]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:25.715608 18735 tablet_service.cc:1467] Processing CreateTablet for tablet 0b4d9e860d7c45fd91d7a566605c9283 (DEFAULT_TABLE table=default.Table_80b9e5521fff4a119f3ce4977b3e44eb [id=d5cea90cbc2c48df9d8b80da1eccc949]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:25.716049 18735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b4d9e860d7c45fd91d7a566605c9283. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:25.721594 18146 master_service.cc:946] ReplaceTablet: received request to replace tablet fe01b95104174279a7d64c0d4c0af982 from {username='slave'} at 127.0.0.1:54524
I20240430 20:11:25.723270 18482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81328e904ca94e2eb5ec94dd3ffb3c90" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:25.723537 18482 raft_consensus.cc:2454] T 81328e904ca94e2eb5ec94dd3ffb3c90 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:25.723923 18451 tablet_service.cc:1514] Processing DeleteTablet for tablet b0e6cc37c6b1488fa88359d16b12a576 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:23 UTC) from {username='slave'} at 127.0.0.1:41678
I20240430 20:11:25.724098 18453 tablet_service.cc:1514] Processing DeleteTablet for tablet 15674aeb56b74210a4a2809608fd4457 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:25 UTC) from {username='slave'} at 127.0.0.1:41678
I20240430 20:11:25.729388 18459 tablet_service.cc:1467] Processing CreateTablet for tablet 0b4d9e860d7c45fd91d7a566605c9283 (DEFAULT_TABLE table=default.Table_80b9e5521fff4a119f3ce4977b3e44eb [id=d5cea90cbc2c48df9d8b80da1eccc949]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:25.729857 18459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b4d9e860d7c45fd91d7a566605c9283. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:25.729948 18985 raft_consensus.cc:348] T 5809e7c9c2204ba293426c24129fb16e P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:25.730629 18985 raft_consensus.cc:374] T 5809e7c9c2204ba293426c24129fb16e P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:25.730736 18985 raft_consensus.cc:729] T 5809e7c9c2204ba293426c24129fb16e P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:25.731643 18985 consensus_queue.cc:260] T 5809e7c9c2204ba293426c24129fb16e P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:25.732775 18454 tablet_service.cc:1467] Processing CreateTablet for tablet 02176eee8e044a7db8cac48439e9f594 (DEFAULT_TABLE table=default.Table_1bb7ee3d454b4c1eac6646e9b3eae7e1 [id=1fb10bddcfe94ec488ecc71e43ca339d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:25.734191 18985 ts_tablet_manager.cc:1398] T 5809e7c9c2204ba293426c24129fb16e P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.041s user 0.000s sys 0.002s
I20240430 20:11:25.734617 18454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02176eee8e044a7db8cac48439e9f594. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:25.748891 18985 tablet_bootstrap.cc:492] T fe01b95104174279a7d64c0d4c0af982 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:25.758966 18985 tablet_bootstrap.cc:654] T fe01b95104174279a7d64c0d4c0af982 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:25.760242 18603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1398d4d2305b48e08fdb12432bd6c4b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:25.761080 18600 tablet_service.cc:1467] Processing CreateTablet for tablet 02176eee8e044a7db8cac48439e9f594 (DEFAULT_TABLE table=default.Table_1bb7ee3d454b4c1eac6646e9b3eae7e1 [id=1fb10bddcfe94ec488ecc71e43ca339d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:25.763890 18621 raft_consensus.cc:2454] T 81328e904ca94e2eb5ec94dd3ffb3c90 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:25.768193 18598 tablet_service.cc:1514] Processing DeleteTablet for tablet 15674aeb56b74210a4a2809608fd4457 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:25 UTC) from {username='slave'} at 127.0.0.1:40202
I20240430 20:11:25.769371 18600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02176eee8e044a7db8cac48439e9f594. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:25.771603 18149 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_062e0e190e17433395f8f47234987d92" } modify_external_catalogs: true
I20240430 20:11:25.771684 18147 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54524:
name: "default.Table_276f62c2338b46a0aab8905430e8b61b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240430 20:11:25.771927 18149 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_062e0e190e17433395f8f47234987d92" } modify_external_catalogs: true
W20240430 20:11:25.772531 18147 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_276f62c2338b46a0aab8905430e8b61b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:25.773761 18150 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54524:
name: "default.Table_f136f0a535fc4860811d81fb1d439f3c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240430 20:11:25.774305 18148 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_2f1d187ff53442efbe9a5266b89f1267" } modify_external_catalogs: true
W20240430 20:11:25.774524 18150 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f136f0a535fc4860811d81fb1d439f3c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:25.774557 18148 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_2f1d187ff53442efbe9a5266b89f1267" } modify_external_catalogs: true
I20240430 20:11:25.776160 18151 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_7a920899dad84902aed804e1990aec4e" } modify_external_catalogs: true
I20240430 20:11:25.776191 18154 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54524:
name: "default.Table_8ddf4569d2f245469f55cb9fef2f5fb1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240430 20:11:25.776427 18151 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_7a920899dad84902aed804e1990aec4e" } modify_external_catalogs: true
W20240430 20:11:25.776893 18154 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8ddf4569d2f245469f55cb9fef2f5fb1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:25.777840 18155 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_8502447753b04761815bf9602fd28e37" } new_table_name: "default.Table_5fc77d5f60ae44f4bb4ca9f7f183672c" modify_external_catalogs: true
I20240430 20:11:25.778717 18146 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_b74297eb78cc4912a25657bbf0180340" } new_table_name: "default.Table_c1669746f9114bf7b37a6f6a68fed20e" modify_external_catalogs: true
I20240430 20:11:25.780371 18152 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54524:
name: "default.Table_95220daaf6d44c08beff4ed6abdbea79"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:25.780833 18152 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_95220daaf6d44c08beff4ed6abdbea79 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:25.792611 18148 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet 215b799255814eb5a9ddccf7b3ae03ca
I20240430 20:11:25.793501 18149 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet d57e59f225114af1b1522be6cb805991
I20240430 20:11:25.797088 18596 tablet_service.cc:1467] Processing CreateTablet for tablet 0b4d9e860d7c45fd91d7a566605c9283 (DEFAULT_TABLE table=default.Table_80b9e5521fff4a119f3ce4977b3e44eb [id=d5cea90cbc2c48df9d8b80da1eccc949]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:25.797981 18699 leader_election.cc:302] T 81328e904ca94e2eb5ec94dd3ffb3c90 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters:
W20240430 20:11:25.798975 18142 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 609a1a7d22e84b69bd0a53577b62b69b (table default.Table_062e0e190e17433395f8f47234987d92 [id=c092ea63bd4742bd9876634a1dd2e2b3], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 609a1a7d22e84b69bd0a53577b62b69b consensus state has no leader
I20240430 20:11:25.801138 18981 ts_tablet_manager.cc:1888] T c3f8f9ada98b40bc8359a00e193c3e5f P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:25.801789 19185 raft_consensus.cc:482] T a01ee128ba8a4c9fb8e125ba97239d4c P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:25.801965 18981 log.cc:1196] T c3f8f9ada98b40bc8359a00e193c3e5f P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/c3f8f9ada98b40bc8359a00e193c3e5f
I20240430 20:11:25.801991 19185 raft_consensus.cc:504] T a01ee128ba8a4c9fb8e125ba97239d4c P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:25.802570 18981 ts_tablet_manager.cc:1909] T c3f8f9ada98b40bc8359a00e193c3e5f P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:25.803484 19185 leader_election.cc:288] T a01ee128ba8a4c9fb8e125ba97239d4c P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:25.803692 18142 catalog_manager.cc:4830] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): tablet c3f8f9ada98b40bc8359a00e193c3e5f (table default.Table_c21d1f6f0bfb4f9ea6978a36db7a3701 [id=70551d80c9d8445fa158eee362758714]) successfully deleted
I20240430 20:11:25.807938 18981 tablet_replica.cc:330] T f79497bf22ed417d988530f6e69cc07f P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:25.808219 18981 raft_consensus.cc:2229] T f79497bf22ed417d988530f6e69cc07f P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:25.808449 18981 raft_consensus.cc:2258] T f79497bf22ed417d988530f6e69cc07f P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:25.809141 18981 ts_tablet_manager.cc:1875] T f79497bf22ed417d988530f6e69cc07f P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:25.812968 18592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aacabead3c874e77a47d2ecf19dbfa82. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:25.813431 18985 tablet_bootstrap.cc:492] T fe01b95104174279a7d64c0d4c0af982 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:25.813632 18985 ts_tablet_manager.cc:1367] T fe01b95104174279a7d64c0d4c0af982 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.079s user 0.008s sys 0.014s
I20240430 20:11:25.814739 18621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a01ee128ba8a4c9fb8e125ba97239d4c" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
W20240430 20:11:25.815078 18696 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.177.2:46027, user_credentials={real_user=slave}} blocked reactor thread for 50451.9us
I20240430 20:11:25.818219 18981 ts_tablet_manager.cc:1888] T f79497bf22ed417d988530f6e69cc07f P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:25.818459 18985 raft_consensus.cc:348] T fe01b95104174279a7d64c0d4c0af982 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:25.818971 18985 raft_consensus.cc:374] T fe01b95104174279a7d64c0d4c0af982 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:25.819108 18985 raft_consensus.cc:729] T fe01b95104174279a7d64c0d4c0af982 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:25.819368 18985 consensus_queue.cc:260] T fe01b95104174279a7d64c0d4c0af982 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:25.818724 18981 log.cc:1196] T f79497bf22ed417d988530f6e69cc07f P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/f79497bf22ed417d988530f6e69cc07f
I20240430 20:11:25.821803 18981 ts_tablet_manager.cc:1909] T f79497bf22ed417d988530f6e69cc07f P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:25.822808 18142 catalog_manager.cc:4830] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): tablet f79497bf22ed417d988530f6e69cc07f (table default.Table_b78b1b2c95694d1faee32321f845047b [id=2a3033894afd47ed8d907628a2e9712d]) successfully deleted
I20240430 20:11:25.820050 18985 ts_tablet_manager.cc:1398] T fe01b95104174279a7d64c0d4c0af982 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20240430 20:11:25.823359 18985 tablet_bootstrap.cc:492] T bd7392f3dfd643cdb1634ee10ad41651 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:25.825672 18985 tablet_bootstrap.cc:654] T bd7392f3dfd643cdb1634ee10ad41651 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:25.825965 18981 tablet_replica.cc:330] T 0d6a4a63c004480b82d84381867bf8a4 P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:25.827212 18981 raft_consensus.cc:2229] T 0d6a4a63c004480b82d84381867bf8a4 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:25.827474 18981 raft_consensus.cc:2258] T 0d6a4a63c004480b82d84381867bf8a4 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:25.827602 19191 raft_consensus.cc:2781] T 81328e904ca94e2eb5ec94dd3ffb3c90 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20240430 20:11:25.827872 18699 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.177.1:41217, user_credentials={real_user=slave}} blocked reactor thread for 74235.2us
I20240430 20:11:25.828619 19191 raft_consensus.cc:482] T 81328e904ca94e2eb5ec94dd3ffb3c90 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:25.828783 19191 raft_consensus.cc:3037] T 81328e904ca94e2eb5ec94dd3ffb3c90 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:25.834650 18981 ts_tablet_manager.cc:1875] T 0d6a4a63c004480b82d84381867bf8a4 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:25.837440 18763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a01ee128ba8a4c9fb8e125ba97239d4c" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
W20240430 20:11:25.838403 18417 leader_election.cc:341] T a01ee128ba8a4c9fb8e125ba97239d4c P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:25.845098 18985 tablet_bootstrap.cc:492] T bd7392f3dfd643cdb1634ee10ad41651 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:25.845286 18985 ts_tablet_manager.cc:1367] T bd7392f3dfd643cdb1634ee10ad41651 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.022s user 0.001s sys 0.007s
I20240430 20:11:25.846076 18985 raft_consensus.cc:348] T bd7392f3dfd643cdb1634ee10ad41651 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:25.846329 18985 raft_consensus.cc:374] T bd7392f3dfd643cdb1634ee10ad41651 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:25.846403 18985 raft_consensus.cc:729] T bd7392f3dfd643cdb1634ee10ad41651 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:25.846678 18985 consensus_queue.cc:260] T bd7392f3dfd643cdb1634ee10ad41651 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:25.847220 18985 ts_tablet_manager.cc:1398] T bd7392f3dfd643cdb1634ee10ad41651 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240430 20:11:25.847551 18985 tablet_bootstrap.cc:492] T b4bc111e183f40a0acc9dd377195f6d9 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:25.849228 18985 tablet_bootstrap.cc:654] T b4bc111e183f40a0acc9dd377195f6d9 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:25.858733 18621 raft_consensus.cc:2454] T a01ee128ba8a4c9fb8e125ba97239d4c P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 0.
I20240430 20:11:25.859184 18985 tablet_bootstrap.cc:492] T b4bc111e183f40a0acc9dd377195f6d9 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:25.859345 18985 ts_tablet_manager.cc:1367] T b4bc111e183f40a0acc9dd377195f6d9 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.002s
I20240430 20:11:25.860191 18985 raft_consensus.cc:348] T b4bc111e183f40a0acc9dd377195f6d9 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:25.860452 18985 raft_consensus.cc:374] T b4bc111e183f40a0acc9dd377195f6d9 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:25.860565 18985 raft_consensus.cc:729] T b4bc111e183f40a0acc9dd377195f6d9 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:25.860760 18415 leader_election.cc:302] T a01ee128ba8a4c9fb8e125ba97239d4c P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters: 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:25.860797 18985 consensus_queue.cc:260] T b4bc111e183f40a0acc9dd377195f6d9 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:25.861142 19186 raft_consensus.cc:2781] T a01ee128ba8a4c9fb8e125ba97239d4c P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:25.812867 18980 ts_tablet_manager.cc:1888] T 26a015250eeb4750a42df93840e8d47f P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:25.861231 19186 raft_consensus.cc:482] T a01ee128ba8a4c9fb8e125ba97239d4c P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:25.861380 19186 raft_consensus.cc:3037] T a01ee128ba8a4c9fb8e125ba97239d4c P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:25.864672 18985 ts_tablet_manager.cc:1398] T b4bc111e183f40a0acc9dd377195f6d9 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240430 20:11:25.865160 18985 tablet_bootstrap.cc:492] T ffe059eeac91476aaeef8c56d2f49ee8 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:25.867084 18985 tablet_bootstrap.cc:654] T ffe059eeac91476aaeef8c56d2f49ee8 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:25.871814 18981 ts_tablet_manager.cc:1888] T 0d6a4a63c004480b82d84381867bf8a4 P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:25.871973 19186 raft_consensus.cc:504] T a01ee128ba8a4c9fb8e125ba97239d4c P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:25.873039 18981 log.cc:1196] T 0d6a4a63c004480b82d84381867bf8a4 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/0d6a4a63c004480b82d84381867bf8a4
I20240430 20:11:25.874305 18763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a01ee128ba8a4c9fb8e125ba97239d4c" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:25.874876 18596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b4d9e860d7c45fd91d7a566605c9283. 1 dirs total, 0 dirs full, 0 dirs failed
W20240430 20:11:25.876154 18417 leader_election.cc:341] T a01ee128ba8a4c9fb8e125ba97239d4c P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:25.878522 19191 raft_consensus.cc:504] T 81328e904ca94e2eb5ec94dd3ffb3c90 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:25.878703 18981 ts_tablet_manager.cc:1909] T 0d6a4a63c004480b82d84381867bf8a4 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:25.879519 19191 leader_election.cc:288] T 81328e904ca94e2eb5ec94dd3ffb3c90 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Requested vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:25.879681 18980 log.cc:1196] T 26a015250eeb4750a42df93840e8d47f P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/26a015250eeb4750a42df93840e8d47f
I20240430 20:11:25.879761 18623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81328e904ca94e2eb5ec94dd3ffb3c90" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:25.880403 18623 raft_consensus.cc:3037] T 81328e904ca94e2eb5ec94dd3ffb3c90 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:25.882642 19186 leader_election.cc:288] T a01ee128ba8a4c9fb8e125ba97239d4c P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:25.885999 18482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81328e904ca94e2eb5ec94dd3ffb3c90" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:25.886255 18482 raft_consensus.cc:3037] T 81328e904ca94e2eb5ec94dd3ffb3c90 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:25.887282 18988 tablet_bootstrap.cc:492] T b0e6cc37c6b1488fa88359d16b12a576 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:25.887506 18988 ts_tablet_manager.cc:1367] T b0e6cc37c6b1488fa88359d16b12a576 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.190s user 0.070s sys 0.088s
I20240430 20:11:25.888463 18988 raft_consensus.cc:348] T b0e6cc37c6b1488fa88359d16b12a576 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:25.888845 18988 raft_consensus.cc:374] T b0e6cc37c6b1488fa88359d16b12a576 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:25.888978 18988 raft_consensus.cc:729] T b0e6cc37c6b1488fa88359d16b12a576 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:25.889151 18742 tablet_service.cc:1467] Processing CreateTablet for tablet 0e503a898406412ab977d3e048ea4b5d (DEFAULT_TABLE table=default.Table_fe1773cf77654e2a86e95444437b7201 [id=fe2222c2b62f4360a5ade622ae8b23e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:25.889300 18988 consensus_queue.cc:260] T b0e6cc37c6b1488fa88359d16b12a576 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:25.890033 18988 ts_tablet_manager.cc:1398] T b0e6cc37c6b1488fa88359d16b12a576 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20240430 20:11:25.890663 18988 tablet_bootstrap.cc:492] T a01ee128ba8a4c9fb8e125ba97239d4c P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:25.892683 18988 tablet_bootstrap.cc:654] T a01ee128ba8a4c9fb8e125ba97239d4c P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:25.893805 18461 tablet_service.cc:1467] Processing CreateTablet for tablet 0e503a898406412ab977d3e048ea4b5d (DEFAULT_TABLE table=default.Table_fe1773cf77654e2a86e95444437b7201 [id=fe2222c2b62f4360a5ade622ae8b23e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:25.894239 18461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e503a898406412ab977d3e048ea4b5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:25.899427 18742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e503a898406412ab977d3e048ea4b5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:25.900866 18988 tablet_bootstrap.cc:492] T a01ee128ba8a4c9fb8e125ba97239d4c P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:25.901063 18988 ts_tablet_manager.cc:1367] T a01ee128ba8a4c9fb8e125ba97239d4c P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.003s
I20240430 20:11:25.901821 18988 raft_consensus.cc:348] T a01ee128ba8a4c9fb8e125ba97239d4c P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:25.902060 18988 raft_consensus.cc:374] T a01ee128ba8a4c9fb8e125ba97239d4c P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:25.902166 18988 raft_consensus.cc:729] T a01ee128ba8a4c9fb8e125ba97239d4c P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:25.902426 18988 consensus_queue.cc:260] T a01ee128ba8a4c9fb8e125ba97239d4c P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:25.903043 18988 ts_tablet_manager.cc:1398] T a01ee128ba8a4c9fb8e125ba97239d4c P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240430 20:11:25.903409 18988 tablet_bootstrap.cc:492] T 81787eccca6c42c1b421377a3bf9ebe6 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:25.903328 18459 tablet_service.cc:1467] Processing CreateTablet for tablet ec060e847f024f9fa369cff320e71558 (DEFAULT_TABLE table=default.Table_2e9c7c683df84a41942d31600a265641 [id=aca6fb02776a40bbac312191adbdc136]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:25.903750 18459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec060e847f024f9fa369cff320e71558. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:25.901232 18621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a01ee128ba8a4c9fb8e125ba97239d4c" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:25.905316 18980 ts_tablet_manager.cc:1909] T 26a015250eeb4750a42df93840e8d47f P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:25.906126 18621 raft_consensus.cc:3037] T a01ee128ba8a4c9fb8e125ba97239d4c P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:25.909133 18988 tablet_bootstrap.cc:654] T 81787eccca6c42c1b421377a3bf9ebe6 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:25.915447 18142 catalog_manager.cc:4830] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): tablet 0d6a4a63c004480b82d84381867bf8a4 (table default.Table_3c5df662747b48bf8beb6ebe4d2ea618 [id=663c8efbd7ac4a148d5f7626f1522033]) successfully deleted
I20240430 20:11:25.916002 18981 tablet_replica.cc:330] T 9750608faeab4be38ddce4e2e4d46964 P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:25.916299 18981 raft_consensus.cc:2229] T 9750608faeab4be38ddce4e2e4d46964 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:25.916538 18981 raft_consensus.cc:2258] T 9750608faeab4be38ddce4e2e4d46964 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:25.917220 18981 ts_tablet_manager.cc:1875] T 9750608faeab4be38ddce4e2e4d46964 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:25.922133 18735 tablet_service.cc:1467] Processing CreateTablet for tablet ec060e847f024f9fa369cff320e71558 (DEFAULT_TABLE table=default.Table_2e9c7c683df84a41942d31600a265641 [id=aca6fb02776a40bbac312191adbdc136]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:25.922756 18988 tablet_bootstrap.cc:492] T 81787eccca6c42c1b421377a3bf9ebe6 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:25.922811 18735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec060e847f024f9fa369cff320e71558. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:25.922941 18988 ts_tablet_manager.cc:1367] T 81787eccca6c42c1b421377a3bf9ebe6 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.020s user 0.011s sys 0.004s
I20240430 20:11:25.923028 18985 tablet_bootstrap.cc:492] T ffe059eeac91476aaeef8c56d2f49ee8 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:25.923249 18985 ts_tablet_manager.cc:1367] T ffe059eeac91476aaeef8c56d2f49ee8 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.058s user 0.007s sys 0.001s
I20240430 20:11:25.923938 18988 raft_consensus.cc:348] T 81787eccca6c42c1b421377a3bf9ebe6 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:25.924036 18985 raft_consensus.cc:348] T ffe059eeac91476aaeef8c56d2f49ee8 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:25.924234 18988 raft_consensus.cc:374] T 81787eccca6c42c1b421377a3bf9ebe6 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:25.924245 18985 raft_consensus.cc:374] T ffe059eeac91476aaeef8c56d2f49ee8 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:25.924324 18988 raft_consensus.cc:729] T 81787eccca6c42c1b421377a3bf9ebe6 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:25.924337 18985 raft_consensus.cc:729] T ffe059eeac91476aaeef8c56d2f49ee8 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:25.924633 18985 consensus_queue.cc:260] T ffe059eeac91476aaeef8c56d2f49ee8 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:25.924633 18988 consensus_queue.cc:260] T 81787eccca6c42c1b421377a3bf9ebe6 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:25.925211 18985 ts_tablet_manager.cc:1398] T ffe059eeac91476aaeef8c56d2f49ee8 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240430 20:11:25.925642 18988 ts_tablet_manager.cc:1398] T 81787eccca6c42c1b421377a3bf9ebe6 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240430 20:11:25.925971 18985 tablet_bootstrap.cc:492] T 81787eccca6c42c1b421377a3bf9ebe6 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:25.926638 18988 tablet_bootstrap.cc:492] T bd7392f3dfd643cdb1634ee10ad41651 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:25.929913 18985 tablet_bootstrap.cc:654] T 81787eccca6c42c1b421377a3bf9ebe6 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:25.929472 18988 tablet_bootstrap.cc:654] T bd7392f3dfd643cdb1634ee10ad41651 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:25.936803 18139 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet 26a015250eeb4750a42df93840e8d47f (table default.Table_2e9c7c683df84a41942d31600a265641 [id=aca6fb02776a40bbac312191adbdc136]) successfully deleted
W20240430 20:11:25.937144 18139 catalog_manager.cc:4810] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): delete failed for tablet 27f94f2d8f50495dababe417a9f82982 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 27f94f2d8f50495dababe417a9f82982 already in progress: creating tablet
W20240430 20:11:25.937403 18139 catalog_manager.cc:4810] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): delete failed for tablet c5fd0730133c472299c93f4b0830e527 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c5fd0730133c472299c93f4b0830e527 already in progress: creating tablet
I20240430 20:11:25.937757 18742 tablet_service.cc:1467] Processing CreateTablet for tablet 34ab40d8d91e401db299ddee7175ba28 (DEFAULT_TABLE table=default.Table_0b4b99c1c03a4e779046063159ce6861 [id=43d7bdc84bab4c54bac58dd1dd0f74ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:25.938210 18742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34ab40d8d91e401db299ddee7175ba28. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:25.943305 18454 tablet_service.cc:1467] Processing CreateTablet for tablet 34ab40d8d91e401db299ddee7175ba28 (DEFAULT_TABLE table=default.Table_0b4b99c1c03a4e779046063159ce6861 [id=43d7bdc84bab4c54bac58dd1dd0f74ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:25.945168 18595 tablet_service.cc:1467] Processing CreateTablet for tablet ec060e847f024f9fa369cff320e71558 (DEFAULT_TABLE table=default.Table_2e9c7c683df84a41942d31600a265641 [id=aca6fb02776a40bbac312191adbdc136]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:25.945536 18482 raft_consensus.cc:2454] T 81328e904ca94e2eb5ec94dd3ffb3c90 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:25.943240 18603 tablet_service.cc:1467] Processing CreateTablet for tablet 0e503a898406412ab977d3e048ea4b5d (DEFAULT_TABLE table=default.Table_fe1773cf77654e2a86e95444437b7201 [id=fe2222c2b62f4360a5ade622ae8b23e5]), partition=RANGE (key) PARTITION UNBOUNDED
W20240430 20:11:25.947863 18139 catalog_manager.cc:4810] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): delete failed for tablet 187a27a6449342f99de3c5a2f93d49fc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 187a27a6449342f99de3c5a2f93d49fc already in progress: creating tablet
I20240430 20:11:25.948666 18603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e503a898406412ab977d3e048ea4b5d. 1 dirs total, 0 dirs full, 0 dirs failed
W20240430 20:11:25.949767 18139 catalog_manager.cc:4810] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): delete failed for tablet 93d158de64994f3c9fa45a1250a4f533 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 93d158de64994f3c9fa45a1250a4f533 already in progress: creating tablet
I20240430 20:11:25.951516 18699 leader_election.cc:302] T 81328e904ca94e2eb5ec94dd3ffb3c90 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters:
I20240430 20:11:25.952003 18595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec060e847f024f9fa369cff320e71558. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:25.954792 19191 raft_consensus.cc:2781] T 81328e904ca94e2eb5ec94dd3ffb3c90 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:25.955505 19191 raft_consensus.cc:686] T 81328e904ca94e2eb5ec94dd3ffb3c90 P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Becoming Leader. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Running, Role: LEADER
I20240430 20:11:25.956032 18454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34ab40d8d91e401db299ddee7175ba28. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:25.956120 19191 consensus_queue.cc:237] T 81328e904ca94e2eb5ec94dd3ffb3c90 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:25.959544 18985 tablet_bootstrap.cc:492] T 81787eccca6c42c1b421377a3bf9ebe6 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:25.959648 18980 tablet_replica.cc:330] T a01ee128ba8a4c9fb8e125ba97239d4c P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:25.959717 18985 ts_tablet_manager.cc:1367] T 81787eccca6c42c1b421377a3bf9ebe6 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.034s user 0.004s sys 0.003s
I20240430 20:11:25.961771 18985 raft_consensus.cc:348] T 81787eccca6c42c1b421377a3bf9ebe6 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:25.962050 18985 raft_consensus.cc:374] T 81787eccca6c42c1b421377a3bf9ebe6 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:25.962149 18985 raft_consensus.cc:729] T 81787eccca6c42c1b421377a3bf9ebe6 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:25.962388 18985 consensus_queue.cc:260] T 81787eccca6c42c1b421377a3bf9ebe6 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:25.963003 18985 ts_tablet_manager.cc:1398] T 81787eccca6c42c1b421377a3bf9ebe6 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20240430 20:11:25.963312 18985 tablet_bootstrap.cc:492] T 3f49eda7aeca42308d01b83a19d7d037 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:25.969224 18598 tablet_service.cc:1467] Processing CreateTablet for tablet 34ab40d8d91e401db299ddee7175ba28 (DEFAULT_TABLE table=default.Table_0b4b99c1c03a4e779046063159ce6861 [id=43d7bdc84bab4c54bac58dd1dd0f74ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:25.971099 18149 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54524:
name: "default.Table_a5743a67104f449d8d26a63b9f85eb8b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240430 20:11:25.971688 18988 tablet_bootstrap.cc:492] T bd7392f3dfd643cdb1634ee10ad41651 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
W20240430 20:11:25.971774 18149 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a5743a67104f449d8d26a63b9f85eb8b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:25.971853 18988 ts_tablet_manager.cc:1367] T bd7392f3dfd643cdb1634ee10ad41651 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.045s user 0.007s sys 0.001s
I20240430 20:11:25.972075 18598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34ab40d8d91e401db299ddee7175ba28. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:25.972692 18988 raft_consensus.cc:348] T bd7392f3dfd643cdb1634ee10ad41651 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:25.972908 18988 raft_consensus.cc:374] T bd7392f3dfd643cdb1634ee10ad41651 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:25.973006 18988 raft_consensus.cc:729] T bd7392f3dfd643cdb1634ee10ad41651 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:25.973253 18988 consensus_queue.cc:260] T bd7392f3dfd643cdb1634ee10ad41651 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:25.973798 18988 ts_tablet_manager.cc:1398] T bd7392f3dfd643cdb1634ee10ad41651 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240430 20:11:25.974115 18988 tablet_bootstrap.cc:492] T ffe059eeac91476aaeef8c56d2f49ee8 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:25.974857 18147 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_529678d6f1444c9294902598ac7c551e" } new_table_name: "default.Table_4d213ee516bb406798377b6b90fa8cc1" modify_external_catalogs: true
I20240430 20:11:25.975328 18150 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_3ad9d7307b80412f87283e58ffb6f290" } new_table_name: "default.Table_19a91a7f17ff456ebb145e8dd89c0940" modify_external_catalogs: true
I20240430 20:11:25.976631 18148 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54524:
name: "default.Table_5d3cb1a25c884804b05b07531e1b6965"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:25.977438 18148 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5d3cb1a25c884804b05b07531e1b6965 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:25.978014 18985 tablet_bootstrap.cc:654] T 3f49eda7aeca42308d01b83a19d7d037 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:25.979897 18742 tablet_service.cc:1514] Processing DeleteTablet for tablet b0e6cc37c6b1488fa88359d16b12a576 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:23 UTC) from {username='slave'} at 127.0.0.1:59818
I20240430 20:11:25.980252 18987 tablet_replica.cc:330] T b0e6cc37c6b1488fa88359d16b12a576 P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:25.980461 18987 raft_consensus.cc:2229] T b0e6cc37c6b1488fa88359d16b12a576 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:25.980629 19180 raft_consensus.cc:482] T 5809e7c9c2204ba293426c24129fb16e P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:25.980685 18987 raft_consensus.cc:2258] T b0e6cc37c6b1488fa88359d16b12a576 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:25.980818 19180 raft_consensus.cc:504] T 5809e7c9c2204ba293426c24129fb16e P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:25.981024 18156 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 81787eccca6c42c1b421377a3bf9ebe6 (table default.Table_b74297eb78cc4912a25657bbf0180340 [id=7f602584cef9486eb7d9b4389be3f43e]). Expected version 1 got 0
I20240430 20:11:25.981734 19180 leader_election.cc:288] T 5809e7c9c2204ba293426c24129fb16e P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:25.982641 18482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5809e7c9c2204ba293426c24129fb16e" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:25.982870 18482 raft_consensus.cc:2454] T 5809e7c9c2204ba293426c24129fb16e P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 0.
I20240430 20:11:25.983358 18558 leader_election.cc:302] T 5809e7c9c2204ba293426c24129fb16e P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:25.983276 18763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5809e7c9c2204ba293426c24129fb16e" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:25.984225 18763 raft_consensus.cc:2454] T 5809e7c9c2204ba293426c24129fb16e P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 0.
I20240430 20:11:25.984251 18987 ts_tablet_manager.cc:1875] T b0e6cc37c6b1488fa88359d16b12a576 P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:25.986883 18988 tablet_bootstrap.cc:654] T ffe059eeac91476aaeef8c56d2f49ee8 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:25.987215 18742 tablet_service.cc:1467] Processing CreateTablet for tablet 5f56ac2afd444d63a6bfd96988464bbb (DEFAULT_TABLE table=default.Table_3ad9d7307b80412f87283e58ffb6f290 [id=67283dd3995a44c6a7a7bc8722e4f955]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:25.987680 18742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f56ac2afd444d63a6bfd96988464bbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:25.989297 19180 raft_consensus.cc:2781] T 5809e7c9c2204ba293426c24129fb16e P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:25.989421 19180 raft_consensus.cc:482] T 5809e7c9c2204ba293426c24129fb16e P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:25.989516 19180 raft_consensus.cc:3037] T 5809e7c9c2204ba293426c24129fb16e P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:25.991959 18597 tablet_service.cc:1467] Processing CreateTablet for tablet 5f56ac2afd444d63a6bfd96988464bbb (DEFAULT_TABLE table=default.Table_3ad9d7307b80412f87283e58ffb6f290 [id=67283dd3995a44c6a7a7bc8722e4f955]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:25.993275 18454 tablet_service.cc:1467] Processing CreateTablet for tablet 5f56ac2afd444d63a6bfd96988464bbb (DEFAULT_TABLE table=default.Table_3ad9d7307b80412f87283e58ffb6f290 [id=67283dd3995a44c6a7a7bc8722e4f955]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:25.993386 18597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f56ac2afd444d63a6bfd96988464bbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:25.994977 18602 tablet_service.cc:1467] Processing CreateTablet for tablet 29974dc2c7dd45b99b6cc0712378fc30 (DEFAULT_TABLE table=default.Table_28b63364f56b484eb48763e662dda489 [id=cc553a8f7052423b9724a3b28c63592d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:25.995400 18602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29974dc2c7dd45b99b6cc0712378fc30. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:26.002228 18151 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet 3bd9890236ea4f3da20e59ee50f14247
I20240430 20:11:26.002324 18744 tablet_service.cc:1467] Processing CreateTablet for tablet 29974dc2c7dd45b99b6cc0712378fc30 (DEFAULT_TABLE table=default.Table_28b63364f56b484eb48763e662dda489 [id=cc553a8f7052423b9724a3b28c63592d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:26.007426 18461 tablet_service.cc:1467] Processing CreateTablet for tablet 29974dc2c7dd45b99b6cc0712378fc30 (DEFAULT_TABLE table=default.Table_28b63364f56b484eb48763e662dda489 [id=cc553a8f7052423b9724a3b28c63592d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:26.008446 18151 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 81787eccca6c42c1b421377a3bf9ebe6 (table default.Table_b74297eb78cc4912a25657bbf0180340 [id=7f602584cef9486eb7d9b4389be3f43e]). Expected version 1 got 0
I20240430 20:11:26.008749 18151 catalog_manager.cc:5484] T 81328e904ca94e2eb5ec94dd3ffb3c90 P 3c51c33d5616482cafb68e26688820f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c51c33d5616482cafb68e26688820f4 (127.17.177.3). New cstate: current_term: 1 leader_uuid: "3c51c33d5616482cafb68e26688820f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } health_report { overall_health: HEALTHY } } }
I20240430 20:11:26.011158 18593 tablet_service.cc:1467] Processing CreateTablet for tablet 77285afcafe04a829e485998bbbe00f0 (DEFAULT_TABLE table=default.Table_ddd446dad29647018f226059e9933e56 [id=040dec3a3d804cdfbdada359bbe5123b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:26.011669 18593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77285afcafe04a829e485998bbbe00f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:26.011561 18738 tablet_service.cc:1467] Processing CreateTablet for tablet 77285afcafe04a829e485998bbbe00f0 (DEFAULT_TABLE table=default.Table_ddd446dad29647018f226059e9933e56 [id=040dec3a3d804cdfbdada359bbe5123b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:26.013691 18459 tablet_service.cc:1467] Processing CreateTablet for tablet 77285afcafe04a829e485998bbbe00f0 (DEFAULT_TABLE table=default.Table_ddd446dad29647018f226059e9933e56 [id=040dec3a3d804cdfbdada359bbe5123b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:26.015344 18591 tablet_service.cc:1467] Processing CreateTablet for tablet d9e578c99058424a9b86390641f9c34f (DEFAULT_TABLE table=default.Table_40857db02a904104a4850e51d5996da2 [id=64a1f78e3f3f425d9c350ced9e169ac5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:26.015928 18591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9e578c99058424a9b86390641f9c34f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:26.016075 18735 tablet_service.cc:1467] Processing CreateTablet for tablet d9e578c99058424a9b86390641f9c34f (DEFAULT_TABLE table=default.Table_40857db02a904104a4850e51d5996da2 [id=64a1f78e3f3f425d9c350ced9e169ac5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:26.017864 18463 tablet_service.cc:1467] Processing CreateTablet for tablet d9e578c99058424a9b86390641f9c34f (DEFAULT_TABLE table=default.Table_40857db02a904104a4850e51d5996da2 [id=64a1f78e3f3f425d9c350ced9e169ac5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:26.021090 18983 tablet_bootstrap.cc:492] T fe01b95104174279a7d64c0d4c0af982 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:26.025568 18983 ts_tablet_manager.cc:1367] T fe01b95104174279a7d64c0d4c0af982 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.386s user 0.009s sys 0.019s
I20240430 20:11:26.027108 18983 raft_consensus.cc:348] T fe01b95104174279a7d64c0d4c0af982 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:26.027348 18983 raft_consensus.cc:374] T fe01b95104174279a7d64c0d4c0af982 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:26.027433 18983 raft_consensus.cc:729] T fe01b95104174279a7d64c0d4c0af982 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:26.028795 18983 consensus_queue.cc:260] T fe01b95104174279a7d64c0d4c0af982 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:26.035557 18985 tablet_bootstrap.cc:492] T 3f49eda7aeca42308d01b83a19d7d037 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:26.037039 18985 ts_tablet_manager.cc:1367] T 3f49eda7aeca42308d01b83a19d7d037 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.074s user 0.014s sys 0.007s
I20240430 20:11:26.037398 18988 tablet_bootstrap.cc:492] T ffe059eeac91476aaeef8c56d2f49ee8 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:26.038784 18988 ts_tablet_manager.cc:1367] T ffe059eeac91476aaeef8c56d2f49ee8 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.065s user 0.005s sys 0.020s
I20240430 20:11:26.039803 18988 raft_consensus.cc:348] T ffe059eeac91476aaeef8c56d2f49ee8 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:26.038404 18985 raft_consensus.cc:348] T 3f49eda7aeca42308d01b83a19d7d037 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:26.040462 18985 raft_consensus.cc:374] T 3f49eda7aeca42308d01b83a19d7d037 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:26.040603 18985 raft_consensus.cc:729] T 3f49eda7aeca42308d01b83a19d7d037 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:26.040879 18154 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_40e4655ab0b449cb8f9688a222e92813" } new_table_name: "default.Table_30f70b4e38614496ae012d519a7338d1" modify_external_catalogs: true
I20240430 20:11:26.041137 18988 raft_consensus.cc:374] T ffe059eeac91476aaeef8c56d2f49ee8 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:26.041304 18985 consensus_queue.cc:260] T 3f49eda7aeca42308d01b83a19d7d037 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:26.041575 18988 raft_consensus.cc:729] T ffe059eeac91476aaeef8c56d2f49ee8 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:26.042678 18988 consensus_queue.cc:260] T ffe059eeac91476aaeef8c56d2f49ee8 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:26.043118 18985 ts_tablet_manager.cc:1398] T 3f49eda7aeca42308d01b83a19d7d037 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240430 20:11:26.044868 18985 tablet_bootstrap.cc:492] T 187a27a6449342f99de3c5a2f93d49fc P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:26.046506 18988 ts_tablet_manager.cc:1398] T ffe059eeac91476aaeef8c56d2f49ee8 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240430 20:11:26.046867 18623 raft_consensus.cc:2454] T 81328e904ca94e2eb5ec94dd3ffb3c90 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:26.047950 18988 tablet_bootstrap.cc:492] T 3f49eda7aeca42308d01b83a19d7d037 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:26.051844 18988 tablet_bootstrap.cc:654] T 3f49eda7aeca42308d01b83a19d7d037 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:26.051810 18981 ts_tablet_manager.cc:1888] T 9750608faeab4be38ddce4e2e4d46964 P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:26.051970 18981 log.cc:1196] T 9750608faeab4be38ddce4e2e4d46964 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/9750608faeab4be38ddce4e2e4d46964
I20240430 20:11:26.052384 18981 ts_tablet_manager.cc:1909] T 9750608faeab4be38ddce4e2e4d46964 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:26.052989 18454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f56ac2afd444d63a6bfd96988464bbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:26.053568 18461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29974dc2c7dd45b99b6cc0712378fc30. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:26.054136 18459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77285afcafe04a829e485998bbbe00f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:26.054733 18463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9e578c99058424a9b86390641f9c34f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:26.054912 18985 tablet_bootstrap.cc:654] T 187a27a6449342f99de3c5a2f93d49fc P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:26.056370 19186 raft_consensus.cc:482] T b0e6cc37c6b1488fa88359d16b12a576 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:26.056618 19186 raft_consensus.cc:504] T b0e6cc37c6b1488fa88359d16b12a576 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:26.057543 19186 leader_election.cc:288] T b0e6cc37c6b1488fa88359d16b12a576 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:26.058576 18763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0e6cc37c6b1488fa88359d16b12a576" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:26.059903 18623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0e6cc37c6b1488fa88359d16b12a576" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:26.061097 18988 tablet_bootstrap.cc:492] T 3f49eda7aeca42308d01b83a19d7d037 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
W20240430 20:11:26.061249 18417 leader_election.cc:341] T b0e6cc37c6b1488fa88359d16b12a576 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: Tablet not RUNNING: STOPPED
I20240430 20:11:26.061502 18988 ts_tablet_manager.cc:1367] T 3f49eda7aeca42308d01b83a19d7d037 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.005s
W20240430 20:11:26.061630 18415 leader_election.cc:341] T b0e6cc37c6b1488fa88359d16b12a576 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:26.061761 18415 leader_election.cc:302] T b0e6cc37c6b1488fa88359d16b12a576 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a; no voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710
I20240430 20:11:26.062369 18988 raft_consensus.cc:348] T 3f49eda7aeca42308d01b83a19d7d037 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:26.062610 18988 raft_consensus.cc:374] T 3f49eda7aeca42308d01b83a19d7d037 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:26.069903 19186 raft_consensus.cc:2726] T b0e6cc37c6b1488fa88359d16b12a576 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:26.074787 18988 raft_consensus.cc:729] T 3f49eda7aeca42308d01b83a19d7d037 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:26.075199 18988 consensus_queue.cc:260] T 3f49eda7aeca42308d01b83a19d7d037 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:26.079690 18155 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54524:
name: "default.Table_54f6ac44723049d3bf4d7cd8c313fe44"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:26.080525 18155 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_54f6ac44723049d3bf4d7cd8c313fe44 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:26.081153 18988 ts_tablet_manager.cc:1398] T 3f49eda7aeca42308d01b83a19d7d037 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.020s user 0.002s sys 0.000s
I20240430 20:11:26.082067 18987 ts_tablet_manager.cc:1888] T b0e6cc37c6b1488fa88359d16b12a576 P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:26.082226 18987 log.cc:1196] T b0e6cc37c6b1488fa88359d16b12a576 P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/b0e6cc37c6b1488fa88359d16b12a576
I20240430 20:11:26.082677 18987 ts_tablet_manager.cc:1909] T b0e6cc37c6b1488fa88359d16b12a576 P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:26.087035 18988 tablet_bootstrap.cc:492] T 187a27a6449342f99de3c5a2f93d49fc P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:26.088840 18988 tablet_bootstrap.cc:654] T 187a27a6449342f99de3c5a2f93d49fc P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:26.090319 18744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29974dc2c7dd45b99b6cc0712378fc30. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:26.090950 18738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77285afcafe04a829e485998bbbe00f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:26.091326 18735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9e578c99058424a9b86390641f9c34f. 1 dirs total, 0 dirs full, 0 dirs failed
W20240430 20:11:26.094767 18146 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 81787eccca6c42c1b421377a3bf9ebe6 consensus state has no leader
I20240430 20:11:26.098549 18452 tablet_service.cc:1514] Processing DeleteTablet for tablet fe01b95104174279a7d64c0d4c0af982 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:23 UTC) from {username='slave'} at 127.0.0.1:41678
I20240430 20:11:26.098924 18455 tablet_service.cc:1514] Processing DeleteTablet for tablet bd7392f3dfd643cdb1634ee10ad41651 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:24 UTC) from {username='slave'} at 127.0.0.1:41678
I20240430 20:11:26.099591 18455 tablet_service.cc:1514] Processing DeleteTablet for tablet b4bc111e183f40a0acc9dd377195f6d9 with delete_type TABLET_DATA_DELETED (replaced by tablet 0b4d9e860d7c45fd91d7a566605c9283) from {username='slave'} at 127.0.0.1:41678
I20240430 20:11:26.099777 18452 tablet_service.cc:1514] Processing DeleteTablet for tablet ffe059eeac91476aaeef8c56d2f49ee8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:24 UTC) from {username='slave'} at 127.0.0.1:41678
I20240430 20:11:26.109426 18146 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54524:
name: "default.Table_6213f980d6a44e099a7a989ceb23c08a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:26.110321 18146 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6213f980d6a44e099a7a989ceb23c08a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:26.140120 18152 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_40857db02a904104a4850e51d5996da2" } modify_external_catalogs: true
I20240430 20:11:26.140498 18152 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_40857db02a904104a4850e51d5996da2" } modify_external_catalogs: true
I20240430 20:11:26.143721 18141 catalog_manager.cc:4830] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): tablet b0e6cc37c6b1488fa88359d16b12a576 (table default.Table_32647411b85344b490966d64f135631b [id=8cd243a627f84916902191f774d1e8cc]) successfully deleted
I20240430 20:11:26.145267 18592 tablet_service.cc:1514] Processing DeleteTablet for tablet fe01b95104174279a7d64c0d4c0af982 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:23 UTC) from {username='slave'} at 127.0.0.1:40202
I20240430 20:11:26.146050 18981 tablet_replica.cc:330] T 6f729a6e8dc14de9a2057cfe09ba9257 P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:26.146404 18981 raft_consensus.cc:2229] T 6f729a6e8dc14de9a2057cfe09ba9257 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:26.154780 18621 raft_consensus.cc:2454] T a01ee128ba8a4c9fb8e125ba97239d4c P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 1.
I20240430 20:11:26.157999 18980 raft_consensus.cc:2229] T a01ee128ba8a4c9fb8e125ba97239d4c P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:26.158514 18980 raft_consensus.cc:2258] T a01ee128ba8a4c9fb8e125ba97239d4c P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:26.159319 18980 ts_tablet_manager.cc:1875] T a01ee128ba8a4c9fb8e125ba97239d4c P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:26.163479 18415 leader_election.cc:302] T a01ee128ba8a4c9fb8e125ba97239d4c P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters: 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:26.164268 19186 raft_consensus.cc:2781] T a01ee128ba8a4c9fb8e125ba97239d4c P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:26.164484 19186 raft_consensus.cc:686] T a01ee128ba8a4c9fb8e125ba97239d4c P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Becoming Leader. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Running, Role: LEADER
I20240430 20:11:26.164885 19186 consensus_queue.cc:237] T a01ee128ba8a4c9fb8e125ba97239d4c P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:26.166523 19185 raft_consensus.cc:482] T bd7392f3dfd643cdb1634ee10ad41651 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:26.166776 19185 raft_consensus.cc:504] T bd7392f3dfd643cdb1634ee10ad41651 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:26.167680 19185 leader_election.cc:288] T bd7392f3dfd643cdb1634ee10ad41651 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:26.170143 18621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd7392f3dfd643cdb1634ee10ad41651" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:26.172281 18762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd7392f3dfd643cdb1634ee10ad41651" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:26.172643 18762 raft_consensus.cc:2454] T bd7392f3dfd643cdb1634ee10ad41651 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 0.
I20240430 20:11:26.172931 18149 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54524:
name: "default.Table_fd3ecbe6c2cc4148a41593f6d47cf1ec"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:26.173805 18149 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fd3ecbe6c2cc4148a41593f6d47cf1ec in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240430 20:11:26.174033 18415 leader_election.cc:341] T bd7392f3dfd643cdb1634ee10ad41651 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:26.174448 18417 leader_election.cc:302] T bd7392f3dfd643cdb1634ee10ad41651 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters: 5e1c6397f09747f091534faf93542710
I20240430 20:11:26.175105 19185 raft_consensus.cc:2781] T bd7392f3dfd643cdb1634ee10ad41651 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:26.175272 19185 raft_consensus.cc:482] T bd7392f3dfd643cdb1634ee10ad41651 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:26.175383 19185 raft_consensus.cc:3037] T bd7392f3dfd643cdb1634ee10ad41651 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:26.146646 18981 raft_consensus.cc:2258] T 6f729a6e8dc14de9a2057cfe09ba9257 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:26.180470 18981 ts_tablet_manager.cc:1875] T 6f729a6e8dc14de9a2057cfe09ba9257 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240430 20:11:26.186491 18147 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c66c142419ca4e1399ecd1c8d1e62a59 has no consensus state
I20240430 20:11:26.192339 18142 catalog_manager.cc:4830] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): tablet 9750608faeab4be38ddce4e2e4d46964 (table default.Table_c06b54647f004ff7a7f7beafee702b4c [id=a6a27dcbd0e648a8bf29a8cd5b3a2a11]) successfully deleted
I20240430 20:11:26.194365 18147 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet e6c0c808bfc94be3837ab835292b3c1b (table default.Table_5fc77d5f60ae44f4bb4ca9f7f183672c [id=6ca223ca6354449c959d85ed2ac6a9c8]). Expected version 1 got 0
I20240430 20:11:26.195637 18147 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54524:
name: "default.Table_85072f20d90d4470bf3f65189b19fd51"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:26.196473 18147 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_85072f20d90d4470bf3f65189b19fd51 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:26.206583 18983 ts_tablet_manager.cc:1398] T fe01b95104174279a7d64c0d4c0af982 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.181s user 0.001s sys 0.005s
I20240430 20:11:26.209211 19235 tablet.cc:1722] T e6c0c808bfc94be3837ab835292b3c1b P 2efb542aa55e439b99ccb98d1c5c884a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240430 20:11:26.212298 19234 tablet.cc:1722] T e6c0c808bfc94be3837ab835292b3c1b P 5e1c6397f09747f091534faf93542710: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20240430 20:11:26.214475 18150 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5f56ac2afd444d63a6bfd96988464bbb consensus state has no leader
I20240430 20:11:26.224412 18150 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_14776edb17d64a83be63345e68995156" } modify_external_catalogs: true
I20240430 20:11:26.224720 18150 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_14776edb17d64a83be63345e68995156" } modify_external_catalogs: true
I20240430 20:11:26.225569 18983 tablet_bootstrap.cc:492] T b0e6cc37c6b1488fa88359d16b12a576 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:26.232048 18983 tablet_bootstrap.cc:654] T b0e6cc37c6b1488fa88359d16b12a576 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:26.234573 18988 tablet_bootstrap.cc:492] T 187a27a6449342f99de3c5a2f93d49fc P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:26.234640 18983 tablet_bootstrap.cc:492] T b0e6cc37c6b1488fa88359d16b12a576 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:26.234803 18983 ts_tablet_manager.cc:1367] T b0e6cc37c6b1488fa88359d16b12a576 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20240430 20:11:26.234823 18988 ts_tablet_manager.cc:1367] T 187a27a6449342f99de3c5a2f93d49fc P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.148s user 0.053s sys 0.055s
I20240430 20:11:26.235610 18983 raft_consensus.cc:348] T b0e6cc37c6b1488fa88359d16b12a576 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:26.235586 18988 raft_consensus.cc:348] T 187a27a6449342f99de3c5a2f93d49fc P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:26.235807 18983 raft_consensus.cc:374] T b0e6cc37c6b1488fa88359d16b12a576 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:26.235841 18988 raft_consensus.cc:374] T 187a27a6449342f99de3c5a2f93d49fc P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:26.235919 18988 raft_consensus.cc:729] T 187a27a6449342f99de3c5a2f93d49fc P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:26.235901 18983 raft_consensus.cc:729] T b0e6cc37c6b1488fa88359d16b12a576 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:26.236202 18983 consensus_queue.cc:260] T b0e6cc37c6b1488fa88359d16b12a576 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:26.237412 18148 master_service.cc:946] ReplaceTablet: received request to replace tablet a01ee128ba8a4c9fb8e125ba97239d4c from {username='slave'} at 127.0.0.1:54524
I20240430 20:11:26.237977 18983 ts_tablet_manager.cc:1398] T b0e6cc37c6b1488fa88359d16b12a576 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240430 20:11:26.238293 18983 tablet_bootstrap.cc:492] T bd7392f3dfd643cdb1634ee10ad41651 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:26.238994 18988 consensus_queue.cc:260] T 187a27a6449342f99de3c5a2f93d49fc P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:26.239614 18988 ts_tablet_manager.cc:1398] T 187a27a6449342f99de3c5a2f93d49fc P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240430 20:11:26.239953 18988 tablet_bootstrap.cc:492] T b4bc111e183f40a0acc9dd377195f6d9 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:26.241688 18988 tablet_bootstrap.cc:654] T b4bc111e183f40a0acc9dd377195f6d9 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:26.250022 18983 tablet_bootstrap.cc:654] T bd7392f3dfd643cdb1634ee10ad41651 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:26.257478 18151 master_service.cc:946] ReplaceTablet: received request to replace tablet a01ee128ba8a4c9fb8e125ba97239d4c from {username='slave'} at 127.0.0.1:54524
I20240430 20:11:26.259779 18744 tablet_service.cc:1514] Processing DeleteTablet for tablet b0e6cc37c6b1488fa88359d16b12a576 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:23 UTC) from {username='slave'} at 127.0.0.1:59818
I20240430 20:11:26.259949 19180 raft_consensus.cc:504] T 5809e7c9c2204ba293426c24129fb16e P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:26.260095 18735 tablet_service.cc:1514] Processing DeleteTablet for tablet a01ee128ba8a4c9fb8e125ba97239d4c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:24 UTC) from {username='slave'} at 127.0.0.1:59818
I20240430 20:11:26.260270 18738 tablet_service.cc:1514] Processing DeleteTablet for tablet bd7392f3dfd643cdb1634ee10ad41651 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:24 UTC) from {username='slave'} at 127.0.0.1:59818
I20240430 20:11:26.260649 18987 tablet_replica.cc:330] T a01ee128ba8a4c9fb8e125ba97239d4c P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:26.260730 19180 leader_election.cc:288] T 5809e7c9c2204ba293426c24129fb16e P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Requested vote from peers 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:26.260875 18987 raft_consensus.cc:2229] T a01ee128ba8a4c9fb8e125ba97239d4c P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:26.261060 18987 raft_consensus.cc:2258] T a01ee128ba8a4c9fb8e125ba97239d4c P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:26.262681 18764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5809e7c9c2204ba293426c24129fb16e" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:26.262909 18764 raft_consensus.cc:3037] T 5809e7c9c2204ba293426c24129fb16e P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:26.264459 18987 ts_tablet_manager.cc:1875] T a01ee128ba8a4c9fb8e125ba97239d4c P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:26.264626 19185 raft_consensus.cc:504] T bd7392f3dfd643cdb1634ee10ad41651 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:26.265475 19185 leader_election.cc:288] T bd7392f3dfd643cdb1634ee10ad41651 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Requested vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
W20240430 20:11:26.266789 18141 catalog_manager.cc:4797] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): delete failed for tablet b0e6cc37c6b1488fa88359d16b12a576 because the tablet was not found. No further retry: Not found: Tablet not found: b0e6cc37c6b1488fa88359d16b12a576
I20240430 20:11:26.268052 18592 tablet_service.cc:1514] Processing DeleteTablet for tablet b0e6cc37c6b1488fa88359d16b12a576 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:23 UTC) from {username='slave'} at 127.0.0.1:40202
I20240430 20:11:26.272157 18621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd7392f3dfd643cdb1634ee10ad41651" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
W20240430 20:11:26.272857 18415 leader_election.cc:341] T bd7392f3dfd643cdb1634ee10ad41651 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:26.273435 18763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd7392f3dfd643cdb1634ee10ad41651" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:26.273648 18763 raft_consensus.cc:3037] T bd7392f3dfd643cdb1634ee10ad41651 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:26.275051 18482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5809e7c9c2204ba293426c24129fb16e" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:26.275336 18482 raft_consensus.cc:3037] T 5809e7c9c2204ba293426c24129fb16e P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:26.306308 19182 catalog_manager.cc:1259] Loaded cluster ID: 889e0faede1f41d1ab74633d6826c0d4
I20240430 20:11:26.308727 18985 tablet_bootstrap.cc:492] T 187a27a6449342f99de3c5a2f93d49fc P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
W20240430 20:11:26.309026 18154 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet aacabead3c874e77a47d2ecf19dbfa82 consensus state has no leader
I20240430 20:11:26.309099 18985 ts_tablet_manager.cc:1367] T 187a27a6449342f99de3c5a2f93d49fc P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.264s user 0.045s sys 0.071s
I20240430 20:11:26.310281 18155 master_service.cc:946] ReplaceTablet: received request to replace tablet cace1398492f4d968dcb6899385d527c from {username='slave'} at 127.0.0.1:54524
I20240430 20:11:26.310626 18985 raft_consensus.cc:348] T 187a27a6449342f99de3c5a2f93d49fc P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:26.310870 18985 raft_consensus.cc:374] T 187a27a6449342f99de3c5a2f93d49fc P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:26.311194 19185 raft_consensus.cc:482] T fe01b95104174279a7d64c0d4c0af982 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:26.311401 19185 raft_consensus.cc:504] T fe01b95104174279a7d64c0d4c0af982 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:26.312296 19185 leader_election.cc:288] T fe01b95104174279a7d64c0d4c0af982 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:26.312934 18621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe01b95104174279a7d64c0d4c0af982" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:26.313230 18621 raft_consensus.cc:2454] T fe01b95104174279a7d64c0d4c0af982 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 0.
W20240430 20:11:26.314465 18417 leader_election.cc:341] T fe01b95104174279a7d64c0d4c0af982 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Not found: Tablet not found: fe01b95104174279a7d64c0d4c0af982
I20240430 20:11:26.315058 18415 leader_election.cc:302] T fe01b95104174279a7d64c0d4c0af982 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters: 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:26.315598 19185 raft_consensus.cc:2781] T fe01b95104174279a7d64c0d4c0af982 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:26.315735 19185 raft_consensus.cc:482] T fe01b95104174279a7d64c0d4c0af982 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:26.315835 19185 raft_consensus.cc:3037] T fe01b95104174279a7d64c0d4c0af982 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:26.310971 18985 raft_consensus.cc:729] T 187a27a6449342f99de3c5a2f93d49fc P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:26.320045 18985 consensus_queue.cc:260] T 187a27a6449342f99de3c5a2f93d49fc P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:26.323550 18146 master_service.cc:946] ReplaceTablet: received request to replace tablet 17843ea77664447eaaa19666e0445ada from {username='slave'} at 127.0.0.1:54524
I20240430 20:11:26.324055 18152 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: Sending DeleteTablet for 3 replicas of tablet d9e578c99058424a9b86390641f9c34f
I20240430 20:11:26.326243 18598 tablet_service.cc:1514] Processing DeleteTablet for tablet d9e578c99058424a9b86390641f9c34f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:26 UTC) from {username='slave'} at 127.0.0.1:40202
I20240430 20:11:26.326259 18738 tablet_service.cc:1514] Processing DeleteTablet for tablet d9e578c99058424a9b86390641f9c34f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:26 UTC) from {username='slave'} at 127.0.0.1:59818
I20240430 20:11:26.326367 18985 ts_tablet_manager.cc:1398] T 187a27a6449342f99de3c5a2f93d49fc P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.017s user 0.000s sys 0.003s
I20240430 20:11:26.326557 19182 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: loading cluster ID for follower catalog manager: success
I20240430 20:11:26.327124 18463 tablet_service.cc:1514] Processing DeleteTablet for tablet d9e578c99058424a9b86390641f9c34f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:26 UTC) from {username='slave'} at 127.0.0.1:41678
I20240430 20:11:26.330785 18152 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54524:
name: "default.Table_4bce1dae3391497ebea5f1b536700d62"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:26.331450 18152 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4bce1dae3391497ebea5f1b536700d62 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:26.336105 18149 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_fcab4e9721f7470abc1df37d83ea299f" } modify_external_catalogs: true
I20240430 20:11:26.336321 18149 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_fcab4e9721f7470abc1df37d83ea299f" } modify_external_catalogs: true
I20240430 20:11:26.337109 18154 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_3acf40c6d58741d4a8e94ca96d642249" } modify_external_catalogs: true
I20240430 20:11:26.337294 18154 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_3acf40c6d58741d4a8e94ca96d642249" } modify_external_catalogs: true
I20240430 20:11:26.338351 18151 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54524:
name: "default.Table_1dc3c3ef217d4f8b9e8fdb60180c5d0f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:26.338915 18151 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1dc3c3ef217d4f8b9e8fdb60180c5d0f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:26.339686 18985 tablet_bootstrap.cc:492] T 609a1a7d22e84b69bd0a53577b62b69b P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:26.342206 18147 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_28b63364f56b484eb48763e662dda489" } modify_external_catalogs: true
I20240430 20:11:26.342406 18147 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_28b63364f56b484eb48763e662dda489" } modify_external_catalogs: true
I20240430 20:11:26.344276 18988 tablet_bootstrap.cc:492] T b4bc111e183f40a0acc9dd377195f6d9 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:26.344442 18988 ts_tablet_manager.cc:1367] T b4bc111e183f40a0acc9dd377195f6d9 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.105s user 0.016s sys 0.044s
I20240430 20:11:26.345263 18988 raft_consensus.cc:348] T b4bc111e183f40a0acc9dd377195f6d9 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:26.345489 18988 raft_consensus.cc:374] T b4bc111e183f40a0acc9dd377195f6d9 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:26.345605 18988 raft_consensus.cc:729] T b4bc111e183f40a0acc9dd377195f6d9 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:26.345875 18988 consensus_queue.cc:260] T b4bc111e183f40a0acc9dd377195f6d9 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:26.346454 18988 ts_tablet_manager.cc:1398] T b4bc111e183f40a0acc9dd377195f6d9 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240430 20:11:26.348836 18985 tablet_bootstrap.cc:654] T 609a1a7d22e84b69bd0a53577b62b69b P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:26.349200 18988 tablet_bootstrap.cc:492] T 0879b02a0bcc4f619fb34cff7b9d7c4c P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:26.350615 18150 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: Sending DeleteTablet for 3 replicas of tablet fd47db5ca8b24c2b827df0a64e195866
I20240430 20:11:26.351176 18988 tablet_bootstrap.cc:654] T 0879b02a0bcc4f619fb34cff7b9d7c4c P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:26.351692 18146 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: Sending DeleteTablet for 3 replicas of tablet 17843ea77664447eaaa19666e0445ada
I20240430 20:11:26.352178 18146 catalog_manager.cc:6554] ReplaceTablet: tablet 17843ea77664447eaaa19666e0445ada deleted and replaced by tablet b72fe1f0713b4076a81a3fc1c74f7fa7
I20240430 20:11:26.352392 19240 tablet.cc:1722] T e6c0c808bfc94be3837ab835292b3c1b P 3c51c33d5616482cafb68e26688820f4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240430 20:11:26.354820 18764 raft_consensus.cc:2454] T 5809e7c9c2204ba293426c24129fb16e P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
I20240430 20:11:26.357635 18763 raft_consensus.cc:2454] T bd7392f3dfd643cdb1634ee10ad41651 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 1.
I20240430 20:11:26.358147 18557 leader_election.cc:302] T 5809e7c9c2204ba293426c24129fb16e P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:26.358585 18417 leader_election.cc:302] T bd7392f3dfd643cdb1634ee10ad41651 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters: 5e1c6397f09747f091534faf93542710
I20240430 20:11:26.359369 19186 raft_consensus.cc:2781] T bd7392f3dfd643cdb1634ee10ad41651 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:26.359504 18482 raft_consensus.cc:2454] T 5809e7c9c2204ba293426c24129fb16e P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
I20240430 20:11:26.359532 19186 raft_consensus.cc:686] T bd7392f3dfd643cdb1634ee10ad41651 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Becoming Leader. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Running, Role: LEADER
I20240430 20:11:26.359814 19180 raft_consensus.cc:2781] T 5809e7c9c2204ba293426c24129fb16e P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:26.359833 19186 consensus_queue.cc:237] T bd7392f3dfd643cdb1634ee10ad41651 P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:26.359977 19180 raft_consensus.cc:686] T 5809e7c9c2204ba293426c24129fb16e P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Becoming Leader. State: Replica: 5e1c6397f09747f091534faf93542710, State: Running, Role: LEADER
I20240430 20:11:26.360234 19180 consensus_queue.cc:237] T 5809e7c9c2204ba293426c24129fb16e P 5e1c6397f09747f091534faf93542710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:26.360702 18598 tablet_service.cc:1514] Processing DeleteTablet for tablet fd47db5ca8b24c2b827df0a64e195866 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:26 UTC) from {username='slave'} at 127.0.0.1:40202
I20240430 20:11:26.360922 18603 tablet_service.cc:1514] Processing DeleteTablet for tablet 17843ea77664447eaaa19666e0445ada with delete_type TABLET_DATA_DELETED (replaced by tablet b72fe1f0713b4076a81a3fc1c74f7fa7) from {username='slave'} at 127.0.0.1:40202
I20240430 20:11:26.361541 18152 master_service.cc:946] ReplaceTablet: received request to replace tablet fe01b95104174279a7d64c0d4c0af982 from {username='slave'} at 127.0.0.1:54524
I20240430 20:11:26.361995 18738 tablet_service.cc:1514] Processing DeleteTablet for tablet fd47db5ca8b24c2b827df0a64e195866 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:26 UTC) from {username='slave'} at 127.0.0.1:59818
I20240430 20:11:26.362206 18735 tablet_service.cc:1514] Processing DeleteTablet for tablet 17843ea77664447eaaa19666e0445ada with delete_type TABLET_DATA_DELETED (replaced by tablet b72fe1f0713b4076a81a3fc1c74f7fa7) from {username='slave'} at 127.0.0.1:59818
I20240430 20:11:26.363162 18463 tablet_service.cc:1514] Processing DeleteTablet for tablet fd47db5ca8b24c2b827df0a64e195866 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:26 UTC) from {username='slave'} at 127.0.0.1:41678
I20240430 20:11:26.363617 18461 tablet_service.cc:1514] Processing DeleteTablet for tablet 17843ea77664447eaaa19666e0445ada with delete_type TABLET_DATA_DELETED (replaced by tablet b72fe1f0713b4076a81a3fc1c74f7fa7) from {username='slave'} at 127.0.0.1:41678
I20240430 20:11:26.374243 18980 ts_tablet_manager.cc:1888] T a01ee128ba8a4c9fb8e125ba97239d4c P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:26.374423 18980 log.cc:1196] T a01ee128ba8a4c9fb8e125ba97239d4c P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/a01ee128ba8a4c9fb8e125ba97239d4c
I20240430 20:11:26.375053 18983 tablet_bootstrap.cc:492] T bd7392f3dfd643cdb1634ee10ad41651 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:26.375214 18983 ts_tablet_manager.cc:1367] T bd7392f3dfd643cdb1634ee10ad41651 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.137s user 0.020s sys 0.014s
I20240430 20:11:26.376009 18983 raft_consensus.cc:348] T bd7392f3dfd643cdb1634ee10ad41651 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:26.376235 18983 raft_consensus.cc:374] T bd7392f3dfd643cdb1634ee10ad41651 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:26.377565 18980 ts_tablet_manager.cc:1909] T a01ee128ba8a4c9fb8e125ba97239d4c P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:26.377660 18983 raft_consensus.cc:729] T bd7392f3dfd643cdb1634ee10ad41651 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:26.377969 18983 consensus_queue.cc:260] T bd7392f3dfd643cdb1634ee10ad41651 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:26.378198 18152 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_085ba776e84d4d66bc187679878700cf" } new_table_name: "default.Table_e68dc52988f24f1788517de94d7d2a0e" modify_external_catalogs: true
I20240430 20:11:26.378650 18983 ts_tablet_manager.cc:1398] T bd7392f3dfd643cdb1634ee10ad41651 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240430 20:11:26.378966 18983 tablet_bootstrap.cc:492] T 81787eccca6c42c1b421377a3bf9ebe6 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:26.380606 18983 tablet_bootstrap.cc:654] T 81787eccca6c42c1b421377a3bf9ebe6 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:26.380828 18150 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54524:
name: "default.Table_388d774462294cab8342c6de331f9712"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:26.381567 18150 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_388d774462294cab8342c6de331f9712 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:26.383607 18146 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_a67165ab67884e57955b604826a006d6" } new_table_name: "default.Table_31d8d978e96c4e12a0fbd5500c7b93cd" modify_external_catalogs: true
I20240430 20:11:26.392120 18983 tablet_bootstrap.cc:492] T 81787eccca6c42c1b421377a3bf9ebe6 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:26.392315 18983 ts_tablet_manager.cc:1367] T 81787eccca6c42c1b421377a3bf9ebe6 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.000s
I20240430 20:11:26.393682 18983 raft_consensus.cc:348] T 81787eccca6c42c1b421377a3bf9ebe6 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:26.393952 18983 raft_consensus.cc:374] T 81787eccca6c42c1b421377a3bf9ebe6 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:26.394043 18983 raft_consensus.cc:729] T 81787eccca6c42c1b421377a3bf9ebe6 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:26.394260 18983 consensus_queue.cc:260] T 81787eccca6c42c1b421377a3bf9ebe6 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:26.394843 18983 ts_tablet_manager.cc:1398] T 81787eccca6c42c1b421377a3bf9ebe6 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240430 20:11:26.395169 18983 tablet_bootstrap.cc:492] T ffe059eeac91476aaeef8c56d2f49ee8 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:26.396992 18983 tablet_bootstrap.cc:654] T ffe059eeac91476aaeef8c56d2f49ee8 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:26.399197 18983 tablet_bootstrap.cc:492] T ffe059eeac91476aaeef8c56d2f49ee8 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:26.399248 18985 tablet_bootstrap.cc:492] T 609a1a7d22e84b69bd0a53577b62b69b P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:26.399372 18983 ts_tablet_manager.cc:1367] T ffe059eeac91476aaeef8c56d2f49ee8 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240430 20:11:26.399405 18985 ts_tablet_manager.cc:1367] T 609a1a7d22e84b69bd0a53577b62b69b P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.060s user 0.001s sys 0.002s
I20240430 20:11:26.400172 18983 raft_consensus.cc:348] T ffe059eeac91476aaeef8c56d2f49ee8 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:26.400200 18985 raft_consensus.cc:348] T 609a1a7d22e84b69bd0a53577b62b69b P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:26.400393 18983 raft_consensus.cc:374] T ffe059eeac91476aaeef8c56d2f49ee8 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:26.400401 18985 raft_consensus.cc:374] T 609a1a7d22e84b69bd0a53577b62b69b P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:26.400475 18983 raft_consensus.cc:729] T ffe059eeac91476aaeef8c56d2f49ee8 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:26.400487 18985 raft_consensus.cc:729] T 609a1a7d22e84b69bd0a53577b62b69b P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:26.400734 18983 consensus_queue.cc:260] T ffe059eeac91476aaeef8c56d2f49ee8 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:26.401239 18983 ts_tablet_manager.cc:1398] T ffe059eeac91476aaeef8c56d2f49ee8 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240430 20:11:26.401496 18983 tablet_bootstrap.cc:492] T b4bc111e183f40a0acc9dd377195f6d9 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:26.403070 18983 tablet_bootstrap.cc:654] T b4bc111e183f40a0acc9dd377195f6d9 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:26.406924 18983 tablet_bootstrap.cc:492] T b4bc111e183f40a0acc9dd377195f6d9 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:26.407080 18983 ts_tablet_manager.cc:1367] T b4bc111e183f40a0acc9dd377195f6d9 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240430 20:11:26.407871 18983 raft_consensus.cc:348] T b4bc111e183f40a0acc9dd377195f6d9 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:26.408093 18983 raft_consensus.cc:374] T b4bc111e183f40a0acc9dd377195f6d9 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:26.408177 18983 raft_consensus.cc:729] T b4bc111e183f40a0acc9dd377195f6d9 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:26.408449 18983 consensus_queue.cc:260] T b4bc111e183f40a0acc9dd377195f6d9 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:26.409044 18983 ts_tablet_manager.cc:1398] T b4bc111e183f40a0acc9dd377195f6d9 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240430 20:11:26.409955 18983 tablet_bootstrap.cc:492] T 3f49eda7aeca42308d01b83a19d7d037 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:26.410384 18988 tablet_bootstrap.cc:492] T 0879b02a0bcc4f619fb34cff7b9d7c4c P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:26.410570 18988 ts_tablet_manager.cc:1367] T 0879b02a0bcc4f619fb34cff7b9d7c4c P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.062s user 0.011s sys 0.010s
I20240430 20:11:26.411399 18988 raft_consensus.cc:348] T 0879b02a0bcc4f619fb34cff7b9d7c4c P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:26.411621 18983 tablet_bootstrap.cc:654] T 3f49eda7aeca42308d01b83a19d7d037 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:26.411614 18988 raft_consensus.cc:374] T 0879b02a0bcc4f619fb34cff7b9d7c4c P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:26.411695 18988 raft_consensus.cc:729] T 0879b02a0bcc4f619fb34cff7b9d7c4c P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:26.411986 18988 consensus_queue.cc:260] T 0879b02a0bcc4f619fb34cff7b9d7c4c P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:26.412575 18988 ts_tablet_manager.cc:1398] T 0879b02a0bcc4f619fb34cff7b9d7c4c P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240430 20:11:26.415663 18735 tablet_service.cc:1467] Processing CreateTablet for tablet d4508e4f6dbd42109d951c648b3dc7df (DEFAULT_TABLE table=default.Table_f136f0a535fc4860811d81fb1d439f3c [id=f146a15e6e8a4acbb4c734c1c863da57]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:26.415719 18603 tablet_service.cc:1467] Processing CreateTablet for tablet d4508e4f6dbd42109d951c648b3dc7df (DEFAULT_TABLE table=default.Table_f136f0a535fc4860811d81fb1d439f3c [id=f146a15e6e8a4acbb4c734c1c863da57]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:26.416834 18598 tablet_service.cc:1467] Processing CreateTablet for tablet dd6083f12c7e4d1e8a2006b3ddf9aa3a (DEFAULT_TABLE table=default.Table_276f62c2338b46a0aab8905430e8b61b [id=789b538b500642d89721e6028b9ac48e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:26.416129 18461 tablet_service.cc:1467] Processing CreateTablet for tablet d4508e4f6dbd42109d951c648b3dc7df (DEFAULT_TABLE table=default.Table_f136f0a535fc4860811d81fb1d439f3c [id=f146a15e6e8a4acbb4c734c1c863da57]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:26.418864 18988 tablet_bootstrap.cc:492] T a0e8e07d363f48c698c7e2b17edc1545 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:26.408661 18985 consensus_queue.cc:260] T 609a1a7d22e84b69bd0a53577b62b69b P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:26.420348 19182 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: acquiring CA information for follower catalog manager: success
W20240430 20:11:26.421234 18152 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet aec9c09cc0454acab6006bb3e10982b7 consensus state has no leader
I20240430 20:11:26.421633 18152 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_ddd446dad29647018f226059e9933e56" } new_table_name: "default.Table_e63f64a411094f8db2ef641f2257ca79" modify_external_catalogs: true
I20240430 20:11:26.422562 18988 tablet_bootstrap.cc:654] T a0e8e07d363f48c698c7e2b17edc1545 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:26.423415 18156 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54524:
name: "default.Table_3106434588bd4241a314064d6a8749c3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:26.424124 18156 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3106434588bd4241a314064d6a8749c3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:26.427088 18151 catalog_manager.cc:5484] T 5809e7c9c2204ba293426c24129fb16e P 5e1c6397f09747f091534faf93542710 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e1c6397f09747f091534faf93542710 (127.17.177.2). New cstate: current_term: 1 leader_uuid: "5e1c6397f09747f091534faf93542710" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } health_report { overall_health: UNKNOWN } } }
I20240430 20:11:26.429421 18738 tablet_service.cc:1467] Processing CreateTablet for tablet dd6083f12c7e4d1e8a2006b3ddf9aa3a (DEFAULT_TABLE table=default.Table_276f62c2338b46a0aab8905430e8b61b [id=789b538b500642d89721e6028b9ac48e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:26.429620 18985 ts_tablet_manager.cc:1398] T 609a1a7d22e84b69bd0a53577b62b69b P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.030s user 0.000s sys 0.003s
I20240430 20:11:26.429673 18988 tablet_bootstrap.cc:492] T a0e8e07d363f48c698c7e2b17edc1545 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:26.429811 18988 ts_tablet_manager.cc:1367] T a0e8e07d363f48c698c7e2b17edc1545 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.001s
I20240430 20:11:26.430684 18988 raft_consensus.cc:348] T a0e8e07d363f48c698c7e2b17edc1545 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:26.430896 18988 raft_consensus.cc:374] T a0e8e07d363f48c698c7e2b17edc1545 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:26.430831 18463 tablet_service.cc:1467] Processing CreateTablet for tablet dd6083f12c7e4d1e8a2006b3ddf9aa3a (DEFAULT_TABLE table=default.Table_276f62c2338b46a0aab8905430e8b61b [id=789b538b500642d89721e6028b9ac48e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:26.430981 18988 raft_consensus.cc:729] T a0e8e07d363f48c698c7e2b17edc1545 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:26.431210 18988 consensus_queue.cc:260] T a0e8e07d363f48c698c7e2b17edc1545 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:26.432241 18988 ts_tablet_manager.cc:1398] T a0e8e07d363f48c698c7e2b17edc1545 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240430 20:11:26.432538 18988 tablet_bootstrap.cc:492] T 609a1a7d22e84b69bd0a53577b62b69b P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:26.433660 19182 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240430 20:11:26.434211 18988 tablet_bootstrap.cc:654] T 609a1a7d22e84b69bd0a53577b62b69b P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
W20240430 20:11:26.437968 18146 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a1aeee1b02214975a376cfd36957afe8 consensus state has no leader
I20240430 20:11:26.438709 18146 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54524:
name: "default.Table_b331bf8aeaae4911ac01ed3fbd2627bd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240430 20:11:26.439728 18985 tablet_bootstrap.cc:492] T 27f94f2d8f50495dababe417a9f82982 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
W20240430 20:11:26.439992 18146 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b331bf8aeaae4911ac01ed3fbd2627bd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:26.442095 18985 tablet_bootstrap.cc:654] T 27f94f2d8f50495dababe417a9f82982 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:26.442878 18988 tablet_bootstrap.cc:492] T 609a1a7d22e84b69bd0a53577b62b69b P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:26.443071 18988 ts_tablet_manager.cc:1367] T 609a1a7d22e84b69bd0a53577b62b69b P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20240430 20:11:26.444286 18602 tablet_service.cc:1467] Processing CreateTablet for tablet f77086a5675f44e99e8a867b694df4b8 (DEFAULT_TABLE table=default.Table_cd719a43207f4bf983e1b8e6d236bb16 [id=47e0b04b17834deb833a8f31c123edc5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:26.444723 18988 raft_consensus.cc:348] T 609a1a7d22e84b69bd0a53577b62b69b P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:26.445019 18988 raft_consensus.cc:374] T 609a1a7d22e84b69bd0a53577b62b69b P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:26.445122 18988 raft_consensus.cc:729] T 609a1a7d22e84b69bd0a53577b62b69b P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:26.445072 18744 tablet_service.cc:1467] Processing CreateTablet for tablet f77086a5675f44e99e8a867b694df4b8 (DEFAULT_TABLE table=default.Table_cd719a43207f4bf983e1b8e6d236bb16 [id=47e0b04b17834deb833a8f31c123edc5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:26.445385 18983 tablet_bootstrap.cc:492] T 3f49eda7aeca42308d01b83a19d7d037 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:26.445376 18988 consensus_queue.cc:260] T 609a1a7d22e84b69bd0a53577b62b69b P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:26.446954 18988 ts_tablet_manager.cc:1398] T 609a1a7d22e84b69bd0a53577b62b69b P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240430 20:11:26.447299 18988 tablet_bootstrap.cc:492] T 680553d4d0bc4fcb839235c1832757e1 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:26.445608 18983 ts_tablet_manager.cc:1367] T 3f49eda7aeca42308d01b83a19d7d037 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.036s user 0.006s sys 0.005s
I20240430 20:11:26.448226 18983 raft_consensus.cc:348] T 3f49eda7aeca42308d01b83a19d7d037 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:26.448467 18983 raft_consensus.cc:374] T 3f49eda7aeca42308d01b83a19d7d037 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:26.448534 18983 raft_consensus.cc:729] T 3f49eda7aeca42308d01b83a19d7d037 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:26.448689 19186 raft_consensus.cc:482] T ffe059eeac91476aaeef8c56d2f49ee8 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:26.448791 18983 consensus_queue.cc:260] T 3f49eda7aeca42308d01b83a19d7d037 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:26.448859 19186 raft_consensus.cc:504] T ffe059eeac91476aaeef8c56d2f49ee8 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:26.449009 18988 tablet_bootstrap.cc:654] T 680553d4d0bc4fcb839235c1832757e1 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:26.449313 18983 ts_tablet_manager.cc:1398] T 3f49eda7aeca42308d01b83a19d7d037 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240430 20:11:26.449558 19186 leader_election.cc:288] T ffe059eeac91476aaeef8c56d2f49ee8 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:26.449636 18983 tablet_bootstrap.cc:492] T 187a27a6449342f99de3c5a2f93d49fc P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:26.450007 18459 tablet_service.cc:1467] Processing CreateTablet for tablet f77086a5675f44e99e8a867b694df4b8 (DEFAULT_TABLE table=default.Table_cd719a43207f4bf983e1b8e6d236bb16 [id=47e0b04b17834deb833a8f31c123edc5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:26.451143 18595 tablet_service.cc:1467] Processing CreateTablet for tablet 5bde72e2791a497db7dd08686ca36d80 (DEFAULT_TABLE table=default.Table_a7add9298f1a44c6a2b2a9d73405c834 [id=5227206f44d04fa89b4842b899f33afb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:26.451500 18983 tablet_bootstrap.cc:654] T 187a27a6449342f99de3c5a2f93d49fc P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:26.452296 18621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffe059eeac91476aaeef8c56d2f49ee8" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:26.452504 18621 raft_consensus.cc:2454] T ffe059eeac91476aaeef8c56d2f49ee8 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 0.
I20240430 20:11:26.453224 18147 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: Sending DeleteTablet for 3 replicas of tablet 29974dc2c7dd45b99b6cc0712378fc30
I20240430 20:11:26.453277 18415 leader_election.cc:302] T ffe059eeac91476aaeef8c56d2f49ee8 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:26.453812 19072 raft_consensus.cc:2781] T ffe059eeac91476aaeef8c56d2f49ee8 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:26.453904 19072 raft_consensus.cc:482] T ffe059eeac91476aaeef8c56d2f49ee8 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:26.453999 19072 raft_consensus.cc:3037] T ffe059eeac91476aaeef8c56d2f49ee8 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:26.454493 18742 tablet_service.cc:1514] Processing DeleteTablet for tablet 29974dc2c7dd45b99b6cc0712378fc30 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:26 UTC) from {username='slave'} at 127.0.0.1:59818
I20240430 20:11:26.455394 18762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffe059eeac91476aaeef8c56d2f49ee8" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:26.455410 18147 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_80b9e5521fff4a119f3ce4977b3e44eb" } new_table_name: "default.Table_44426457fd7045dfa1c149f0a5ed5129" modify_external_catalogs: true
I20240430 20:11:26.455525 18454 tablet_service.cc:1514] Processing DeleteTablet for tablet 29974dc2c7dd45b99b6cc0712378fc30 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:26 UTC) from {username='slave'} at 127.0.0.1:41678
I20240430 20:11:26.455636 18762 raft_consensus.cc:2454] T ffe059eeac91476aaeef8c56d2f49ee8 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 0.
W20240430 20:11:26.456432 18417 consensus_peers.cc:485] T 59548d560b4e4f7ca034d730aca79364 P 2efb542aa55e439b99ccb98d1c5c884a -> Peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Couldn't send request to peer 3c51c33d5616482cafb68e26688820f4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 59548d560b4e4f7ca034d730aca79364. This is attempt 1: this message will repeat every 5th retry.
I20240430 20:11:26.457257 18600 tablet_service.cc:1514] Processing DeleteTablet for tablet 29974dc2c7dd45b99b6cc0712378fc30 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:26 UTC) from {username='slave'} at 127.0.0.1:40202
I20240430 20:11:26.459729 18738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd6083f12c7e4d1e8a2006b3ddf9aa3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:26.459738 18987 ts_tablet_manager.cc:1888] T a01ee128ba8a4c9fb8e125ba97239d4c P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:26.460327 18987 log.cc:1196] T a01ee128ba8a4c9fb8e125ba97239d4c P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/a01ee128ba8a4c9fb8e125ba97239d4c
I20240430 20:11:26.460428 18735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4508e4f6dbd42109d951c648b3dc7df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:26.460745 18987 ts_tablet_manager.cc:1909] T a01ee128ba8a4c9fb8e125ba97239d4c P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:26.461001 18744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f77086a5675f44e99e8a867b694df4b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:26.463476 18742 tablet_service.cc:1467] Processing CreateTablet for tablet 5bde72e2791a497db7dd08686ca36d80 (DEFAULT_TABLE table=default.Table_a7add9298f1a44c6a2b2a9d73405c834 [id=5227206f44d04fa89b4842b899f33afb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:26.465934 18454 tablet_service.cc:1467] Processing CreateTablet for tablet 5bde72e2791a497db7dd08686ca36d80 (DEFAULT_TABLE table=default.Table_a7add9298f1a44c6a2b2a9d73405c834 [id=5227206f44d04fa89b4842b899f33afb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:26.467164 18981 ts_tablet_manager.cc:1888] T 6f729a6e8dc14de9a2057cfe09ba9257 P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:26.467320 18981 log.cc:1196] T 6f729a6e8dc14de9a2057cfe09ba9257 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/6f729a6e8dc14de9a2057cfe09ba9257
I20240430 20:11:26.467691 18981 ts_tablet_manager.cc:1909] T 6f729a6e8dc14de9a2057cfe09ba9257 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:26.467988 19185 raft_consensus.cc:504] T fe01b95104174279a7d64c0d4c0af982 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:26.468731 19185 leader_election.cc:288] T fe01b95104174279a7d64c0d4c0af982 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:26.469755 18600 tablet_service.cc:1467] Processing CreateTablet for tablet 7dfdecba959543a795dfcd3eeda29e58 (DEFAULT_TABLE table=default.Table_da5216d025844afda5a83ada555bc120 [id=dc55b25c3ca14863a21042e805e3bf9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:26.470189 18600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7dfdecba959543a795dfcd3eeda29e58. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:26.471554 18461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4508e4f6dbd42109d951c648b3dc7df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:26.473381 18980 tablet_replica.cc:330] T fe01b95104174279a7d64c0d4c0af982 P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:26.473628 18980 raft_consensus.cc:2229] T fe01b95104174279a7d64c0d4c0af982 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:26.474184 18980 raft_consensus.cc:2258] T fe01b95104174279a7d64c0d4c0af982 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:26.475517 18602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f77086a5675f44e99e8a867b694df4b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:26.476444 18737 tablet_service.cc:1467] Processing CreateTablet for tablet 7dfdecba959543a795dfcd3eeda29e58 (DEFAULT_TABLE table=default.Table_da5216d025844afda5a83ada555bc120 [id=dc55b25c3ca14863a21042e805e3bf9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:26.476644 18595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bde72e2791a497db7dd08686ca36d80. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:26.478935 18603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4508e4f6dbd42109d951c648b3dc7df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:26.479441 18980 ts_tablet_manager.cc:1875] T fe01b95104174279a7d64c0d4c0af982 P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:26.479724 18598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd6083f12c7e4d1e8a2006b3ddf9aa3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:26.485757 18463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd6083f12c7e4d1e8a2006b3ddf9aa3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:26.486984 18621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe01b95104174279a7d64c0d4c0af982" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:26.487491 18459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f77086a5675f44e99e8a867b694df4b8. 1 dirs total, 0 dirs full, 0 dirs failed
W20240430 20:11:26.491339 18415 leader_election.cc:341] T fe01b95104174279a7d64c0d4c0af982 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Illegal state: Tablet not RUNNING: STOPPED
I20240430 20:11:26.493171 18988 tablet_bootstrap.cc:492] T 680553d4d0bc4fcb839235c1832757e1 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:26.493345 18988 ts_tablet_manager.cc:1367] T 680553d4d0bc4fcb839235c1832757e1 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.046s user 0.004s sys 0.005s
I20240430 20:11:26.494865 18454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bde72e2791a497db7dd08686ca36d80. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:26.495213 18988 raft_consensus.cc:348] T 680553d4d0bc4fcb839235c1832757e1 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:26.495429 18988 raft_consensus.cc:374] T 680553d4d0bc4fcb839235c1832757e1 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:26.495524 18988 raft_consensus.cc:729] T 680553d4d0bc4fcb839235c1832757e1 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:26.495806 18988 consensus_queue.cc:260] T 680553d4d0bc4fcb839235c1832757e1 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:26.496373 18988 ts_tablet_manager.cc:1398] T 680553d4d0bc4fcb839235c1832757e1 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240430 20:11:26.507649 18149 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: Sending DeleteTablet for 3 replicas of tablet 81328e904ca94e2eb5ec94dd3ffb3c90
I20240430 20:11:26.509238 18736 tablet_service.cc:1514] Processing DeleteTablet for tablet 81328e904ca94e2eb5ec94dd3ffb3c90 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:26 UTC) from {username='slave'} at 127.0.0.1:59818
W20240430 20:11:26.510131 18417 leader_election.cc:341] T fe01b95104174279a7d64c0d4c0af982 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Not found: Tablet not found: fe01b95104174279a7d64c0d4c0af982
I20240430 20:11:26.510291 18417 leader_election.cc:302] T fe01b95104174279a7d64c0d4c0af982 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a; no voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710
I20240430 20:11:26.511065 18452 tablet_service.cc:1467] Processing CreateTablet for tablet 7dfdecba959543a795dfcd3eeda29e58 (DEFAULT_TABLE table=default.Table_da5216d025844afda5a83ada555bc120 [id=dc55b25c3ca14863a21042e805e3bf9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:26.511548 18452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7dfdecba959543a795dfcd3eeda29e58. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:26.513207 18455 tablet_service.cc:1514] Processing DeleteTablet for tablet 81328e904ca94e2eb5ec94dd3ffb3c90 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:26 UTC) from {username='slave'} at 127.0.0.1:41678
I20240430 20:11:26.514889 19185 raft_consensus.cc:2726] T fe01b95104174279a7d64c0d4c0af982 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240430 20:11:26.514868 18743 tablet_service.cc:1514] Processing DeleteTablet for tablet ffe059eeac91476aaeef8c56d2f49ee8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:24 UTC) from {username='slave'} at 127.0.0.1:59818
I20240430 20:11:26.514838 18736 tablet_service.cc:1514] Processing DeleteTablet for tablet 187a27a6449342f99de3c5a2f93d49fc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:25 UTC) from {username='slave'} at 127.0.0.1:59818
I20240430 20:11:26.516027 18154 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: Sending DeleteTablet for 3 replicas of tablet bbe9a2429a434f3f95d6e7f314fcfe4c
I20240430 20:11:26.517112 18736 tablet_service.cc:1514] Processing DeleteTablet for tablet bbe9a2429a434f3f95d6e7f314fcfe4c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:26 UTC) from {username='slave'} at 127.0.0.1:59818
I20240430 20:11:26.518177 18987 tablet_replica.cc:330] T bd7392f3dfd643cdb1634ee10ad41651 P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:26.518402 18987 raft_consensus.cc:2229] T bd7392f3dfd643cdb1634ee10ad41651 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:26.518625 18987 raft_consensus.cc:2258] T bd7392f3dfd643cdb1634ee10ad41651 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:26.519145 18987 ts_tablet_manager.cc:1875] T bd7392f3dfd643cdb1634ee10ad41651 P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:26.522228 18988 tablet_bootstrap.cc:492] T 27f94f2d8f50495dababe417a9f82982 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:26.523176 18141 catalog_manager.cc:4830] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): tablet a01ee128ba8a4c9fb8e125ba97239d4c (table default.Table_23b82186d14a44eeb81cad606fb8e9cc [id=172b1ea31b8142108ae466a018ecab47]) successfully deleted
I20240430 20:11:26.523924 18988 tablet_bootstrap.cc:654] T 27f94f2d8f50495dababe417a9f82982 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:26.525214 18455 tablet_service.cc:1514] Processing DeleteTablet for tablet bbe9a2429a434f3f95d6e7f314fcfe4c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:26 UTC) from {username='slave'} at 127.0.0.1:41678
I20240430 20:11:26.529249 18155 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: Sending DeleteTablet for 3 replicas of tablet cace1398492f4d968dcb6899385d527c
I20240430 20:11:26.529695 18155 catalog_manager.cc:6554] ReplaceTablet: tablet cace1398492f4d968dcb6899385d527c deleted and replaced by tablet 01436e5b6232494babbb414342f03090
I20240430 20:11:26.531741 18744 tablet_service.cc:1514] Processing DeleteTablet for tablet cace1398492f4d968dcb6899385d527c with delete_type TABLET_DATA_DELETED (replaced by tablet 01436e5b6232494babbb414342f03090) from {username='slave'} at 127.0.0.1:59818
I20240430 20:11:26.533900 18455 tablet_service.cc:1514] Processing DeleteTablet for tablet cace1398492f4d968dcb6899385d527c with delete_type TABLET_DATA_DELETED (replaced by tablet 01436e5b6232494babbb414342f03090) from {username='slave'} at 127.0.0.1:41678
I20240430 20:11:26.534968 18597 tablet_service.cc:1514] Processing DeleteTablet for tablet 81328e904ca94e2eb5ec94dd3ffb3c90 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:26 UTC) from {username='slave'} at 127.0.0.1:40202
W20240430 20:11:26.535959 18152 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 77285afcafe04a829e485998bbbe00f0 consensus state has no leader
I20240430 20:11:26.536211 18597 tablet_service.cc:1514] Processing DeleteTablet for tablet bbe9a2429a434f3f95d6e7f314fcfe4c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:26 UTC) from {username='slave'} at 127.0.0.1:40202
I20240430 20:11:26.537060 18597 tablet_service.cc:1514] Processing DeleteTablet for tablet cace1398492f4d968dcb6899385d527c with delete_type TABLET_DATA_DELETED (replaced by tablet 01436e5b6232494babbb414342f03090) from {username='slave'} at 127.0.0.1:40202
I20240430 20:11:26.538177 18148 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54524:
name: "default.Table_71075ae6b4d545889273c891648c70d1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:26.538887 18148 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_71075ae6b4d545889273c891648c70d1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240430 20:11:26.538986 18147 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0b4d9e860d7c45fd91d7a566605c9283 consensus state has no leader
I20240430 20:11:26.539095 18139 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet a01ee128ba8a4c9fb8e125ba97239d4c (table default.Table_23b82186d14a44eeb81cad606fb8e9cc [id=172b1ea31b8142108ae466a018ecab47]) successfully deleted
W20240430 20:11:26.539496 18139 catalog_manager.cc:4810] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): delete failed for tablet 1398d4d2305b48e08fdb12432bd6c4b0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1398d4d2305b48e08fdb12432bd6c4b0 already in progress: creating tablet
W20240430 20:11:26.539721 18139 catalog_manager.cc:4810] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): delete failed for tablet 15674aeb56b74210a4a2809608fd4457 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 15674aeb56b74210a4a2809608fd4457 already in progress: creating tablet
I20240430 20:11:26.540503 18155 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_0c6652c070534fbbaab6c2b8d175f139" } new_table_name: "default.Table_b8388e076be44f45ab6ed2ab3bce74a8" modify_external_catalogs: true
I20240430 20:11:26.541497 18147 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 81787eccca6c42c1b421377a3bf9ebe6 (table default.Table_c1669746f9114bf7b37a6f6a68fed20e [id=7f602584cef9486eb7d9b4389be3f43e]). Expected version 2 got 0
I20240430 20:11:26.542855 18156 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54524:
name: "default.Table_aa42ff1beec14dd6baf999b8962907e0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:26.543439 18156 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_aa42ff1beec14dd6baf999b8962907e0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:26.545367 18152 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_31d8d978e96c4e12a0fbd5500c7b93cd" } modify_external_catalogs: true
I20240430 20:11:26.545586 18152 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_31d8d978e96c4e12a0fbd5500c7b93cd" } modify_external_catalogs: true
I20240430 20:11:26.547241 18149 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54524:
name: "default.Table_bdb66daeadd54058b7c006c6514acd6c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:26.547816 18149 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bdb66daeadd54058b7c006c6514acd6c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:26.548141 18146 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54524:
name: "default.Table_ad1a80db9c774df6add2b60e93577217"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:26.548637 18146 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ad1a80db9c774df6add2b60e93577217 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:26.549489 18981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41678 (request call id 104) took 1108 ms. Trace:
I20240430 20:11:26.549646 18981 rpcz_store.cc:276] 0430 20:11:25.440651 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:25.440735 (+ 84us) service_pool.cc:225] Handling call
0430 20:11:26.145941 (+705206us) ts_tablet_manager.cc:1120] Deleting tablet 6f729a6e8dc14de9a2057cfe09ba9257
0430 20:11:26.145955 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:26.265780 (+119825us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:26.467027 (+201247us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:26.549420 (+ 82393us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":704982}
I20240430 20:11:26.550935 18154 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet a0e8e07d363f48c698c7e2b17edc1545 (table default.Table_a058c17d17cb4a68a597fe273e1c1dfd [id=50771665d40d466f92f3a1babd60e39b]). Expected version 2 got 0
I20240430 20:11:26.551155 18154 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 680553d4d0bc4fcb839235c1832757e1 (table default.Table_49a012a7c4c8457fadb0a8f0fc9654c6 [id=eecb2f8eeef547c5ab1abe6a2430e592]). Expected version 1 got 0
I20240430 20:11:26.551306 18981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41678 (request call id 106) took 1082 ms. Trace:
I20240430 20:11:26.551349 18142 catalog_manager.cc:4830] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): tablet 6f729a6e8dc14de9a2057cfe09ba9257 (table default.Table_5feaab34a5cd439abdc705b85c22313b [id=1cefefb9abd64daa914f73fb65c4b201]) successfully deleted
I20240430 20:11:26.551410 18981 rpcz_store.cc:276] 0430 20:11:25.469015 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:25.469097 (+ 82us) service_pool.cc:225] Handling call
0430 20:11:26.551123 (+1082026us) ts_tablet_manager.cc:1120] Deleting tablet 27f94f2d8f50495dababe417a9f82982
0430 20:11:26.551135 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:26.551243 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1081485}
I20240430 20:11:26.551725 18981 tablet_replica.cc:330] T 187a27a6449342f99de3c5a2f93d49fc P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:26.551936 18981 raft_consensus.cc:2229] T 187a27a6449342f99de3c5a2f93d49fc P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:26.551883 18597 tablet_service.cc:1467] Processing CreateTablet for tablet c66c142419ca4e1399ecd1c8d1e62a59 (DEFAULT_TABLE table=default.Table_4d213ee516bb406798377b6b90fa8cc1 [id=93c44abdf6b147228224388398cd7aeb]), partition=RANGE (key) PARTITION UNBOUNDED
W20240430 20:11:26.551929 18142 catalog_manager.cc:4810] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): delete failed for tablet 27f94f2d8f50495dababe417a9f82982 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 27f94f2d8f50495dababe417a9f82982 already in progress: creating tablet
I20240430 20:11:26.552119 18981 raft_consensus.cc:2258] T 187a27a6449342f99de3c5a2f93d49fc P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:26.553164 18981 ts_tablet_manager.cc:1875] T 187a27a6449342f99de3c5a2f93d49fc P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:26.554347 18985 tablet_bootstrap.cc:492] T 27f94f2d8f50495dababe417a9f82982 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:26.554562 18985 ts_tablet_manager.cc:1367] T 27f94f2d8f50495dababe417a9f82982 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.116s user 0.005s sys 0.002s
I20240430 20:11:26.555891 18985 raft_consensus.cc:348] T 27f94f2d8f50495dababe417a9f82982 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:26.556263 18985 raft_consensus.cc:374] T 27f94f2d8f50495dababe417a9f82982 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:26.556391 18985 raft_consensus.cc:729] T 27f94f2d8f50495dababe417a9f82982 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:26.556685 18985 consensus_queue.cc:260] T 27f94f2d8f50495dababe417a9f82982 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:26.559299 18742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bde72e2791a497db7dd08686ca36d80. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:26.559965 18737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7dfdecba959543a795dfcd3eeda29e58. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:26.565382 19072 raft_consensus.cc:504] T ffe059eeac91476aaeef8c56d2f49ee8 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:26.566498 19072 leader_election.cc:288] T ffe059eeac91476aaeef8c56d2f49ee8 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Requested vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:26.567654 18621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffe059eeac91476aaeef8c56d2f49ee8" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:26.568015 18621 raft_consensus.cc:3037] T ffe059eeac91476aaeef8c56d2f49ee8 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:26.578852 18597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c66c142419ca4e1399ecd1c8d1e62a59. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:26.579821 18742 tablet_service.cc:1467] Processing CreateTablet for tablet c66c142419ca4e1399ecd1c8d1e62a59 (DEFAULT_TABLE table=default.Table_4d213ee516bb406798377b6b90fa8cc1 [id=93c44abdf6b147228224388398cd7aeb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:26.580502 18151 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54524:
name: "default.Table_e233554a5e1d41389948fecaf498a856"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:26.581375 18151 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e233554a5e1d41389948fecaf498a856 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:26.585945 18744 tablet_service.cc:1467] Processing CreateTablet for tablet 724e51ff05ec4baebc2bb55e05c8eddc (DEFAULT_TABLE table=default.Table_73ee808b1dda40dbafb28cf5669d53bc [id=4dce8a767f0541e1beb5464b2ed95d06]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:26.586941 18595 tablet_service.cc:1467] Processing CreateTablet for tablet 724e51ff05ec4baebc2bb55e05c8eddc (DEFAULT_TABLE table=default.Table_73ee808b1dda40dbafb28cf5669d53bc [id=4dce8a767f0541e1beb5464b2ed95d06]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:26.587515 18595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 724e51ff05ec4baebc2bb55e05c8eddc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:26.588279 18600 tablet_service.cc:1467] Processing CreateTablet for tablet 68ca48351cd4412381f5195d81109016 (DEFAULT_TABLE table=default.Table_172b295132814722b733e25f446730a7 [id=7168f3d7e00348ce8fa33100c98652f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:26.588550 18738 tablet_service.cc:1467] Processing CreateTablet for tablet 68ca48351cd4412381f5195d81109016 (DEFAULT_TABLE table=default.Table_172b295132814722b733e25f446730a7 [id=7168f3d7e00348ce8fa33100c98652f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:26.590127 18600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68ca48351cd4412381f5195d81109016. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:26.590322 19072 raft_consensus.cc:482] T 3f49eda7aeca42308d01b83a19d7d037 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:26.590490 19072 raft_consensus.cc:504] T 3f49eda7aeca42308d01b83a19d7d037 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:26.591228 19072 leader_election.cc:288] T 3f49eda7aeca42308d01b83a19d7d037 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:26.591897 18623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f49eda7aeca42308d01b83a19d7d037" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:26.592144 18623 raft_consensus.cc:2454] T 3f49eda7aeca42308d01b83a19d7d037 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 0.
I20240430 20:11:26.592592 18415 leader_election.cc:302] T 3f49eda7aeca42308d01b83a19d7d037 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:26.593712 19072 raft_consensus.cc:2781] T 3f49eda7aeca42308d01b83a19d7d037 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:26.593801 19072 raft_consensus.cc:482] T 3f49eda7aeca42308d01b83a19d7d037 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:26.593897 19072 raft_consensus.cc:3037] T 3f49eda7aeca42308d01b83a19d7d037 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:26.596403 18602 tablet_service.cc:1467] Processing CreateTablet for tablet e5111d7607dd42188a0176cf5be2ec05 (DEFAULT_TABLE table=default.Table_f61a45518e134c4496e4ac81fbe40fce [id=02d9fe0426694ee8b5fd70589fb66ead]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:26.596880 18602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5111d7607dd42188a0176cf5be2ec05. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:26.602471 18763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffe059eeac91476aaeef8c56d2f49ee8" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:26.602749 18763 raft_consensus.cc:3037] T ffe059eeac91476aaeef8c56d2f49ee8 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:26.603868 19191 raft_consensus.cc:482] T 3f49eda7aeca42308d01b83a19d7d037 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:26.604110 19191 raft_consensus.cc:504] T 3f49eda7aeca42308d01b83a19d7d037 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:26.605036 19191 leader_election.cc:288] T 3f49eda7aeca42308d01b83a19d7d037 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:26.605324 19185 raft_consensus.cc:482] T b4bc111e183f40a0acc9dd377195f6d9 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:26.605465 18623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f49eda7aeca42308d01b83a19d7d037" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:26.605522 19185 raft_consensus.cc:504] T b4bc111e183f40a0acc9dd377195f6d9 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:26.605726 18623 raft_consensus.cc:2454] T 3f49eda7aeca42308d01b83a19d7d037 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:26.606195 19185 leader_election.cc:288] T b4bc111e183f40a0acc9dd377195f6d9 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:26.606174 18696 leader_election.cc:302] T 3f49eda7aeca42308d01b83a19d7d037 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:26.606487 19191 raft_consensus.cc:2781] T 3f49eda7aeca42308d01b83a19d7d037 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:26.606580 19191 raft_consensus.cc:482] T 3f49eda7aeca42308d01b83a19d7d037 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:26.606698 19191 raft_consensus.cc:3037] T 3f49eda7aeca42308d01b83a19d7d037 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:26.607187 18482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f49eda7aeca42308d01b83a19d7d037" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:26.607690 18623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4bc111e183f40a0acc9dd377195f6d9" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:26.607978 18623 raft_consensus.cc:2454] T b4bc111e183f40a0acc9dd377195f6d9 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 0.
I20240430 20:11:26.608330 18735 tablet_service.cc:1467] Processing CreateTablet for tablet e5111d7607dd42188a0176cf5be2ec05 (DEFAULT_TABLE table=default.Table_f61a45518e134c4496e4ac81fbe40fce [id=02d9fe0426694ee8b5fd70589fb66ead]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:26.608558 18415 leader_election.cc:302] T b4bc111e183f40a0acc9dd377195f6d9 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:26.608961 19185 raft_consensus.cc:2781] T b4bc111e183f40a0acc9dd377195f6d9 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:26.609067 19185 raft_consensus.cc:482] T b4bc111e183f40a0acc9dd377195f6d9 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:26.609190 19185 raft_consensus.cc:3037] T b4bc111e183f40a0acc9dd377195f6d9 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:26.612668 18598 tablet_service.cc:1467] Processing CreateTablet for tablet 44fa6917a85c46f3aaa363df46ea586b (DEFAULT_TABLE table=default.Table_5924e4520c6c49988ccef21ca04d25d4 [id=68c1eb7252974258aabd9873e6654b10]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:26.613982 18762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f49eda7aeca42308d01b83a19d7d037" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:26.614468 18736 tablet_service.cc:1467] Processing CreateTablet for tablet 44fa6917a85c46f3aaa363df46ea586b (DEFAULT_TABLE table=default.Table_5924e4520c6c49988ccef21ca04d25d4 [id=68c1eb7252974258aabd9873e6654b10]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:26.615429 18461 tablet_service.cc:1467] Processing CreateTablet for tablet c66c142419ca4e1399ecd1c8d1e62a59 (DEFAULT_TABLE table=default.Table_4d213ee516bb406798377b6b90fa8cc1 [id=93c44abdf6b147228224388398cd7aeb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:26.616055 18985 ts_tablet_manager.cc:1398] T 27f94f2d8f50495dababe417a9f82982 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.061s user 0.003s sys 0.000s
I20240430 20:11:26.616182 18454 tablet_service.cc:1467] Processing CreateTablet for tablet 724e51ff05ec4baebc2bb55e05c8eddc (DEFAULT_TABLE table=default.Table_73ee808b1dda40dbafb28cf5669d53bc [id=4dce8a767f0541e1beb5464b2ed95d06]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:26.616549 18985 tablet_bootstrap.cc:492] T 0879b02a0bcc4f619fb34cff7b9d7c4c P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:26.617908 18983 tablet_bootstrap.cc:492] T 187a27a6449342f99de3c5a2f93d49fc P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:26.618064 18983 ts_tablet_manager.cc:1367] T 187a27a6449342f99de3c5a2f93d49fc P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.169s user 0.042s sys 0.030s
I20240430 20:11:26.618839 18455 tablet_service.cc:1467] Processing CreateTablet for tablet 68ca48351cd4412381f5195d81109016 (DEFAULT_TABLE table=default.Table_172b295132814722b733e25f446730a7 [id=7168f3d7e00348ce8fa33100c98652f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:26.618885 18983 raft_consensus.cc:348] T 187a27a6449342f99de3c5a2f93d49fc P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:26.619074 18983 raft_consensus.cc:374] T 187a27a6449342f99de3c5a2f93d49fc P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:26.619156 18983 raft_consensus.cc:729] T 187a27a6449342f99de3c5a2f93d49fc P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:26.619665 18457 tablet_service.cc:1467] Processing CreateTablet for tablet e5111d7607dd42188a0176cf5be2ec05 (DEFAULT_TABLE table=default.Table_f61a45518e134c4496e4ac81fbe40fce [id=02d9fe0426694ee8b5fd70589fb66ead]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:26.620395 18453 tablet_service.cc:1467] Processing CreateTablet for tablet 44fa6917a85c46f3aaa363df46ea586b (DEFAULT_TABLE table=default.Table_5924e4520c6c49988ccef21ca04d25d4 [id=68c1eb7252974258aabd9873e6654b10]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:26.621245 18764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4bc111e183f40a0acc9dd377195f6d9" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:26.621505 18764 raft_consensus.cc:2454] T b4bc111e183f40a0acc9dd377195f6d9 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 0.
I20240430 20:11:26.622232 18150 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: default.Table_5fc77d5f60ae44f4bb4ca9f7f183672c [id=6ca223ca6354449c959d85ed2ac6a9c8] alter complete (version 1)
W20240430 20:11:26.622400 18155 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f5236574672b477a9026e91adcaf70ec consensus state has no leader
I20240430 20:11:26.622735 18150 master_service.cc:946] ReplaceTablet: received request to replace tablet aacabead3c874e77a47d2ecf19dbfa82 from {username='slave'} at 127.0.0.1:54524
I20240430 20:11:26.623530 18155 master_service.cc:946] ReplaceTablet: received request to replace tablet 3f49eda7aeca42308d01b83a19d7d037 from {username='slave'} at 127.0.0.1:54524
I20240430 20:11:26.623716 18985 tablet_bootstrap.cc:654] T 0879b02a0bcc4f619fb34cff7b9d7c4c P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:26.624290 18591 tablet_service.cc:1514] Processing DeleteTablet for tablet ffe059eeac91476aaeef8c56d2f49ee8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:24 UTC) from {username='slave'} at 127.0.0.1:40202
I20240430 20:11:26.625387 18596 tablet_service.cc:1514] Processing DeleteTablet for tablet b4bc111e183f40a0acc9dd377195f6d9 with delete_type TABLET_DATA_DELETED (replaced by tablet 0b4d9e860d7c45fd91d7a566605c9283) from {username='slave'} at 127.0.0.1:40202
I20240430 20:11:26.625612 18591 tablet_service.cc:1514] Processing DeleteTablet for tablet bd7392f3dfd643cdb1634ee10ad41651 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:24 UTC) from {username='slave'} at 127.0.0.1:40202
I20240430 20:11:26.627424 18147 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54524:
name: "default.Table_b36ff4b419a94a0e8f84c416f3007ffe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:26.627985 18147 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b36ff4b419a94a0e8f84c416f3007ffe in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:26.628973 18983 consensus_queue.cc:260] T 187a27a6449342f99de3c5a2f93d49fc P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:26.629659 18983 ts_tablet_manager.cc:1398] T 187a27a6449342f99de3c5a2f93d49fc P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240430 20:11:26.633026 18983 tablet_bootstrap.cc:492] T 680553d4d0bc4fcb839235c1832757e1 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:26.634816 18983 tablet_bootstrap.cc:654] T 680553d4d0bc4fcb839235c1832757e1 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:26.635748 18743 tablet_service.cc:1514] Processing DeleteTablet for tablet b4bc111e183f40a0acc9dd377195f6d9 with delete_type TABLET_DATA_DELETED (replaced by tablet 0b4d9e860d7c45fd91d7a566605c9283) from {username='slave'} at 127.0.0.1:59818
I20240430 20:11:26.636915 18734 tablet_service.cc:1514] Processing DeleteTablet for tablet 609a1a7d22e84b69bd0a53577b62b69b with delete_type TABLET_DATA_DELETED (replaced by tablet d57e59f225114af1b1522be6cb805991) from {username='slave'} at 127.0.0.1:59818
I20240430 20:11:26.648607 18985 tablet_bootstrap.cc:492] T 0879b02a0bcc4f619fb34cff7b9d7c4c P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:26.648794 18985 ts_tablet_manager.cc:1367] T 0879b02a0bcc4f619fb34cff7b9d7c4c P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.032s user 0.005s sys 0.001s
I20240430 20:11:26.649518 18985 raft_consensus.cc:348] T 0879b02a0bcc4f619fb34cff7b9d7c4c P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:26.649742 18985 raft_consensus.cc:374] T 0879b02a0bcc4f619fb34cff7b9d7c4c P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:26.649813 18985 raft_consensus.cc:729] T 0879b02a0bcc4f619fb34cff7b9d7c4c P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:26.650125 18985 consensus_queue.cc:260] T 0879b02a0bcc4f619fb34cff7b9d7c4c P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:26.650892 18591 tablet_service.cc:1514] Processing DeleteTablet for tablet 187a27a6449342f99de3c5a2f93d49fc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:25 UTC) from {username='slave'} at 127.0.0.1:40202
I20240430 20:11:26.653357 18154 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 680553d4d0bc4fcb839235c1832757e1 (table default.Table_49a012a7c4c8457fadb0a8f0fc9654c6 [id=eecb2f8eeef547c5ab1abe6a2430e592]). Expected version 1 got 0
I20240430 20:11:26.657896 18985 ts_tablet_manager.cc:1398] T 0879b02a0bcc4f619fb34cff7b9d7c4c P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20240430 20:11:26.663265 18451 tablet_service.cc:1514] Processing DeleteTablet for tablet 27f94f2d8f50495dababe417a9f82982 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:24 UTC) from {username='slave'} at 127.0.0.1:41678
I20240430 20:11:26.665809 18987 ts_tablet_manager.cc:1888] T bd7392f3dfd643cdb1634ee10ad41651 P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:26.666043 18987 log.cc:1196] T bd7392f3dfd643cdb1634ee10ad41651 P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/bd7392f3dfd643cdb1634ee10ad41651
I20240430 20:11:26.666426 18987 ts_tablet_manager.cc:1909] T bd7392f3dfd643cdb1634ee10ad41651 P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:26.666893 18981 ts_tablet_manager.cc:1888] T 187a27a6449342f99de3c5a2f93d49fc P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:26.667057 18981 log.cc:1196] T 187a27a6449342f99de3c5a2f93d49fc P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/187a27a6449342f99de3c5a2f93d49fc
I20240430 20:11:26.667479 18981 ts_tablet_manager.cc:1909] T 187a27a6449342f99de3c5a2f93d49fc P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:26.667774 19072 raft_consensus.cc:504] T 3f49eda7aeca42308d01b83a19d7d037 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:26.668543 19072 leader_election.cc:288] T 3f49eda7aeca42308d01b83a19d7d037 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Requested vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:26.668901 18621 raft_consensus.cc:2454] T ffe059eeac91476aaeef8c56d2f49ee8 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 1.
I20240430 20:11:26.669472 18980 ts_tablet_manager.cc:1888] T fe01b95104174279a7d64c0d4c0af982 P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:26.669598 18980 log.cc:1196] T fe01b95104174279a7d64c0d4c0af982 P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/fe01b95104174279a7d64c0d4c0af982
I20240430 20:11:26.669994 18980 ts_tablet_manager.cc:1909] T fe01b95104174279a7d64c0d4c0af982 P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:26.670320 18763 raft_consensus.cc:2454] T ffe059eeac91476aaeef8c56d2f49ee8 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 1.
I20240430 20:11:26.670702 18482 raft_consensus.cc:2379] T 3f49eda7aeca42308d01b83a19d7d037 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c51c33d5616482cafb68e26688820f4 in current term 1: Already voted for candidate 2efb542aa55e439b99ccb98d1c5c884a in this term.
I20240430 20:11:26.674096 18461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c66c142419ca4e1399ecd1c8d1e62a59. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:26.675483 18454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 724e51ff05ec4baebc2bb55e05c8eddc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:26.676000 18455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68ca48351cd4412381f5195d81109016. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:26.676455 18457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5111d7607dd42188a0176cf5be2ec05. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:26.676898 18453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44fa6917a85c46f3aaa363df46ea586b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:26.677350 18983 tablet_bootstrap.cc:492] T 680553d4d0bc4fcb839235c1832757e1 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:26.677529 18983 ts_tablet_manager.cc:1367] T 680553d4d0bc4fcb839235c1832757e1 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.045s user 0.021s sys 0.021s
I20240430 20:11:26.678994 18983 raft_consensus.cc:348] T 680553d4d0bc4fcb839235c1832757e1 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:26.679247 18983 raft_consensus.cc:374] T 680553d4d0bc4fcb839235c1832757e1 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:26.679313 18983 raft_consensus.cc:729] T 680553d4d0bc4fcb839235c1832757e1 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:26.679545 18983 consensus_queue.cc:260] T 680553d4d0bc4fcb839235c1832757e1 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:26.680074 18983 ts_tablet_manager.cc:1398] T 680553d4d0bc4fcb839235c1832757e1 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240430 20:11:26.662302 18156 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_172b295132814722b733e25f446730a7" } modify_external_catalogs: true
I20240430 20:11:26.675732 18152 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: Sending DeleteTablet for 3 replicas of tablet a1aeee1b02214975a376cfd36957afe8
I20240430 20:11:26.681457 18988 tablet_bootstrap.cc:492] T 27f94f2d8f50495dababe417a9f82982 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:26.681641 18988 ts_tablet_manager.cc:1367] T 27f94f2d8f50495dababe417a9f82982 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.160s user 0.011s sys 0.001s
I20240430 20:11:26.682189 18734 tablet_service.cc:1514] Processing DeleteTablet for tablet a1aeee1b02214975a376cfd36957afe8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:26 UTC) from {username='slave'} at 127.0.0.1:59818
I20240430 20:11:26.682521 18988 raft_consensus.cc:348] T 27f94f2d8f50495dababe417a9f82982 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:26.682766 18988 raft_consensus.cc:374] T 27f94f2d8f50495dababe417a9f82982 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:26.682845 18988 raft_consensus.cc:729] T 27f94f2d8f50495dababe417a9f82982 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:26.682952 18591 tablet_service.cc:1514] Processing DeleteTablet for tablet a1aeee1b02214975a376cfd36957afe8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:26 UTC) from {username='slave'} at 127.0.0.1:40202
I20240430 20:11:26.683109 18988 consensus_queue.cc:260] T 27f94f2d8f50495dababe417a9f82982 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:26.683853 18459 tablet_service.cc:1514] Processing DeleteTablet for tablet a1aeee1b02214975a376cfd36957afe8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:26 UTC) from {username='slave'} at 127.0.0.1:41678
I20240430 20:11:26.684942 18156 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_172b295132814722b733e25f446730a7" } modify_external_catalogs: true
I20240430 20:11:26.687378 18742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c66c142419ca4e1399ecd1c8d1e62a59. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:26.687424 18983 tablet_bootstrap.cc:492] T 27f94f2d8f50495dababe417a9f82982 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:26.688011 18744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 724e51ff05ec4baebc2bb55e05c8eddc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:26.688499 18738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68ca48351cd4412381f5195d81109016. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:26.688933 18735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5111d7607dd42188a0176cf5be2ec05. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:26.689172 18983 tablet_bootstrap.cc:654] T 27f94f2d8f50495dababe417a9f82982 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:26.690014 18417 leader_election.cc:302] T ffe059eeac91476aaeef8c56d2f49ee8 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters:
I20240430 20:11:26.695940 19072 raft_consensus.cc:2781] T ffe059eeac91476aaeef8c56d2f49ee8 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:26.696158 19072 raft_consensus.cc:686] T ffe059eeac91476aaeef8c56d2f49ee8 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Becoming Leader. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Running, Role: LEADER
I20240430 20:11:26.696513 19072 consensus_queue.cc:237] T ffe059eeac91476aaeef8c56d2f49ee8 P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:26.697369 18598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44fa6917a85c46f3aaa363df46ea586b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:26.702427 18151 master_service.cc:946] ReplaceTablet: received request to replace tablet bd7392f3dfd643cdb1634ee10ad41651 from {username='slave'} at 127.0.0.1:54524
I20240430 20:11:26.704903 18150 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: Sending DeleteTablet for 3 replicas of tablet aacabead3c874e77a47d2ecf19dbfa82
I20240430 20:11:26.705435 18150 catalog_manager.cc:6554] ReplaceTablet: tablet aacabead3c874e77a47d2ecf19dbfa82 deleted and replaced by tablet fd40967ab4a04001819e16c06a6c4300
I20240430 20:11:26.706183 18734 tablet_service.cc:1514] Processing DeleteTablet for tablet aacabead3c874e77a47d2ecf19dbfa82 with delete_type TABLET_DATA_DELETED (replaced by tablet fd40967ab4a04001819e16c06a6c4300) from {username='slave'} at 127.0.0.1:59818
I20240430 20:11:26.707405 18155 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: Sending DeleteTablet for 3 replicas of tablet 3f49eda7aeca42308d01b83a19d7d037
I20240430 20:11:26.707947 18155 catalog_manager.cc:6554] ReplaceTablet: tablet 3f49eda7aeca42308d01b83a19d7d037 deleted and replaced by tablet 76d6e9d547ca4a848eae852944b1d117
I20240430 20:11:26.708563 18734 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f49eda7aeca42308d01b83a19d7d037 with delete_type TABLET_DATA_DELETED (replaced by tablet 76d6e9d547ca4a848eae852944b1d117) from {username='slave'} at 127.0.0.1:59818
I20240430 20:11:26.708998 18459 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f49eda7aeca42308d01b83a19d7d037 with delete_type TABLET_DATA_DELETED (replaced by tablet 76d6e9d547ca4a848eae852944b1d117) from {username='slave'} at 127.0.0.1:41678
I20240430 20:11:26.709878 18452 tablet_service.cc:1514] Processing DeleteTablet for tablet aacabead3c874e77a47d2ecf19dbfa82 with delete_type TABLET_DATA_DELETED (replaced by tablet fd40967ab4a04001819e16c06a6c4300) from {username='slave'} at 127.0.0.1:41678
I20240430 20:11:26.713929 18763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f49eda7aeca42308d01b83a19d7d037" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:26.714743 18736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44fa6917a85c46f3aaa363df46ea586b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:26.715747 18149 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54524:
name: "default.Table_d997b723a0dd48f6bc2d42c9bfd6bc14"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:26.716552 18149 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d997b723a0dd48f6bc2d42c9bfd6bc14 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:26.718070 18146 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_f136f0a535fc4860811d81fb1d439f3c" } new_table_name: "default.Table_276bbdc7cfe542d1a2936485a2aad1ec" modify_external_catalogs: true
I20240430 20:11:26.722883 18152 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_af1c8032c64245218a81d5bda4c68302" } modify_external_catalogs: true
I20240430 20:11:26.723109 18152 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_af1c8032c64245218a81d5bda4c68302" } modify_external_catalogs: true
I20240430 20:11:26.723634 18148 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54524:
name: "default.Table_e689526f90344c6f91dae34f22d60295"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240430 20:11:26.725975 18154 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_31f8c79337974bd9957c65f54d3c74e5" } new_table_name: "default.Table_ee86b8fcf2064d8b8c3e327e357d3395" modify_external_catalogs: true
W20240430 20:11:26.727224 18148 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e689526f90344c6f91dae34f22d60295 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:26.729504 18147 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_b8388e076be44f45ab6ed2ab3bce74a8" } modify_external_catalogs: true
I20240430 20:11:26.729717 18147 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_b8388e076be44f45ab6ed2ab3bce74a8" } modify_external_catalogs: true
I20240430 20:11:26.743419 18150 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54524:
name: "default.Table_0a397bb1e5ea4cbe87a088c7fc84e9c1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:26.744136 18150 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0a397bb1e5ea4cbe87a088c7fc84e9c1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240430 20:11:26.747859 18154 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 372b135dd01f44f98cd713ca8857ba61 consensus state has no leader
W20240430 20:11:26.750020 18146 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d4508e4f6dbd42109d951c648b3dc7df consensus state has no leader
I20240430 20:11:26.754055 18980 tablet_replica.cc:330] T b0e6cc37c6b1488fa88359d16b12a576 P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:26.754308 18980 raft_consensus.cc:2229] T b0e6cc37c6b1488fa88359d16b12a576 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:26.754549 18980 raft_consensus.cc:2258] T b0e6cc37c6b1488fa88359d16b12a576 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:26.743435 18151 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_b36ff4b419a94a0e8f84c416f3007ffe" } modify_external_catalogs: true
I20240430 20:11:26.754976 18151 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_b36ff4b419a94a0e8f84c416f3007ffe" } modify_external_catalogs: true
I20240430 20:11:26.759048 18156 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: Sending DeleteTablet for 3 replicas of tablet 68ca48351cd4412381f5195d81109016
I20240430 20:11:26.762482 18452 tablet_service.cc:1514] Processing DeleteTablet for tablet 68ca48351cd4412381f5195d81109016 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:26 UTC) from {username='slave'} at 127.0.0.1:41678
I20240430 20:11:26.769752 18980 ts_tablet_manager.cc:1875] T b0e6cc37c6b1488fa88359d16b12a576 P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:26.773873 18600 tablet_service.cc:1514] Processing DeleteTablet for tablet 68ca48351cd4412381f5195d81109016 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:26 UTC) from {username='slave'} at 127.0.0.1:40202
I20240430 20:11:26.774384 18139 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet fe01b95104174279a7d64c0d4c0af982 (table default.Table_52f64b8de0684649aa824da65de3416f [id=41ea2be4643c4e75a2c20b7f9f0a825e]) successfully deleted
I20240430 20:11:26.777923 18621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f49eda7aeca42308d01b83a19d7d037" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:26.778182 18621 raft_consensus.cc:3037] T 3f49eda7aeca42308d01b83a19d7d037 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:26.778769 18147 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: Sending DeleteTablet for 3 replicas of tablet f5236574672b477a9026e91adcaf70ec
I20240430 20:11:26.779645 18983 tablet_bootstrap.cc:492] T 27f94f2d8f50495dababe417a9f82982 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:26.779619 18452 tablet_service.cc:1514] Processing DeleteTablet for tablet f5236574672b477a9026e91adcaf70ec with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:26 UTC) from {username='slave'} at 127.0.0.1:41678
I20240430 20:11:26.779815 18983 ts_tablet_manager.cc:1367] T 27f94f2d8f50495dababe417a9f82982 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.092s user 0.038s sys 0.018s
I20240430 20:11:26.780777 18983 raft_consensus.cc:348] T 27f94f2d8f50495dababe417a9f82982 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:26.781009 18983 raft_consensus.cc:374] T 27f94f2d8f50495dababe417a9f82982 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:26.781107 18983 raft_consensus.cc:729] T 27f94f2d8f50495dababe417a9f82982 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:26.781366 18983 consensus_queue.cc:260] T 27f94f2d8f50495dababe417a9f82982 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:26.781976 18983 ts_tablet_manager.cc:1398] T 27f94f2d8f50495dababe417a9f82982 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240430 20:11:26.782297 18983 tablet_bootstrap.cc:492] T 609a1a7d22e84b69bd0a53577b62b69b P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:26.784211 19185 raft_consensus.cc:504] T b4bc111e183f40a0acc9dd377195f6d9 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:26.785059 18983 tablet_bootstrap.cc:654] T 609a1a7d22e84b69bd0a53577b62b69b P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:26.785221 19185 leader_election.cc:288] T b4bc111e183f40a0acc9dd377195f6d9 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Requested vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:26.785383 19185 raft_consensus.cc:482] T 27f94f2d8f50495dababe417a9f82982 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:26.785553 19185 raft_consensus.cc:504] T 27f94f2d8f50495dababe417a9f82982 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:26.787011 18981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41678 (request call id 107) took 1250 ms. Trace:
I20240430 20:11:26.787140 18981 rpcz_store.cc:276] 0430 20:11:25.536488 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:25.537793 (+ 1305us) service_pool.cc:225] Handling call
0430 20:11:26.551639 (+1013846us) ts_tablet_manager.cc:1120] Deleting tablet 187a27a6449342f99de3c5a2f93d49fc
0430 20:11:26.551648 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:26.574720 (+ 23072us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:26.666732 (+ 92012us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:26.786973 (+120241us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1013565}
I20240430 20:11:26.787532 18981 tablet_replica.cc:330] T 609a1a7d22e84b69bd0a53577b62b69b P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:26.787741 18981 raft_consensus.cc:2229] T 609a1a7d22e84b69bd0a53577b62b69b P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:26.787757 18597 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f49eda7aeca42308d01b83a19d7d037 with delete_type TABLET_DATA_DELETED (replaced by tablet 76d6e9d547ca4a848eae852944b1d117) from {username='slave'} at 127.0.0.1:40202
I20240430 20:11:26.787922 18981 raft_consensus.cc:2258] T 609a1a7d22e84b69bd0a53577b62b69b P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:26.789183 18142 catalog_manager.cc:4830] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): tablet 187a27a6449342f99de3c5a2f93d49fc (table default.Table_173d3907261b4390aa5647f01794d901 [id=f61d04c63e844a7b99d503770e5fcd87]) successfully deleted
I20240430 20:11:26.789922 18981 ts_tablet_manager.cc:1875] T 609a1a7d22e84b69bd0a53577b62b69b P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:26.790400 18597 tablet_service.cc:1514] Processing DeleteTablet for tablet f5236574672b477a9026e91adcaf70ec with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:26 UTC) from {username='slave'} at 127.0.0.1:40202
I20240430 20:11:26.791139 18154 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_54f6ac44723049d3bf4d7cd8c313fe44" } new_table_name: "default.Table_e9655558850747968a52e5885a113f59" modify_external_catalogs: true
I20240430 20:11:26.792096 18623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4bc111e183f40a0acc9dd377195f6d9" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:26.792343 18623 raft_consensus.cc:3037] T b4bc111e183f40a0acc9dd377195f6d9 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:26.792728 18622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27f94f2d8f50495dababe417a9f82982" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:26.792929 18622 raft_consensus.cc:2454] T 27f94f2d8f50495dababe417a9f82982 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 0.
I20240430 20:11:26.793120 18149 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_f196eb2d3b014b878b9640fbc44cd952" } new_table_name: "default.Table_ab061310f439426dbd5a65210ab9b573" modify_external_catalogs: true
I20240430 20:11:26.797777 18595 tablet_service.cc:1514] Processing DeleteTablet for tablet aacabead3c874e77a47d2ecf19dbfa82 with delete_type TABLET_DATA_DELETED (replaced by tablet fd40967ab4a04001819e16c06a6c4300) from {username='slave'} at 127.0.0.1:40202
I20240430 20:11:26.802983 18415 leader_election.cc:302] T 27f94f2d8f50495dababe417a9f82982 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:26.804059 19186 raft_consensus.cc:2781] T 27f94f2d8f50495dababe417a9f82982 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:26.804101 18147 master_service.cc:946] ReplaceTablet: received request to replace tablet c9ff972ade694938888514979e1638aa from {username='slave'} at 127.0.0.1:54524
I20240430 20:11:26.804214 19186 raft_consensus.cc:482] T 27f94f2d8f50495dababe417a9f82982 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:26.804363 19186 raft_consensus.cc:3037] T 27f94f2d8f50495dababe417a9f82982 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
W20240430 20:11:26.807274 18415 consensus_peers.cc:485] T 59548d560b4e4f7ca034d730aca79364 P 2efb542aa55e439b99ccb98d1c5c884a -> Peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Couldn't send request to peer 5e1c6397f09747f091534faf93542710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 59548d560b4e4f7ca034d730aca79364. This is attempt 1: this message will repeat every 5th retry.
I20240430 20:11:26.807657 18156 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54524:
name: "default.Table_a5ab8fc47d4e4d128ba4370b3249d8e5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:26.807926 18142 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet c66c142419ca4e1399ecd1c8d1e62a59 (table default.Table_4d213ee516bb406798377b6b90fa8cc1 [id=93c44abdf6b147228224388398cd7aeb], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet c66c142419ca4e1399ecd1c8d1e62a59 consensus state has no leader
W20240430 20:11:26.808485 18156 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a5ab8fc47d4e4d128ba4370b3249d8e5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:26.810899 19185 leader_election.cc:288] T 27f94f2d8f50495dababe417a9f82982 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
W20240430 20:11:26.811244 18149 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0879b02a0bcc4f619fb34cff7b9d7c4c consensus state has no leader
I20240430 20:11:26.813333 18141 catalog_manager.cc:4830] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): tablet bd7392f3dfd643cdb1634ee10ad41651 (table default.Table_b29130df231a4316bcb8f9c2be002c92 [id=5e77a9e6adfa4bf0b1c1f81be22e6499]) successfully deleted
W20240430 20:11:26.813637 18141 catalog_manager.cc:4810] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): delete failed for tablet d9e578c99058424a9b86390641f9c34f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d9e578c99058424a9b86390641f9c34f already in progress: creating tablet
I20240430 20:11:26.813875 18983 tablet_bootstrap.cc:492] T 609a1a7d22e84b69bd0a53577b62b69b P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
W20240430 20:11:26.813866 18141 catalog_manager.cc:4810] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): delete failed for tablet fd47db5ca8b24c2b827df0a64e195866 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fd47db5ca8b24c2b827df0a64e195866 already in progress: creating tablet
I20240430 20:11:26.814065 18983 ts_tablet_manager.cc:1367] T 609a1a7d22e84b69bd0a53577b62b69b P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.032s user 0.015s sys 0.015s
I20240430 20:11:26.814908 18983 raft_consensus.cc:348] T 609a1a7d22e84b69bd0a53577b62b69b P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:26.815196 18983 raft_consensus.cc:374] T 609a1a7d22e84b69bd0a53577b62b69b P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:26.815295 18983 raft_consensus.cc:729] T 609a1a7d22e84b69bd0a53577b62b69b P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:26.815569 18983 consensus_queue.cc:260] T 609a1a7d22e84b69bd0a53577b62b69b P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:26.816183 18983 ts_tablet_manager.cc:1398] T 609a1a7d22e84b69bd0a53577b62b69b P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240430 20:11:26.821457 18151 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet c1c538b753374ac9ad7d4ef7c25182c2
I20240430 20:11:26.822010 18985 tablet_bootstrap.cc:492] T 680553d4d0bc4fcb839235c1832757e1 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:26.823488 18983 tablet_bootstrap.cc:492] T a0e8e07d363f48c698c7e2b17edc1545 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:26.825322 18983 tablet_bootstrap.cc:654] T a0e8e07d363f48c698c7e2b17edc1545 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:26.832235 18985 tablet_bootstrap.cc:654] T 680553d4d0bc4fcb839235c1832757e1 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:26.833101 18151 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_4d213ee516bb406798377b6b90fa8cc1" } new_table_name: "default.Table_0443359dbeff437fb6cffb0db33cbb30" modify_external_catalogs: true
W20240430 20:11:26.811266 18154 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 75ebf051383e42529dcb5f7dd0770a4b has no consensus state
I20240430 20:11:26.773852 19264 raft_consensus.cc:482] T 81787eccca6c42c1b421377a3bf9ebe6 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:26.842005 19264 raft_consensus.cc:504] T 81787eccca6c42c1b421377a3bf9ebe6 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:26.843026 19264 leader_election.cc:288] T 81787eccca6c42c1b421377a3bf9ebe6 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:26.776922 18734 tablet_service.cc:1514] Processing DeleteTablet for tablet 68ca48351cd4412381f5195d81109016 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:26 UTC) from {username='slave'} at 127.0.0.1:59818
I20240430 20:11:26.843475 18620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81787eccca6c42c1b421377a3bf9ebe6" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:26.843739 18620 raft_consensus.cc:2454] T 81787eccca6c42c1b421377a3bf9ebe6 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:26.843719 18482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81787eccca6c42c1b421377a3bf9ebe6" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:26.844053 18482 raft_consensus.cc:2454] T 81787eccca6c42c1b421377a3bf9ebe6 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:26.778767 18988 ts_tablet_manager.cc:1398] T 27f94f2d8f50495dababe417a9f82982 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.097s user 0.006s sys 0.022s
I20240430 20:11:26.780543 18743 tablet_service.cc:1514] Processing DeleteTablet for tablet f5236574672b477a9026e91adcaf70ec with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:26 UTC) from {username='slave'} at 127.0.0.1:59818
I20240430 20:11:26.844740 18988 tablet_bootstrap.cc:492] T fd47db5ca8b24c2b827df0a64e195866 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:26.788939 18764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27f94f2d8f50495dababe417a9f82982" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:26.844998 18764 raft_consensus.cc:2454] T 27f94f2d8f50495dababe417a9f82982 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 0.
I20240430 20:11:26.789034 18763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4bc111e183f40a0acc9dd377195f6d9" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:26.845502 18763 raft_consensus.cc:3037] T b4bc111e183f40a0acc9dd377195f6d9 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:26.846552 18988 tablet_bootstrap.cc:654] T fd47db5ca8b24c2b827df0a64e195866 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:26.784680 19261 consensus_queue.cc:1035] T 81328e904ca94e2eb5ec94dd3ffb3c90 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.063s
I20240430 20:11:26.822857 19267 raft_consensus.cc:482] T 0879b02a0bcc4f619fb34cff7b9d7c4c P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:26.844458 18696 leader_election.cc:302] T 81787eccca6c42c1b421377a3bf9ebe6 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:26.849745 19267 raft_consensus.cc:504] T 0879b02a0bcc4f619fb34cff7b9d7c4c P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:26.803273 18740 tablet_service.cc:1514] Processing DeleteTablet for tablet 27f94f2d8f50495dababe417a9f82982 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:24 UTC) from {username='slave'} at 127.0.0.1:59818
I20240430 20:11:26.850636 19267 leader_election.cc:288] T 0879b02a0bcc4f619fb34cff7b9d7c4c P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:26.850797 19267 raft_consensus.cc:2781] T 81787eccca6c42c1b421377a3bf9ebe6 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:26.850890 19267 raft_consensus.cc:482] T 81787eccca6c42c1b421377a3bf9ebe6 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:26.850968 19267 raft_consensus.cc:3037] T 81787eccca6c42c1b421377a3bf9ebe6 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:26.811661 19191 raft_consensus.cc:504] T 3f49eda7aeca42308d01b83a19d7d037 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:26.851797 19191 leader_election.cc:288] T 3f49eda7aeca42308d01b83a19d7d037 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Requested vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:26.852031 18762 raft_consensus.cc:2379] T 3f49eda7aeca42308d01b83a19d7d037 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2efb542aa55e439b99ccb98d1c5c884a in current term 1: Already voted for candidate 3c51c33d5616482cafb68e26688820f4 in this term.
W20240430 20:11:26.853055 18151 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c66c142419ca4e1399ecd1c8d1e62a59 consensus state has no leader
I20240430 20:11:26.853794 18481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0879b02a0bcc4f619fb34cff7b9d7c4c" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:26.853806 18483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f49eda7aeca42308d01b83a19d7d037" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:26.854118 18481 raft_consensus.cc:2454] T 0879b02a0bcc4f619fb34cff7b9d7c4c P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:26.854311 18483 raft_consensus.cc:2379] T 3f49eda7aeca42308d01b83a19d7d037 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 3c51c33d5616482cafb68e26688820f4 in current term 1: Already voted for candidate 2efb542aa55e439b99ccb98d1c5c884a in this term.
I20240430 20:11:26.855091 18622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f49eda7aeca42308d01b83a19d7d037" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:26.855715 18620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0879b02a0bcc4f619fb34cff7b9d7c4c" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:26.856433 18699 leader_election.cc:302] T 0879b02a0bcc4f619fb34cff7b9d7c4c P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters:
I20240430 20:11:26.856951 19261 raft_consensus.cc:2781] T 0879b02a0bcc4f619fb34cff7b9d7c4c P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:26.857079 19261 raft_consensus.cc:482] T 0879b02a0bcc4f619fb34cff7b9d7c4c P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:26.857175 19261 raft_consensus.cc:3037] T 0879b02a0bcc4f619fb34cff7b9d7c4c P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:26.857829 18696 leader_election.cc:302] T 3f49eda7aeca42308d01b83a19d7d037 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710
W20240430 20:11:26.858268 18696 leader_election.cc:341] T 0879b02a0bcc4f619fb34cff7b9d7c4c P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:26.858556 19191 raft_consensus.cc:2726] T 3f49eda7aeca42308d01b83a19d7d037 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240430 20:11:26.813207 18987 tablet_replica.cc:330] T 17843ea77664447eaaa19666e0445ada P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:26.858968 18987 raft_consensus.cc:2229] T 17843ea77664447eaaa19666e0445ada P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:26.859184 18987 raft_consensus.cc:2258] T 17843ea77664447eaaa19666e0445ada P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:26.859777 18987 ts_tablet_manager.cc:1875] T 17843ea77664447eaaa19666e0445ada P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:26.860114 19186 raft_consensus.cc:504] T 27f94f2d8f50495dababe417a9f82982 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:26.860942 19186 leader_election.cc:288] T 27f94f2d8f50495dababe417a9f82982 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Requested vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:26.861004 18621 raft_consensus.cc:2454] T 3f49eda7aeca42308d01b83a19d7d037 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 1.
I20240430 20:11:26.864306 18415 leader_election.cc:302] T 3f49eda7aeca42308d01b83a19d7d037 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters: 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:26.864746 18621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27f94f2d8f50495dababe417a9f82982" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:26.865022 18621 raft_consensus.cc:3037] T 27f94f2d8f50495dababe417a9f82982 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:26.865952 19186 raft_consensus.cc:2781] T 3f49eda7aeca42308d01b83a19d7d037 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:26.866122 19186 raft_consensus.cc:686] T 3f49eda7aeca42308d01b83a19d7d037 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Becoming Leader. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Running, Role: LEADER
I20240430 20:11:26.866393 19186 consensus_queue.cc:237] T 3f49eda7aeca42308d01b83a19d7d037 P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:26.867029 18155 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 680553d4d0bc4fcb839235c1832757e1 (table default.Table_49a012a7c4c8457fadb0a8f0fc9654c6 [id=eecb2f8eeef547c5ab1abe6a2430e592]). Expected version 1 got 0
I20240430 20:11:26.868566 18148 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 0879b02a0bcc4f619fb34cff7b9d7c4c (table default.Table_ab061310f439426dbd5a65210ab9b573 [id=6d6fa10e4bcc437aaee661cd8abb441f]). Expected version 1 got 0
I20240430 20:11:26.869848 18762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27f94f2d8f50495dababe417a9f82982" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:26.870079 18762 raft_consensus.cc:3037] T 27f94f2d8f50495dababe417a9f82982 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:26.872341 18595 tablet_service.cc:1467] Processing CreateTablet for tablet 12c6e131fa904d23865b8673b58a7e33 (DEFAULT_TABLE table=default.Table_3106434588bd4241a314064d6a8749c3 [id=19f4e6decedf4ee59fccc0a24a05cedd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:26.872340 18454 tablet_service.cc:1467] Processing CreateTablet for tablet eed6338ad3d64859af087ee15e1ec798 (DEFAULT_TABLE table=default.Table_1dc3c3ef217d4f8b9e8fdb60180c5d0f [id=bc29c76ef4a448a2863c278309b548e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:26.872920 18455 tablet_service.cc:1467] Processing CreateTablet for tablet 20d166756819406ba85044f4fee9c4ce (DEFAULT_TABLE table=default.Table_b331bf8aeaae4911ac01ed3fbd2627bd [id=1bf5c29b08134670984b6569db298129]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:26.873328 18600 tablet_service.cc:1467] Processing CreateTablet for tablet 47c37876694c4a4abb1091c631dab7a5 (DEFAULT_TABLE table=default.Table_85072f20d90d4470bf3f65189b19fd51 [id=0fd2754282af4b2b9fcc7952be704cd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:26.873394 18457 tablet_service.cc:1467] Processing CreateTablet for tablet 12c6e131fa904d23865b8673b58a7e33 (DEFAULT_TABLE table=default.Table_3106434588bd4241a314064d6a8749c3 [id=19f4e6decedf4ee59fccc0a24a05cedd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:26.873770 18602 tablet_service.cc:1467] Processing CreateTablet for tablet eed6338ad3d64859af087ee15e1ec798 (DEFAULT_TABLE table=default.Table_1dc3c3ef217d4f8b9e8fdb60180c5d0f [id=bc29c76ef4a448a2863c278309b548e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:26.874009 19180 raft_consensus.cc:482] T 187a27a6449342f99de3c5a2f93d49fc P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:26.874089 18461 tablet_service.cc:1467] Processing CreateTablet for tablet 1555924b5f1444398f12ee82c3cb5b13 (DEFAULT_TABLE table=default.Table_388d774462294cab8342c6de331f9712 [id=aca48e01301e4f5f9562df8c87829b9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:26.874292 19180 raft_consensus.cc:504] T 187a27a6449342f99de3c5a2f93d49fc P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:26.874543 18454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eed6338ad3d64859af087ee15e1ec798. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:26.767424 19268 raft_consensus.cc:482] T 187a27a6449342f99de3c5a2f93d49fc P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:26.874866 19268 raft_consensus.cc:504] T 187a27a6449342f99de3c5a2f93d49fc P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:26.875581 18457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12c6e131fa904d23865b8673b58a7e33. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:26.876219 18455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20d166756819406ba85044f4fee9c4ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:26.876010 19265 raft_consensus.cc:482] T a0e8e07d363f48c698c7e2b17edc1545 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:26.876379 19265 raft_consensus.cc:504] T a0e8e07d363f48c698c7e2b17edc1545 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:26.876893 18981 ts_tablet_manager.cc:1888] T 609a1a7d22e84b69bd0a53577b62b69b P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:26.877055 18981 log.cc:1196] T 609a1a7d22e84b69bd0a53577b62b69b P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/609a1a7d22e84b69bd0a53577b62b69b
I20240430 20:11:26.877561 18981 ts_tablet_manager.cc:1909] T 609a1a7d22e84b69bd0a53577b62b69b P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:26.878125 18743 tablet_service.cc:1467] Processing CreateTablet for tablet 12c6e131fa904d23865b8673b58a7e33 (DEFAULT_TABLE table=default.Table_3106434588bd4241a314064d6a8749c3 [id=19f4e6decedf4ee59fccc0a24a05cedd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:26.878326 18597 tablet_service.cc:1467] Processing CreateTablet for tablet 1555924b5f1444398f12ee82c3cb5b13 (DEFAULT_TABLE table=default.Table_388d774462294cab8342c6de331f9712 [id=aca48e01301e4f5f9562df8c87829b9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:26.878476 18740 tablet_service.cc:1467] Processing CreateTablet for tablet 20d166756819406ba85044f4fee9c4ce (DEFAULT_TABLE table=default.Table_b331bf8aeaae4911ac01ed3fbd2627bd [id=1bf5c29b08134670984b6569db298129]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:26.878722 18461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1555924b5f1444398f12ee82c3cb5b13. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:26.878767 18598 tablet_service.cc:1467] Processing CreateTablet for tablet 20d166756819406ba85044f4fee9c4ce (DEFAULT_TABLE table=default.Table_b331bf8aeaae4911ac01ed3fbd2627bd [id=1bf5c29b08134670984b6569db298129]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:26.879210 18734 tablet_service.cc:1467] Processing CreateTablet for tablet 1555924b5f1444398f12ee82c3cb5b13 (DEFAULT_TABLE table=default.Table_388d774462294cab8342c6de331f9712 [id=aca48e01301e4f5f9562df8c87829b9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:26.881178 19180 leader_election.cc:288] T 187a27a6449342f99de3c5a2f93d49fc P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:26.881465 18155 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54524:
name: "default.Table_2ec9639eca4a47439b7859929cd705bc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240430 20:11:26.881633 18622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0e8e07d363f48c698c7e2b17edc1545" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:26.882054 18620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "187a27a6449342f99de3c5a2f93d49fc" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
W20240430 20:11:26.882195 18155 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2ec9639eca4a47439b7859929cd705bc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:26.882287 18620 raft_consensus.cc:2454] T 187a27a6449342f99de3c5a2f93d49fc P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
W20240430 20:11:26.882958 18696 leader_election.cc:341] T a0e8e07d363f48c698c7e2b17edc1545 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:26.883219 18696 leader_election.cc:302] T 187a27a6449342f99de3c5a2f93d49fc P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:26.883652 19191 raft_consensus.cc:2781] T 187a27a6449342f99de3c5a2f93d49fc P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:26.883749 19191 raft_consensus.cc:482] T 187a27a6449342f99de3c5a2f93d49fc P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:26.883859 19191 raft_consensus.cc:3037] T 187a27a6449342f99de3c5a2f93d49fc P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:26.884268 18764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "187a27a6449342f99de3c5a2f93d49fc" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:26.885038 18148 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54524:
name: "default.Table_3f42877693b04aaa837c89b8983d5a72"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:26.885830 18148 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3f42877693b04aaa837c89b8983d5a72 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:26.886443 18738 tablet_service.cc:1467] Processing CreateTablet for tablet eed6338ad3d64859af087ee15e1ec798 (DEFAULT_TABLE table=default.Table_1dc3c3ef217d4f8b9e8fdb60180c5d0f [id=bc29c76ef4a448a2863c278309b548e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:26.887020 18742 tablet_service.cc:1467] Processing CreateTablet for tablet 47c37876694c4a4abb1091c631dab7a5 (DEFAULT_TABLE table=default.Table_85072f20d90d4470bf3f65189b19fd51 [id=0fd2754282af4b2b9fcc7952be704cd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:26.888511 19268 leader_election.cc:288] T 187a27a6449342f99de3c5a2f93d49fc P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:26.888813 19265 leader_election.cc:288] T a0e8e07d363f48c698c7e2b17edc1545 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:26.890805 18603 tablet_service.cc:1467] Processing CreateTablet for tablet 294644d7a16b4ca2bbb6d106bcb8c7ff (DEFAULT_TABLE table=default.Table_fd3ecbe6c2cc4148a41593f6d47cf1ec [id=97d8a36ddfae4bb5b99c04a7a08544af]), partition=RANGE (key) PARTITION UNBOUNDED
W20240430 20:11:26.891376 18558 leader_election.cc:341] T 187a27a6449342f99de3c5a2f93d49fc P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Not found: Tablet not found: 187a27a6449342f99de3c5a2f93d49fc
I20240430 20:11:26.892735 18744 tablet_service.cc:1467] Processing CreateTablet for tablet 294644d7a16b4ca2bbb6d106bcb8c7ff (DEFAULT_TABLE table=default.Table_fd3ecbe6c2cc4148a41593f6d47cf1ec [id=97d8a36ddfae4bb5b99c04a7a08544af]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:26.894007 18152 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: Sending DeleteTablet for 3 replicas of tablet 4e97f7315f8b4ac29db16e00fc35bf99
I20240430 20:11:26.894546 18763 raft_consensus.cc:2454] T b4bc111e183f40a0acc9dd377195f6d9 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 1.
I20240430 20:11:26.897569 18591 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e97f7315f8b4ac29db16e00fc35bf99 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:26 UTC) from {username='slave'} at 127.0.0.1:40202
I20240430 20:11:26.899044 18453 tablet_service.cc:1467] Processing CreateTablet for tablet 47c37876694c4a4abb1091c631dab7a5 (DEFAULT_TABLE table=default.Table_85072f20d90d4470bf3f65189b19fd51 [id=0fd2754282af4b2b9fcc7952be704cd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:26.899947 18154 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_1dc3c3ef217d4f8b9e8fdb60180c5d0f" } modify_external_catalogs: true
I20240430 20:11:26.900175 18154 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_1dc3c3ef217d4f8b9e8fdb60180c5d0f" } modify_external_catalogs: true
I20240430 20:11:26.900846 18591 tablet_service.cc:1467] Processing CreateTablet for tablet b76a4105a2974fe280e759349037d027 (DEFAULT_TABLE table=default.Table_95220daaf6d44c08beff4ed6abdbea79 [id=75f0256acc45402d8b6809d92c38a1d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:26.900981 18736 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e97f7315f8b4ac29db16e00fc35bf99 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:26 UTC) from {username='slave'} at 127.0.0.1:59818
I20240430 20:11:26.901239 18453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47c37876694c4a4abb1091c631dab7a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:26.901615 18149 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54524:
name: "default.Table_eaa091213b264ad09f7ea16af281479f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240430 20:11:26.901847 18988 tablet_bootstrap.cc:492] T fd47db5ca8b24c2b827df0a64e195866 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:26.901988 18988 ts_tablet_manager.cc:1367] T fd47db5ca8b24c2b827df0a64e195866 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.057s user 0.008s sys 0.000s
W20240430 20:11:26.902190 18149 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_eaa091213b264ad09f7ea16af281479f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:26.902850 18988 raft_consensus.cc:348] T fd47db5ca8b24c2b827df0a64e195866 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:26.903059 18988 raft_consensus.cc:374] T fd47db5ca8b24c2b827df0a64e195866 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:26.903412 18151 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54524:
name: "default.Table_b6ff64dbfc324fb199b8e6eebd51fd50"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:26.903901 18151 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b6ff64dbfc324fb199b8e6eebd51fd50 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:26.903149 18988 raft_consensus.cc:729] T fd47db5ca8b24c2b827df0a64e195866 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:26.904255 18988 consensus_queue.cc:260] T fd47db5ca8b24c2b827df0a64e195866 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:26.904413 18481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0e8e07d363f48c698c7e2b17edc1545" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:26.904886 18988 ts_tablet_manager.cc:1398] T fd47db5ca8b24c2b827df0a64e195866 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240430 20:11:26.905536 18152 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54524:
name: "default.Table_7b72c19b4765456dba543151abaf1d44"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:26.906028 18152 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7b72c19b4765456dba543151abaf1d44 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240430 20:11:26.906049 18699 leader_election.cc:341] T a0e8e07d363f48c698c7e2b17edc1545 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:26.906237 18699 leader_election.cc:302] T a0e8e07d363f48c698c7e2b17edc1545 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710
W20240430 20:11:26.906903 18699 leader_election.cc:341] T 187a27a6449342f99de3c5a2f93d49fc P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Not found: Tablet not found: 187a27a6449342f99de3c5a2f93d49fc
I20240430 20:11:26.909314 19268 raft_consensus.cc:2726] T a0e8e07d363f48c698c7e2b17edc1545 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:26.911864 18417 leader_election.cc:302] T b4bc111e183f40a0acc9dd377195f6d9 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters:
I20240430 20:11:26.912590 18452 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e97f7315f8b4ac29db16e00fc35bf99 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:26 UTC) from {username='slave'} at 127.0.0.1:41678
I20240430 20:11:26.913762 18736 tablet_service.cc:1514] Processing DeleteTablet for tablet fd47db5ca8b24c2b827df0a64e195866 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:26 UTC) from {username='slave'} at 127.0.0.1:59818
I20240430 20:11:26.913985 18459 tablet_service.cc:1467] Processing CreateTablet for tablet 294644d7a16b4ca2bbb6d106bcb8c7ff (DEFAULT_TABLE table=default.Table_fd3ecbe6c2cc4148a41593f6d47cf1ec [id=97d8a36ddfae4bb5b99c04a7a08544af]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:26.914402 18459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 294644d7a16b4ca2bbb6d106bcb8c7ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:26.916394 19180 raft_consensus.cc:482] T 680553d4d0bc4fcb839235c1832757e1 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:26.916556 19180 raft_consensus.cc:504] T 680553d4d0bc4fcb839235c1832757e1 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:26.917268 19180 leader_election.cc:288] T 680553d4d0bc4fcb839235c1832757e1 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:26.918516 18763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680553d4d0bc4fcb839235c1832757e1" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:26.918915 18763 raft_consensus.cc:2454] T 680553d4d0bc4fcb839235c1832757e1 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 0.
I20240430 20:11:26.919499 18557 leader_election.cc:302] T 680553d4d0bc4fcb839235c1832757e1 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:26.919867 19180 raft_consensus.cc:2781] T 680553d4d0bc4fcb839235c1832757e1 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:26.920034 19180 raft_consensus.cc:482] T 680553d4d0bc4fcb839235c1832757e1 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:26.920146 19180 raft_consensus.cc:3037] T 680553d4d0bc4fcb839235c1832757e1 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:26.921627 18155 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54524:
name: "default.Table_8c7f3f5e74f64d409fbecfce64acf648"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240430 20:11:26.921907 18988 tablet_bootstrap.cc:492] T 15674aeb56b74210a4a2809608fd4457 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
W20240430 20:11:26.922215 18155 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8c7f3f5e74f64d409fbecfce64acf648 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:26.923822 18736 tablet_service.cc:1467] Processing CreateTablet for tablet b76a4105a2974fe280e759349037d027 (DEFAULT_TABLE table=default.Table_95220daaf6d44c08beff4ed6abdbea79 [id=75f0256acc45402d8b6809d92c38a1d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:26.925168 18735 tablet_service.cc:1467] Processing CreateTablet for tablet dd9b893d48684cfea7c242b93ecba01f (DEFAULT_TABLE table=default.Table_4bce1dae3391497ebea5f1b536700d62 [id=8f0cb0ef65d84fb8964209dd501c0b5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:26.926199 18596 tablet_service.cc:1467] Processing CreateTablet for tablet dd9b893d48684cfea7c242b93ecba01f (DEFAULT_TABLE table=default.Table_4bce1dae3391497ebea5f1b536700d62 [id=8f0cb0ef65d84fb8964209dd501c0b5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:26.928438 18147 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: Sending DeleteTablet for 3 replicas of tablet c9ff972ade694938888514979e1638aa
I20240430 20:11:26.928553 18483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680553d4d0bc4fcb839235c1832757e1" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:26.929195 18593 tablet_service.cc:1514] Processing DeleteTablet for tablet c9ff972ade694938888514979e1638aa with delete_type TABLET_DATA_DELETED (replaced by tablet a9e0659564894b0e8cb7f600bb07a085) from {username='slave'} at 127.0.0.1:40202
I20240430 20:11:26.929710 18463 tablet_service.cc:1467] Processing CreateTablet for tablet dd9b893d48684cfea7c242b93ecba01f (DEFAULT_TABLE table=default.Table_4bce1dae3391497ebea5f1b536700d62 [id=8f0cb0ef65d84fb8964209dd501c0b5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:26.930043 18737 tablet_service.cc:1514] Processing DeleteTablet for tablet c9ff972ade694938888514979e1638aa with delete_type TABLET_DATA_DELETED (replaced by tablet a9e0659564894b0e8cb7f600bb07a085) from {username='slave'} at 127.0.0.1:59818
I20240430 20:11:26.930567 18147 catalog_manager.cc:6554] ReplaceTablet: tablet c9ff972ade694938888514979e1638aa deleted and replaced by tablet a9e0659564894b0e8cb7f600bb07a085
W20240430 20:11:26.930799 18558 leader_election.cc:341] T 680553d4d0bc4fcb839235c1832757e1 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:26.932322 18593 tablet_service.cc:1467] Processing CreateTablet for tablet a05e32a33aa7488eb287cc6314c09c44 (DEFAULT_TABLE table=default.Table_8ddf4569d2f245469f55cb9fef2f5fb1 [id=955d833ff8f14bf6957abadf9ef669e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:26.932701 18463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd9b893d48684cfea7c242b93ecba01f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:26.934266 19186 raft_consensus.cc:2781] T b4bc111e183f40a0acc9dd377195f6d9 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:26.935026 18737 tablet_service.cc:1467] Processing CreateTablet for tablet a05e32a33aa7488eb287cc6314c09c44 (DEFAULT_TABLE table=default.Table_8ddf4569d2f245469f55cb9fef2f5fb1 [id=955d833ff8f14bf6957abadf9ef669e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:26.935338 18988 tablet_bootstrap.cc:654] T 15674aeb56b74210a4a2809608fd4457 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:26.935245 18601 tablet_service.cc:1467] Processing CreateTablet for tablet b72fe1f0713b4076a81a3fc1c74f7fa7 (DEFAULT_TABLE table=default.Table_16b082d140234e1c9a1e70654839b4cd [id=c0d8dc87d1f9426a85f6176f161595a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:26.936319 18451 tablet_service.cc:1514] Processing DeleteTablet for tablet c9ff972ade694938888514979e1638aa with delete_type TABLET_DATA_DELETED (replaced by tablet a9e0659564894b0e8cb7f600bb07a085) from {username='slave'} at 127.0.0.1:41678
I20240430 20:11:26.936844 18599 tablet_service.cc:1467] Processing CreateTablet for tablet 53a899fa5a424a0d99f1b47925e8c7ad (DEFAULT_TABLE table=default.Table_6213f980d6a44e099a7a989ceb23c08a [id=608890834da843fa9af0cfc0d548029e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:26.938704 19186 raft_consensus.cc:686] T b4bc111e183f40a0acc9dd377195f6d9 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Becoming Leader. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Running, Role: LEADER
I20240430 20:11:26.940215 19230 raft_consensus.cc:482] T 609a1a7d22e84b69bd0a53577b62b69b P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:26.940389 19230 raft_consensus.cc:504] T 609a1a7d22e84b69bd0a53577b62b69b P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:26.941213 19230 leader_election.cc:288] T 609a1a7d22e84b69bd0a53577b62b69b P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:26.941993 18983 tablet_bootstrap.cc:492] T a0e8e07d363f48c698c7e2b17edc1545 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:26.942229 18592 tablet_service.cc:1467] Processing CreateTablet for tablet e3e74e76ab2b4fe492ffcc1838656fd1 (DEFAULT_TABLE table=default.Table_5d3cb1a25c884804b05b07531e1b6965 [id=55e60456c89d441a8b5874a2063d52ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:26.938472 18452 tablet_service.cc:1467] Processing CreateTablet for tablet b76a4105a2974fe280e759349037d027 (DEFAULT_TABLE table=default.Table_95220daaf6d44c08beff4ed6abdbea79 [id=75f0256acc45402d8b6809d92c38a1d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:26.942973 18452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b76a4105a2974fe280e759349037d027. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:26.943220 18983 ts_tablet_manager.cc:1367] T a0e8e07d363f48c698c7e2b17edc1545 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.120s user 0.051s sys 0.030s
I20240430 20:11:26.943328 18590 tablet_service.cc:1467] Processing CreateTablet for tablet 01436e5b6232494babbb414342f03090 (DEFAULT_TABLE table=default.Table_a8d5ac73c6d047698d87b3d1ba34e66f [id=9c80193cde2346c982cf84ae9612e2f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:26.944172 18983 raft_consensus.cc:348] T a0e8e07d363f48c698c7e2b17edc1545 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:26.945019 18589 tablet_service.cc:1467] Processing CreateTablet for tablet 75ebf051383e42529dcb5f7dd0770a4b (DEFAULT_TABLE table=default.Table_e9655558850747968a52e5885a113f59 [id=9c272f6ad56440b682f12f027af57bc4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:26.946897 18983 raft_consensus.cc:374] T a0e8e07d363f48c698c7e2b17edc1545 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:26.947032 18983 raft_consensus.cc:729] T a0e8e07d363f48c698c7e2b17edc1545 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:26.946691 18588 tablet_service.cc:1467] Processing CreateTablet for tablet acd23f0a63d5448aaa7ed78422e67bad (DEFAULT_TABLE table=default.Table_a5743a67104f449d8d26a63b9f85eb8b [id=69f37db4de634a3185613e40013e0ed7]), partition=RANGE (key) PARTITION UNBOUNDED
W20240430 20:11:26.947181 18558 leader_election.cc:341] T 609a1a7d22e84b69bd0a53577b62b69b P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Not found: Tablet not found: 609a1a7d22e84b69bd0a53577b62b69b
I20240430 20:11:26.947310 18983 consensus_queue.cc:260] T a0e8e07d363f48c698c7e2b17edc1545 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:26.948616 18983 ts_tablet_manager.cc:1398] T a0e8e07d363f48c698c7e2b17edc1545 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240430 20:11:26.948899 18983 tablet_bootstrap.cc:492] T 0879b02a0bcc4f619fb34cff7b9d7c4c P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:26.953967 18596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd9b893d48684cfea7c242b93ecba01f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:26.954015 18980 ts_tablet_manager.cc:1888] T b0e6cc37c6b1488fa88359d16b12a576 P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:26.954149 18980 log.cc:1196] T b0e6cc37c6b1488fa88359d16b12a576 P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/b0e6cc37c6b1488fa88359d16b12a576
I20240430 20:11:26.954604 18980 ts_tablet_manager.cc:1909] T b0e6cc37c6b1488fa88359d16b12a576 P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:26.954864 18623 raft_consensus.cc:2454] T b4bc111e183f40a0acc9dd377195f6d9 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 1.
I20240430 20:11:26.955199 18595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12c6e131fa904d23865b8673b58a7e33. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:26.955701 18600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47c37876694c4a4abb1091c631dab7a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:26.956149 18602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eed6338ad3d64859af087ee15e1ec798. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:26.956550 18597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1555924b5f1444398f12ee82c3cb5b13. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:26.957002 18598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20d166756819406ba85044f4fee9c4ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:26.956946 18147 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_30f70b4e38614496ae012d519a7338d1" } new_table_name: "default.Table_fda07fd7279c484db2bee67e8c859855" modify_external_catalogs: true
I20240430 20:11:26.957420 18603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 294644d7a16b4ca2bbb6d106bcb8c7ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:26.957851 18591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b76a4105a2974fe280e759349037d027. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:26.958310 18593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a05e32a33aa7488eb287cc6314c09c44. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:26.958755 18601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b72fe1f0713b4076a81a3fc1c74f7fa7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:26.959156 18599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53a899fa5a424a0d99f1b47925e8c7ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:26.959117 19267 raft_consensus.cc:504] T 81787eccca6c42c1b421377a3bf9ebe6 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:26.959517 18592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3e74e76ab2b4fe492ffcc1838656fd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:26.959935 18590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01436e5b6232494babbb414342f03090. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:26.959966 19267 leader_election.cc:288] T 81787eccca6c42c1b421377a3bf9ebe6 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Requested vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:26.960340 18589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75ebf051383e42529dcb5f7dd0770a4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:26.960781 18588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acd23f0a63d5448aaa7ed78422e67bad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:26.966980 18623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81787eccca6c42c1b421377a3bf9ebe6" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:26.967254 18623 raft_consensus.cc:3037] T 81787eccca6c42c1b421377a3bf9ebe6 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:26.971149 18983 tablet_bootstrap.cc:654] T 0879b02a0bcc4f619fb34cff7b9d7c4c P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:26.973271 18154 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: Sending DeleteTablet for 3 replicas of tablet eed6338ad3d64859af087ee15e1ec798
I20240430 20:11:26.979933 18587 tablet_service.cc:1514] Processing DeleteTablet for tablet eed6338ad3d64859af087ee15e1ec798 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:26 UTC) from {username='slave'} at 127.0.0.1:40202
W20240430 20:11:26.986186 18147 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fd40967ab4a04001819e16c06a6c4300 has no consensus state
I20240430 20:11:26.987754 18152 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54524:
name: "default.Table_a7f63e5c33fd4651becff171d438c66d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240430 20:11:26.987834 18155 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_52dcbd45fadf4a4d85c4533f1d69f976" } modify_external_catalogs: true
I20240430 20:11:26.988104 18155 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_52dcbd45fadf4a4d85c4533f1d69f976" } modify_external_catalogs: true
W20240430 20:11:26.988500 18152 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a7f63e5c33fd4651becff171d438c66d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:26.991400 18151 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_ab061310f439426dbd5a65210ab9b573" } modify_external_catalogs: true
I20240430 20:11:26.991643 18151 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_ab061310f439426dbd5a65210ab9b573" } modify_external_catalogs: true
I20240430 20:11:26.992375 18147 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54524:
name: "default.Table_c5646eaabbf5443894dab99237416cc7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:26.993036 18147 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c5646eaabbf5443894dab99237416cc7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:26.994669 18743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12c6e131fa904d23865b8673b58a7e33. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:26.996968 18739 tablet_service.cc:1467] Processing CreateTablet for tablet e3e74e76ab2b4fe492ffcc1838656fd1 (DEFAULT_TABLE table=default.Table_5d3cb1a25c884804b05b07531e1b6965 [id=55e60456c89d441a8b5874a2063d52ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:26.997177 18732 tablet_service.cc:1467] Processing CreateTablet for tablet 01436e5b6232494babbb414342f03090 (DEFAULT_TABLE table=default.Table_a8d5ac73c6d047698d87b3d1ba34e66f [id=9c80193cde2346c982cf84ae9612e2f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:26.997702 18731 tablet_service.cc:1467] Processing CreateTablet for tablet 75ebf051383e42529dcb5f7dd0770a4b (DEFAULT_TABLE table=default.Table_e9655558850747968a52e5885a113f59 [id=9c272f6ad56440b682f12f027af57bc4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:26.998236 18733 tablet_service.cc:1467] Processing CreateTablet for tablet acd23f0a63d5448aaa7ed78422e67bad (DEFAULT_TABLE table=default.Table_a5743a67104f449d8d26a63b9f85eb8b [id=69f37db4de634a3185613e40013e0ed7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:26.998981 18741 tablet_service.cc:1467] Processing CreateTablet for tablet b72fe1f0713b4076a81a3fc1c74f7fa7 (DEFAULT_TABLE table=default.Table_16b082d140234e1c9a1e70654839b4cd [id=c0d8dc87d1f9426a85f6176f161595a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:27.001410 19267 consensus_queue.cc:1035] T 81328e904ca94e2eb5ec94dd3ffb3c90 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240430 20:11:27.001956 18729 tablet_service.cc:1514] Processing DeleteTablet for tablet eed6338ad3d64859af087ee15e1ec798 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:26 UTC) from {username='slave'} at 127.0.0.1:59818
I20240430 20:11:27.002714 18451 tablet_service.cc:1467] Processing CreateTablet for tablet a05e32a33aa7488eb287cc6314c09c44 (DEFAULT_TABLE table=default.Table_8ddf4569d2f245469f55cb9fef2f5fb1 [id=955d833ff8f14bf6957abadf9ef669e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:27.002907 18734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1555924b5f1444398f12ee82c3cb5b13. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:27.003191 18451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a05e32a33aa7488eb287cc6314c09c44. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:27.004645 18730 tablet_service.cc:1467] Processing CreateTablet for tablet 53a899fa5a424a0d99f1b47925e8c7ad (DEFAULT_TABLE table=default.Table_6213f980d6a44e099a7a989ceb23c08a [id=608890834da843fa9af0cfc0d548029e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:27.005836 18737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a05e32a33aa7488eb287cc6314c09c44. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:27.006886 18738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eed6338ad3d64859af087ee15e1ec798. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:27.006011 19186 consensus_queue.cc:237] T b4bc111e183f40a0acc9dd377195f6d9 P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:27.011129 18744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 294644d7a16b4ca2bbb6d106bcb8c7ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:27.013316 18152 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54524:
name: "default.Table_772ce91c27ce4119b6d1b1993e417f21"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:27.014024 18152 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_772ce91c27ce4119b6d1b1993e417f21 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:27.016048 18148 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_1bb7ee3d454b4c1eac6646e9b3eae7e1" } new_table_name: "default.Table_3b327aece0ea4ec4b882f2264e3f5f0c" modify_external_catalogs: true
I20240430 20:11:27.016119 18149 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_fd3ecbe6c2cc4148a41593f6d47cf1ec" } new_table_name: "default.Table_e3a0afbced214062a8b026abe2320f3e" modify_external_catalogs: true
I20240430 20:11:27.018980 18983 tablet_bootstrap.cc:492] T 0879b02a0bcc4f619fb34cff7b9d7c4c P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:27.019245 18983 ts_tablet_manager.cc:1367] T 0879b02a0bcc4f619fb34cff7b9d7c4c P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.070s user 0.000s sys 0.008s
I20240430 20:11:27.020155 18983 raft_consensus.cc:348] T 0879b02a0bcc4f619fb34cff7b9d7c4c P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:27.021278 18983 raft_consensus.cc:374] T 0879b02a0bcc4f619fb34cff7b9d7c4c P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:27.021528 18983 raft_consensus.cc:729] T 0879b02a0bcc4f619fb34cff7b9d7c4c P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:27.022071 18983 consensus_queue.cc:260] T 0879b02a0bcc4f619fb34cff7b9d7c4c P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:27.023509 18983 ts_tablet_manager.cc:1398] T 0879b02a0bcc4f619fb34cff7b9d7c4c P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240430 20:11:27.024657 18981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41678 (request call id 109) took 1389 ms. Trace:
I20240430 20:11:27.026544 18983 tablet_bootstrap.cc:492] T 15674aeb56b74210a4a2809608fd4457 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:27.027680 18147 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_a5ab8fc47d4e4d128ba4370b3249d8e5" } modify_external_catalogs: true
I20240430 20:11:27.027959 18147 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_a5ab8fc47d4e4d128ba4370b3249d8e5" } modify_external_catalogs: true
I20240430 20:11:27.029697 18483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81787eccca6c42c1b421377a3bf9ebe6" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:27.030515 18483 raft_consensus.cc:3037] T 81787eccca6c42c1b421377a3bf9ebe6 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:27.032207 18983 tablet_bootstrap.cc:654] T 15674aeb56b74210a4a2809608fd4457 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:27.030997 18981 rpcz_store.cc:276] 0430 20:11:25.548060 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:25.548116 (+ 56us) service_pool.cc:225] Handling call
0430 20:11:26.787457 (+1239341us) ts_tablet_manager.cc:1120] Deleting tablet 609a1a7d22e84b69bd0a53577b62b69b
0430 20:11:26.787468 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:26.856225 (+ 68757us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:26.876748 (+ 20523us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:26.937221 (+ 60473us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1239089}
I20240430 20:11:27.035068 18736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b76a4105a2974fe280e759349037d027. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:27.036306 18985 tablet_bootstrap.cc:492] T 680553d4d0bc4fcb839235c1832757e1 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:27.039331 18739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3e74e76ab2b4fe492ffcc1838656fd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:27.040062 18587 tablet_service.cc:1467] Processing CreateTablet for tablet 26ed12f498644d15bf45e884dcad172e (DEFAULT_TABLE table=default.Table_3f42877693b04aaa837c89b8983d5a72 [id=6d499a625cee4df2a4f40ee189cc6220]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:27.040609 18731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75ebf051383e42529dcb5f7dd0770a4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:27.040719 18587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26ed12f498644d15bf45e884dcad172e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:27.041585 18741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b72fe1f0713b4076a81a3fc1c74f7fa7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:27.041996 18742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47c37876694c4a4abb1091c631dab7a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:27.041914 18458 tablet_service.cc:1467] Processing CreateTablet for tablet b72fe1f0713b4076a81a3fc1c74f7fa7 (DEFAULT_TABLE table=default.Table_16b082d140234e1c9a1e70654839b4cd [id=c0d8dc87d1f9426a85f6176f161595a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:27.042402 18735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd9b893d48684cfea7c242b93ecba01f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:27.042860 18981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41678 (request call id 108) took 1494 ms. Trace:
I20240430 20:11:27.043000 18981 rpcz_store.cc:276] 0430 20:11:25.547310 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:25.548520 (+ 1210us) service_pool.cc:225] Handling call
0430 20:11:27.039625 (+1491105us) ts_tablet_manager.cc:1120] Deleting tablet c5fd0730133c472299c93f4b0830e527
0430 20:11:27.039637 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:27.041256 (+ 1619us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1490835}
I20240430 20:11:27.044653 18985 ts_tablet_manager.cc:1367] T 680553d4d0bc4fcb839235c1832757e1 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.215s user 0.006s sys 0.000s
I20240430 20:11:27.046710 18983 tablet_bootstrap.cc:492] T 15674aeb56b74210a4a2809608fd4457 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:27.046957 18983 ts_tablet_manager.cc:1367] T 15674aeb56b74210a4a2809608fd4457 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.021s user 0.006s sys 0.005s
I20240430 20:11:27.047610 18586 tablet_service.cc:1467] Processing CreateTablet for tablet b688e03ea71940c69bda5520b62c1fc3 (DEFAULT_TABLE table=default.Table_2ec9639eca4a47439b7859929cd705bc [id=6ffe241384bd42b7a22d41154ac59e3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:27.047981 18729 tablet_service.cc:1467] Processing CreateTablet for tablet 26ed12f498644d15bf45e884dcad172e (DEFAULT_TABLE table=default.Table_3f42877693b04aaa837c89b8983d5a72 [id=6d499a625cee4df2a4f40ee189cc6220]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:27.048053 18983 raft_consensus.cc:348] T 15674aeb56b74210a4a2809608fd4457 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:27.048264 18586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b688e03ea71940c69bda5520b62c1fc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:27.048280 18983 raft_consensus.cc:374] T 15674aeb56b74210a4a2809608fd4457 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:27.048462 18983 raft_consensus.cc:729] T 15674aeb56b74210a4a2809608fd4457 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:27.048930 18733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acd23f0a63d5448aaa7ed78422e67bad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:27.048828 18983 consensus_queue.cc:260] T 15674aeb56b74210a4a2809608fd4457 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:27.049587 18983 ts_tablet_manager.cc:1398] T 15674aeb56b74210a4a2809608fd4457 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240430 20:11:27.049973 18983 tablet_bootstrap.cc:492] T 372b135dd01f44f98cd713ca8857ba61 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:27.046061 18462 tablet_service.cc:1467] Processing CreateTablet for tablet e3e74e76ab2b4fe492ffcc1838656fd1 (DEFAULT_TABLE table=default.Table_5d3cb1a25c884804b05b07531e1b6965 [id=55e60456c89d441a8b5874a2063d52ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:27.051219 18621 raft_consensus.cc:2454] T 27f94f2d8f50495dababe417a9f82982 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 1.
I20240430 20:11:27.051192 18456 tablet_service.cc:1467] Processing CreateTablet for tablet 75ebf051383e42529dcb5f7dd0770a4b (DEFAULT_TABLE table=default.Table_e9655558850747968a52e5885a113f59 [id=9c272f6ad56440b682f12f027af57bc4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:27.053593 18458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b72fe1f0713b4076a81a3fc1c74f7fa7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:27.054208 19180 raft_consensus.cc:504] T 680553d4d0bc4fcb839235c1832757e1 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:27.055440 18981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41678 (request call id 110) took 1492 ms. Trace:
W20240430 20:11:27.056093 18142 catalog_manager.cc:4810] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): delete failed for tablet c5fd0730133c472299c93f4b0830e527 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c5fd0730133c472299c93f4b0830e527 already in progress: creating tablet
I20240430 20:11:27.056077 18450 tablet_service.cc:1467] Processing CreateTablet for tablet acd23f0a63d5448aaa7ed78422e67bad (DEFAULT_TABLE table=default.Table_a5743a67104f449d8d26a63b9f85eb8b [id=69f37db4de634a3185613e40013e0ed7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:27.056687 18732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01436e5b6232494babbb414342f03090. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:27.057180 18980 tablet_replica.cc:330] T 17843ea77664447eaaa19666e0445ada P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:27.057399 18980 raft_consensus.cc:2229] T 17843ea77664447eaaa19666e0445ada P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:27.057648 18980 raft_consensus.cc:2258] T 17843ea77664447eaaa19666e0445ada P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:27.058277 18980 ts_tablet_manager.cc:1875] T 17843ea77664447eaaa19666e0445ada P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:27.058223 18454 tablet_service.cc:1467] Processing CreateTablet for tablet 53a899fa5a424a0d99f1b47925e8c7ad (DEFAULT_TABLE table=default.Table_6213f980d6a44e099a7a989ceb23c08a [id=608890834da843fa9af0cfc0d548029e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:27.058833 18462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3e74e76ab2b4fe492ffcc1838656fd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:27.059401 18981 rpcz_store.cc:276] 0430 20:11:25.551405 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:25.551471 (+ 66us) service_pool.cc:225] Handling call
0430 20:11:27.043380 (+1491909us) ts_tablet_manager.cc:1120] Deleting tablet 93d158de64994f3c9fa45a1250a4f533
0430 20:11:27.043392 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:27.043558 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1491615}
I20240430 20:11:27.060897 18150 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_73ee808b1dda40dbafb28cf5669d53bc" } new_table_name: "default.Table_0c5d4686f2ed40fb8d62c31b0166dc05" modify_external_catalogs: true
W20240430 20:11:27.061470 18142 catalog_manager.cc:4810] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): delete failed for tablet 93d158de64994f3c9fa45a1250a4f533 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 93d158de64994f3c9fa45a1250a4f533 already in progress: creating tablet
I20240430 20:11:27.062328 18156 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_2d41f75890eb40199519c2937c89e658" } new_table_name: "default.Table_210b7eaff7144b2eb7db1c3b5f7d9505" modify_external_catalogs: true
I20240430 20:11:27.067196 18740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20d166756819406ba85044f4fee9c4ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:27.067077 18728 tablet_service.cc:1467] Processing CreateTablet for tablet b688e03ea71940c69bda5520b62c1fc3 (DEFAULT_TABLE table=default.Table_2ec9639eca4a47439b7859929cd705bc [id=6ffe241384bd42b7a22d41154ac59e3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:27.073242 18730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53a899fa5a424a0d99f1b47925e8c7ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:27.076781 18623 raft_consensus.cc:2454] T 81787eccca6c42c1b421377a3bf9ebe6 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:27.081924 18696 leader_election.cc:302] T 81787eccca6c42c1b421377a3bf9ebe6 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:27.059841 18448 tablet_service.cc:1467] Processing CreateTablet for tablet 26ed12f498644d15bf45e884dcad172e (DEFAULT_TABLE table=default.Table_3f42877693b04aaa837c89b8983d5a72 [id=6d499a625cee4df2a4f40ee189cc6220]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:27.082305 18450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acd23f0a63d5448aaa7ed78422e67bad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:27.084533 18762 raft_consensus.cc:2454] T 27f94f2d8f50495dababe417a9f82982 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 1.
I20240430 20:11:27.084869 18449 tablet_service.cc:1514] Processing DeleteTablet for tablet eed6338ad3d64859af087ee15e1ec798 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:26 UTC) from {username='slave'} at 127.0.0.1:41678
I20240430 20:11:27.085747 18455 tablet_service.cc:1467] Processing CreateTablet for tablet 01436e5b6232494babbb414342f03090 (DEFAULT_TABLE table=default.Table_a8d5ac73c6d047698d87b3d1ba34e66f [id=9c80193cde2346c982cf84ae9612e2f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:27.085995 18139 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet b0e6cc37c6b1488fa88359d16b12a576 (table default.Table_32647411b85344b490966d64f135631b [id=8cd243a627f84916902191f774d1e8cc]) successfully deleted
W20240430 20:11:27.086292 18139 catalog_manager.cc:4810] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): delete failed for tablet d9e578c99058424a9b86390641f9c34f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d9e578c99058424a9b86390641f9c34f already in progress: creating tablet
W20240430 20:11:27.086547 18139 catalog_manager.cc:4810] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): delete failed for tablet fd47db5ca8b24c2b827df0a64e195866 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fd47db5ca8b24c2b827df0a64e195866 already in progress: creating tablet
I20240430 20:11:27.086799 18985 raft_consensus.cc:348] T 680553d4d0bc4fcb839235c1832757e1 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:27.087031 18985 raft_consensus.cc:374] T 680553d4d0bc4fcb839235c1832757e1 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:27.087335 18456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75ebf051383e42529dcb5f7dd0770a4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:27.088130 18985 raft_consensus.cc:729] T 680553d4d0bc4fcb839235c1832757e1 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:27.088393 18985 consensus_queue.cc:260] T 680553d4d0bc4fcb839235c1832757e1 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:27.089586 18985 ts_tablet_manager.cc:1398] T 680553d4d0bc4fcb839235c1832757e1 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.044s user 0.002s sys 0.000s
I20240430 20:11:27.090860 18415 leader_election.cc:302] T 27f94f2d8f50495dababe417a9f82982 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:27.092036 18152 master_service.cc:946] ReplaceTablet: received request to replace tablet 34ab40d8d91e401db299ddee7175ba28 from {username='slave'} at 127.0.0.1:54524
I20240430 20:11:27.092392 18447 tablet_service.cc:1467] Processing CreateTablet for tablet b688e03ea71940c69bda5520b62c1fc3 (DEFAULT_TABLE table=default.Table_2ec9639eca4a47439b7859929cd705bc [id=6ffe241384bd42b7a22d41154ac59e3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:27.093915 18154 master_service.cc:946] ReplaceTablet: received request to replace tablet 02176eee8e044a7db8cac48439e9f594 from {username='slave'} at 127.0.0.1:54524
I20240430 20:11:27.094028 18981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41678 (request call id 111) took 1485 ms. Trace:
I20240430 20:11:27.094130 18981 rpcz_store.cc:276] 0430 20:11:25.600492 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:25.600567 (+ 75us) service_pool.cc:225] Handling call
0430 20:11:27.060851 (+1460284us) ts_tablet_manager.cc:1120] Deleting tablet 1398d4d2305b48e08fdb12432bd6c4b0
0430 20:11:27.060860 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:27.086094 (+ 25234us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1459905}
I20240430 20:11:27.094533 19185 raft_consensus.cc:2781] T 27f94f2d8f50495dababe417a9f82982 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:27.094684 19185 raft_consensus.cc:686] T 27f94f2d8f50495dababe417a9f82982 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Becoming Leader. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Running, Role: LEADER
W20240430 20:11:27.095033 18417 consensus_peers.cc:485] T 17843ea77664447eaaa19666e0445ada P 2efb542aa55e439b99ccb98d1c5c884a -> Peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Couldn't send request to peer 3c51c33d5616482cafb68e26688820f4. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240430 20:11:27.096029 18446 tablet_service.cc:1467] Processing CreateTablet for tablet 4e69ca12bd10494fb013614741bc166c (DEFAULT_TABLE table=default.Table_0a397bb1e5ea4cbe87a088c7fc84e9c1 [id=9abcfa9eaaa84cd2b679fd29ec6e5ab7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:27.096823 18449 tablet_service.cc:1467] Processing CreateTablet for tablet 255c13ad8fae4ea4aa8e7b7aabb7602e (DEFAULT_TABLE table=default.Table_a5ab8fc47d4e4d128ba4370b3249d8e5 [id=1238beeef04b476b8f4dc5af2fb68012]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:27.096918 18445 tablet_service.cc:1467] Processing CreateTablet for tablet 24dcc20832804229a2664873bf6ddaa7 (DEFAULT_TABLE table=default.Table_e689526f90344c6f91dae34f22d60295 [id=2d44ec94131148048021b2ee6366ba30]), partition=RANGE (key) PARTITION UNBOUNDED
W20240430 20:11:27.097280 18142 catalog_manager.cc:4810] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): delete failed for tablet 1398d4d2305b48e08fdb12432bd6c4b0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1398d4d2305b48e08fdb12432bd6c4b0 already in progress: creating tablet
I20240430 20:11:27.099179 18585 tablet_service.cc:1467] Processing CreateTablet for tablet 802336602d8f4ac4bd7e0872d6f3dae9 (DEFAULT_TABLE table=default.Table_d997b723a0dd48f6bc2d42c9bfd6bc14 [id=7737ec4807974640aa8556cb1c3ca3ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:27.099435 18584 tablet_service.cc:1467] Processing CreateTablet for tablet c9e7660eccc448dbb99d135f82bfd261 (DEFAULT_TABLE table=default.Table_e233554a5e1d41389948fecaf498a856 [id=51d177e8498848b38c43e3f8d11f092c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:27.099820 18985 tablet_bootstrap.cc:492] T a0e8e07d363f48c698c7e2b17edc1545 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:27.099908 18585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 802336602d8f4ac4bd7e0872d6f3dae9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:27.100118 18584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9e7660eccc448dbb99d135f82bfd261. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:27.100975 18981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41678 (request call id 119) took 1408 ms. Trace:
I20240430 20:11:27.101075 18981 rpcz_store.cc:276] 0430 20:11:25.692904 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:25.693103 (+ 199us) service_pool.cc:225] Handling call
0430 20:11:27.094426 (+1401323us) ts_tablet_manager.cc:1120] Deleting tablet 26a015250eeb4750a42df93840e8d47f
0430 20:11:27.100779 (+ 6353us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:27.100914 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1397821}
I20240430 20:11:27.101414 18981 tablet_replica.cc:330] T 59548d560b4e4f7ca034d730aca79364 P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:27.101598 18981 raft_consensus.cc:2229] T 59548d560b4e4f7ca034d730aca79364 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Raft consensus shutting down.
I20240430 20:11:27.101678 18597 tablet_service.cc:1467] Processing CreateTablet for tablet 2a1e50f99c5d4b44bdbb51b58a92bc75 (DEFAULT_TABLE table=default.Table_71075ae6b4d545889273c891648c70d1 [id=0fb93f36bc17408ea56bc4cafcfd8580]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:27.101948 18981 pending_rounds.cc:62] T 59548d560b4e4f7ca034d730aca79364 P 2efb542aa55e439b99ccb98d1c5c884a: Trying to abort 1 pending ops.
I20240430 20:11:27.102010 18981 pending_rounds.cc:68] T 59548d560b4e4f7ca034d730aca79364 P 2efb542aa55e439b99ccb98d1c5c884a: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7022624299293831168 op_type: NO_OP noop_request { }
I20240430 20:11:27.102097 18597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a1e50f99c5d4b44bdbb51b58a92bc75. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:27.102173 18981 raft_consensus.cc:2866] T 59548d560b4e4f7ca034d730aca79364 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240430 20:11:27.102265 18981 raft_consensus.cc:2258] T 59548d560b4e4f7ca034d730aca79364 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:27.102870 18981 ts_tablet_manager.cc:1875] T 59548d560b4e4f7ca034d730aca79364 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:27.103036 18454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53a899fa5a424a0d99f1b47925e8c7ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:27.102991 18583 tablet_service.cc:1467] Processing CreateTablet for tablet fcc9698abda4491b98e1e879589913a8 (DEFAULT_TABLE table=default.Table_bdb66daeadd54058b7c006c6514acd6c [id=1e785b6c159b4312bd1b064271f55a49]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:27.103538 18583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcc9698abda4491b98e1e879589913a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:27.103706 18446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e69ca12bd10494fb013614741bc166c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:27.103873 18603 tablet_service.cc:1467] Processing CreateTablet for tablet 255c13ad8fae4ea4aa8e7b7aabb7602e (DEFAULT_TABLE table=default.Table_a5ab8fc47d4e4d128ba4370b3249d8e5 [id=1238beeef04b476b8f4dc5af2fb68012]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:27.104300 18603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 255c13ad8fae4ea4aa8e7b7aabb7602e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:27.104578 18447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b688e03ea71940c69bda5520b62c1fc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:27.105319 18448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26ed12f498644d15bf45e884dcad172e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:27.106277 18449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 255c13ad8fae4ea4aa8e7b7aabb7602e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:27.106901 18586 tablet_service.cc:1467] Processing CreateTablet for tablet 24dcc20832804229a2664873bf6ddaa7 (DEFAULT_TABLE table=default.Table_e689526f90344c6f91dae34f22d60295 [id=2d44ec94131148048021b2ee6366ba30]), partition=RANGE (key) PARTITION UNBOUNDED
W20240430 20:11:27.107148 18415 consensus_peers.cc:485] T a01ee128ba8a4c9fb8e125ba97239d4c P 2efb542aa55e439b99ccb98d1c5c884a -> Peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Couldn't send request to peer 5e1c6397f09747f091534faf93542710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a01ee128ba8a4c9fb8e125ba97239d4c. This is attempt 1: this message will repeat every 5th retry.
I20240430 20:11:27.107304 18586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24dcc20832804229a2664873bf6ddaa7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:27.107337 19072 raft_consensus.cc:482] T 680553d4d0bc4fcb839235c1832757e1 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:27.107484 19072 raft_consensus.cc:504] T 680553d4d0bc4fcb839235c1832757e1 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:27.108180 19072 leader_election.cc:288] T 680553d4d0bc4fcb839235c1832757e1 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:27.108453 18445 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24dcc20832804229a2664873bf6ddaa7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:27.109252 18455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01436e5b6232494babbb414342f03090. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:27.121251 18763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "609a1a7d22e84b69bd0a53577b62b69b" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:27.121353 19265 raft_consensus.cc:2781] T 81787eccca6c42c1b421377a3bf9ebe6 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:27.124514 18483 raft_consensus.cc:2454] T 81787eccca6c42c1b421377a3bf9ebe6 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:27.097579 18726 tablet_service.cc:1467] Processing CreateTablet for tablet 255c13ad8fae4ea4aa8e7b7aabb7602e (DEFAULT_TABLE table=default.Table_a5ab8fc47d4e4d128ba4370b3249d8e5 [id=1238beeef04b476b8f4dc5af2fb68012]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:27.126582 18987 ts_tablet_manager.cc:1888] T 17843ea77664447eaaa19666e0445ada P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240430 20:11:27.127375 18763 raft_consensus.cc:2454] T 609a1a7d22e84b69bd0a53577b62b69b P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 0.
W20240430 20:11:27.132434 18148 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 02176eee8e044a7db8cac48439e9f594 consensus state has no leader
I20240430 20:11:27.132820 18148 master_service.cc:946] ReplaceTablet: received request to replace tablet bd7392f3dfd643cdb1634ee10ad41651 from {username='slave'} at 127.0.0.1:54524
W20240430 20:11:27.133350 18149 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 294644d7a16b4ca2bbb6d106bcb8c7ff consensus state has no leader
I20240430 20:11:27.133445 18148 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_aa42ff1beec14dd6baf999b8962907e0" } modify_external_catalogs: true
I20240430 20:11:27.134483 18149 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54524:
name: "default.Table_4119d46f13a74c0b950aecb0744d429d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240430 20:11:27.134713 18148 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_aa42ff1beec14dd6baf999b8962907e0" } modify_external_catalogs: true
I20240430 20:11:27.134858 18146 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54524:
name: "default.Table_27f3d711beeb4387a9088ab291168bc8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:27.135685 18149 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4119d46f13a74c0b950aecb0744d429d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240430 20:11:27.136569 18146 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_27f3d711beeb4387a9088ab291168bc8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:27.137012 18147 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: Sending DeleteTablet for 3 replicas of tablet 255c13ad8fae4ea4aa8e7b7aabb7602e
I20240430 20:11:27.137605 18151 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: Sending DeleteTablet for 3 replicas of tablet 0879b02a0bcc4f619fb34cff7b9d7c4c
I20240430 20:11:27.140012 18155 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: Sending DeleteTablet for 3 replicas of tablet 13c1107ea4d04eb78a74d3f658c72fb3
I20240430 20:11:27.142550 18590 tablet_service.cc:1467] Processing CreateTablet for tablet 4e69ca12bd10494fb013614741bc166c (DEFAULT_TABLE table=default.Table_0a397bb1e5ea4cbe87a088c7fc84e9c1 [id=9abcfa9eaaa84cd2b679fd29ec6e5ab7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:27.143115 18590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e69ca12bd10494fb013614741bc166c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:27.143662 19261 raft_consensus.cc:504] T 0879b02a0bcc4f619fb34cff7b9d7c4c P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:27.146778 18147 master_service.cc:946] ReplaceTablet: received request to replace tablet 81328e904ca94e2eb5ec94dd3ffb3c90 from {username='slave'} at 127.0.0.1:54524
I20240430 20:11:27.146831 19185 consensus_queue.cc:237] T 27f94f2d8f50495dababe417a9f82982 P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:27.147552 18147 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54524:
name: "default.Table_999c55e7bef74339b7b3ab1142599447"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:27.148165 18147 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_999c55e7bef74339b7b3ab1142599447 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:27.149237 18452 tablet_service.cc:1514] Processing DeleteTablet for tablet 13c1107ea4d04eb78a74d3f658c72fb3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:26 UTC) from {username='slave'} at 127.0.0.1:41678
I20240430 20:11:27.125097 19191 raft_consensus.cc:504] T 187a27a6449342f99de3c5a2f93d49fc P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:27.150214 18457 tablet_service.cc:1514] Processing DeleteTablet for tablet 0879b02a0bcc4f619fb34cff7b9d7c4c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:26 UTC) from {username='slave'} at 127.0.0.1:41678
I20240430 20:11:27.150686 18461 tablet_service.cc:1467] Processing CreateTablet for tablet fcc9698abda4491b98e1e879589913a8 (DEFAULT_TABLE table=default.Table_bdb66daeadd54058b7c006c6514acd6c [id=1e785b6c159b4312bd1b064271f55a49]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:27.150832 18623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "187a27a6449342f99de3c5a2f93d49fc" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:27.151048 18623 raft_consensus.cc:3037] T 187a27a6449342f99de3c5a2f93d49fc P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:27.151134 18461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcc9698abda4491b98e1e879589913a8. 1 dirs total, 0 dirs full, 0 dirs failed
W20240430 20:11:27.151139 18142 catalog_manager.cc:4797] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): delete failed for tablet 26a015250eeb4750a42df93840e8d47f because the tablet was not found. No further retry: Not found: Tablet not found: 26a015250eeb4750a42df93840e8d47f
I20240430 20:11:27.153097 18457 tablet_service.cc:1514] Processing DeleteTablet for tablet 255c13ad8fae4ea4aa8e7b7aabb7602e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:27 UTC) from {username='slave'} at 127.0.0.1:41678
I20240430 20:11:27.154556 18457 tablet_service.cc:1467] Processing CreateTablet for tablet 2a1e50f99c5d4b44bdbb51b58a92bc75 (DEFAULT_TABLE table=default.Table_71075ae6b4d545889273c891648c70d1 [id=0fb93f36bc17408ea56bc4cafcfd8580]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:27.154978 18457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a1e50f99c5d4b44bdbb51b58a92bc75. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:27.156613 18557 leader_election.cc:302] T 609a1a7d22e84b69bd0a53577b62b69b P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a
I20240430 20:11:27.163478 19180 leader_election.cc:288] T 680553d4d0bc4fcb839235c1832757e1 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Requested vote from peers 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:27.165550 19230 raft_consensus.cc:2781] T 609a1a7d22e84b69bd0a53577b62b69b P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:27.166718 18729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26ed12f498644d15bf45e884dcad172e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:27.166810 19265 raft_consensus.cc:686] T 81787eccca6c42c1b421377a3bf9ebe6 P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Becoming Leader. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Running, Role: LEADER
W20240430 20:11:27.177301 18150 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 724e51ff05ec4baebc2bb55e05c8eddc consensus state has no leader
I20240430 20:11:27.181142 18463 tablet_service.cc:1467] Processing CreateTablet for tablet 802336602d8f4ac4bd7e0872d6f3dae9 (DEFAULT_TABLE table=default.Table_d997b723a0dd48f6bc2d42c9bfd6bc14 [id=7737ec4807974640aa8556cb1c3ca3ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:27.187461 18482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680553d4d0bc4fcb839235c1832757e1" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:27.189044 18983 tablet_bootstrap.cc:654] T 372b135dd01f44f98cd713ca8857ba61 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:27.190357 18155 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet a0e8e07d363f48c698c7e2b17edc1545 (table default.Table_a058c17d17cb4a68a597fe273e1c1dfd [id=50771665d40d466f92f3a1babd60e39b]). Expected version 2 got 0
I20240430 20:11:27.191890 19230 raft_consensus.cc:482] T 609a1a7d22e84b69bd0a53577b62b69b P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:27.192298 18621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680553d4d0bc4fcb839235c1832757e1" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:27.188290 18463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 802336602d8f4ac4bd7e0872d6f3dae9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:27.192587 18621 raft_consensus.cc:2379] T 680553d4d0bc4fcb839235c1832757e1 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2efb542aa55e439b99ccb98d1c5c884a in current term 1: Already voted for candidate 5e1c6397f09747f091534faf93542710 in this term.
I20240430 20:11:27.192679 18985 tablet_bootstrap.cc:654] T a0e8e07d363f48c698c7e2b17edc1545 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:27.193161 18482 raft_consensus.cc:3037] T 680553d4d0bc4fcb839235c1832757e1 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:27.194435 18987 log.cc:1196] T 17843ea77664447eaaa19666e0445ada P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/17843ea77664447eaaa19666e0445ada
I20240430 20:11:27.194696 18600 tablet_service.cc:1514] Processing DeleteTablet for tablet 255c13ad8fae4ea4aa8e7b7aabb7602e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:27 UTC) from {username='slave'} at 127.0.0.1:40202
I20240430 20:11:27.194725 18593 tablet_service.cc:1514] Processing DeleteTablet for tablet 609a1a7d22e84b69bd0a53577b62b69b with delete_type TABLET_DATA_DELETED (replaced by tablet d57e59f225114af1b1522be6cb805991) from {username='slave'} at 127.0.0.1:40202
I20240430 20:11:27.196516 18583 tablet_service.cc:1514] Processing DeleteTablet for tablet 13c1107ea4d04eb78a74d3f658c72fb3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:26 UTC) from {username='slave'} at 127.0.0.1:40202
I20240430 20:11:27.197171 19230 raft_consensus.cc:3037] T 609a1a7d22e84b69bd0a53577b62b69b P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:27.198037 18150 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54524:
name: "default.Table_0965d6f90dc0474091a9d7a550fcb3cb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:27.198766 18150 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0965d6f90dc0474091a9d7a550fcb3cb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:27.200706 18151 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_eaa091213b264ad09f7ea16af281479f" } new_table_name: "default.Table_e8c73c89623d4b49b8d7f0a01d729bdd" modify_external_catalogs: true
I20240430 20:11:27.202080 18589 tablet_service.cc:1514] Processing DeleteTablet for tablet 27f94f2d8f50495dababe417a9f82982 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:24 UTC) from {username='slave'} at 127.0.0.1:40202
I20240430 20:11:27.203598 18593 tablet_service.cc:1467] Processing CreateTablet for tablet e6eac92dfa5f4456bc49571878f0f8dd (DEFAULT_TABLE table=default.Table_ad1a80db9c774df6add2b60e93577217 [id=19ffa77f540b499482d5dc7b64b12b79]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:27.203609 18155 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_a7f63e5c33fd4651becff171d438c66d" } modify_external_catalogs: true
I20240430 20:11:27.205092 18593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6eac92dfa5f4456bc49571878f0f8dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:27.204715 18155 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_a7f63e5c33fd4651becff171d438c66d" } modify_external_catalogs: true
I20240430 20:11:27.206049 18583 tablet_service.cc:1514] Processing DeleteTablet for tablet 15674aeb56b74210a4a2809608fd4457 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:25 UTC) from {username='slave'} at 127.0.0.1:40202
I20240430 20:11:27.209264 18980 ts_tablet_manager.cc:1888] T 17843ea77664447eaaa19666e0445ada P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240430 20:11:27.209435 18980 log.cc:1196] T 17843ea77664447eaaa19666e0445ada P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/17843ea77664447eaaa19666e0445ada
I20240430 20:11:27.209857 18980 ts_tablet_manager.cc:1909] T 17843ea77664447eaaa19666e0445ada P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
W20240430 20:11:27.219588 18156 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 76d6e9d547ca4a848eae852944b1d117 consensus state has no leader
I20240430 20:11:27.220191 18156 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54524:
name: "default.Table_49e612b137f343d1ac3ec06ded7d42f3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:27.220919 18156 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_49e612b137f343d1ac3ec06ded7d42f3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:27.222817 19180 raft_consensus.cc:482] T bd7392f3dfd643cdb1634ee10ad41651 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:27.222997 19180 raft_consensus.cc:504] T bd7392f3dfd643cdb1634ee10ad41651 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:27.223166 18462 tablet_service.cc:1467] Processing CreateTablet for tablet c9e7660eccc448dbb99d135f82bfd261 (DEFAULT_TABLE table=default.Table_e233554a5e1d41389948fecaf498a856 [id=51d177e8498848b38c43e3f8d11f092c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:27.223677 19180 leader_election.cc:288] T bd7392f3dfd643cdb1634ee10ad41651 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:27.189780 18457 tablet_service.cc:1467] Processing CreateTablet for tablet e6eac92dfa5f4456bc49571878f0f8dd (DEFAULT_TABLE table=default.Table_ad1a80db9c774df6add2b60e93577217 [id=19ffa77f540b499482d5dc7b64b12b79]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:27.231665 18623 raft_consensus.cc:2454] T 187a27a6449342f99de3c5a2f93d49fc P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:27.231781 18139 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet 17843ea77664447eaaa19666e0445ada (table default.Table_16b082d140234e1c9a1e70654839b4cd [id=c0d8dc87d1f9426a85f6176f161595a1]) successfully deleted
I20240430 20:11:27.235764 18583 tablet_service.cc:1467] Processing CreateTablet for tablet a9e0659564894b0e8cb7f600bb07a085 (DEFAULT_TABLE table=default.Table_3c5df662747b48bf8beb6ebe4d2ea618 [id=663c8efbd7ac4a148d5f7626f1522033]), partition=RANGE (key) PARTITION UNBOUNDED
W20240430 20:11:27.237929 18139 catalog_manager.cc:4810] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): delete failed for tablet 29974dc2c7dd45b99b6cc0712378fc30 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 29974dc2c7dd45b99b6cc0712378fc30 already in progress: creating tablet
I20240430 20:11:27.239272 18764 raft_consensus.cc:2379] T 187a27a6449342f99de3c5a2f93d49fc P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e1c6397f09747f091534faf93542710 in current term 1: Already voted for candidate 3c51c33d5616482cafb68e26688820f4 in this term.
I20240430 20:11:27.240291 19191 leader_election.cc:288] T 187a27a6449342f99de3c5a2f93d49fc P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Requested vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:27.240320 18725 tablet_service.cc:1467] Processing CreateTablet for tablet 4e69ca12bd10494fb013614741bc166c (DEFAULT_TABLE table=default.Table_0a397bb1e5ea4cbe87a088c7fc84e9c1 [id=9abcfa9eaaa84cd2b679fd29ec6e5ab7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:27.240690 19230 raft_consensus.cc:504] T 609a1a7d22e84b69bd0a53577b62b69b P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:27.241467 19230 leader_election.cc:288] T 609a1a7d22e84b69bd0a53577b62b69b P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Requested vote from peers 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:27.242462 18599 tablet_service.cc:1467] Processing CreateTablet for tablet f5546f9edeea44d7a6c5cb954f5157ba (DEFAULT_TABLE table=default.Table_aa42ff1beec14dd6baf999b8962907e0 [id=836a4da2ef3a455db26af8149ca5415b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:27.242672 18590 tablet_service.cc:1514] Processing DeleteTablet for tablet 0879b02a0bcc4f619fb34cff7b9d7c4c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:26 UTC) from {username='slave'} at 127.0.0.1:40202
I20240430 20:11:27.243285 18980 tablet_replica.cc:330] T 81328e904ca94e2eb5ec94dd3ffb3c90 P 5e1c6397f09747f091534faf93542710: stopping tablet replica
W20240430 20:11:27.243721 18558 leader_election.cc:341] T 609a1a7d22e84b69bd0a53577b62b69b P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Not found: Tablet not found: 609a1a7d22e84b69bd0a53577b62b69b
I20240430 20:11:27.244060 18583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9e0659564894b0e8cb7f600bb07a085. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:27.246644 18980 raft_consensus.cc:2229] T 81328e904ca94e2eb5ec94dd3ffb3c90 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:27.246822 18980 pending_rounds.cc:62] T 81328e904ca94e2eb5ec94dd3ffb3c90 P 5e1c6397f09747f091534faf93542710: Trying to abort 1 pending ops.
I20240430 20:11:27.248214 18980 pending_rounds.cc:68] T 81328e904ca94e2eb5ec94dd3ffb3c90 P 5e1c6397f09747f091534faf93542710: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7022624300879077376 op_type: NO_OP noop_request { }
I20240430 20:11:27.248369 18980 raft_consensus.cc:2866] T 81328e904ca94e2eb5ec94dd3ffb3c90 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20240430 20:11:27.248903 18980 raft_consensus.cc:2258] T 81328e904ca94e2eb5ec94dd3ffb3c90 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:27.249130 18599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5546f9edeea44d7a6c5cb954f5157ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:27.250216 18980 ts_tablet_manager.cc:1875] T 81328e904ca94e2eb5ec94dd3ffb3c90 P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:27.259085 18983 tablet_bootstrap.cc:492] T 372b135dd01f44f98cd713ca8857ba61 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:27.259564 18483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd7392f3dfd643cdb1634ee10ad41651" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:27.259646 18983 ts_tablet_manager.cc:1367] T 372b135dd01f44f98cd713ca8857ba61 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.209s user 0.004s sys 0.001s
I20240430 20:11:27.261796 18763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680553d4d0bc4fcb839235c1832757e1" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:27.262058 18763 raft_consensus.cc:2454] T 680553d4d0bc4fcb839235c1832757e1 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 0.
I20240430 20:11:27.262483 18983 raft_consensus.cc:348] T 372b135dd01f44f98cd713ca8857ba61 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:27.263238 18983 raft_consensus.cc:374] T 372b135dd01f44f98cd713ca8857ba61 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:27.263345 18983 raft_consensus.cc:729] T 372b135dd01f44f98cd713ca8857ba61 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:27.263895 18728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b688e03ea71940c69bda5520b62c1fc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:27.264314 18983 consensus_queue.cc:260] T 372b135dd01f44f98cd713ca8857ba61 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:27.265041 18987 ts_tablet_manager.cc:1909] T 17843ea77664447eaaa19666e0445ada P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:27.267592 18983 ts_tablet_manager.cc:1398] T 372b135dd01f44f98cd713ca8857ba61 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240430 20:11:27.268069 18983 tablet_bootstrap.cc:492] T fd47db5ca8b24c2b827df0a64e195866 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:27.268633 19261 leader_election.cc:288] T 0879b02a0bcc4f619fb34cff7b9d7c4c P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Requested vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:27.268715 18557 leader_election.cc:302] T 187a27a6449342f99de3c5a2f93d49fc P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:27.270920 18983 tablet_bootstrap.cc:654] T fd47db5ca8b24c2b827df0a64e195866 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:27.271342 19230 raft_consensus.cc:2726] T 187a27a6449342f99de3c5a2f93d49fc P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:27.269264 18764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680553d4d0bc4fcb839235c1832757e1" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
W20240430 20:11:27.277829 18699 leader_election.cc:341] T 187a27a6449342f99de3c5a2f93d49fc P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Not found: Tablet not found: 187a27a6449342f99de3c5a2f93d49fc
I20240430 20:11:27.276383 19265 consensus_queue.cc:237] T 81787eccca6c42c1b421377a3bf9ebe6 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:27.283445 18725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e69ca12bd10494fb013614741bc166c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:27.287189 18141 catalog_manager.cc:4830] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): tablet 17843ea77664447eaaa19666e0445ada (table default.Table_16b082d140234e1c9a1e70654839b4cd [id=c0d8dc87d1f9426a85f6176f161595a1]) successfully deleted
W20240430 20:11:27.287508 18141 catalog_manager.cc:4810] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): delete failed for tablet 29974dc2c7dd45b99b6cc0712378fc30 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 29974dc2c7dd45b99b6cc0712378fc30 already in progress: creating tablet
I20240430 20:11:27.287739 18987 tablet_replica.cc:330] T 81328e904ca94e2eb5ec94dd3ffb3c90 P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:27.287873 18724 tablet_service.cc:1467] Processing CreateTablet for tablet 24dcc20832804229a2664873bf6ddaa7 (DEFAULT_TABLE table=default.Table_e689526f90344c6f91dae34f22d60295 [id=2d44ec94131148048021b2ee6366ba30]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:27.288424 18696 leader_election.cc:302] T 187a27a6449342f99de3c5a2f93d49fc P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a
I20240430 20:11:27.289156 19191 raft_consensus.cc:2781] T 187a27a6449342f99de3c5a2f93d49fc P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:27.289315 19191 raft_consensus.cc:686] T 187a27a6449342f99de3c5a2f93d49fc P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Becoming Leader. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Running, Role: LEADER
I20240430 20:11:27.289589 19191 consensus_queue.cc:237] T 187a27a6449342f99de3c5a2f93d49fc P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:27.289979 18980 ts_tablet_manager.cc:1888] T 81328e904ca94e2eb5ec94dd3ffb3c90 P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:27.290120 18980 log.cc:1196] T 81328e904ca94e2eb5ec94dd3ffb3c90 P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/81328e904ca94e2eb5ec94dd3ffb3c90
I20240430 20:11:27.291469 18620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0879b02a0bcc4f619fb34cff7b9d7c4c" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:27.291702 18620 raft_consensus.cc:3037] T 0879b02a0bcc4f619fb34cff7b9d7c4c P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
W20240430 20:11:27.292990 18696 consensus_peers.cc:485] T 81328e904ca94e2eb5ec94dd3ffb3c90 P 3c51c33d5616482cafb68e26688820f4 -> Peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Couldn't send request to peer 5e1c6397f09747f091534faf93542710. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240430 20:11:27.294728 18620 raft_consensus.cc:2454] T 0879b02a0bcc4f619fb34cff7b9d7c4c P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:27.294854 18726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 255c13ad8fae4ea4aa8e7b7aabb7602e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:27.295967 18980 ts_tablet_manager.cc:1909] T 81328e904ca94e2eb5ec94dd3ffb3c90 P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:27.298228 18764 raft_consensus.cc:3037] T 680553d4d0bc4fcb839235c1832757e1 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:27.299450 18983 tablet_bootstrap.cc:492] T fd47db5ca8b24c2b827df0a64e195866 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:27.299610 18983 ts_tablet_manager.cc:1367] T fd47db5ca8b24c2b827df0a64e195866 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.032s user 0.003s sys 0.004s
I20240430 20:11:27.300439 18983 raft_consensus.cc:348] T fd47db5ca8b24c2b827df0a64e195866 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:27.300652 18983 raft_consensus.cc:374] T fd47db5ca8b24c2b827df0a64e195866 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:27.300726 18983 raft_consensus.cc:729] T fd47db5ca8b24c2b827df0a64e195866 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:27.300997 18983 consensus_queue.cc:260] T fd47db5ca8b24c2b827df0a64e195866 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:27.303200 18696 leader_election.cc:302] T 0879b02a0bcc4f619fb34cff7b9d7c4c P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
W20240430 20:11:27.304904 18557 leader_election.cc:341] T bd7392f3dfd643cdb1634ee10ad41651 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Not found: Tablet not found: bd7392f3dfd643cdb1634ee10ad41651
I20240430 20:11:27.305048 18983 ts_tablet_manager.cc:1398] T fd47db5ca8b24c2b827df0a64e195866 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240430 20:11:27.305070 18557 leader_election.cc:302] T bd7392f3dfd643cdb1634ee10ad41651 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:27.305492 19230 raft_consensus.cc:2726] T bd7392f3dfd643cdb1634ee10ad41651 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:27.306629 19230 raft_consensus.cc:482] T 372b135dd01f44f98cd713ca8857ba61 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:27.306767 19230 raft_consensus.cc:504] T 372b135dd01f44f98cd713ca8857ba61 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:27.307497 19230 leader_election.cc:288] T 372b135dd01f44f98cd713ca8857ba61 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:27.308627 18983 tablet_bootstrap.cc:492] T c9ff972ade694938888514979e1638aa P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:27.310443 18983 tablet_bootstrap.cc:654] T c9ff972ade694938888514979e1638aa P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:27.310719 19261 raft_consensus.cc:2781] T 0879b02a0bcc4f619fb34cff7b9d7c4c P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:27.310971 19261 raft_consensus.cc:686] T 0879b02a0bcc4f619fb34cff7b9d7c4c P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Becoming Leader. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Running, Role: LEADER
I20240430 20:11:27.311314 19261 consensus_queue.cc:237] T 0879b02a0bcc4f619fb34cff7b9d7c4c P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:27.312213 18980 tablet_replica.cc:330] T ffe059eeac91476aaeef8c56d2f49ee8 P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:27.312451 18980 raft_consensus.cc:2229] T ffe059eeac91476aaeef8c56d2f49ee8 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:27.312603 18980 raft_consensus.cc:2258] T ffe059eeac91476aaeef8c56d2f49ee8 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:27.313167 18980 ts_tablet_manager.cc:1875] T ffe059eeac91476aaeef8c56d2f49ee8 P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:27.314019 18763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "609a1a7d22e84b69bd0a53577b62b69b" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:27.314224 18763 raft_consensus.cc:3037] T 609a1a7d22e84b69bd0a53577b62b69b P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:27.323683 18148 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: Sending DeleteTablet for 3 replicas of tablet f5546f9edeea44d7a6c5cb954f5157ba
I20240430 20:11:27.327574 18983 tablet_bootstrap.cc:492] T c9ff972ade694938888514979e1638aa P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:27.328682 18983 ts_tablet_manager.cc:1367] T c9ff972ade694938888514979e1638aa P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.000s
I20240430 20:11:27.329739 18983 raft_consensus.cc:348] T c9ff972ade694938888514979e1638aa P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:27.331343 18983 raft_consensus.cc:374] T c9ff972ade694938888514979e1638aa P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:27.331429 18983 raft_consensus.cc:729] T c9ff972ade694938888514979e1638aa P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:27.334061 18983 consensus_queue.cc:260] T c9ff972ade694938888514979e1638aa P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:27.336119 18139 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet 81328e904ca94e2eb5ec94dd3ffb3c90 (table default.Table_fcab4e9721f7470abc1df37d83ea299f [id=e197b6008d9a40269e645c15412b21b4]) successfully deleted
W20240430 20:11:27.336438 18139 catalog_manager.cc:4810] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): delete failed for tablet bbe9a2429a434f3f95d6e7f314fcfe4c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bbe9a2429a434f3f95d6e7f314fcfe4c already in progress: creating tablet
W20240430 20:11:27.336639 18139 catalog_manager.cc:4810] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): delete failed for tablet cace1398492f4d968dcb6899385d527c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cace1398492f4d968dcb6899385d527c already in progress: creating tablet
I20240430 20:11:27.336823 18980 ts_tablet_manager.cc:1888] T ffe059eeac91476aaeef8c56d2f49ee8 P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:27.336990 18980 log.cc:1196] T ffe059eeac91476aaeef8c56d2f49ee8 P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/ffe059eeac91476aaeef8c56d2f49ee8
I20240430 20:11:27.337383 18980 ts_tablet_manager.cc:1909] T ffe059eeac91476aaeef8c56d2f49ee8 P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:27.338943 18586 tablet_service.cc:1467] Processing CreateTablet for tablet fd40967ab4a04001819e16c06a6c4300 (DEFAULT_TABLE table=default.Table_fda07fd7279c484db2bee67e8c859855 [id=da9ad991953a40c4b77b31317597ad1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:27.339394 18586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd40967ab4a04001819e16c06a6c4300. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:27.340680 18980 tablet_replica.cc:330] T b4bc111e183f40a0acc9dd377195f6d9 P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:27.340987 18980 raft_consensus.cc:2229] T b4bc111e183f40a0acc9dd377195f6d9 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:27.343508 18149 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54524:
name: "default.Table_7a9f9a8c27bb4392acddd2a7ea3cc3b2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:27.344066 18149 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7a9f9a8c27bb4392acddd2a7ea3cc3b2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:27.344542 18983 ts_tablet_manager.cc:1398] T c9ff972ade694938888514979e1638aa P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.016s user 0.001s sys 0.001s
I20240430 20:11:27.345737 18146 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54524:
name: "default.Table_3e3885add2344068a5da1614890073e0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:27.346272 18146 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3e3885add2344068a5da1614890073e0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:27.354931 18148 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 372b135dd01f44f98cd713ca8857ba61 (table default.Table_ee86b8fcf2064d8b8c3e327e357d3395 [id=7f3a15a442984827a7521d5d1c98dc3d]). Expected version 2 got 0
I20240430 20:11:27.356020 18148 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54524:
name: "default.Table_db56d63aef494592b69246a0c13006f2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240430 20:11:27.358779 18139 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet ffe059eeac91476aaeef8c56d2f49ee8 (table default.Table_73903d0d2a4746f38fdb73932bde6597 [id=6b56400601e6453495da25f608db780f]) successfully deleted
I20240430 20:11:27.371608 18724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24dcc20832804229a2664873bf6ddaa7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:27.377638 18462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9e7660eccc448dbb99d135f82bfd261. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:27.377946 18457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6eac92dfa5f4456bc49571878f0f8dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:27.378901 18154 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: Sending DeleteTablet for 3 replicas of tablet 02176eee8e044a7db8cac48439e9f594
I20240430 20:11:27.379313 18597 tablet_service.cc:1467] Processing CreateTablet for tablet 76d6e9d547ca4a848eae852944b1d117 (DEFAULT_TABLE table=default.Table_210b7eaff7144b2eb7db1c3b5f7d9505 [id=ff86c0357c164dd0991098c416087773]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:27.379503 18154 catalog_manager.cc:6554] ReplaceTablet: tablet 02176eee8e044a7db8cac48439e9f594 deleted and replaced by tablet 09d92a4a6ebe44cbb3553ed660f143d0
I20240430 20:11:27.379807 18597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76d6e9d547ca4a848eae852944b1d117. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:27.381101 18154 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 372b135dd01f44f98cd713ca8857ba61 (table default.Table_ee86b8fcf2064d8b8c3e327e357d3395 [id=7f3a15a442984827a7521d5d1c98dc3d]). Expected version 2 got 0
I20240430 20:11:27.382722 18983 tablet_bootstrap.cc:492] T 4e97f7315f8b4ac29db16e00fc35bf99 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:27.382689 18583 tablet_service.cc:1514] Processing DeleteTablet for tablet f5546f9edeea44d7a6c5cb954f5157ba with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:27 UTC) from {username='slave'} at 127.0.0.1:40202
W20240430 20:11:27.382849 18415 consensus_peers.cc:485] T 17843ea77664447eaaa19666e0445ada P 2efb542aa55e439b99ccb98d1c5c884a -> Peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Couldn't send request to peer 5e1c6397f09747f091534faf93542710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 17843ea77664447eaaa19666e0445ada. This is attempt 1: this message will repeat every 5th retry.
I20240430 20:11:27.382884 18152 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: Sending DeleteTablet for 3 replicas of tablet 34ab40d8d91e401db299ddee7175ba28
I20240430 20:11:27.383846 18152 catalog_manager.cc:6554] ReplaceTablet: tablet 34ab40d8d91e401db299ddee7175ba28 deleted and replaced by tablet 5f02d8862e1f4ccfa9dbda73dbe18615
I20240430 20:11:27.384121 18152 master_service.cc:946] ReplaceTablet: received request to replace tablet a1aeee1b02214975a376cfd36957afe8 from {username='slave'} at 127.0.0.1:54524
I20240430 20:11:27.384395 18450 tablet_service.cc:1467] Processing CreateTablet for tablet a9e0659564894b0e8cb7f600bb07a085 (DEFAULT_TABLE table=default.Table_3c5df662747b48bf8beb6ebe4d2ea618 [id=663c8efbd7ac4a148d5f7626f1522033]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:27.384927 18983 tablet_bootstrap.cc:654] T 4e97f7315f8b4ac29db16e00fc35bf99 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:27.384783 18152 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54524:
name: "default.Table_f5a3451cd2284390be4357d771c03390"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:27.385442 18152 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f5a3451cd2284390be4357d771c03390 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240430 20:11:27.387496 18151 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 32c823d1700949ab8b22adcd83b7bfd1 has no consensus state
I20240430 20:11:27.387812 18151 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_a5743a67104f449d8d26a63b9f85eb8b" } new_table_name: "default.Table_972f13be4dca4cd19b9a04534dc59f51" modify_external_catalogs: true
I20240430 20:11:27.388098 18980 raft_consensus.cc:2258] T b4bc111e183f40a0acc9dd377195f6d9 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:27.388727 18980 ts_tablet_manager.cc:1875] T b4bc111e183f40a0acc9dd377195f6d9 P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:27.389515 18150 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_0a397bb1e5ea4cbe87a088c7fc84e9c1" } modify_external_catalogs: true
I20240430 20:11:27.389724 18150 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_0a397bb1e5ea4cbe87a088c7fc84e9c1" } modify_external_catalogs: true
I20240430 20:11:27.390897 18147 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_e689526f90344c6f91dae34f22d60295" } new_table_name: "default.Table_17b7c8e853bd427db0c3fa2555a9067b" modify_external_catalogs: true
I20240430 20:11:27.391135 18481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "372b135dd01f44f98cd713ca8857ba61" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:27.393231 18483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0879b02a0bcc4f619fb34cff7b9d7c4c" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:27.393345 18156 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_a058c17d17cb4a68a597fe273e1c1dfd" } modify_external_catalogs: true
I20240430 20:11:27.393533 18156 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_a058c17d17cb4a68a597fe273e1c1dfd" } modify_external_catalogs: true
I20240430 20:11:27.394044 18450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9e0659564894b0e8cb7f600bb07a085. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:27.396337 18980 ts_tablet_manager.cc:1888] T b4bc111e183f40a0acc9dd377195f6d9 P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:27.396617 18980 log.cc:1196] T b4bc111e183f40a0acc9dd377195f6d9 P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/b4bc111e183f40a0acc9dd377195f6d9
I20240430 20:11:27.397097 18980 ts_tablet_manager.cc:1909] T b4bc111e183f40a0acc9dd377195f6d9 P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:27.397241 18149 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_e9655558850747968a52e5885a113f59" } modify_external_catalogs: true
I20240430 20:11:27.391781 18417 leader_election.cc:302] T 680553d4d0bc4fcb839235c1832757e1 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters: 5e1c6397f09747f091534faf93542710
I20240430 20:11:27.397468 18149 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_e9655558850747968a52e5885a113f59" } modify_external_catalogs: true
W20240430 20:11:27.398172 19283 log.cc:924] Time spent T 27f94f2d8f50495dababe417a9f82982 P 2efb542aa55e439b99ccb98d1c5c884a: Append to log took a long time: real 0.170s user 0.000s sys 0.000s
I20240430 20:11:27.398316 18980 tablet_replica.cc:330] T bd7392f3dfd643cdb1634ee10ad41651 P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:27.398530 18980 raft_consensus.cc:2229] T bd7392f3dfd643cdb1634ee10ad41651 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:27.398722 18980 raft_consensus.cc:2258] T bd7392f3dfd643cdb1634ee10ad41651 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Raft consensus is shut down!
W20240430 20:11:27.398936 18558 leader_election.cc:341] T 372b135dd01f44f98cd713ca8857ba61 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:27.400041 18146 master_service.cc:946] ReplaceTablet: received request to replace tablet b4bc111e183f40a0acc9dd377195f6d9 from {username='slave'} at 127.0.0.1:54524
I20240430 20:11:27.400113 18980 ts_tablet_manager.cc:1875] T bd7392f3dfd643cdb1634ee10ad41651 P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240430 20:11:27.400359 18417 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.177.3:44867, user_credentials={real_user=slave}} blocked reactor thread for 127917us
I20240430 20:11:27.382493 18154 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54524:
name: "default.Table_8af52c0f1d404b42bc53287f64e20c2d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:27.401770 18154 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8af52c0f1d404b42bc53287f64e20c2d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:27.402578 18483 raft_consensus.cc:3037] T 0879b02a0bcc4f619fb34cff7b9d7c4c P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:27.402964 18155 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet 68e5a7ea035b41b6ac9f582e56a14edd
W20240430 20:11:27.402217 18415 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.177.2:46027, user_credentials={real_user=slave}} blocked reactor thread for 100474us
I20240430 20:11:27.405985 18146 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_c5646eaabbf5443894dab99237416cc7" } new_table_name: "default.Table_208b12b8e9bd4f3e8bc9c6d422ada649" modify_external_catalogs: true
I20240430 20:11:27.406435 18447 tablet_service.cc:1467] Processing CreateTablet for tablet fd40967ab4a04001819e16c06a6c4300 (DEFAULT_TABLE table=default.Table_fda07fd7279c484db2bee67e8c859855 [id=da9ad991953a40c4b77b31317597ad1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:27.406944 18980 ts_tablet_manager.cc:1888] T bd7392f3dfd643cdb1634ee10ad41651 P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:27.407164 18980 log.cc:1196] T bd7392f3dfd643cdb1634ee10ad41651 P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/bd7392f3dfd643cdb1634ee10ad41651
W20240430 20:11:27.407488 18417 consensus_peers.cc:485] T a01ee128ba8a4c9fb8e125ba97239d4c P 2efb542aa55e439b99ccb98d1c5c884a -> Peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Couldn't send request to peer 3c51c33d5616482cafb68e26688820f4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a01ee128ba8a4c9fb8e125ba97239d4c. This is attempt 1: this message will repeat every 5th retry.
I20240430 20:11:27.407634 18980 ts_tablet_manager.cc:1909] T bd7392f3dfd643cdb1634ee10ad41651 P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:27.407760 18155 master_service.cc:946] ReplaceTablet: received request to replace tablet eed6338ad3d64859af087ee15e1ec798 from {username='slave'} at 127.0.0.1:54524
I20240430 20:11:27.408375 18448 tablet_service.cc:1514] Processing DeleteTablet for tablet f5546f9edeea44d7a6c5cb954f5157ba with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:27 UTC) from {username='slave'} at 127.0.0.1:41678
I20240430 20:11:27.408555 18452 tablet_service.cc:1514] Processing DeleteTablet for tablet 34ab40d8d91e401db299ddee7175ba28 with delete_type TABLET_DATA_DELETED (replaced by tablet 5f02d8862e1f4ccfa9dbda73dbe18615) from {username='slave'} at 127.0.0.1:41678
I20240430 20:11:27.408799 18980 tablet_replica.cc:330] T 187a27a6449342f99de3c5a2f93d49fc P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:27.409013 18980 raft_consensus.cc:2229] T 187a27a6449342f99de3c5a2f93d49fc P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:27.409456 18459 tablet_service.cc:1514] Processing DeleteTablet for tablet 02176eee8e044a7db8cac48439e9f594 with delete_type TABLET_DATA_DELETED (replaced by tablet 09d92a4a6ebe44cbb3553ed660f143d0) from {username='slave'} at 127.0.0.1:41678
I20240430 20:11:27.409770 18586 tablet_service.cc:1514] Processing DeleteTablet for tablet 02176eee8e044a7db8cac48439e9f594 with delete_type TABLET_DATA_DELETED (replaced by tablet 09d92a4a6ebe44cbb3553ed660f143d0) from {username='slave'} at 127.0.0.1:40202
I20240430 20:11:27.409888 18447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd40967ab4a04001819e16c06a6c4300. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:27.410015 18597 tablet_service.cc:1514] Processing DeleteTablet for tablet fd47db5ca8b24c2b827df0a64e195866 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:26 UTC) from {username='slave'} at 127.0.0.1:40202
I20240430 20:11:27.410136 18139 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet b4bc111e183f40a0acc9dd377195f6d9 (table default.Table_44426457fd7045dfa1c149f0a5ed5129 [id=d5cea90cbc2c48df9d8b80da1eccc949]) successfully deleted
I20240430 20:11:27.410202 18583 tablet_service.cc:1514] Processing DeleteTablet for tablet 34ab40d8d91e401db299ddee7175ba28 with delete_type TABLET_DATA_DELETED (replaced by tablet 5f02d8862e1f4ccfa9dbda73dbe18615) from {username='slave'} at 127.0.0.1:40202
I20240430 20:11:27.410353 18139 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet bd7392f3dfd643cdb1634ee10ad41651 (table default.Table_b29130df231a4316bcb8f9c2be002c92 [id=5e77a9e6adfa4bf0b1c1f81be22e6499]) successfully deleted
I20240430 20:11:27.410434 18980 raft_consensus.cc:2258] T 187a27a6449342f99de3c5a2f93d49fc P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:27.411314 18980 ts_tablet_manager.cc:1875] T 187a27a6449342f99de3c5a2f93d49fc P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:27.417953 18980 ts_tablet_manager.cc:1888] T 187a27a6449342f99de3c5a2f93d49fc P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:27.418213 18980 log.cc:1196] T 187a27a6449342f99de3c5a2f93d49fc P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/187a27a6449342f99de3c5a2f93d49fc
I20240430 20:11:27.418710 18980 ts_tablet_manager.cc:1909] T 187a27a6449342f99de3c5a2f93d49fc P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:27.420012 18139 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet 187a27a6449342f99de3c5a2f93d49fc (table default.Table_173d3907261b4390aa5647f01794d901 [id=f61d04c63e844a7b99d503770e5fcd87]) successfully deleted
W20240430 20:11:27.420907 18139 catalog_manager.cc:4810] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): delete failed for tablet a1aeee1b02214975a376cfd36957afe8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a1aeee1b02214975a376cfd36957afe8 already in progress: creating tablet
W20240430 20:11:27.421694 18139 catalog_manager.cc:4810] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): delete failed for tablet 68ca48351cd4412381f5195d81109016 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 68ca48351cd4412381f5195d81109016 already in progress: creating tablet
I20240430 20:11:27.421586 18454 tablet_service.cc:1467] Processing CreateTablet for tablet 76d6e9d547ca4a848eae852944b1d117 (DEFAULT_TABLE table=default.Table_210b7eaff7144b2eb7db1c3b5f7d9505 [id=ff86c0357c164dd0991098c416087773]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:27.422104 18980 tablet_replica.cc:330] T 3f49eda7aeca42308d01b83a19d7d037 P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:27.422163 18454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76d6e9d547ca4a848eae852944b1d117. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:27.422410 18980 raft_consensus.cc:2229] T 3f49eda7aeca42308d01b83a19d7d037 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:27.422683 18980 raft_consensus.cc:2258] T 3f49eda7aeca42308d01b83a19d7d037 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:27.423663 18980 ts_tablet_manager.cc:1875] T 3f49eda7aeca42308d01b83a19d7d037 P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:27.427062 18744 tablet_service.cc:1467] Processing CreateTablet for tablet f5546f9edeea44d7a6c5cb954f5157ba (DEFAULT_TABLE table=default.Table_aa42ff1beec14dd6baf999b8962907e0 [id=836a4da2ef3a455db26af8149ca5415b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:27.428534 19230 raft_consensus.cc:482] T 15674aeb56b74210a4a2809608fd4457 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240430 20:11:27.428520 18417 consensus_peers.cc:485] T bd7392f3dfd643cdb1634ee10ad41651 P 2efb542aa55e439b99ccb98d1c5c884a -> Peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Couldn't send request to peer 3c51c33d5616482cafb68e26688820f4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bd7392f3dfd643cdb1634ee10ad41651. This is attempt 1: this message will repeat every 5th retry.
I20240430 20:11:27.428710 19230 raft_consensus.cc:504] T 15674aeb56b74210a4a2809608fd4457 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:27.429396 19230 leader_election.cc:288] T 15674aeb56b74210a4a2809608fd4457 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:27.432478 18456 tablet_service.cc:1467] Processing CreateTablet for tablet f5546f9edeea44d7a6c5cb954f5157ba (DEFAULT_TABLE table=default.Table_aa42ff1beec14dd6baf999b8962907e0 [id=836a4da2ef3a455db26af8149ca5415b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:27.433007 18456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5546f9edeea44d7a6c5cb954f5157ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:27.434047 18980 ts_tablet_manager.cc:1888] T 3f49eda7aeca42308d01b83a19d7d037 P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:27.434273 18980 log.cc:1196] T 3f49eda7aeca42308d01b83a19d7d037 P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/3f49eda7aeca42308d01b83a19d7d037
I20240430 20:11:27.434870 18980 ts_tablet_manager.cc:1909] T 3f49eda7aeca42308d01b83a19d7d037 P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:27.436216 18139 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet 3f49eda7aeca42308d01b83a19d7d037 (table default.Table_210b7eaff7144b2eb7db1c3b5f7d9505 [id=ff86c0357c164dd0991098c416087773]) successfully deleted
W20240430 20:11:27.437044 18139 catalog_manager.cc:4810] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): delete failed for tablet f5236574672b477a9026e91adcaf70ec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f5236574672b477a9026e91adcaf70ec already in progress: creating tablet
W20240430 20:11:27.437731 18139 catalog_manager.cc:4810] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): delete failed for tablet aacabead3c874e77a47d2ecf19dbfa82 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aacabead3c874e77a47d2ecf19dbfa82 already in progress: creating tablet
I20240430 20:11:27.437599 18155 catalog_manager.cc:5484] T 81787eccca6c42c1b421377a3bf9ebe6 P 3c51c33d5616482cafb68e26688820f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c51c33d5616482cafb68e26688820f4 (127.17.177.3). New cstate: current_term: 1 leader_uuid: "3c51c33d5616482cafb68e26688820f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } health_report { overall_health: HEALTHY } } }
I20240430 20:11:27.438102 18155 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 81787eccca6c42c1b421377a3bf9ebe6 (table default.Table_c1669746f9114bf7b37a6f6a68fed20e [id=7f602584cef9486eb7d9b4389be3f43e]). Expected version 2 got 0
W20240430 20:11:27.438550 18139 catalog_manager.cc:4810] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): delete failed for tablet 4e97f7315f8b4ac29db16e00fc35bf99 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4e97f7315f8b4ac29db16e00fc35bf99 already in progress: creating tablet
I20240430 20:11:27.438952 18980 tablet_replica.cc:330] T c9ff972ade694938888514979e1638aa P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:27.439188 18980 raft_consensus.cc:2229] T c9ff972ade694938888514979e1638aa P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:27.439428 18980 raft_consensus.cc:2258] T c9ff972ade694938888514979e1638aa P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:27.440234 18980 ts_tablet_manager.cc:1875] T c9ff972ade694938888514979e1638aa P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:27.436771 18481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15674aeb56b74210a4a2809608fd4457" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:27.436957 18981 ts_tablet_manager.cc:1888] T 59548d560b4e4f7ca034d730aca79364 P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20240430 20:11:27.437037 18415 consensus_peers.cc:485] T bd7392f3dfd643cdb1634ee10ad41651 P 2efb542aa55e439b99ccb98d1c5c884a -> Peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Couldn't send request to peer 5e1c6397f09747f091534faf93542710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bd7392f3dfd643cdb1634ee10ad41651. This is attempt 1: this message will repeat every 5th retry.
W20240430 20:11:27.443306 18558 leader_election.cc:341] T 15674aeb56b74210a4a2809608fd4457 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:27.447634 18980 ts_tablet_manager.cc:1888] T c9ff972ade694938888514979e1638aa P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:27.447855 18980 log.cc:1196] T c9ff972ade694938888514979e1638aa P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/c9ff972ade694938888514979e1638aa
I20240430 20:11:27.448321 18980 ts_tablet_manager.cc:1909] T c9ff972ade694938888514979e1638aa P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:27.448472 18482 raft_consensus.cc:2454] T 680553d4d0bc4fcb839235c1832757e1 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
I20240430 20:11:27.449080 18558 leader_election.cc:302] T 680553d4d0bc4fcb839235c1832757e1 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:27.449409 19230 raft_consensus.cc:2781] T 680553d4d0bc4fcb839235c1832757e1 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:27.449434 18139 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet c9ff972ade694938888514979e1638aa (table default.Table_3c5df662747b48bf8beb6ebe4d2ea618 [id=663c8efbd7ac4a148d5f7626f1522033]) successfully deleted
I20240430 20:11:27.449532 19230 raft_consensus.cc:686] T 680553d4d0bc4fcb839235c1832757e1 P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Becoming Leader. State: Replica: 5e1c6397f09747f091534faf93542710, State: Running, Role: LEADER
I20240430 20:11:27.441381 18730 tablet_service.cc:1467] Processing CreateTablet for tablet fcc9698abda4491b98e1e879589913a8 (DEFAULT_TABLE table=default.Table_bdb66daeadd54058b7c006c6514acd6c [id=1e785b6c159b4312bd1b064271f55a49]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:27.449819 19230 consensus_queue.cc:237] T 680553d4d0bc4fcb839235c1832757e1 P 5e1c6397f09747f091534faf93542710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
W20240430 20:11:27.357896 18148 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_db56d63aef494592b69246a0c13006f2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:27.459534 19185 raft_consensus.cc:2741] T 680553d4d0bc4fcb839235c1832757e1 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20240430 20:11:27.464879 18743 tablet_service.cc:1467] Processing CreateTablet for tablet 802336602d8f4ac4bd7e0872d6f3dae9 (DEFAULT_TABLE table=default.Table_d997b723a0dd48f6bc2d42c9bfd6bc14 [id=7737ec4807974640aa8556cb1c3ca3ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:27.466521 18730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcc9698abda4491b98e1e879589913a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:27.444864 18736 tablet_service.cc:1467] Processing CreateTablet for tablet c9e7660eccc448dbb99d135f82bfd261 (DEFAULT_TABLE table=default.Table_e233554a5e1d41389948fecaf498a856 [id=51d177e8498848b38c43e3f8d11f092c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:27.442823 18762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "372b135dd01f44f98cd713ca8857ba61" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:27.465157 18981 log.cc:1196] T 59548d560b4e4f7ca034d730aca79364 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/59548d560b4e4f7ca034d730aca79364
I20240430 20:11:27.469920 18736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9e7660eccc448dbb99d135f82bfd261. 1 dirs total, 0 dirs full, 0 dirs failed
W20240430 20:11:27.472466 18151 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet acd23f0a63d5448aaa7ed78422e67bad consensus state has no leader
I20240430 20:11:27.472899 18151 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_bdb66daeadd54058b7c006c6514acd6c" } new_table_name: "default.Table_e431f94220aa44ecba799a54dfb612ae" modify_external_catalogs: true
I20240430 20:11:27.473718 18760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15674aeb56b74210a4a2809608fd4457" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
W20240430 20:11:27.474557 18147 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 24dcc20832804229a2664873bf6ddaa7 consensus state has no leader
I20240430 20:11:27.475554 18988 tablet_bootstrap.cc:492] T 15674aeb56b74210a4a2809608fd4457 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
W20240430 20:11:27.475900 18557 leader_election.cc:341] T 372b135dd01f44f98cd713ca8857ba61 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:27.476091 18557 leader_election.cc:302] T 372b135dd01f44f98cd713ca8857ba61 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:27.476390 19180 raft_consensus.cc:2726] T 372b135dd01f44f98cd713ca8857ba61 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240430 20:11:27.476568 18557 leader_election.cc:341] T 15674aeb56b74210a4a2809608fd4457 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:27.476703 18557 leader_election.cc:302] T 15674aeb56b74210a4a2809608fd4457 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:27.476946 19180 raft_consensus.cc:2726] T 15674aeb56b74210a4a2809608fd4457 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:27.478046 18156 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: Sending DeleteTablet for 3 replicas of tablet a0e8e07d363f48c698c7e2b17edc1545
I20240430 20:11:27.478329 18743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 802336602d8f4ac4bd7e0872d6f3dae9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:27.479929 18583 tablet_service.cc:1514] Processing DeleteTablet for tablet a0e8e07d363f48c698c7e2b17edc1545 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:27 UTC) from {username='slave'} at 127.0.0.1:40202
W20240430 20:11:27.480623 18146 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6e754af64cb64fd2a082adae7d5f24da has no consensus state
I20240430 20:11:27.480753 18149 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: Sending DeleteTablet for 3 replicas of tablet 75ebf051383e42529dcb5f7dd0770a4b
I20240430 20:11:27.481638 18583 tablet_service.cc:1514] Processing DeleteTablet for tablet 75ebf051383e42529dcb5f7dd0770a4b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:27 UTC) from {username='slave'} at 127.0.0.1:40202
I20240430 20:11:27.486829 18988 ts_tablet_manager.cc:1367] T 15674aeb56b74210a4a2809608fd4457 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.554s user 0.053s sys 0.019s
I20240430 20:11:27.491233 18737 tablet_service.cc:1514] Processing DeleteTablet for tablet 255c13ad8fae4ea4aa8e7b7aabb7602e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:27 UTC) from {username='slave'} at 127.0.0.1:59818
I20240430 20:11:27.491436 18737 tablet_service.cc:1514] Processing DeleteTablet for tablet 0879b02a0bcc4f619fb34cff7b9d7c4c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:26 UTC) from {username='slave'} at 127.0.0.1:59818
I20240430 20:11:27.491586 18737 tablet_service.cc:1514] Processing DeleteTablet for tablet 13c1107ea4d04eb78a74d3f658c72fb3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:26 UTC) from {username='slave'} at 127.0.0.1:59818
I20240430 20:11:27.491940 18737 tablet_service.cc:1467] Processing CreateTablet for tablet 2a1e50f99c5d4b44bdbb51b58a92bc75 (DEFAULT_TABLE table=default.Table_71075ae6b4d545889273c891648c70d1 [id=0fb93f36bc17408ea56bc4cafcfd8580]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:27.492870 18737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a1e50f99c5d4b44bdbb51b58a92bc75. 1 dirs total, 0 dirs full, 0 dirs failed
W20240430 20:11:27.498337 18139 catalog_manager.cc:4810] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): delete failed for tablet eed6338ad3d64859af087ee15e1ec798 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet eed6338ad3d64859af087ee15e1ec798 already in progress: creating tablet
W20240430 20:11:27.499009 18139 catalog_manager.cc:4810] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): delete failed for tablet 255c13ad8fae4ea4aa8e7b7aabb7602e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 255c13ad8fae4ea4aa8e7b7aabb7602e already in progress: creating tablet
I20240430 20:11:27.499498 18980 tablet_replica.cc:330] T 609a1a7d22e84b69bd0a53577b62b69b P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:27.499764 18980 raft_consensus.cc:2229] T 609a1a7d22e84b69bd0a53577b62b69b P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:27.499989 18980 raft_consensus.cc:2258] T 609a1a7d22e84b69bd0a53577b62b69b P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:27.500622 18980 ts_tablet_manager.cc:1875] T 609a1a7d22e84b69bd0a53577b62b69b P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:27.500787 18988 raft_consensus.cc:348] T 15674aeb56b74210a4a2809608fd4457 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:27.501010 18988 raft_consensus.cc:374] T 15674aeb56b74210a4a2809608fd4457 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:27.501096 18988 raft_consensus.cc:729] T 15674aeb56b74210a4a2809608fd4457 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:27.501374 18988 consensus_queue.cc:260] T 15674aeb56b74210a4a2809608fd4457 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:27.501917 18988 ts_tablet_manager.cc:1398] T 15674aeb56b74210a4a2809608fd4457 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.015s user 0.001s sys 0.001s
I20240430 20:11:27.502234 18988 tablet_bootstrap.cc:492] T f5236574672b477a9026e91adcaf70ec P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
W20240430 20:11:27.503232 18151 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fcc9698abda4491b98e1e879589913a8 consensus state has no leader
I20240430 20:11:27.504200 18483 raft_consensus.cc:2454] T 0879b02a0bcc4f619fb34cff7b9d7c4c P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:27.514762 18981 ts_tablet_manager.cc:1909] T 59548d560b4e4f7ca034d730aca79364 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:27.515120 19230 consensus_queue.cc:1035] T 5809e7c9c2204ba293426c24129fb16e P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:27.515947 18738 tablet_service.cc:1467] Processing CreateTablet for tablet e6eac92dfa5f4456bc49571878f0f8dd (DEFAULT_TABLE table=default.Table_ad1a80db9c774df6add2b60e93577217 [id=19ffa77f540b499482d5dc7b64b12b79]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:27.516379 18738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6eac92dfa5f4456bc49571878f0f8dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:27.521937 18980 ts_tablet_manager.cc:1888] T 609a1a7d22e84b69bd0a53577b62b69b P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:27.522244 18980 log.cc:1196] T 609a1a7d22e84b69bd0a53577b62b69b P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/609a1a7d22e84b69bd0a53577b62b69b
I20240430 20:11:27.522755 18980 ts_tablet_manager.cc:1909] T 609a1a7d22e84b69bd0a53577b62b69b P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:27.523669 18139 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet 609a1a7d22e84b69bd0a53577b62b69b (table default.Table_062e0e190e17433395f8f47234987d92 [id=c092ea63bd4742bd9876634a1dd2e2b3]) successfully deleted
W20240430 20:11:27.524561 18139 catalog_manager.cc:4810] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): delete failed for tablet 13c1107ea4d04eb78a74d3f658c72fb3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 13c1107ea4d04eb78a74d3f658c72fb3 already in progress: creating tablet
I20240430 20:11:27.525326 18980 tablet_replica.cc:330] T 27f94f2d8f50495dababe417a9f82982 P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:27.525619 18980 raft_consensus.cc:2229] T 27f94f2d8f50495dababe417a9f82982 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:27.525878 18980 raft_consensus.cc:2258] T 27f94f2d8f50495dababe417a9f82982 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:27.526564 18980 ts_tablet_manager.cc:1875] T 27f94f2d8f50495dababe417a9f82982 P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:27.531169 18981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41678 (request call id 120) took 1830 ms. Trace:
I20240430 20:11:27.531312 18981 rpcz_store.cc:276] 0430 20:11:25.696306 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:25.696388 (+ 82us) service_pool.cc:225] Handling call
0430 20:11:27.101329 (+1404941us) ts_tablet_manager.cc:1120] Deleting tablet 59548d560b4e4f7ca034d730aca79364
0430 20:11:27.101336 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:27.211221 (+109885us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:27.434689 (+223468us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:27.526550 (+ 91861us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1403788}
I20240430 20:11:27.540717 18447 tablet_service.cc:1514] Processing DeleteTablet for tablet 75ebf051383e42529dcb5f7dd0770a4b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:27 UTC) from {username='slave'} at 127.0.0.1:41678
I20240430 20:11:27.540740 18149 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_85072f20d90d4470bf3f65189b19fd51" } modify_external_catalogs: true
I20240430 20:11:27.540961 18149 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_85072f20d90d4470bf3f65189b19fd51" } modify_external_catalogs: true
I20240430 20:11:27.541042 18457 tablet_service.cc:1514] Processing DeleteTablet for tablet a0e8e07d363f48c698c7e2b17edc1545 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:27 UTC) from {username='slave'} at 127.0.0.1:41678
I20240430 20:11:27.541157 18981 tablet_replica.cc:330] T a01ee128ba8a4c9fb8e125ba97239d4c P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:27.541128 18142 catalog_manager.cc:4830] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): tablet 59548d560b4e4f7ca034d730aca79364 (table default.Table_f6d2f14f89184a32a523563bdc6ba478 [id=f31f8f1f8e164a9d946c46164ba12f44]) successfully deleted
I20240430 20:11:27.542296 18981 raft_consensus.cc:2229] T a01ee128ba8a4c9fb8e125ba97239d4c P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Raft consensus shutting down.
I20240430 20:11:27.542987 18981 pending_rounds.cc:62] T a01ee128ba8a4c9fb8e125ba97239d4c P 2efb542aa55e439b99ccb98d1c5c884a: Trying to abort 1 pending ops.
I20240430 20:11:27.543159 18981 pending_rounds.cc:68] T a01ee128ba8a4c9fb8e125ba97239d4c P 2efb542aa55e439b99ccb98d1c5c884a: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7022624301735428096 op_type: NO_OP noop_request { }
I20240430 20:11:27.543759 18980 ts_tablet_manager.cc:1888] T 27f94f2d8f50495dababe417a9f82982 P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:27.544015 18980 log.cc:1196] T 27f94f2d8f50495dababe417a9f82982 P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/27f94f2d8f50495dababe417a9f82982
I20240430 20:11:27.544494 18980 ts_tablet_manager.cc:1909] T 27f94f2d8f50495dababe417a9f82982 P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:27.545001 18981 raft_consensus.cc:2866] T a01ee128ba8a4c9fb8e125ba97239d4c P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240430 20:11:27.545178 18981 raft_consensus.cc:2258] T a01ee128ba8a4c9fb8e125ba97239d4c P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:27.545727 18139 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet 27f94f2d8f50495dababe417a9f82982 (table default.Table_156650e4b399419c97dcf5160b761ee1 [id=b93708ff15ef4c8299967c8c91045c8f]) successfully deleted
I20240430 20:11:27.545881 18981 ts_tablet_manager.cc:1875] T a01ee128ba8a4c9fb8e125ba97239d4c P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:27.546209 18980 tablet_replica.cc:330] T 15674aeb56b74210a4a2809608fd4457 P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:27.546499 18980 raft_consensus.cc:2229] T 15674aeb56b74210a4a2809608fd4457 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:27.546990 18980 raft_consensus.cc:2258] T 15674aeb56b74210a4a2809608fd4457 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:27.548457 18739 tablet_service.cc:1514] Processing DeleteTablet for tablet f5546f9edeea44d7a6c5cb954f5157ba with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:27 UTC) from {username='slave'} at 127.0.0.1:59818
I20240430 20:11:27.548518 18980 ts_tablet_manager.cc:1875] T 15674aeb56b74210a4a2809608fd4457 P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:27.550191 18764 raft_consensus.cc:2454] T 680553d4d0bc4fcb839235c1832757e1 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
I20240430 20:11:27.555248 18980 ts_tablet_manager.cc:1888] T 15674aeb56b74210a4a2809608fd4457 P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:27.555506 18980 log.cc:1196] T 15674aeb56b74210a4a2809608fd4457 P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/15674aeb56b74210a4a2809608fd4457
I20240430 20:11:27.555939 18980 ts_tablet_manager.cc:1909] T 15674aeb56b74210a4a2809608fd4457 P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:27.557127 18139 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet 15674aeb56b74210a4a2809608fd4457 (table default.Table_7974c4f995c047469ccc69cc0d0521f3 [id=efc0011648254789a4f3b41a1fffdf4f]) successfully deleted
I20240430 20:11:27.557322 19230 consensus_queue.cc:1035] T 5809e7c9c2204ba293426c24129fb16e P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:27.557543 18980 tablet_replica.cc:330] T 0879b02a0bcc4f619fb34cff7b9d7c4c P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:27.557768 18980 raft_consensus.cc:2229] T 0879b02a0bcc4f619fb34cff7b9d7c4c P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:27.558004 18980 raft_consensus.cc:2258] T 0879b02a0bcc4f619fb34cff7b9d7c4c P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:27.558717 18980 ts_tablet_manager.cc:1875] T 0879b02a0bcc4f619fb34cff7b9d7c4c P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:27.565992 18980 ts_tablet_manager.cc:1888] T 0879b02a0bcc4f619fb34cff7b9d7c4c P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:27.566239 18980 log.cc:1196] T 0879b02a0bcc4f619fb34cff7b9d7c4c P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/0879b02a0bcc4f619fb34cff7b9d7c4c
I20240430 20:11:27.566735 18980 ts_tablet_manager.cc:1909] T 0879b02a0bcc4f619fb34cff7b9d7c4c P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:27.567685 18734 tablet_service.cc:1467] Processing CreateTablet for tablet a9e0659564894b0e8cb7f600bb07a085 (DEFAULT_TABLE table=default.Table_3c5df662747b48bf8beb6ebe4d2ea618 [id=663c8efbd7ac4a148d5f7626f1522033]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:27.568339 18985 tablet_bootstrap.cc:492] T a0e8e07d363f48c698c7e2b17edc1545 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:27.568509 18985 ts_tablet_manager.cc:1367] T a0e8e07d363f48c698c7e2b17edc1545 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.469s user 0.014s sys 0.010s
I20240430 20:11:27.571880 18734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9e0659564894b0e8cb7f600bb07a085. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:27.576570 18154 catalog_manager.cc:5484] T 680553d4d0bc4fcb839235c1832757e1 P 5e1c6397f09747f091534faf93542710 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e1c6397f09747f091534faf93542710 (127.17.177.2). New cstate: current_term: 1 leader_uuid: "5e1c6397f09747f091534faf93542710" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } health_report { overall_health: UNKNOWN } } }
I20240430 20:11:27.577148 18154 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 680553d4d0bc4fcb839235c1832757e1 (table default.Table_49a012a7c4c8457fadb0a8f0fc9654c6 [id=eecb2f8eeef547c5ab1abe6a2430e592]). Expected version 1 got 0
I20240430 20:11:27.580168 18985 raft_consensus.cc:348] T a0e8e07d363f48c698c7e2b17edc1545 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:27.580660 18583 tablet_service.cc:1467] Processing CreateTablet for tablet acf3b6fead41483a82c43dc0286fab7c (DEFAULT_TABLE table=default.Table_8c7f3f5e74f64d409fbecfce64acf648 [id=ce9911e0a91949b1997ee47321800255]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:27.581110 18583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acf3b6fead41483a82c43dc0286fab7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:27.582914 18597 tablet_service.cc:1467] Processing CreateTablet for tablet 0a9066d895744c5bb7d9532b2940603c (DEFAULT_TABLE table=default.Table_7b72c19b4765456dba543151abaf1d44 [id=ec7831570fea4ff2944bc6304fc17bc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:27.583371 18597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a9066d895744c5bb7d9532b2940603c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:27.584233 18985 raft_consensus.cc:374] T a0e8e07d363f48c698c7e2b17edc1545 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:27.584416 18985 raft_consensus.cc:729] T a0e8e07d363f48c698c7e2b17edc1545 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:27.587558 18985 consensus_queue.cc:260] T a0e8e07d363f48c698c7e2b17edc1545 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:27.588169 18985 ts_tablet_manager.cc:1398] T a0e8e07d363f48c698c7e2b17edc1545 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.020s user 0.001s sys 0.001s
I20240430 20:11:27.588496 18985 tablet_bootstrap.cc:492] T bbe9a2429a434f3f95d6e7f314fcfe4c P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:27.590514 18151 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 680553d4d0bc4fcb839235c1832757e1 (table default.Table_49a012a7c4c8457fadb0a8f0fc9654c6 [id=eecb2f8eeef547c5ab1abe6a2430e592]). Expected version 1 got 0
I20240430 20:11:27.591042 18593 tablet_service.cc:1467] Processing CreateTablet for tablet 32c823d1700949ab8b22adcd83b7bfd1 (DEFAULT_TABLE table=default.Table_e8c73c89623d4b49b8d7f0a01d729bdd [id=5cd10682ce52414cb5a103e4433a6a0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:27.591572 18593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32c823d1700949ab8b22adcd83b7bfd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:27.591738 18586 tablet_service.cc:1467] Processing CreateTablet for tablet 0eb4f3e7ece6496293e01b63620d644b (DEFAULT_TABLE table=default.Table_b6ff64dbfc324fb199b8e6eebd51fd50 [id=4b13edffc2a4446dba14079d2b0f2ca2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:27.592031 18139 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet 0879b02a0bcc4f619fb34cff7b9d7c4c (table default.Table_ab061310f439426dbd5a65210ab9b573 [id=6d6fa10e4bcc437aaee661cd8abb441f]) successfully deleted
I20240430 20:11:27.592154 18586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0eb4f3e7ece6496293e01b63620d644b. 1 dirs total, 0 dirs full, 0 dirs failed
W20240430 20:11:27.592298 18139 catalog_manager.cc:4810] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): delete failed for tablet f5546f9edeea44d7a6c5cb954f5157ba because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f5546f9edeea44d7a6c5cb954f5157ba already in progress: creating tablet
I20240430 20:11:27.592985 18599 tablet_service.cc:1467] Processing CreateTablet for tablet 5d09baed576e4136bf6d756ca39d08b2 (DEFAULT_TABLE table=default.Table_772ce91c27ce4119b6d1b1993e417f21 [id=29b41dafb4334b14a09602eabac2f031]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:27.593448 18599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d09baed576e4136bf6d756ca39d08b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:27.595872 18981 ts_tablet_manager.cc:1888] T a01ee128ba8a4c9fb8e125ba97239d4c P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:27.596096 18981 log.cc:1196] T a01ee128ba8a4c9fb8e125ba97239d4c P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/a01ee128ba8a4c9fb8e125ba97239d4c
I20240430 20:11:27.596519 18981 ts_tablet_manager.cc:1909] T a01ee128ba8a4c9fb8e125ba97239d4c P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:27.599912 18981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41678 (request call id 118) took 1907 ms. Trace:
I20240430 20:11:27.600107 18981 rpcz_store.cc:276] 0430 20:11:25.692800 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:25.693081 (+ 281us) service_pool.cc:225] Handling call
0430 20:11:27.531626 (+1838545us) ts_tablet_manager.cc:1120] Deleting tablet a01ee128ba8a4c9fb8e125ba97239d4c
0430 20:11:27.531637 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:27.572048 (+ 40411us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:27.595757 (+ 23709us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:27.599835 (+ 4078us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1824710}
I20240430 20:11:27.600363 18763 raft_consensus.cc:2454] T 609a1a7d22e84b69bd0a53577b62b69b P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
I20240430 20:11:27.601038 18981 tablet_replica.cc:330] T b0e6cc37c6b1488fa88359d16b12a576 P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:27.601570 18588 tablet_service.cc:1467] Processing CreateTablet for tablet 6e754af64cb64fd2a082adae7d5f24da (DEFAULT_TABLE table=default.Table_208b12b8e9bd4f3e8bc9c6d422ada649 [id=1df4e95702cd46f59ab21b101049a9ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:27.601742 18150 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: Sending DeleteTablet for 3 replicas of tablet 4e69ca12bd10494fb013614741bc166c
I20240430 20:11:27.601853 18980 tablet_replica.cc:330] T fd47db5ca8b24c2b827df0a64e195866 P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:27.602183 18981 raft_consensus.cc:2229] T b0e6cc37c6b1488fa88359d16b12a576 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:27.602660 18149 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: Sending DeleteTablet for 3 replicas of tablet 47c37876694c4a4abb1091c631dab7a5
I20240430 20:11:27.603165 18981 raft_consensus.cc:2258] T b0e6cc37c6b1488fa88359d16b12a576 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Raft consensus is shut down!
W20240430 20:11:27.604061 18139 catalog_manager.cc:4810] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): delete failed for tablet 02176eee8e044a7db8cac48439e9f594 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 02176eee8e044a7db8cac48439e9f594 already in progress: creating tablet
I20240430 20:11:27.604954 18732 tablet_service.cc:1467] Processing CreateTablet for tablet fd40967ab4a04001819e16c06a6c4300 (DEFAULT_TABLE table=default.Table_fda07fd7279c484db2bee67e8c859855 [id=da9ad991953a40c4b77b31317597ad1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:27.606243 18588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e754af64cb64fd2a082adae7d5f24da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:27.606670 18732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd40967ab4a04001819e16c06a6c4300. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:27.608883 18981 ts_tablet_manager.cc:1875] T b0e6cc37c6b1488fa88359d16b12a576 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:27.610062 18739 tablet_service.cc:1467] Processing CreateTablet for tablet 76d6e9d547ca4a848eae852944b1d117 (DEFAULT_TABLE table=default.Table_210b7eaff7144b2eb7db1c3b5f7d9505 [id=ff86c0357c164dd0991098c416087773]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:27.610567 18739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76d6e9d547ca4a848eae852944b1d117. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:27.615898 18744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5546f9edeea44d7a6c5cb954f5157ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:27.618817 18454 tablet_service.cc:1467] Processing CreateTablet for tablet 6e754af64cb64fd2a082adae7d5f24da (DEFAULT_TABLE table=default.Table_208b12b8e9bd4f3e8bc9c6d422ada649 [id=1df4e95702cd46f59ab21b101049a9ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:27.619014 18456 tablet_service.cc:1467] Processing CreateTablet for tablet acf3b6fead41483a82c43dc0286fab7c (DEFAULT_TABLE table=default.Table_8c7f3f5e74f64d409fbecfce64acf648 [id=ce9911e0a91949b1997ee47321800255]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:27.619401 18455 tablet_service.cc:1467] Processing CreateTablet for tablet 0a9066d895744c5bb7d9532b2940603c (DEFAULT_TABLE table=default.Table_7b72c19b4765456dba543151abaf1d44 [id=ec7831570fea4ff2944bc6304fc17bc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:27.619557 18447 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e69ca12bd10494fb013614741bc166c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:27 UTC) from {username='slave'} at 127.0.0.1:41678
I20240430 20:11:27.619396 18457 tablet_service.cc:1467] Processing CreateTablet for tablet 5d09baed576e4136bf6d756ca39d08b2 (DEFAULT_TABLE table=default.Table_772ce91c27ce4119b6d1b1993e417f21 [id=29b41dafb4334b14a09602eabac2f031]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:27.619946 18450 tablet_service.cc:1467] Processing CreateTablet for tablet 0eb4f3e7ece6496293e01b63620d644b (DEFAULT_TABLE table=default.Table_b6ff64dbfc324fb199b8e6eebd51fd50 [id=4b13edffc2a4446dba14079d2b0f2ca2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:27.620340 18462 tablet_service.cc:1467] Processing CreateTablet for tablet 32c823d1700949ab8b22adcd83b7bfd1 (DEFAULT_TABLE table=default.Table_e8c73c89623d4b49b8d7f0a01d729bdd [id=5cd10682ce52414cb5a103e4433a6a0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:27.621506 18985 tablet_bootstrap.cc:654] T bbe9a2429a434f3f95d6e7f314fcfe4c P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:27.625094 18980 raft_consensus.cc:2229] T fd47db5ca8b24c2b827df0a64e195866 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:27.625334 18980 raft_consensus.cc:2258] T fd47db5ca8b24c2b827df0a64e195866 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:27.626070 18735 tablet_service.cc:1514] Processing DeleteTablet for tablet 02176eee8e044a7db8cac48439e9f594 with delete_type TABLET_DATA_DELETED (replaced by tablet 09d92a4a6ebe44cbb3553ed660f143d0) from {username='slave'} at 127.0.0.1:59818
I20240430 20:11:27.626384 18735 tablet_service.cc:1514] Processing DeleteTablet for tablet 34ab40d8d91e401db299ddee7175ba28 with delete_type TABLET_DATA_DELETED (replaced by tablet 5f02d8862e1f4ccfa9dbda73dbe18615) from {username='slave'} at 127.0.0.1:59818
I20240430 20:11:27.627143 18735 tablet_service.cc:1514] Processing DeleteTablet for tablet a0e8e07d363f48c698c7e2b17edc1545 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:27 UTC) from {username='slave'} at 127.0.0.1:59818
I20240430 20:11:27.627414 18735 tablet_service.cc:1514] Processing DeleteTablet for tablet 75ebf051383e42529dcb5f7dd0770a4b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:27 UTC) from {username='slave'} at 127.0.0.1:59818
I20240430 20:11:27.628913 18735 tablet_service.cc:1467] Processing CreateTablet for tablet acf3b6fead41483a82c43dc0286fab7c (DEFAULT_TABLE table=default.Table_8c7f3f5e74f64d409fbecfce64acf648 [id=ce9911e0a91949b1997ee47321800255]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:27.630363 18980 ts_tablet_manager.cc:1875] T fd47db5ca8b24c2b827df0a64e195866 P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:27.631327 18557 leader_election.cc:302] T 609a1a7d22e84b69bd0a53577b62b69b P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a
I20240430 20:11:27.632167 18591 tablet_service.cc:1514] Processing DeleteTablet for tablet 47c37876694c4a4abb1091c631dab7a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:27 UTC) from {username='slave'} at 127.0.0.1:40202
I20240430 20:11:27.632903 18597 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e69ca12bd10494fb013614741bc166c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:27 UTC) from {username='slave'} at 127.0.0.1:40202
W20240430 20:11:27.633175 18557 raft_consensus.cc:2670] T 609a1a7d22e84b69bd0a53577b62b69b P 5e1c6397f09747f091534faf93542710: unable to run election callback: Service unavailable: Thread pool token was shut down
I20240430 20:11:27.635457 18456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acf3b6fead41483a82c43dc0286fab7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:27.635689 18457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d09baed576e4136bf6d756ca39d08b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:27.636092 18455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a9066d895744c5bb7d9532b2940603c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:27.636291 18454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e754af64cb64fd2a082adae7d5f24da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:27.636466 18450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0eb4f3e7ece6496293e01b63620d644b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:27.636853 18462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32c823d1700949ab8b22adcd83b7bfd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:27.631444 18731 tablet_service.cc:1467] Processing CreateTablet for tablet 0a9066d895744c5bb7d9532b2940603c (DEFAULT_TABLE table=default.Table_7b72c19b4765456dba543151abaf1d44 [id=ec7831570fea4ff2944bc6304fc17bc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:27.645907 18988 tablet_bootstrap.cc:654] T f5236574672b477a9026e91adcaf70ec P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:27.666608 18142 catalog_manager.cc:4830] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): tablet a01ee128ba8a4c9fb8e125ba97239d4c (table default.Table_23b82186d14a44eeb81cad606fb8e9cc [id=172b1ea31b8142108ae466a018ecab47]) successfully deleted
I20240430 20:11:27.666989 18447 tablet_service.cc:1514] Processing DeleteTablet for tablet 47c37876694c4a4abb1091c631dab7a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:27 UTC) from {username='slave'} at 127.0.0.1:41678
I20240430 20:11:27.674409 18736 tablet_service.cc:1467] Processing CreateTablet for tablet 32c823d1700949ab8b22adcd83b7bfd1 (DEFAULT_TABLE table=default.Table_e8c73c89623d4b49b8d7f0a01d729bdd [id=5cd10682ce52414cb5a103e4433a6a0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:27.678481 18724 tablet_service.cc:1467] Processing CreateTablet for tablet 0eb4f3e7ece6496293e01b63620d644b (DEFAULT_TABLE table=default.Table_b6ff64dbfc324fb199b8e6eebd51fd50 [id=4b13edffc2a4446dba14079d2b0f2ca2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:27.690289 18731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a9066d895744c5bb7d9532b2940603c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:27.692407 18980 ts_tablet_manager.cc:1888] T fd47db5ca8b24c2b827df0a64e195866 P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:27.692582 18980 log.cc:1196] T fd47db5ca8b24c2b827df0a64e195866 P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/fd47db5ca8b24c2b827df0a64e195866
I20240430 20:11:27.692955 18980 ts_tablet_manager.cc:1909] T fd47db5ca8b24c2b827df0a64e195866 P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:27.693609 18139 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet fd47db5ca8b24c2b827df0a64e195866 (table default.Table_14776edb17d64a83be63345e68995156 [id=6544b0a3871d41c9bf3e524f669aa0fa]) successfully deleted
I20240430 20:11:27.696579 18981 ts_tablet_manager.cc:1888] T b0e6cc37c6b1488fa88359d16b12a576 P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20240430 20:11:27.697515 18139 catalog_manager.cc:4810] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): delete failed for tablet 34ab40d8d91e401db299ddee7175ba28 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 34ab40d8d91e401db299ddee7175ba28 already in progress: creating tablet
I20240430 20:11:27.697893 18980 tablet_replica.cc:330] T a0e8e07d363f48c698c7e2b17edc1545 P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:27.698134 18980 raft_consensus.cc:2229] T a0e8e07d363f48c698c7e2b17edc1545 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:27.698356 18980 raft_consensus.cc:2258] T a0e8e07d363f48c698c7e2b17edc1545 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:27.698715 18743 tablet_service.cc:1514] Processing DeleteTablet for tablet 47c37876694c4a4abb1091c631dab7a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:27 UTC) from {username='slave'} at 127.0.0.1:59818
I20240430 20:11:27.698982 18980 ts_tablet_manager.cc:1875] T a0e8e07d363f48c698c7e2b17edc1545 P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:27.705864 18981 log.cc:1196] T b0e6cc37c6b1488fa88359d16b12a576 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/b0e6cc37c6b1488fa88359d16b12a576
I20240430 20:11:27.707460 18149 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54524:
name: "default.Table_327b9bc7f9c64a128209a7a24d2a410a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:27.708149 18149 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_327b9bc7f9c64a128209a7a24d2a410a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:27.713588 18730 tablet_service.cc:1467] Processing CreateTablet for tablet 5d09baed576e4136bf6d756ca39d08b2 (DEFAULT_TABLE table=default.Table_772ce91c27ce4119b6d1b1993e417f21 [id=29b41dafb4334b14a09602eabac2f031]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:27.713927 18724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0eb4f3e7ece6496293e01b63620d644b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:27.714852 18150 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54524:
name: "default.Table_08dc70f541b842c78a3a946d2d02e207"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:27.715473 18150 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_08dc70f541b842c78a3a946d2d02e207 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:27.716145 18981 ts_tablet_manager.cc:1909] T b0e6cc37c6b1488fa88359d16b12a576 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:27.716617 19265 raft_consensus.cc:482] T 15674aeb56b74210a4a2809608fd4457 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:27.716794 19265 raft_consensus.cc:504] T 15674aeb56b74210a4a2809608fd4457 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:27.699604 18741 tablet_service.cc:1467] Processing CreateTablet for tablet 6e754af64cb64fd2a082adae7d5f24da (DEFAULT_TABLE table=default.Table_208b12b8e9bd4f3e8bc9c6d422ada649 [id=1df4e95702cd46f59ab21b101049a9ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:27.719120 18483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15674aeb56b74210a4a2809608fd4457" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:27.719763 18981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41678 (request call id 121) took 2012 ms. Trace:
I20240430 20:11:27.719902 18981 rpcz_store.cc:276] 0430 20:11:25.707290 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:25.707401 (+ 111us) service_pool.cc:225] Handling call
0430 20:11:27.600945 (+1893544us) ts_tablet_manager.cc:1120] Deleting tablet b0e6cc37c6b1488fa88359d16b12a576
0430 20:11:27.600955 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:27.686168 (+ 85213us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:27.696446 (+ 10278us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:27.719722 (+ 23276us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1876879}
I20240430 20:11:27.721213 18980 ts_tablet_manager.cc:1888] T a0e8e07d363f48c698c7e2b17edc1545 P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:27.721370 18980 log.cc:1196] T a0e8e07d363f48c698c7e2b17edc1545 P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/a0e8e07d363f48c698c7e2b17edc1545
I20240430 20:11:27.721836 18980 ts_tablet_manager.cc:1909] T a0e8e07d363f48c698c7e2b17edc1545 P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:27.722878 18139 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet a0e8e07d363f48c698c7e2b17edc1545 (table default.Table_a058c17d17cb4a68a597fe273e1c1dfd [id=50771665d40d466f92f3a1babd60e39b]) successfully deleted
I20240430 20:11:27.722894 18981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41678 (request call id 117) took 2030 ms. Trace:
I20240430 20:11:27.723045 18981 rpcz_store.cc:276] 0430 20:11:25.692636 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:25.693051 (+ 415us) service_pool.cc:225] Handling call
0430 20:11:27.722724 (+2029673us) ts_tablet_manager.cc:1120] Deleting tablet 15674aeb56b74210a4a2809608fd4457
0430 20:11:27.722736 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:27.722855 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1998524}
I20240430 20:11:27.723573 18981 tablet_replica.cc:330] T fe01b95104174279a7d64c0d4c0af982 P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:27.723843 18981 raft_consensus.cc:2229] T fe01b95104174279a7d64c0d4c0af982 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:27.724040 18981 raft_consensus.cc:2258] T fe01b95104174279a7d64c0d4c0af982 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:27.724730 18981 ts_tablet_manager.cc:1875] T fe01b95104174279a7d64c0d4c0af982 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:27.725692 18116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNG3HqZ/build/asan/bin/kudu pid 18124
W20240430 20:11:27.727176 18139 catalog_manager.cc:4810] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): delete failed for tablet 75ebf051383e42529dcb5f7dd0770a4b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 75ebf051383e42529dcb5f7dd0770a4b already in progress: creating tablet
W20240430 20:11:27.727481 18139 catalog_manager.cc:4810] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): delete failed for tablet 47c37876694c4a4abb1091c631dab7a5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 47c37876694c4a4abb1091c631dab7a5 already in progress: creating tablet
W20240430 20:11:27.727676 18139 catalog_manager.cc:4810] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): delete failed for tablet 4e69ca12bd10494fb013614741bc166c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4e69ca12bd10494fb013614741bc166c already in progress: creating tablet
I20240430 20:11:27.728279 18186 generic_service.cc:196] Checking for leaks (request via RPC)
I20240430 20:11:27.731793 18985 tablet_bootstrap.cc:492] T bbe9a2429a434f3f95d6e7f314fcfe4c P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:27.732028 18985 ts_tablet_manager.cc:1367] T bbe9a2429a434f3f95d6e7f314fcfe4c P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.144s user 0.002s sys 0.002s
I20240430 20:11:27.733060 18985 raft_consensus.cc:348] T bbe9a2429a434f3f95d6e7f314fcfe4c P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:27.733299 18985 raft_consensus.cc:374] T bbe9a2429a434f3f95d6e7f314fcfe4c P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:27.733386 18985 raft_consensus.cc:729] T bbe9a2429a434f3f95d6e7f314fcfe4c P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:27.733644 19265 leader_election.cc:288] T 15674aeb56b74210a4a2809608fd4457 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:27.733616 18985 consensus_queue.cc:260] T bbe9a2429a434f3f95d6e7f314fcfe4c P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
W20240430 20:11:27.734040 18696 leader_election.cc:341] T 15674aeb56b74210a4a2809608fd4457 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Not found: Tablet not found: 15674aeb56b74210a4a2809608fd4457
I20240430 20:11:27.734273 18985 ts_tablet_manager.cc:1398] T bbe9a2429a434f3f95d6e7f314fcfe4c P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20240430 20:11:27.734638 18985 tablet_bootstrap.cc:492] T fd47db5ca8b24c2b827df0a64e195866 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:27.734988 18735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acf3b6fead41483a82c43dc0286fab7c. 1 dirs total, 0 dirs full, 0 dirs failed
W20240430 20:11:27.735581 18699 leader_election.cc:341] T 15674aeb56b74210a4a2809608fd4457 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:27.735726 18699 leader_election.cc:302] T 15674aeb56b74210a4a2809608fd4457 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710
I20240430 20:11:27.736006 19265 raft_consensus.cc:2726] T 15674aeb56b74210a4a2809608fd4457 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:27.736356 18985 tablet_bootstrap.cc:654] T fd47db5ca8b24c2b827df0a64e195866 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:27.738992 18985 tablet_bootstrap.cc:492] T fd47db5ca8b24c2b827df0a64e195866 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:27.739187 18985 ts_tablet_manager.cc:1367] T fd47db5ca8b24c2b827df0a64e195866 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240430 20:11:27.739975 18985 raft_consensus.cc:348] T fd47db5ca8b24c2b827df0a64e195866 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:27.740864 18985 raft_consensus.cc:374] T fd47db5ca8b24c2b827df0a64e195866 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:27.740972 18985 raft_consensus.cc:729] T fd47db5ca8b24c2b827df0a64e195866 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:27.741744 18985 consensus_queue.cc:260] T fd47db5ca8b24c2b827df0a64e195866 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:27.743880 18985 ts_tablet_manager.cc:1398] T fd47db5ca8b24c2b827df0a64e195866 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240430 20:11:27.744269 18985 tablet_bootstrap.cc:492] T 13c1107ea4d04eb78a74d3f658c72fb3 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:27.746835 18985 tablet_bootstrap.cc:654] T 13c1107ea4d04eb78a74d3f658c72fb3 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:27.747831 18983 tablet_bootstrap.cc:492] T 4e97f7315f8b4ac29db16e00fc35bf99 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:27.748039 18983 ts_tablet_manager.cc:1367] T 4e97f7315f8b4ac29db16e00fc35bf99 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.396s user 0.021s sys 0.053s
I20240430 20:11:27.749349 18983 raft_consensus.cc:348] T 4e97f7315f8b4ac29db16e00fc35bf99 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:27.749586 18983 raft_consensus.cc:374] T 4e97f7315f8b4ac29db16e00fc35bf99 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:27.749686 18983 raft_consensus.cc:729] T 4e97f7315f8b4ac29db16e00fc35bf99 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:27.749953 18983 consensus_queue.cc:260] T 4e97f7315f8b4ac29db16e00fc35bf99 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:27.753206 18981 ts_tablet_manager.cc:1888] T fe01b95104174279a7d64c0d4c0af982 P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:27.753449 18729 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e69ca12bd10494fb013614741bc166c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:27 UTC) from {username='slave'} at 127.0.0.1:59818
I20240430 20:11:27.763731 18983 ts_tablet_manager.cc:1398] T 4e97f7315f8b4ac29db16e00fc35bf99 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.016s user 0.002s sys 0.002s
I20240430 20:11:27.771466 18983 tablet_bootstrap.cc:492] T fae9a35a2ac844b4bc2ebdf214aa7171 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:27.774013 19185 raft_consensus.cc:482] T fd47db5ca8b24c2b827df0a64e195866 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:27.774214 19185 raft_consensus.cc:504] T fd47db5ca8b24c2b827df0a64e195866 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:27.776831 19185 leader_election.cc:288] T fd47db5ca8b24c2b827df0a64e195866 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
W20240430 20:11:27.777144 18415 leader_election.cc:341] T fd47db5ca8b24c2b827df0a64e195866 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Not found: Tablet not found: fd47db5ca8b24c2b827df0a64e195866
I20240430 20:11:27.777374 18981 log.cc:1196] T fe01b95104174279a7d64c0d4c0af982 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/fe01b95104174279a7d64c0d4c0af982
I20240430 20:11:27.777878 18981 ts_tablet_manager.cc:1909] T fe01b95104174279a7d64c0d4c0af982 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:27.778900 18981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41678 (request call id 131) took 1680 ms. Trace:
I20240430 20:11:27.779073 18981 rpcz_store.cc:276] 0430 20:11:26.098044 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:26.098402 (+ 358us) service_pool.cc:225] Handling call
0430 20:11:27.723491 (+1625089us) ts_tablet_manager.cc:1120] Deleting tablet fe01b95104174279a7d64c0d4c0af982
0430 20:11:27.723503 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:27.744318 (+ 20815us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:27.753042 (+ 8724us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:27.778823 (+ 25781us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1624684}
I20240430 20:11:27.779789 18981 tablet_replica.cc:330] T bd7392f3dfd643cdb1634ee10ad41651 P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:27.780465 18981 raft_consensus.cc:2229] T bd7392f3dfd643cdb1634ee10ad41651 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Raft consensus shutting down.
I20240430 20:11:27.780952 18981 pending_rounds.cc:62] T bd7392f3dfd643cdb1634ee10ad41651 P 2efb542aa55e439b99ccb98d1c5c884a: Trying to abort 1 pending ops.
I20240430 20:11:27.781129 18981 pending_rounds.cc:68] T bd7392f3dfd643cdb1634ee10ad41651 P 2efb542aa55e439b99ccb98d1c5c884a: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7022624302532632576 op_type: NO_OP noop_request { }
I20240430 20:11:27.781415 18981 raft_consensus.cc:2866] T bd7392f3dfd643cdb1634ee10ad41651 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240430 20:11:27.781613 18981 raft_consensus.cc:2258] T bd7392f3dfd643cdb1634ee10ad41651 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:27.781139 18985 tablet_bootstrap.cc:492] T 13c1107ea4d04eb78a74d3f658c72fb3 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:27.782667 18736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32c823d1700949ab8b22adcd83b7bfd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:27.782894 18763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd47db5ca8b24c2b827df0a64e195866" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:27.785341 19191 raft_consensus.cc:482] T fd47db5ca8b24c2b827df0a64e195866 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240430 20:11:27.785499 19268 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.654s user 0.018s sys 0.040s
I20240430 20:11:27.786841 18741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e754af64cb64fd2a082adae7d5f24da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:27.788779 18981 ts_tablet_manager.cc:1875] T bd7392f3dfd643cdb1634ee10ad41651 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:27.789443 18985 ts_tablet_manager.cc:1367] T 13c1107ea4d04eb78a74d3f658c72fb3 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.038s user 0.017s sys 0.015s
I20240430 20:11:27.784773 18983 tablet_bootstrap.cc:654] T fae9a35a2ac844b4bc2ebdf214aa7171 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:27.791062 18620 raft_consensus.cc:1261] T 81787eccca6c42c1b421377a3bf9ebe6 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Refusing update from remote peer 3c51c33d5616482cafb68e26688820f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240430 20:11:27.794628 18483 raft_consensus.cc:1261] T 81787eccca6c42c1b421377a3bf9ebe6 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Refusing update from remote peer 3c51c33d5616482cafb68e26688820f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240430 20:11:27.794632 18985 raft_consensus.cc:348] T 13c1107ea4d04eb78a74d3f658c72fb3 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:27.795472 18985 raft_consensus.cc:374] T 13c1107ea4d04eb78a74d3f658c72fb3 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:27.795573 18985 raft_consensus.cc:729] T 13c1107ea4d04eb78a74d3f658c72fb3 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:27.795797 18985 consensus_queue.cc:260] T 13c1107ea4d04eb78a74d3f658c72fb3 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:27.796383 18985 ts_tablet_manager.cc:1398] T 13c1107ea4d04eb78a74d3f658c72fb3 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
W20240430 20:11:27.796911 19268 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.703s user 0.018s sys 0.041s
I20240430 20:11:27.797622 19267 consensus_queue.cc:1035] T 81787eccca6c42c1b421377a3bf9ebe6 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.006s
W20240430 20:11:27.800136 18415 consensus_peers.cc:485] T ffe059eeac91476aaeef8c56d2f49ee8 P 2efb542aa55e439b99ccb98d1c5c884a -> Peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Couldn't send request to peer 5e1c6397f09747f091534faf93542710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ffe059eeac91476aaeef8c56d2f49ee8. This is attempt 1: this message will repeat every 5th retry.
I20240430 20:11:27.800268 18988 tablet_bootstrap.cc:492] T f5236574672b477a9026e91adcaf70ec P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:27.802803 18985 tablet_bootstrap.cc:492] T 4e97f7315f8b4ac29db16e00fc35bf99 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:27.804543 18985 tablet_bootstrap.cc:654] T 4e97f7315f8b4ac29db16e00fc35bf99 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:27.800401 19191 raft_consensus.cc:504] T fd47db5ca8b24c2b827df0a64e195866 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:27.801960 19265 consensus_queue.cc:1035] T 81787eccca6c42c1b421377a3bf9ebe6 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.016s
I20240430 20:11:27.815037 18730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d09baed576e4136bf6d756ca39d08b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:27.816561 18988 ts_tablet_manager.cc:1367] T f5236574672b477a9026e91adcaf70ec P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.299s user 0.005s sys 0.001s
I20240430 20:11:27.819526 18482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd47db5ca8b24c2b827df0a64e195866" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:27.819887 18482 raft_consensus.cc:2454] T fd47db5ca8b24c2b827df0a64e195866 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:27.820605 18763 raft_consensus.cc:2454] T fd47db5ca8b24c2b827df0a64e195866 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 0.
I20240430 20:11:27.821799 18699 leader_election.cc:302] T fd47db5ca8b24c2b827df0a64e195866 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters:
I20240430 20:11:27.823101 18417 leader_election.cc:302] T fd47db5ca8b24c2b827df0a64e195866 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters: 5e1c6397f09747f091534faf93542710
I20240430 20:11:27.823407 19185 raft_consensus.cc:2781] T fd47db5ca8b24c2b827df0a64e195866 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:27.823539 19185 raft_consensus.cc:482] T fd47db5ca8b24c2b827df0a64e195866 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:27.823643 19185 raft_consensus.cc:3037] T fd47db5ca8b24c2b827df0a64e195866 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:27.827469 18983 tablet_bootstrap.cc:492] T fae9a35a2ac844b4bc2ebdf214aa7171 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:27.827667 18983 ts_tablet_manager.cc:1367] T fae9a35a2ac844b4bc2ebdf214aa7171 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.057s user 0.003s sys 0.001s
I20240430 20:11:27.828505 18983 raft_consensus.cc:348] T fae9a35a2ac844b4bc2ebdf214aa7171 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:27.828763 18983 raft_consensus.cc:374] T fae9a35a2ac844b4bc2ebdf214aa7171 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:27.828860 18983 raft_consensus.cc:729] T fae9a35a2ac844b4bc2ebdf214aa7171 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:27.826486 19185 raft_consensus.cc:504] T fd47db5ca8b24c2b827df0a64e195866 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
W20240430 20:11:27.830392 18415 leader_election.cc:341] T fd47db5ca8b24c2b827df0a64e195866 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Not found: Tablet not found: fd47db5ca8b24c2b827df0a64e195866
I20240430 20:11:27.830945 19185 leader_election.cc:288] T fd47db5ca8b24c2b827df0a64e195866 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:27.826514 18981 ts_tablet_manager.cc:1888] T bd7392f3dfd643cdb1634ee10ad41651 P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:27.833995 18981 log.cc:1196] T bd7392f3dfd643cdb1634ee10ad41651 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/bd7392f3dfd643cdb1634ee10ad41651
W20240430 20:11:27.835721 18415 consensus_peers.cc:485] T b4bc111e183f40a0acc9dd377195f6d9 P 2efb542aa55e439b99ccb98d1c5c884a -> Peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Couldn't send request to peer 5e1c6397f09747f091534faf93542710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b4bc111e183f40a0acc9dd377195f6d9. This is attempt 1: this message will repeat every 5th retry.
I20240430 20:11:27.839143 18981 ts_tablet_manager.cc:1909] T bd7392f3dfd643cdb1634ee10ad41651 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:27.840425 18981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41678 (request call id 132) took 1742 ms. Trace:
I20240430 20:11:27.840783 18983 consensus_queue.cc:260] T fae9a35a2ac844b4bc2ebdf214aa7171 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:27.840531 18981 rpcz_store.cc:276] 0430 20:11:26.098330 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:26.098860 (+ 530us) service_pool.cc:225] Handling call
0430 20:11:27.779695 (+1680835us) ts_tablet_manager.cc:1120] Deleting tablet bd7392f3dfd643cdb1634ee10ad41651
0430 20:11:27.779708 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:27.797647 (+ 17939us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:27.811746 (+ 14099us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:27.840361 (+ 28615us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1680671}
I20240430 20:11:27.841912 18981 tablet_replica.cc:330] T b4bc111e183f40a0acc9dd377195f6d9 P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:27.842165 18981 raft_consensus.cc:2229] T b4bc111e183f40a0acc9dd377195f6d9 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Raft consensus shutting down.
I20240430 20:11:27.842537 18981 pending_rounds.cc:62] T b4bc111e183f40a0acc9dd377195f6d9 P 2efb542aa55e439b99ccb98d1c5c884a: Trying to abort 1 pending ops.
I20240430 20:11:27.842636 18981 pending_rounds.cc:68] T b4bc111e183f40a0acc9dd377195f6d9 P 2efb542aa55e439b99ccb98d1c5c884a: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7022624305251516416 op_type: NO_OP noop_request { }
I20240430 20:11:27.843113 18983 ts_tablet_manager.cc:1398] T fae9a35a2ac844b4bc2ebdf214aa7171 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.015s user 0.003s sys 0.000s
I20240430 20:11:27.843465 18983 tablet_bootstrap.cc:492] T bbe9a2429a434f3f95d6e7f314fcfe4c P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:27.844240 18981 raft_consensus.cc:2866] T b4bc111e183f40a0acc9dd377195f6d9 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240430 20:11:27.844386 18981 raft_consensus.cc:2258] T b4bc111e183f40a0acc9dd377195f6d9 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:27.845083 18763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd47db5ca8b24c2b827df0a64e195866" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:27.846861 18983 tablet_bootstrap.cc:654] T bbe9a2429a434f3f95d6e7f314fcfe4c P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:27.848817 18981 ts_tablet_manager.cc:1875] T b4bc111e183f40a0acc9dd377195f6d9 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:27.850659 19267 raft_consensus.cc:2781] T fd47db5ca8b24c2b827df0a64e195866 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:27.850786 19267 raft_consensus.cc:482] T fd47db5ca8b24c2b827df0a64e195866 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240430 20:11:27.850893 18696 leader_election.cc:341] T fd47db5ca8b24c2b827df0a64e195866 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Not found: Tablet not found: fd47db5ca8b24c2b827df0a64e195866
I20240430 20:11:27.854696 19267 raft_consensus.cc:3037] T fd47db5ca8b24c2b827df0a64e195866 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:27.880264 19267 raft_consensus.cc:504] T fd47db5ca8b24c2b827df0a64e195866 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:27.881522 18483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd47db5ca8b24c2b827df0a64e195866" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:27.881939 18483 raft_consensus.cc:2379] T fd47db5ca8b24c2b827df0a64e195866 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 3c51c33d5616482cafb68e26688820f4 in current term 1: Already voted for candidate 2efb542aa55e439b99ccb98d1c5c884a in this term.
W20240430 20:11:27.882364 18696 leader_election.cc:341] T fd47db5ca8b24c2b827df0a64e195866 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Not found: Tablet not found: fd47db5ca8b24c2b827df0a64e195866
I20240430 20:11:27.882568 18699 leader_election.cc:302] T fd47db5ca8b24c2b827df0a64e195866 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710
I20240430 20:11:27.890239 18983 tablet_bootstrap.cc:492] T bbe9a2429a434f3f95d6e7f314fcfe4c P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:27.890461 18983 ts_tablet_manager.cc:1367] T bbe9a2429a434f3f95d6e7f314fcfe4c P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.047s user 0.001s sys 0.003s
I20240430 20:11:27.891362 18763 raft_consensus.cc:2379] T fd47db5ca8b24c2b827df0a64e195866 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 2efb542aa55e439b99ccb98d1c5c884a in current term 1: Already voted for candidate 3c51c33d5616482cafb68e26688820f4 in this term.
I20240430 20:11:27.891567 18983 raft_consensus.cc:348] T bbe9a2429a434f3f95d6e7f314fcfe4c P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:27.891782 18983 raft_consensus.cc:374] T bbe9a2429a434f3f95d6e7f314fcfe4c P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:27.891901 18983 raft_consensus.cc:729] T bbe9a2429a434f3f95d6e7f314fcfe4c P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:27.891889 19265 raft_consensus.cc:2726] T fd47db5ca8b24c2b827df0a64e195866 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240430 20:11:27.892563 18417 leader_election.cc:302] T fd47db5ca8b24c2b827df0a64e195866 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a; no voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710
I20240430 20:11:27.894483 19185 raft_consensus.cc:2726] T fd47db5ca8b24c2b827df0a64e195866 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240430 20:11:27.895251 19267 leader_election.cc:288] T fd47db5ca8b24c2b827df0a64e195866 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Requested vote from peers 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:27.896008 18988 raft_consensus.cc:348] T f5236574672b477a9026e91adcaf70ec P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:27.896330 18988 raft_consensus.cc:374] T f5236574672b477a9026e91adcaf70ec P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:27.846352 19191 leader_election.cc:288] T fd47db5ca8b24c2b827df0a64e195866 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:27.902187 19186 consensus_queue.cc:1035] T ffe059eeac91476aaeef8c56d2f49ee8 P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:27.902812 18988 raft_consensus.cc:729] T f5236574672b477a9026e91adcaf70ec P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:27.905189 18988 consensus_queue.cc:260] T f5236574672b477a9026e91adcaf70ec P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:27.906703 19311 tablet.cc:1722] T 81787eccca6c42c1b421377a3bf9ebe6 P 2efb542aa55e439b99ccb98d1c5c884a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240430 20:11:27.906456 19312 tablet.cc:1722] T 81787eccca6c42c1b421377a3bf9ebe6 P 3c51c33d5616482cafb68e26688820f4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20240430 20:11:27.909741 18415 consensus_peers.cc:485] T 27f94f2d8f50495dababe417a9f82982 P 2efb542aa55e439b99ccb98d1c5c884a -> Peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Couldn't send request to peer 5e1c6397f09747f091534faf93542710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 27f94f2d8f50495dababe417a9f82982. This is attempt 1: this message will repeat every 5th retry.
I20240430 20:11:27.892132 18983 consensus_queue.cc:260] T bbe9a2429a434f3f95d6e7f314fcfe4c P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:27.918205 18983 ts_tablet_manager.cc:1398] T bbe9a2429a434f3f95d6e7f314fcfe4c P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.028s user 0.004s sys 0.000s
I20240430 20:11:27.919898 18983 tablet_bootstrap.cc:492] T 0fe1247349344863a6273a769ce8ddf6 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:27.923674 18983 tablet_bootstrap.cc:654] T 0fe1247349344863a6273a769ce8ddf6 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:27.925510 18981 ts_tablet_manager.cc:1888] T b4bc111e183f40a0acc9dd377195f6d9 P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:27.925781 18981 log.cc:1196] T b4bc111e183f40a0acc9dd377195f6d9 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/b4bc111e183f40a0acc9dd377195f6d9
I20240430 20:11:27.926328 18981 ts_tablet_manager.cc:1909] T b4bc111e183f40a0acc9dd377195f6d9 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:27.928834 18981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41678 (request call id 133) took 1829 ms. Trace:
I20240430 20:11:27.930656 18988 ts_tablet_manager.cc:1398] T f5236574672b477a9026e91adcaf70ec P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.098s user 0.001s sys 0.001s
I20240430 20:11:27.934079 18981 rpcz_store.cc:276] 0430 20:11:26.099340 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:26.099503 (+ 163us) service_pool.cc:225] Handling call
0430 20:11:27.841807 (+1742304us) ts_tablet_manager.cc:1120] Deleting tablet b4bc111e183f40a0acc9dd377195f6d9
0430 20:11:27.841822 (+ 15us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:27.868225 (+ 26403us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:27.924068 (+ 55843us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:27.928761 (+ 4693us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1742098}
I20240430 20:11:27.934666 18981 tablet_replica.cc:330] T ffe059eeac91476aaeef8c56d2f49ee8 P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:27.934882 18981 raft_consensus.cc:2229] T ffe059eeac91476aaeef8c56d2f49ee8 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Raft consensus shutting down.
I20240430 20:11:27.935191 18981 pending_rounds.cc:62] T ffe059eeac91476aaeef8c56d2f49ee8 P 2efb542aa55e439b99ccb98d1c5c884a: Trying to abort 1 pending ops.
I20240430 20:11:27.935267 18981 pending_rounds.cc:68] T ffe059eeac91476aaeef8c56d2f49ee8 P 2efb542aa55e439b99ccb98d1c5c884a: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7022624303912542208 op_type: NO_OP noop_request { }
I20240430 20:11:27.935418 18981 raft_consensus.cc:2866] T ffe059eeac91476aaeef8c56d2f49ee8 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240430 20:11:27.935525 18981 raft_consensus.cc:2258] T ffe059eeac91476aaeef8c56d2f49ee8 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:27.936131 18981 ts_tablet_manager.cc:1875] T ffe059eeac91476aaeef8c56d2f49ee8 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:27.939108 18988 tablet_bootstrap.cc:492] T 0fe1247349344863a6273a769ce8ddf6 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:27.944068 18983 tablet_bootstrap.cc:492] T 0fe1247349344863a6273a769ce8ddf6 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:27.942988 18988 tablet_bootstrap.cc:654] T 0fe1247349344863a6273a769ce8ddf6 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:27.944268 18983 ts_tablet_manager.cc:1367] T 0fe1247349344863a6273a769ce8ddf6 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.025s user 0.002s sys 0.002s
I20240430 20:11:27.945115 18983 raft_consensus.cc:348] T 0fe1247349344863a6273a769ce8ddf6 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:27.945369 18983 raft_consensus.cc:374] T 0fe1247349344863a6273a769ce8ddf6 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:27.945451 18983 raft_consensus.cc:729] T 0fe1247349344863a6273a769ce8ddf6 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:27.945739 18983 consensus_queue.cc:260] T 0fe1247349344863a6273a769ce8ddf6 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:27.947381 18987 raft_consensus.cc:2229] T 81328e904ca94e2eb5ec94dd3ffb3c90 P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Raft consensus shutting down.
I20240430 20:11:27.948357 18987 raft_consensus.cc:2258] T 81328e904ca94e2eb5ec94dd3ffb3c90 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:27.949213 18983 ts_tablet_manager.cc:1398] T 0fe1247349344863a6273a769ce8ddf6 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240430 20:11:27.949239 18987 ts_tablet_manager.cc:1875] T 81328e904ca94e2eb5ec94dd3ffb3c90 P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:27.949647 18983 tablet_bootstrap.cc:492] T 7bde1d6934784c9aa5911181e8df9eeb P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:27.951411 18983 tablet_bootstrap.cc:654] T 7bde1d6934784c9aa5911181e8df9eeb P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:27.953408 18981 ts_tablet_manager.cc:1888] T ffe059eeac91476aaeef8c56d2f49ee8 P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:27.953655 18981 log.cc:1196] T ffe059eeac91476aaeef8c56d2f49ee8 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/ffe059eeac91476aaeef8c56d2f49ee8
I20240430 20:11:27.954172 18981 ts_tablet_manager.cc:1909] T ffe059eeac91476aaeef8c56d2f49ee8 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:27.955225 18981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41678 (request call id 134) took 1855 ms. Trace:
I20240430 20:11:27.955349 18981 rpcz_store.cc:276] 0430 20:11:26.099463 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:26.099724 (+ 261us) service_pool.cc:225] Handling call
0430 20:11:27.934550 (+1834826us) ts_tablet_manager.cc:1120] Deleting tablet ffe059eeac91476aaeef8c56d2f49ee8
0430 20:11:27.934561 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:27.939471 (+ 4910us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:27.942562 (+ 3091us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:27.955153 (+ 12591us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1834682}
I20240430 20:11:27.956559 18987 ts_tablet_manager.cc:1888] T 81328e904ca94e2eb5ec94dd3ffb3c90 P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:27.956761 18987 log.cc:1196] T 81328e904ca94e2eb5ec94dd3ffb3c90 P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/81328e904ca94e2eb5ec94dd3ffb3c90
I20240430 20:11:27.957202 18987 ts_tablet_manager.cc:1909] T 81328e904ca94e2eb5ec94dd3ffb3c90 P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:27.957981 18981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41678 (request call id 135) took 1631 ms. Trace:
I20240430 20:11:27.958098 18981 rpcz_store.cc:276] 0430 20:11:26.326950 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:26.327031 (+ 81us) service_pool.cc:225] Handling call
0430 20:11:27.957768 (+1630737us) ts_tablet_manager.cc:1120] Deleting tablet d9e578c99058424a9b86390641f9c34f
0430 20:11:27.957781 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:27.957919 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1630492}
I20240430 20:11:27.958032 18987 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59818 (request call id 148) took 1449 ms. Trace:
I20240430 20:11:27.959251 18981 tablet_replica.cc:330] T fd47db5ca8b24c2b827df0a64e195866 P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:27.959457 18981 raft_consensus.cc:2229] T fd47db5ca8b24c2b827df0a64e195866 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:27.959653 18981 raft_consensus.cc:2258] T fd47db5ca8b24c2b827df0a64e195866 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:27.959614 18987 rpcz_store.cc:276] 0430 20:11:26.508977 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:26.509070 (+ 93us) service_pool.cc:225] Handling call
0430 20:11:27.278338 (+769268us) ts_tablet_manager.cc:1120] Deleting tablet 81328e904ca94e2eb5ec94dd3ffb3c90
0430 20:11:27.278352 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:27.947687 (+669335us) spinlock_profiling.cc:243] Waited 638 ms on lock 0x6160000a4b54. stack: 00007f30e7d63756 00007f30e7d63510 00007f30e6d9648d 0000000000955804 00000000009557cd 000000000095519a 00007f30f35d1a00 00007f30f48ab853 00007f30f733e4bc 00007f30f7370814 00007f30f7351b00 00007f30f735188d 0000000000a60bcc 00007f30e7da9b74 00007f30e7dad19b 00007f30e7dad018
0430 20:11:27.952675 (+ 4988us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:27.956410 (+ 3735us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:27.957994 (+ 1584us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":21625,"spinlock_wait_cycles":1466697984,"tablet-delete.queue_time_us":768825}
I20240430 20:11:27.961014 18981 ts_tablet_manager.cc:1875] T fd47db5ca8b24c2b827df0a64e195866 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:27.963554 18988 tablet_bootstrap.cc:492] T 0fe1247349344863a6273a769ce8ddf6 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:27.967095 18988 ts_tablet_manager.cc:1367] T 0fe1247349344863a6273a769ce8ddf6 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.028s user 0.004s sys 0.004s
I20240430 20:11:27.967502 18987 tablet_replica.cc:330] T ffe059eeac91476aaeef8c56d2f49ee8 P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:27.967578 18981 ts_tablet_manager.cc:1888] T fd47db5ca8b24c2b827df0a64e195866 P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:27.967718 18981 log.cc:1196] T fd47db5ca8b24c2b827df0a64e195866 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/fd47db5ca8b24c2b827df0a64e195866
I20240430 20:11:27.968127 18981 ts_tablet_manager.cc:1909] T fd47db5ca8b24c2b827df0a64e195866 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:27.968881 18983 tablet_bootstrap.cc:492] T 7bde1d6934784c9aa5911181e8df9eeb P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:27.968945 18981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41678 (request call id 136) took 1609 ms. Trace:
I20240430 20:11:27.969062 18983 ts_tablet_manager.cc:1367] T 7bde1d6934784c9aa5911181e8df9eeb P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.003s
I20240430 20:11:27.969062 18981 rpcz_store.cc:276] 0430 20:11:26.359126 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:26.363074 (+ 3948us) service_pool.cc:225] Handling call
0430 20:11:27.959159 (+1596085us) ts_tablet_manager.cc:1120] Deleting tablet fd47db5ca8b24c2b827df0a64e195866
0430 20:11:27.959168 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:27.964348 (+ 5180us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:27.967437 (+ 3089us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:27.968892 (+ 1455us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1595882}
I20240430 20:11:27.969669 18981 tablet_replica.cc:330] T 17843ea77664447eaaa19666e0445ada P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:27.969902 18981 raft_consensus.cc:2229] T 17843ea77664447eaaa19666e0445ada P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Raft consensus shutting down.
I20240430 20:11:27.970266 18981 raft_consensus.cc:2258] T 17843ea77664447eaaa19666e0445ada P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:27.970623 18987 raft_consensus.cc:2229] T ffe059eeac91476aaeef8c56d2f49ee8 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:27.970571 18983 raft_consensus.cc:348] T 7bde1d6934784c9aa5911181e8df9eeb P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:27.970834 18983 raft_consensus.cc:374] T 7bde1d6934784c9aa5911181e8df9eeb P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:27.970916 18981 ts_tablet_manager.cc:1875] T 17843ea77664447eaaa19666e0445ada P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:27.970939 18983 raft_consensus.cc:729] T 7bde1d6934784c9aa5911181e8df9eeb P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:27.971199 18983 consensus_queue.cc:260] T 7bde1d6934784c9aa5911181e8df9eeb P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:27.972236 18983 ts_tablet_manager.cc:1398] T 7bde1d6934784c9aa5911181e8df9eeb P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20240430 20:11:27.968012 18988 raft_consensus.cc:348] T 0fe1247349344863a6273a769ce8ddf6 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:27.973170 18988 raft_consensus.cc:374] T 0fe1247349344863a6273a769ce8ddf6 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:27.973275 18988 raft_consensus.cc:729] T 0fe1247349344863a6273a769ce8ddf6 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:27.973579 18988 consensus_queue.cc:260] T 0fe1247349344863a6273a769ce8ddf6 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:27.974162 18988 ts_tablet_manager.cc:1398] T 0fe1247349344863a6273a769ce8ddf6 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20240430 20:11:27.974908 18988 tablet_bootstrap.cc:492] T 7bde1d6934784c9aa5911181e8df9eeb P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:27.976178 18983 tablet_bootstrap.cc:492] T cace1398492f4d968dcb6899385d527c P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:27.976759 18988 tablet_bootstrap.cc:654] T 7bde1d6934784c9aa5911181e8df9eeb P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:27.977947 18981 ts_tablet_manager.cc:1888] T 17843ea77664447eaaa19666e0445ada P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240430 20:11:27.978111 18981 log.cc:1196] T 17843ea77664447eaaa19666e0445ada P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/17843ea77664447eaaa19666e0445ada
I20240430 20:11:27.978570 18981 ts_tablet_manager.cc:1909] T 17843ea77664447eaaa19666e0445ada P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:27.973577 18987 pending_rounds.cc:62] T ffe059eeac91476aaeef8c56d2f49ee8 P 3c51c33d5616482cafb68e26688820f4: Trying to abort 1 pending ops.
I20240430 20:11:27.979465 18981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41678 (request call id 137) took 1620 ms. Trace:
I20240430 20:11:27.979614 18981 rpcz_store.cc:276] 0430 20:11:26.359229 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:26.363528 (+ 4299us) service_pool.cc:225] Handling call
0430 20:11:27.969582 (+1606054us) ts_tablet_manager.cc:1120] Deleting tablet 17843ea77664447eaaa19666e0445ada
0430 20:11:27.969596 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:27.974579 (+ 4983us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:27.977809 (+ 3230us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:27.979378 (+ 1569us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1594678}
I20240430 20:11:27.980264 18981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41678 (request call id 141) took 1525 ms. Trace:
I20240430 20:11:27.980369 18981 rpcz_store.cc:276] 0430 20:11:26.454963 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:26.455432 (+ 469us) service_pool.cc:225] Handling call
0430 20:11:27.980088 (+1524656us) ts_tablet_manager.cc:1120] Deleting tablet 29974dc2c7dd45b99b6cc0712378fc30
0430 20:11:27.980099 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:27.980215 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1524442}
I20240430 20:11:27.979733 18987 pending_rounds.cc:68] T ffe059eeac91476aaeef8c56d2f49ee8 P 3c51c33d5616482cafb68e26688820f4: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7022624303912542208 op_type: NO_OP noop_request { }
I20240430 20:11:27.980811 18981 tablet_replica.cc:330] T 81328e904ca94e2eb5ec94dd3ffb3c90 P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:27.980929 18983 tablet_bootstrap.cc:654] T cace1398492f4d968dcb6899385d527c P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:27.981038 18981 raft_consensus.cc:2229] T 81328e904ca94e2eb5ec94dd3ffb3c90 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:27.981252 18981 raft_consensus.cc:2258] T 81328e904ca94e2eb5ec94dd3ffb3c90 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:27.981510 18985 tablet_bootstrap.cc:492] T 4e97f7315f8b4ac29db16e00fc35bf99 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
W20240430 20:11:27.980327 19313 consensus_queue.cc:1159] T ffe059eeac91476aaeef8c56d2f49ee8 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "3c51c33d5616482cafb68e26688820f4" status { last_received { term: 1 index: 1 } last_committed_idx: 0 last_received_current_leader { term: 1 index: 1 } }
I20240430 20:11:27.980717 18987 raft_consensus.cc:2866] T ffe059eeac91476aaeef8c56d2f49ee8 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20240430 20:11:27.983474 18983 tablet_bootstrap.cc:492] T cace1398492f4d968dcb6899385d527c P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:27.983659 18983 ts_tablet_manager.cc:1367] T cace1398492f4d968dcb6899385d527c P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20240430 20:11:27.983790 18987 raft_consensus.cc:2258] T ffe059eeac91476aaeef8c56d2f49ee8 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:27.984436 18983 raft_consensus.cc:348] T cace1398492f4d968dcb6899385d527c P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:27.984671 18983 raft_consensus.cc:374] T cace1398492f4d968dcb6899385d527c P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:27.984750 18983 raft_consensus.cc:729] T cace1398492f4d968dcb6899385d527c P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:27.985033 18983 consensus_queue.cc:260] T cace1398492f4d968dcb6899385d527c P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:27.985679 18983 ts_tablet_manager.cc:1398] T cace1398492f4d968dcb6899385d527c P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240430 20:11:27.986016 18983 tablet_bootstrap.cc:492] T f5236574672b477a9026e91adcaf70ec P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:27.982206 18981 ts_tablet_manager.cc:1875] T 81328e904ca94e2eb5ec94dd3ffb3c90 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:27.988142 18985 ts_tablet_manager.cc:1367] T 4e97f7315f8b4ac29db16e00fc35bf99 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.185s user 0.008s sys 0.010s
I20240430 20:11:27.992513 18987 ts_tablet_manager.cc:1875] T ffe059eeac91476aaeef8c56d2f49ee8 P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:27.994390 18981 ts_tablet_manager.cc:1888] T 81328e904ca94e2eb5ec94dd3ffb3c90 P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:27.994576 18981 log.cc:1196] T 81328e904ca94e2eb5ec94dd3ffb3c90 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/81328e904ca94e2eb5ec94dd3ffb3c90
I20240430 20:11:27.995138 18981 ts_tablet_manager.cc:1909] T 81328e904ca94e2eb5ec94dd3ffb3c90 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:27.995790 18985 raft_consensus.cc:348] T 4e97f7315f8b4ac29db16e00fc35bf99 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:27.996732 18985 raft_consensus.cc:374] T 4e97f7315f8b4ac29db16e00fc35bf99 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:27.997016 18985 raft_consensus.cc:729] T 4e97f7315f8b4ac29db16e00fc35bf99 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:27.995966 18981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41678 (request call id 144) took 1482 ms. Trace:
I20240430 20:11:27.997455 18981 rpcz_store.cc:276] 0430 20:11:26.513062 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:26.513124 (+ 62us) service_pool.cc:225] Handling call
0430 20:11:27.980714 (+1467590us) ts_tablet_manager.cc:1120] Deleting tablet 81328e904ca94e2eb5ec94dd3ffb3c90
0430 20:11:27.980725 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:27.991020 (+ 10295us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:27.994242 (+ 3222us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:27.995922 (+ 1680us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1467374}
I20240430 20:11:27.998143 18981 tablet_replica.cc:330] T bbe9a2429a434f3f95d6e7f314fcfe4c P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:27.998414 18981 raft_consensus.cc:2229] T bbe9a2429a434f3f95d6e7f314fcfe4c P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:27.998533 18988 tablet_bootstrap.cc:492] T 7bde1d6934784c9aa5911181e8df9eeb P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:27.998713 18988 ts_tablet_manager.cc:1367] T 7bde1d6934784c9aa5911181e8df9eeb P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.024s user 0.001s sys 0.005s
I20240430 20:11:27.998800 18981 raft_consensus.cc:2258] T bbe9a2429a434f3f95d6e7f314fcfe4c P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:27.999168 18985 consensus_queue.cc:260] T 4e97f7315f8b4ac29db16e00fc35bf99 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:27.999740 18985 ts_tablet_manager.cc:1398] T 4e97f7315f8b4ac29db16e00fc35bf99 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240430 20:11:27.999802 18988 raft_consensus.cc:348] T 7bde1d6934784c9aa5911181e8df9eeb P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:28.000169 18988 raft_consensus.cc:374] T 7bde1d6934784c9aa5911181e8df9eeb P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:28.000206 18985 tablet_bootstrap.cc:492] T 0fe1247349344863a6273a769ce8ddf6 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:28.000355 18988 raft_consensus.cc:729] T 7bde1d6934784c9aa5911181e8df9eeb P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:28.000676 18988 consensus_queue.cc:260] T 7bde1d6934784c9aa5911181e8df9eeb P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:28.001421 18988 ts_tablet_manager.cc:1398] T 7bde1d6934784c9aa5911181e8df9eeb P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240430 20:11:28.001858 18988 tablet_bootstrap.cc:492] T 372b135dd01f44f98cd713ca8857ba61 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:28.002334 18987 ts_tablet_manager.cc:1888] T ffe059eeac91476aaeef8c56d2f49ee8 P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:28.002583 18983 tablet_bootstrap.cc:654] T f5236574672b477a9026e91adcaf70ec P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:28.003218 18985 tablet_bootstrap.cc:654] T 0fe1247349344863a6273a769ce8ddf6 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:28.003558 19319 tablet.cc:1722] T 81787eccca6c42c1b421377a3bf9ebe6 P 5e1c6397f09747f091534faf93542710: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240430 20:11:28.009975 18983 tablet_bootstrap.cc:492] T f5236574672b477a9026e91adcaf70ec P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:28.010262 18983 ts_tablet_manager.cc:1367] T f5236574672b477a9026e91adcaf70ec P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.024s user 0.005s sys 0.000s
I20240430 20:11:28.011222 18981 ts_tablet_manager.cc:1875] T bbe9a2429a434f3f95d6e7f314fcfe4c P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:28.011181 18983 raft_consensus.cc:348] T f5236574672b477a9026e91adcaf70ec P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:28.011518 18983 raft_consensus.cc:374] T f5236574672b477a9026e91adcaf70ec P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:28.011656 18983 raft_consensus.cc:729] T f5236574672b477a9026e91adcaf70ec P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:28.012030 18983 consensus_queue.cc:260] T f5236574672b477a9026e91adcaf70ec P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:28.012881 18983 ts_tablet_manager.cc:1398] T f5236574672b477a9026e91adcaf70ec P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240430 20:11:28.013229 18983 tablet_bootstrap.cc:492] T 13c1107ea4d04eb78a74d3f658c72fb3 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:28.015020 18983 tablet_bootstrap.cc:654] T 13c1107ea4d04eb78a74d3f658c72fb3 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:28.020102 18983 tablet_bootstrap.cc:492] T 13c1107ea4d04eb78a74d3f658c72fb3 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:28.020457 18983 ts_tablet_manager.cc:1367] T 13c1107ea4d04eb78a74d3f658c72fb3 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20240430 20:11:28.021350 18983 raft_consensus.cc:348] T 13c1107ea4d04eb78a74d3f658c72fb3 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:28.021734 18983 raft_consensus.cc:374] T 13c1107ea4d04eb78a74d3f658c72fb3 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:28.021854 18983 raft_consensus.cc:729] T 13c1107ea4d04eb78a74d3f658c72fb3 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:28.026964 18981 ts_tablet_manager.cc:1888] T bbe9a2429a434f3f95d6e7f314fcfe4c P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:28.027242 18981 log.cc:1196] T bbe9a2429a434f3f95d6e7f314fcfe4c P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/bbe9a2429a434f3f95d6e7f314fcfe4c
I20240430 20:11:28.027740 18981 ts_tablet_manager.cc:1909] T bbe9a2429a434f3f95d6e7f314fcfe4c P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:28.028587 19186 consensus_queue.cc:1035] T 3f49eda7aeca42308d01b83a19d7d037 P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:28.031482 18987 log.cc:1196] T ffe059eeac91476aaeef8c56d2f49ee8 P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/ffe059eeac91476aaeef8c56d2f49ee8
W20240430 20:11:28.032099 18415 consensus_peers.cc:485] T 3f49eda7aeca42308d01b83a19d7d037 P 2efb542aa55e439b99ccb98d1c5c884a -> Peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Couldn't send request to peer 5e1c6397f09747f091534faf93542710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3f49eda7aeca42308d01b83a19d7d037. This is attempt 1: this message will repeat every 5th retry.
I20240430 20:11:28.033262 18981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41678 (request call id 145) took 1508 ms. Trace:
I20240430 20:11:28.033484 18981 rpcz_store.cc:276] 0430 20:11:26.525023 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:26.525083 (+ 60us) service_pool.cc:225] Handling call
0430 20:11:27.998072 (+1472989us) ts_tablet_manager.cc:1120] Deleting tablet bbe9a2429a434f3f95d6e7f314fcfe4c
0430 20:11:27.998084 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:28.015107 (+ 17023us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:28.025650 (+ 10543us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:28.033183 (+ 7533us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1472719}
I20240430 20:11:28.034113 18981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41678 (request call id 146) took 1500 ms. Trace:
I20240430 20:11:28.034261 18981 rpcz_store.cc:276] 0430 20:11:26.533749 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:26.533804 (+ 55us) service_pool.cc:225] Handling call
0430 20:11:28.033943 (+1500139us) ts_tablet_manager.cc:1120] Deleting tablet cace1398492f4d968dcb6899385d527c
0430 20:11:28.033954 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:28.034054 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1499901}
I20240430 20:11:28.034658 18981 tablet_replica.cc:330] T 27f94f2d8f50495dababe417a9f82982 P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:28.034969 18981 raft_consensus.cc:2229] T 27f94f2d8f50495dababe417a9f82982 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Raft consensus shutting down.
I20240430 20:11:28.035341 18981 pending_rounds.cc:62] T 27f94f2d8f50495dababe417a9f82982 P 2efb542aa55e439b99ccb98d1c5c884a: Trying to abort 1 pending ops.
I20240430 20:11:28.038713 18981 pending_rounds.cc:68] T 27f94f2d8f50495dababe417a9f82982 P 2efb542aa55e439b99ccb98d1c5c884a: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7022624305756319744 op_type: NO_OP noop_request { }
I20240430 20:11:28.039045 18981 raft_consensus.cc:2866] T 27f94f2d8f50495dababe417a9f82982 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240430 20:11:28.039259 18981 raft_consensus.cc:2258] T 27f94f2d8f50495dababe417a9f82982 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:28.041288 18981 ts_tablet_manager.cc:1875] T 27f94f2d8f50495dababe417a9f82982 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:28.047319 18987 ts_tablet_manager.cc:1909] T ffe059eeac91476aaeef8c56d2f49ee8 P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:28.048336 18987 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59818 (request call id 150) took 1533 ms. Trace:
I20240430 20:11:28.048554 18987 rpcz_store.cc:276] 0430 20:11:26.514705 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:26.514774 (+ 69us) service_pool.cc:225] Handling call
0430 20:11:27.967402 (+1452628us) ts_tablet_manager.cc:1120] Deleting tablet ffe059eeac91476aaeef8c56d2f49ee8
0430 20:11:27.967415 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:27.973456 (+ 6041us) spinlock_profiling.cc:243] Waited 2.07 ms on lock 0x6160000f2e54. stack: 00007f30e7d63756 00007f30e7d63510 00007f30e6d9648d 0000000000955804 00000000009557cd 000000000095519a 00007f30f35d1a00 00007f30f48ab853 00007f30f733e4bc 00007f30f7370814 00007f30f7351b00 00007f30f735188d 0000000000a60bcc 00007f30e7da9b74 00007f30e7dad19b 00007f30e7dad018
0430 20:11:27.999063 (+ 25607us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:28.002197 (+ 3134us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:28.048297 (+ 46100us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4756864,"tablet-delete.queue_time_us":1452359}
I20240430 20:11:28.049311 18987 tablet_replica.cc:330] T 187a27a6449342f99de3c5a2f93d49fc P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:28.049597 18987 raft_consensus.cc:2229] T 187a27a6449342f99de3c5a2f93d49fc P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Raft consensus shutting down.
I20240430 20:11:28.049896 18987 pending_rounds.cc:62] T 187a27a6449342f99de3c5a2f93d49fc P 3c51c33d5616482cafb68e26688820f4: Trying to abort 1 pending ops.
I20240430 20:11:28.050215 18987 pending_rounds.cc:68] T 187a27a6449342f99de3c5a2f93d49fc P 3c51c33d5616482cafb68e26688820f4: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7022624306341146624 op_type: NO_OP noop_request { }
I20240430 20:11:28.050397 18987 raft_consensus.cc:2866] T 187a27a6449342f99de3c5a2f93d49fc P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240430 20:11:28.050563 18987 raft_consensus.cc:2258] T 187a27a6449342f99de3c5a2f93d49fc P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:28.051340 18985 tablet_bootstrap.cc:492] T 0fe1247349344863a6273a769ce8ddf6 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:28.051512 18985 ts_tablet_manager.cc:1367] T 0fe1247349344863a6273a769ce8ddf6 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.051s user 0.013s sys 0.033s
I20240430 20:11:28.051524 18987 ts_tablet_manager.cc:1875] T 187a27a6449342f99de3c5a2f93d49fc P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:28.052071 18988 tablet_bootstrap.cc:654] T 372b135dd01f44f98cd713ca8857ba61 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:28.052287 18985 raft_consensus.cc:348] T 0fe1247349344863a6273a769ce8ddf6 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:28.052503 18985 raft_consensus.cc:374] T 0fe1247349344863a6273a769ce8ddf6 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:28.052595 18985 raft_consensus.cc:729] T 0fe1247349344863a6273a769ce8ddf6 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:28.052850 18985 consensus_queue.cc:260] T 0fe1247349344863a6273a769ce8ddf6 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:28.055123 18985 ts_tablet_manager.cc:1398] T 0fe1247349344863a6273a769ce8ddf6 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240430 20:11:28.055408 18985 tablet_bootstrap.cc:492] T 7bde1d6934784c9aa5911181e8df9eeb P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:28.056851 18985 tablet_bootstrap.cc:654] T 7bde1d6934784c9aa5911181e8df9eeb P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:28.057219 18983 consensus_queue.cc:260] T 13c1107ea4d04eb78a74d3f658c72fb3 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:28.067986 18983 ts_tablet_manager.cc:1398] T 13c1107ea4d04eb78a74d3f658c72fb3 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.047s user 0.002s sys 0.000s
I20240430 20:11:28.069455 18983 tablet_bootstrap.cc:492] T 93d158de64994f3c9fa45a1250a4f533 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:28.072322 18983 tablet_bootstrap.cc:654] T 93d158de64994f3c9fa45a1250a4f533 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:28.072856 18981 ts_tablet_manager.cc:1888] T 27f94f2d8f50495dababe417a9f82982 P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:28.073186 18981 log.cc:1196] T 27f94f2d8f50495dababe417a9f82982 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/27f94f2d8f50495dababe417a9f82982
I20240430 20:11:28.073726 18981 ts_tablet_manager.cc:1909] T 27f94f2d8f50495dababe417a9f82982 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:28.074725 18981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41678 (request call id 152) took 1411 ms. Trace:
I20240430 20:11:28.074910 18981 rpcz_store.cc:276] 0430 20:11:26.663028 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:26.663107 (+ 79us) service_pool.cc:225] Handling call
0430 20:11:28.034554 (+1371447us) ts_tablet_manager.cc:1120] Deleting tablet 27f94f2d8f50495dababe417a9f82982
0430 20:11:28.034563 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:28.045836 (+ 11273us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:28.072663 (+ 26827us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:28.074643 (+ 1980us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1371099}
I20240430 20:11:28.075565 18981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41678 (request call id 153) took 1391 ms. Trace:
I20240430 20:11:28.075699 18981 rpcz_store.cc:276] 0430 20:11:26.683676 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:26.683738 (+ 62us) service_pool.cc:225] Handling call
0430 20:11:28.075365 (+1391627us) ts_tablet_manager.cc:1120] Deleting tablet a1aeee1b02214975a376cfd36957afe8
0430 20:11:28.075375 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:28.075506 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1391374}
I20240430 20:11:28.080438 18981 tablet_replica.cc:330] T 3f49eda7aeca42308d01b83a19d7d037 P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:28.080754 18981 raft_consensus.cc:2229] T 3f49eda7aeca42308d01b83a19d7d037 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Raft consensus shutting down.
I20240430 20:11:28.081274 18981 pending_rounds.cc:62] T 3f49eda7aeca42308d01b83a19d7d037 P 2efb542aa55e439b99ccb98d1c5c884a: Trying to abort 1 pending ops.
I20240430 20:11:28.081393 18981 pending_rounds.cc:68] T 3f49eda7aeca42308d01b83a19d7d037 P 2efb542aa55e439b99ccb98d1c5c884a: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7022624304610185216 op_type: NO_OP noop_request { }
I20240430 20:11:28.081622 18981 raft_consensus.cc:2866] T 3f49eda7aeca42308d01b83a19d7d037 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240430 20:11:28.081784 18981 raft_consensus.cc:2258] T 3f49eda7aeca42308d01b83a19d7d037 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:28.082525 18981 ts_tablet_manager.cc:1875] T 3f49eda7aeca42308d01b83a19d7d037 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:28.085078 18985 tablet_bootstrap.cc:492] T 7bde1d6934784c9aa5911181e8df9eeb P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:28.085260 18985 ts_tablet_manager.cc:1367] T 7bde1d6934784c9aa5911181e8df9eeb P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.030s user 0.000s sys 0.024s
I20240430 20:11:28.086117 18985 raft_consensus.cc:348] T 7bde1d6934784c9aa5911181e8df9eeb P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:28.086352 18985 raft_consensus.cc:374] T 7bde1d6934784c9aa5911181e8df9eeb P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:28.086405 18988 tablet_bootstrap.cc:492] T 372b135dd01f44f98cd713ca8857ba61 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:28.086426 18985 raft_consensus.cc:729] T 7bde1d6934784c9aa5911181e8df9eeb P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:28.086728 18985 consensus_queue.cc:260] T 7bde1d6934784c9aa5911181e8df9eeb P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:28.086581 18988 ts_tablet_manager.cc:1367] T 372b135dd01f44f98cd713ca8857ba61 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.085s user 0.006s sys 0.002s
I20240430 20:11:28.089198 18988 raft_consensus.cc:348] T 372b135dd01f44f98cd713ca8857ba61 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:28.089464 18988 raft_consensus.cc:374] T 372b135dd01f44f98cd713ca8857ba61 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:28.089551 18988 raft_consensus.cc:729] T 372b135dd01f44f98cd713ca8857ba61 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:28.092268 18985 ts_tablet_manager.cc:1398] T 7bde1d6934784c9aa5911181e8df9eeb P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240430 20:11:28.095410 18987 ts_tablet_manager.cc:1888] T 187a27a6449342f99de3c5a2f93d49fc P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:28.095564 18987 log.cc:1196] T 187a27a6449342f99de3c5a2f93d49fc P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/187a27a6449342f99de3c5a2f93d49fc
I20240430 20:11:28.096006 18987 ts_tablet_manager.cc:1909] T 187a27a6449342f99de3c5a2f93d49fc P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:28.096840 18987 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59818 (request call id 149) took 1582 ms. Trace:
I20240430 20:11:28.097043 18985 tablet_bootstrap.cc:492] T cace1398492f4d968dcb6899385d527c P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:28.098330 18988 consensus_queue.cc:260] T 372b135dd01f44f98cd713ca8857ba61 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:28.098775 18981 ts_tablet_manager.cc:1888] T 3f49eda7aeca42308d01b83a19d7d037 P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:28.099023 18981 log.cc:1196] T 3f49eda7aeca42308d01b83a19d7d037 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/3f49eda7aeca42308d01b83a19d7d037
I20240430 20:11:28.101050 18981 ts_tablet_manager.cc:1909] T 3f49eda7aeca42308d01b83a19d7d037 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:28.099511 18985 tablet_bootstrap.cc:654] T cace1398492f4d968dcb6899385d527c P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:28.104198 18988 ts_tablet_manager.cc:1398] T 372b135dd01f44f98cd713ca8857ba61 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.016s user 0.002s sys 0.000s
I20240430 20:11:28.104466 19230 raft_consensus.cc:482] T 4e97f7315f8b4ac29db16e00fc35bf99 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:28.104691 18981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41678 (request call id 155) took 1395 ms. Trace:
I20240430 20:11:28.104646 19230 raft_consensus.cc:504] T 4e97f7315f8b4ac29db16e00fc35bf99 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:28.104786 18981 rpcz_store.cc:276] 0430 20:11:26.708805 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:26.708871 (+ 66us) service_pool.cc:225] Handling call
0430 20:11:28.080327 (+1371456us) ts_tablet_manager.cc:1120] Deleting tablet 3f49eda7aeca42308d01b83a19d7d037
0430 20:11:28.080340 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:28.089265 (+ 8925us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:28.095433 (+ 6168us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:28.104620 (+ 9187us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1371160}
I20240430 20:11:28.105305 18981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41678 (request call id 154) took 1397 ms. Trace:
I20240430 20:11:28.105405 19230 leader_election.cc:288] T 4e97f7315f8b4ac29db16e00fc35bf99 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:28.105406 18981 rpcz_store.cc:276] 0430 20:11:26.708035 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:26.709751 (+ 1716us) service_pool.cc:225] Handling call
0430 20:11:28.105082 (+1395331us) ts_tablet_manager.cc:1120] Deleting tablet aacabead3c874e77a47d2ecf19dbfa82
0430 20:11:28.105092 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:28.105241 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1395059}
I20240430 20:11:28.106372 18981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41678 (request call id 156) took 1344 ms. Trace:
I20240430 20:11:28.106474 18981 rpcz_store.cc:276] 0430 20:11:26.762276 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:26.762364 (+ 88us) service_pool.cc:225] Handling call
0430 20:11:28.105626 (+1343262us) ts_tablet_manager.cc:1120] Deleting tablet 68ca48351cd4412381f5195d81109016
0430 20:11:28.105636 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:28.106301 (+ 665us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1342914}
I20240430 20:11:28.106743 18482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e97f7315f8b4ac29db16e00fc35bf99" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:28.106873 18981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41678 (request call id 157) took 1327 ms. Trace:
I20240430 20:11:28.106979 18482 raft_consensus.cc:2454] T 4e97f7315f8b4ac29db16e00fc35bf99 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 0.
I20240430 20:11:28.107685 19186 raft_consensus.cc:482] T 13c1107ea4d04eb78a74d3f658c72fb3 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:28.107662 18558 leader_election.cc:302] T 4e97f7315f8b4ac29db16e00fc35bf99 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:28.107852 19186 raft_consensus.cc:504] T 13c1107ea4d04eb78a74d3f658c72fb3 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:28.108122 19230 raft_consensus.cc:2781] T 4e97f7315f8b4ac29db16e00fc35bf99 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:28.108220 19230 raft_consensus.cc:482] T 4e97f7315f8b4ac29db16e00fc35bf99 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:28.108315 19230 raft_consensus.cc:3037] T 4e97f7315f8b4ac29db16e00fc35bf99 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:28.108458 19186 leader_election.cc:288] T 13c1107ea4d04eb78a74d3f658c72fb3 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:28.109035 18620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13c1107ea4d04eb78a74d3f658c72fb3" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:28.109254 18620 raft_consensus.cc:2454] T 13c1107ea4d04eb78a74d3f658c72fb3 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 0.
I20240430 20:11:28.109683 18415 leader_election.cc:302] T 13c1107ea4d04eb78a74d3f658c72fb3 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:28.110320 18760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13c1107ea4d04eb78a74d3f658c72fb3" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
W20240430 20:11:28.110908 18417 leader_election.cc:341] T 13c1107ea4d04eb78a74d3f658c72fb3 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:28.111299 19186 raft_consensus.cc:2781] T 13c1107ea4d04eb78a74d3f658c72fb3 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:28.111385 19186 raft_consensus.cc:482] T 13c1107ea4d04eb78a74d3f658c72fb3 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:28.111491 19186 raft_consensus.cc:3037] T 13c1107ea4d04eb78a74d3f658c72fb3 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:28.111431 19230 raft_consensus.cc:504] T 4e97f7315f8b4ac29db16e00fc35bf99 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:28.112460 18482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e97f7315f8b4ac29db16e00fc35bf99" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:28.112668 18482 raft_consensus.cc:3037] T 4e97f7315f8b4ac29db16e00fc35bf99 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:28.114445 19186 raft_consensus.cc:504] T 13c1107ea4d04eb78a74d3f658c72fb3 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:28.115334 18760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e97f7315f8b4ac29db16e00fc35bf99" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
W20240430 20:11:28.116096 18557 leader_election.cc:341] T 4e97f7315f8b4ac29db16e00fc35bf99 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:28.116056 18620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13c1107ea4d04eb78a74d3f658c72fb3" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:28.116250 19186 leader_election.cc:288] T 13c1107ea4d04eb78a74d3f658c72fb3 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:28.116374 18620 raft_consensus.cc:3037] T 13c1107ea4d04eb78a74d3f658c72fb3 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:28.106976 18981 rpcz_store.cc:276] 0430 20:11:26.779448 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:26.779505 (+ 57us) service_pool.cc:225] Handling call
0430 20:11:28.106722 (+1327217us) ts_tablet_manager.cc:1120] Deleting tablet f5236574672b477a9026e91adcaf70ec
0430 20:11:28.106734 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:28.106822 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1326959}
I20240430 20:11:28.118860 18981 tablet_replica.cc:330] T 4e97f7315f8b4ac29db16e00fc35bf99 P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:28.119560 18760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13c1107ea4d04eb78a74d3f658c72fb3" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
W20240430 20:11:28.120182 18417 leader_election.cc:341] T 13c1107ea4d04eb78a74d3f658c72fb3 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:28.120308 18985 tablet_bootstrap.cc:492] T cace1398492f4d968dcb6899385d527c P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:28.120466 18985 ts_tablet_manager.cc:1367] T cace1398492f4d968dcb6899385d527c P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.024s user 0.001s sys 0.010s
I20240430 20:11:28.120450 19230 leader_election.cc:288] T 4e97f7315f8b4ac29db16e00fc35bf99 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Requested vote from peers 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:28.121233 18985 raft_consensus.cc:348] T cace1398492f4d968dcb6899385d527c P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:28.121495 18985 raft_consensus.cc:374] T cace1398492f4d968dcb6899385d527c P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:28.121578 18985 raft_consensus.cc:729] T cace1398492f4d968dcb6899385d527c P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:28.121681 18482 raft_consensus.cc:2454] T 4e97f7315f8b4ac29db16e00fc35bf99 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
I20240430 20:11:28.121824 18981 raft_consensus.cc:2229] T 4e97f7315f8b4ac29db16e00fc35bf99 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:28.121855 18985 consensus_queue.cc:260] T cace1398492f4d968dcb6899385d527c P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:28.122205 18981 raft_consensus.cc:2258] T 4e97f7315f8b4ac29db16e00fc35bf99 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:28.122232 18558 leader_election.cc:302] T 4e97f7315f8b4ac29db16e00fc35bf99 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters: 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:28.123085 18981 ts_tablet_manager.cc:1875] T 4e97f7315f8b4ac29db16e00fc35bf99 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:28.123343 18620 raft_consensus.cc:2454] T 13c1107ea4d04eb78a74d3f658c72fb3 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 1.
I20240430 20:11:28.123507 18763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e97f7315f8b4ac29db16e00fc35bf99" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:28.123353 19230 raft_consensus.cc:2781] T 4e97f7315f8b4ac29db16e00fc35bf99 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:28.123889 18985 ts_tablet_manager.cc:1398] T cace1398492f4d968dcb6899385d527c P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240430 20:11:28.123940 18415 leader_election.cc:302] T 13c1107ea4d04eb78a74d3f658c72fb3 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters: 3c51c33d5616482cafb68e26688820f4
W20240430 20:11:28.124075 18557 leader_election.cc:341] T 4e97f7315f8b4ac29db16e00fc35bf99 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:28.124197 18985 tablet_bootstrap.cc:492] T f5236574672b477a9026e91adcaf70ec P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:28.124329 19186 raft_consensus.cc:2781] T 13c1107ea4d04eb78a74d3f658c72fb3 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:28.126753 19186 raft_consensus.cc:686] T 13c1107ea4d04eb78a74d3f658c72fb3 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Becoming Leader. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Running, Role: LEADER
I20240430 20:11:28.127068 19186 consensus_queue.cc:237] T 13c1107ea4d04eb78a74d3f658c72fb3 P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:28.127796 18988 tablet_bootstrap.cc:492] T cace1398492f4d968dcb6899385d527c P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:28.127928 19230 raft_consensus.cc:686] T 4e97f7315f8b4ac29db16e00fc35bf99 P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Becoming Leader. State: Replica: 5e1c6397f09747f091534faf93542710, State: Running, Role: LEADER
I20240430 20:11:28.128250 19230 consensus_queue.cc:237] T 4e97f7315f8b4ac29db16e00fc35bf99 P 5e1c6397f09747f091534faf93542710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:28.096975 18987 rpcz_store.cc:276] 0430 20:11:26.514538 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:26.514759 (+ 221us) service_pool.cc:225] Handling call
0430 20:11:28.049245 (+1534486us) ts_tablet_manager.cc:1120] Deleting tablet 187a27a6449342f99de3c5a2f93d49fc
0430 20:11:28.049255 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:28.073893 (+ 24638us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:28.095234 (+ 21341us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:28.096804 (+ 1570us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":15,"tablet-delete.queue_time_us":1534197}
I20240430 20:11:28.130349 18987 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59818 (request call id 151) took 1613 ms. Trace:
I20240430 20:11:28.130906 18987 rpcz_store.cc:276] 0430 20:11:26.516958 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:26.517007 (+ 49us) service_pool.cc:225] Handling call
0430 20:11:28.130174 (+1613167us) ts_tablet_manager.cc:1120] Deleting tablet bbe9a2429a434f3f95d6e7f314fcfe4c
0430 20:11:28.130186 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:28.130307 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1612940}
I20240430 20:11:28.131536 18987 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59818 (request call id 152) took 1600 ms. Trace:
I20240430 20:11:28.131696 18987 rpcz_store.cc:276] 0430 20:11:26.531573 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:26.531625 (+ 52us) service_pool.cc:225] Handling call
0430 20:11:28.131400 (+1599775us) ts_tablet_manager.cc:1120] Deleting tablet cace1398492f4d968dcb6899385d527c
0430 20:11:28.131411 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:28.131502 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1599485}
I20240430 20:11:28.132058 18983 tablet_bootstrap.cc:492] T 93d158de64994f3c9fa45a1250a4f533 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:28.132220 18983 ts_tablet_manager.cc:1367] T 93d158de64994f3c9fa45a1250a4f533 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.063s user 0.003s sys 0.000s
I20240430 20:11:28.132405 18987 tablet_replica.cc:330] T b4bc111e183f40a0acc9dd377195f6d9 P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:28.132826 18987 raft_consensus.cc:2229] T b4bc111e183f40a0acc9dd377195f6d9 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:28.133121 18983 raft_consensus.cc:348] T 93d158de64994f3c9fa45a1250a4f533 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:28.133376 18983 raft_consensus.cc:374] T 93d158de64994f3c9fa45a1250a4f533 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:28.132623 18988 tablet_bootstrap.cc:654] T cace1398492f4d968dcb6899385d527c P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:28.133483 18983 raft_consensus.cc:729] T 93d158de64994f3c9fa45a1250a4f533 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:28.133723 18983 consensus_queue.cc:260] T 93d158de64994f3c9fa45a1250a4f533 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:28.134397 18983 ts_tablet_manager.cc:1398] T 93d158de64994f3c9fa45a1250a4f533 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240430 20:11:28.134539 18987 raft_consensus.cc:2258] T b4bc111e183f40a0acc9dd377195f6d9 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:28.134749 18983 tablet_bootstrap.cc:492] T a745334c12c545c1ba825c7552c60359 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:28.139070 18987 ts_tablet_manager.cc:1875] T b4bc111e183f40a0acc9dd377195f6d9 P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:28.141817 18983 tablet_bootstrap.cc:654] T a745334c12c545c1ba825c7552c60359 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:28.144191 19180 raft_consensus.cc:482] T bbe9a2429a434f3f95d6e7f314fcfe4c P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:28.144393 19180 raft_consensus.cc:504] T bbe9a2429a434f3f95d6e7f314fcfe4c P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:28.145114 19180 leader_election.cc:288] T bbe9a2429a434f3f95d6e7f314fcfe4c P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:28.146179 18763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbe9a2429a434f3f95d6e7f314fcfe4c" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
W20240430 20:11:28.147168 18557 leader_election.cc:341] T bbe9a2429a434f3f95d6e7f314fcfe4c P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:28.147715 18985 tablet_bootstrap.cc:654] T f5236574672b477a9026e91adcaf70ec P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
W20240430 20:11:28.148517 18558 leader_election.cc:341] T bbe9a2429a434f3f95d6e7f314fcfe4c P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Not found: Tablet not found: bbe9a2429a434f3f95d6e7f314fcfe4c
I20240430 20:11:28.148666 18558 leader_election.cc:302] T bbe9a2429a434f3f95d6e7f314fcfe4c P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:28.149016 19180 raft_consensus.cc:2726] T bbe9a2429a434f3f95d6e7f314fcfe4c P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:28.158097 18987 ts_tablet_manager.cc:1888] T b4bc111e183f40a0acc9dd377195f6d9 P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:28.165478 18981 ts_tablet_manager.cc:1888] T 4e97f7315f8b4ac29db16e00fc35bf99 P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:28.165630 18981 log.cc:1196] T 4e97f7315f8b4ac29db16e00fc35bf99 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/4e97f7315f8b4ac29db16e00fc35bf99
I20240430 20:11:28.166011 18981 ts_tablet_manager.cc:1909] T 4e97f7315f8b4ac29db16e00fc35bf99 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:28.166864 18981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41678 (request call id 163) took 1254 ms. Trace:
I20240430 20:11:28.166996 18981 rpcz_store.cc:276] 0430 20:11:26.912272 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:26.912500 (+ 228us) service_pool.cc:225] Handling call
0430 20:11:28.118754 (+1206254us) ts_tablet_manager.cc:1120] Deleting tablet 4e97f7315f8b4ac29db16e00fc35bf99
0430 20:11:28.118768 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:28.122032 (+ 3264us) spinlock_profiling.cc:243] Waited 2.71 ms on lock 0x616000160354. stack: 00007f9c64594756 00007f9c64594510 00007f9c635c748d 0000000000955804 00000000009557cd 000000000095519a 00007f9c6fe02a00 00007f9c710dc853 00007f9c73b6f4bc 00007f9c73ba1814 00007f9c73b82b00 00007f9c73b8288d 0000000000a60bcc 00007f9c645dab74 00007f9c645de19b 00007f9c645de018
0430 20:11:28.138931 (+ 16899us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:28.165319 (+ 26388us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:28.166826 (+ 1507us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":29,"spinlock_wait_cycles":6225792,"tablet-delete.queue_time_us":1206017}
I20240430 20:11:28.158983 18988 tablet_bootstrap.cc:492] T cace1398492f4d968dcb6899385d527c P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:28.167520 18981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41678 (request call id 167) took 1235 ms. Trace:
I20240430 20:11:28.167552 18988 ts_tablet_manager.cc:1367] T cace1398492f4d968dcb6899385d527c P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.040s user 0.004s sys 0.000s
I20240430 20:11:28.167629 18981 rpcz_store.cc:276] 0430 20:11:26.932389 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:26.932443 (+ 54us) service_pool.cc:225] Handling call
0430 20:11:28.167359 (+1234916us) ts_tablet_manager.cc:1120] Deleting tablet c9ff972ade694938888514979e1638aa
0430 20:11:28.167368 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:28.167487 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1230863}
I20240430 20:11:28.167984 18981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41678 (request call id 175) took 1113 ms. Trace:
I20240430 20:11:28.168114 18981 rpcz_store.cc:276] 0430 20:11:27.054270 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:27.054341 (+ 71us) service_pool.cc:225] Handling call
0430 20:11:28.167870 (+1113529us) ts_tablet_manager.cc:1120] Deleting tablet eed6338ad3d64859af087ee15e1ec798
0430 20:11:28.167878 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:28.167954 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1075207}
I20240430 20:11:28.168397 18981 tablet_replica.cc:330] T 13c1107ea4d04eb78a74d3f658c72fb3 P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:28.168390 18988 raft_consensus.cc:348] T cace1398492f4d968dcb6899385d527c P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:28.168608 18988 raft_consensus.cc:374] T cace1398492f4d968dcb6899385d527c P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:28.168601 18981 raft_consensus.cc:2229] T 13c1107ea4d04eb78a74d3f658c72fb3 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Raft consensus shutting down.
I20240430 20:11:28.168689 18988 raft_consensus.cc:729] T cace1398492f4d968dcb6899385d527c P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:28.168946 18988 consensus_queue.cc:260] T cace1398492f4d968dcb6899385d527c P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:28.171813 18985 tablet_bootstrap.cc:492] T f5236574672b477a9026e91adcaf70ec P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:28.172158 18985 ts_tablet_manager.cc:1367] T f5236574672b477a9026e91adcaf70ec P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.048s user 0.002s sys 0.002s
I20240430 20:11:28.172688 18988 ts_tablet_manager.cc:1398] T cace1398492f4d968dcb6899385d527c P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240430 20:11:28.173362 18988 tablet_bootstrap.cc:492] T 4e97f7315f8b4ac29db16e00fc35bf99 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:28.173434 18985 raft_consensus.cc:348] T f5236574672b477a9026e91adcaf70ec P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:28.173681 18985 raft_consensus.cc:374] T f5236574672b477a9026e91adcaf70ec P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:28.173763 18985 raft_consensus.cc:729] T f5236574672b477a9026e91adcaf70ec P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:28.174048 18985 consensus_queue.cc:260] T f5236574672b477a9026e91adcaf70ec P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:28.176728 18988 tablet_bootstrap.cc:654] T 4e97f7315f8b4ac29db16e00fc35bf99 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:28.178278 18987 log.cc:1196] T b4bc111e183f40a0acc9dd377195f6d9 P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/b4bc111e183f40a0acc9dd377195f6d9
I20240430 20:11:28.178939 18987 ts_tablet_manager.cc:1909] T b4bc111e183f40a0acc9dd377195f6d9 P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:28.174576 18985 ts_tablet_manager.cc:1398] T f5236574672b477a9026e91adcaf70ec P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240430 20:11:28.178175 18981 pending_rounds.cc:62] T 13c1107ea4d04eb78a74d3f658c72fb3 P 2efb542aa55e439b99ccb98d1c5c884a: Trying to abort 1 pending ops.
I20240430 20:11:28.180781 18981 pending_rounds.cc:68] T 13c1107ea4d04eb78a74d3f658c72fb3 P 2efb542aa55e439b99ccb98d1c5c884a: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7022624309778882560 op_type: NO_OP noop_request { }
I20240430 20:11:28.180931 18981 raft_consensus.cc:2866] T 13c1107ea4d04eb78a74d3f658c72fb3 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240430 20:11:28.181077 18981 raft_consensus.cc:2258] T 13c1107ea4d04eb78a74d3f658c72fb3 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:28.181849 18987 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59818 (request call id 158) took 1546 ms. Trace:
I20240430 20:11:28.182137 18981 ts_tablet_manager.cc:1875] T 13c1107ea4d04eb78a74d3f658c72fb3 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:28.182732 18987 rpcz_store.cc:276] 0430 20:11:26.635526 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:26.635609 (+ 83us) service_pool.cc:225] Handling call
0430 20:11:28.132343 (+1496734us) ts_tablet_manager.cc:1120] Deleting tablet b4bc111e183f40a0acc9dd377195f6d9
0430 20:11:28.132353 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:28.146329 (+ 13976us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:28.157945 (+ 11616us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:28.181793 (+ 23848us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1496224}
I20240430 20:11:28.183364 18987 tablet_replica.cc:330] T 609a1a7d22e84b69bd0a53577b62b69b P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:28.183365 18983 tablet_bootstrap.cc:492] T a745334c12c545c1ba825c7552c60359 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:28.183547 18983 ts_tablet_manager.cc:1367] T a745334c12c545c1ba825c7552c60359 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.049s user 0.011s sys 0.002s
I20240430 20:11:28.184346 18987 raft_consensus.cc:2229] T 609a1a7d22e84b69bd0a53577b62b69b P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:28.184346 18983 raft_consensus.cc:348] T a745334c12c545c1ba825c7552c60359 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:28.184590 18983 raft_consensus.cc:374] T a745334c12c545c1ba825c7552c60359 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:28.184670 18983 raft_consensus.cc:729] T a745334c12c545c1ba825c7552c60359 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:28.184827 18987 raft_consensus.cc:2258] T 609a1a7d22e84b69bd0a53577b62b69b P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:28.185020 18983 consensus_queue.cc:260] T a745334c12c545c1ba825c7552c60359 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:28.185624 18983 ts_tablet_manager.cc:1398] T a745334c12c545c1ba825c7552c60359 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240430 20:11:28.185959 18983 tablet_bootstrap.cc:492] T c5fd0730133c472299c93f4b0830e527 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:28.187232 18987 ts_tablet_manager.cc:1875] T 609a1a7d22e84b69bd0a53577b62b69b P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:28.187247 18985 tablet_bootstrap.cc:492] T 15674aeb56b74210a4a2809608fd4457 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:28.188213 18983 tablet_bootstrap.cc:654] T c5fd0730133c472299c93f4b0830e527 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:28.188949 18985 tablet_bootstrap.cc:654] T 15674aeb56b74210a4a2809608fd4457 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:28.190171 18988 tablet_bootstrap.cc:492] T 4e97f7315f8b4ac29db16e00fc35bf99 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:28.190444 18988 ts_tablet_manager.cc:1367] T 4e97f7315f8b4ac29db16e00fc35bf99 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.001s
I20240430 20:11:28.191533 18988 raft_consensus.cc:348] T 4e97f7315f8b4ac29db16e00fc35bf99 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:28.191931 18988 raft_consensus.cc:374] T 4e97f7315f8b4ac29db16e00fc35bf99 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:28.192087 18988 raft_consensus.cc:729] T 4e97f7315f8b4ac29db16e00fc35bf99 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:28.192494 18988 consensus_queue.cc:260] T 4e97f7315f8b4ac29db16e00fc35bf99 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:28.195494 18985 tablet_bootstrap.cc:492] T 15674aeb56b74210a4a2809608fd4457 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:28.195663 18985 ts_tablet_manager.cc:1367] T 15674aeb56b74210a4a2809608fd4457 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.004s
I20240430 20:11:28.196532 18985 raft_consensus.cc:348] T 15674aeb56b74210a4a2809608fd4457 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:28.196769 18985 raft_consensus.cc:374] T 15674aeb56b74210a4a2809608fd4457 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:28.196851 18985 raft_consensus.cc:729] T 15674aeb56b74210a4a2809608fd4457 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:28.197077 18985 consensus_queue.cc:260] T 15674aeb56b74210a4a2809608fd4457 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:28.197458 19180 raft_consensus.cc:482] T 0fe1247349344863a6273a769ce8ddf6 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:28.197646 19180 raft_consensus.cc:504] T 0fe1247349344863a6273a769ce8ddf6 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:28.198895 18988 ts_tablet_manager.cc:1398] T 4e97f7315f8b4ac29db16e00fc35bf99 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240430 20:11:28.199016 19180 leader_election.cc:288] T 0fe1247349344863a6273a769ce8ddf6 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:28.199518 18482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fe1247349344863a6273a769ce8ddf6" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:28.199723 18482 raft_consensus.cc:2454] T 0fe1247349344863a6273a769ce8ddf6 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 0.
I20240430 20:11:28.200244 18558 leader_election.cc:302] T 0fe1247349344863a6273a769ce8ddf6 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:28.200222 18985 ts_tablet_manager.cc:1398] T 15674aeb56b74210a4a2809608fd4457 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240430 20:11:28.200644 19180 raft_consensus.cc:2781] T 0fe1247349344863a6273a769ce8ddf6 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:28.200668 18985 tablet_bootstrap.cc:492] T c9ff972ade694938888514979e1638aa P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:28.200728 19180 raft_consensus.cc:482] T 0fe1247349344863a6273a769ce8ddf6 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:28.200810 19180 raft_consensus.cc:3037] T 0fe1247349344863a6273a769ce8ddf6 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:28.202514 18985 tablet_bootstrap.cc:654] T c9ff972ade694938888514979e1638aa P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:28.203595 19180 raft_consensus.cc:504] T 0fe1247349344863a6273a769ce8ddf6 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:28.204885 18482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fe1247349344863a6273a769ce8ddf6" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:28.205085 18482 raft_consensus.cc:3037] T 0fe1247349344863a6273a769ce8ddf6 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:28.199693 18763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fe1247349344863a6273a769ce8ddf6" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:28.206673 18981 ts_tablet_manager.cc:1888] T 13c1107ea4d04eb78a74d3f658c72fb3 P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:28.206804 18981 log.cc:1196] T 13c1107ea4d04eb78a74d3f658c72fb3 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/13c1107ea4d04eb78a74d3f658c72fb3
I20240430 20:11:28.206771 18763 raft_consensus.cc:2454] T 0fe1247349344863a6273a769ce8ddf6 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 0.
I20240430 20:11:28.207320 18981 ts_tablet_manager.cc:1909] T 13c1107ea4d04eb78a74d3f658c72fb3 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:28.207978 18482 raft_consensus.cc:2454] T 0fe1247349344863a6273a769ce8ddf6 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
I20240430 20:11:28.208573 18558 leader_election.cc:302] T 0fe1247349344863a6273a769ce8ddf6 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:28.210784 19230 raft_consensus.cc:2781] T 0fe1247349344863a6273a769ce8ddf6 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:28.210940 19230 raft_consensus.cc:686] T 0fe1247349344863a6273a769ce8ddf6 P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Becoming Leader. State: Replica: 5e1c6397f09747f091534faf93542710, State: Running, Role: LEADER
I20240430 20:11:28.211272 19230 consensus_queue.cc:237] T 0fe1247349344863a6273a769ce8ddf6 P 5e1c6397f09747f091534faf93542710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:28.212347 19314 raft_consensus.cc:482] T 4e97f7315f8b4ac29db16e00fc35bf99 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:28.212517 19314 raft_consensus.cc:504] T 4e97f7315f8b4ac29db16e00fc35bf99 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:28.213326 19314 leader_election.cc:288] T 4e97f7315f8b4ac29db16e00fc35bf99 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
W20240430 20:11:28.214953 18699 leader_election.cc:341] T 4e97f7315f8b4ac29db16e00fc35bf99 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Not found: Tablet not found: 4e97f7315f8b4ac29db16e00fc35bf99
I20240430 20:11:28.216388 18763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fe1247349344863a6273a769ce8ddf6" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:28.216543 18988 tablet_bootstrap.cc:492] T fae9a35a2ac844b4bc2ebdf214aa7171 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:28.216625 18763 raft_consensus.cc:3037] T 0fe1247349344863a6273a769ce8ddf6 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:28.218309 18988 tablet_bootstrap.cc:654] T fae9a35a2ac844b4bc2ebdf214aa7171 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:28.218662 18981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41678 (request call id 185) took 1069 ms. Trace:
I20240430 20:11:28.219573 18981 rpcz_store.cc:276] 0430 20:11:27.149086 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:27.149133 (+ 47us) service_pool.cc:225] Handling call
0430 20:11:28.168327 (+1019194us) ts_tablet_manager.cc:1120] Deleting tablet 13c1107ea4d04eb78a74d3f658c72fb3
0430 20:11:28.168336 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:28.188581 (+ 20245us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:28.206488 (+ 17907us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:28.218618 (+ 12130us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1018955}
I20240430 20:11:28.220122 18763 raft_consensus.cc:2454] T 0fe1247349344863a6273a769ce8ddf6 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
I20240430 20:11:28.220875 19180 leader_election.cc:288] T 0fe1247349344863a6273a769ce8ddf6 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Requested vote from peers 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:28.220929 18981 tablet_replica.cc:330] T 0879b02a0bcc4f619fb34cff7b9d7c4c P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:28.221154 18981 raft_consensus.cc:2229] T 0879b02a0bcc4f619fb34cff7b9d7c4c P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:28.221340 18981 raft_consensus.cc:2258] T 0879b02a0bcc4f619fb34cff7b9d7c4c P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:28.221925 18981 ts_tablet_manager.cc:1875] T 0879b02a0bcc4f619fb34cff7b9d7c4c P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:28.229158 18987 ts_tablet_manager.cc:1888] T 609a1a7d22e84b69bd0a53577b62b69b P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:28.232435 19186 raft_consensus.cc:482] T cace1398492f4d968dcb6899385d527c P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:28.232666 19186 raft_consensus.cc:504] T cace1398492f4d968dcb6899385d527c P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:28.232908 18983 tablet_bootstrap.cc:492] T c5fd0730133c472299c93f4b0830e527 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:28.233084 18983 ts_tablet_manager.cc:1367] T c5fd0730133c472299c93f4b0830e527 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.047s user 0.003s sys 0.011s
I20240430 20:11:28.233631 19186 leader_election.cc:288] T cace1398492f4d968dcb6899385d527c P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:28.234004 18983 raft_consensus.cc:348] T c5fd0730133c472299c93f4b0830e527 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:28.231699 18988 tablet_bootstrap.cc:492] T fae9a35a2ac844b4bc2ebdf214aa7171 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:28.234447 18988 ts_tablet_manager.cc:1367] T fae9a35a2ac844b4bc2ebdf214aa7171 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.001s
I20240430 20:11:28.234710 18620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cace1398492f4d968dcb6899385d527c" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:28.234995 18620 raft_consensus.cc:2454] T cace1398492f4d968dcb6899385d527c P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 0.
I20240430 20:11:28.235235 18983 raft_consensus.cc:374] T c5fd0730133c472299c93f4b0830e527 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:28.235409 18988 raft_consensus.cc:348] T fae9a35a2ac844b4bc2ebdf214aa7171 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:28.235553 18623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e97f7315f8b4ac29db16e00fc35bf99" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:28.235637 18988 raft_consensus.cc:374] T fae9a35a2ac844b4bc2ebdf214aa7171 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:28.235728 18988 raft_consensus.cc:729] T fae9a35a2ac844b4bc2ebdf214aa7171 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:28.236069 18696 leader_election.cc:302] T 4e97f7315f8b4ac29db16e00fc35bf99 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710
I20240430 20:11:28.236027 18988 consensus_queue.cc:260] T fae9a35a2ac844b4bc2ebdf214aa7171 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:28.236227 18415 leader_election.cc:302] T cace1398492f4d968dcb6899385d527c P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:28.236320 18987 log.cc:1196] T 609a1a7d22e84b69bd0a53577b62b69b P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/609a1a7d22e84b69bd0a53577b62b69b
I20240430 20:11:28.236554 19314 raft_consensus.cc:2726] T 4e97f7315f8b4ac29db16e00fc35bf99 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:28.236847 18983 raft_consensus.cc:729] T c5fd0730133c472299c93f4b0830e527 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:28.236932 18763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cace1398492f4d968dcb6899385d527c" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:28.237138 18763 raft_consensus.cc:2454] T cace1398492f4d968dcb6899385d527c P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 0.
I20240430 20:11:28.237427 18981 ts_tablet_manager.cc:1888] T 0879b02a0bcc4f619fb34cff7b9d7c4c P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:28.237582 18981 log.cc:1196] T 0879b02a0bcc4f619fb34cff7b9d7c4c P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/0879b02a0bcc4f619fb34cff7b9d7c4c
I20240430 20:11:28.237987 18981 ts_tablet_manager.cc:1909] T 0879b02a0bcc4f619fb34cff7b9d7c4c P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:28.238042 19186 raft_consensus.cc:2781] T cace1398492f4d968dcb6899385d527c P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:28.238162 19186 raft_consensus.cc:482] T cace1398492f4d968dcb6899385d527c P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:28.238052 18983 consensus_queue.cc:260] T c5fd0730133c472299c93f4b0830e527 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:28.238257 19186 raft_consensus.cc:3037] T cace1398492f4d968dcb6899385d527c P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:28.238655 18983 ts_tablet_manager.cc:1398] T c5fd0730133c472299c93f4b0830e527 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.005s user 0.002s sys 0.001s
I20240430 20:11:28.238941 18983 tablet_bootstrap.cc:492] T 6e826459c3f4437da81c9e6e09f0c22f P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:28.238909 18981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41678 (request call id 184) took 1089 ms. Trace:
I20240430 20:11:28.239020 18981 rpcz_store.cc:276] 0430 20:11:27.148953 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:27.150116 (+ 1163us) service_pool.cc:225] Handling call
0430 20:11:28.220850 (+1070734us) ts_tablet_manager.cc:1120] Deleting tablet 0879b02a0bcc4f619fb34cff7b9d7c4c
0430 20:11:28.220862 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:28.224850 (+ 3988us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:28.237289 (+ 12439us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:28.238843 (+ 1554us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1070519}
I20240430 20:11:28.242244 18988 ts_tablet_manager.cc:1398] T fae9a35a2ac844b4bc2ebdf214aa7171 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20240430 20:11:28.243244 18985 tablet_bootstrap.cc:492] T c9ff972ade694938888514979e1638aa P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:28.243407 18985 ts_tablet_manager.cc:1367] T c9ff972ade694938888514979e1638aa P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.043s user 0.003s sys 0.009s
I20240430 20:11:28.244203 18985 raft_consensus.cc:348] T c9ff972ade694938888514979e1638aa P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:28.244434 18985 raft_consensus.cc:374] T c9ff972ade694938888514979e1638aa P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:28.244513 18985 raft_consensus.cc:729] T c9ff972ade694938888514979e1638aa P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:28.244761 18985 consensus_queue.cc:260] T c9ff972ade694938888514979e1638aa P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:28.245327 18985 ts_tablet_manager.cc:1398] T c9ff972ade694938888514979e1638aa P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240430 20:11:28.245697 18985 tablet_bootstrap.cc:492] T fae9a35a2ac844b4bc2ebdf214aa7171 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:28.246237 18988 tablet_bootstrap.cc:492] T c9ff972ade694938888514979e1638aa P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:28.247448 18985 tablet_bootstrap.cc:654] T fae9a35a2ac844b4bc2ebdf214aa7171 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:28.248576 18981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41678 (request call id 186) took 1095 ms. Trace:
I20240430 20:11:28.248690 18981 rpcz_store.cc:276] 0430 20:11:27.152958 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:27.153019 (+ 61us) service_pool.cc:225] Handling call
0430 20:11:28.248396 (+1095377us) ts_tablet_manager.cc:1120] Deleting tablet 255c13ad8fae4ea4aa8e7b7aabb7602e
0430 20:11:28.248403 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:28.248514 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1095159}
I20240430 20:11:28.243121 19186 raft_consensus.cc:504] T cace1398492f4d968dcb6899385d527c P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:28.252661 18981 tablet_replica.cc:330] T a0e8e07d363f48c698c7e2b17edc1545 P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:28.252619 18623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cace1398492f4d968dcb6899385d527c" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:28.252851 18981 raft_consensus.cc:2229] T a0e8e07d363f48c698c7e2b17edc1545 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:28.252859 18623 raft_consensus.cc:3037] T cace1398492f4d968dcb6899385d527c P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:28.253048 18981 raft_consensus.cc:2258] T a0e8e07d363f48c698c7e2b17edc1545 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:28.253602 18981 ts_tablet_manager.cc:1875] T a0e8e07d363f48c698c7e2b17edc1545 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:28.236758 18987 ts_tablet_manager.cc:1909] T 609a1a7d22e84b69bd0a53577b62b69b P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:28.255698 18623 raft_consensus.cc:2454] T cace1398492f4d968dcb6899385d527c P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 1.
I20240430 20:11:28.255769 18987 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59818 (request call id 159) took 1620 ms. Trace:
I20240430 20:11:28.255939 18987 rpcz_store.cc:276] 0430 20:11:26.635691 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:26.636805 (+ 1114us) service_pool.cc:225] Handling call
0430 20:11:28.183280 (+1546475us) ts_tablet_manager.cc:1120] Deleting tablet 609a1a7d22e84b69bd0a53577b62b69b
0430 20:11:28.183290 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:28.210571 (+ 27281us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:28.228978 (+ 18407us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:28.255713 (+ 26735us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1546240}
I20240430 20:11:28.256547 18987 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59818 (request call id 160) took 1574 ms. Trace:
I20240430 20:11:28.256690 18415 leader_election.cc:302] T cace1398492f4d968dcb6899385d527c P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:28.256690 18987 rpcz_store.cc:276] 0430 20:11:26.682045 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:26.682104 (+ 59us) service_pool.cc:225] Handling call
0430 20:11:28.256339 (+1574235us) ts_tablet_manager.cc:1120] Deleting tablet a1aeee1b02214975a376cfd36957afe8
0430 20:11:28.256354 (+ 15us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:28.256508 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1574014}
I20240430 20:11:28.257018 19185 raft_consensus.cc:2781] T cace1398492f4d968dcb6899385d527c P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:28.257122 18987 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59818 (request call id 161) took 1551 ms. Trace:
I20240430 20:11:28.257242 18987 rpcz_store.cc:276] 0430 20:11:26.705954 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:26.706023 (+ 69us) service_pool.cc:225] Handling call
0430 20:11:28.256983 (+1550960us) ts_tablet_manager.cc:1120] Deleting tablet aacabead3c874e77a47d2ecf19dbfa82
0430 20:11:28.256993 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:28.257085 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1550601}
I20240430 20:11:28.257535 18987 tablet_replica.cc:330] T 3f49eda7aeca42308d01b83a19d7d037 P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:28.257755 18987 raft_consensus.cc:2229] T 3f49eda7aeca42308d01b83a19d7d037 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:28.257908 18987 pending_rounds.cc:62] T 3f49eda7aeca42308d01b83a19d7d037 P 3c51c33d5616482cafb68e26688820f4: Trying to abort 1 pending ops.
I20240430 20:11:28.257993 18987 pending_rounds.cc:68] T 3f49eda7aeca42308d01b83a19d7d037 P 3c51c33d5616482cafb68e26688820f4: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7022624304610185216 op_type: NO_OP noop_request { }
I20240430 20:11:28.258136 18987 raft_consensus.cc:2866] T 3f49eda7aeca42308d01b83a19d7d037 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20240430 20:11:28.258258 18987 raft_consensus.cc:2258] T 3f49eda7aeca42308d01b83a19d7d037 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:28.258406 18983 tablet_bootstrap.cc:654] T 6e826459c3f4437da81c9e6e09f0c22f P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:28.258656 18988 tablet_bootstrap.cc:654] T c9ff972ade694938888514979e1638aa P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:28.263284 18987 ts_tablet_manager.cc:1875] T 3f49eda7aeca42308d01b83a19d7d037 P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:28.263549 18763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cace1398492f4d968dcb6899385d527c" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:28.263788 18763 raft_consensus.cc:3037] T cace1398492f4d968dcb6899385d527c P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:28.264724 18981 ts_tablet_manager.cc:1888] T a0e8e07d363f48c698c7e2b17edc1545 P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:28.264878 18981 log.cc:1196] T a0e8e07d363f48c698c7e2b17edc1545 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/a0e8e07d363f48c698c7e2b17edc1545
I20240430 20:11:28.266744 19342 raft_consensus.cc:482] T 00000000000000000000000000000000 P 89df84ccf5394c9bb60d4e4ed872fc2e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 88b3f98793354c7aaa5c106e8ec31ed8)
I20240430 20:11:28.266994 19342 raft_consensus.cc:504] T 00000000000000000000000000000000 P 89df84ccf5394c9bb60d4e4ed872fc2e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b3f98793354c7aaa5c106e8ec31ed8" member_type: VOTER last_known_addr { host: "127.17.177.62" port: 36441 } } peers { permanent_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e" member_type: VOTER last_known_addr { host: "127.17.177.61" port: 34777 } } peers { permanent_uuid: "b632d6ef4c104a058a6db208024cab33" member_type: VOTER last_known_addr { host: "127.17.177.60" port: 37225 } }
I20240430 20:11:28.267791 18763 raft_consensus.cc:2454] T cace1398492f4d968dcb6899385d527c P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 1.
I20240430 20:11:28.268992 18981 ts_tablet_manager.cc:1909] T a0e8e07d363f48c698c7e2b17edc1545 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:28.269862 19186 leader_election.cc:288] T cace1398492f4d968dcb6899385d527c P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Requested vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:28.270053 18983 tablet_bootstrap.cc:492] T 6e826459c3f4437da81c9e6e09f0c22f P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:28.270247 18983 ts_tablet_manager.cc:1367] T 6e826459c3f4437da81c9e6e09f0c22f P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.031s user 0.007s sys 0.000s
I20240430 20:11:28.272439 18988 tablet_bootstrap.cc:492] T c9ff972ade694938888514979e1638aa P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:28.272643 18988 ts_tablet_manager.cc:1367] T c9ff972ade694938888514979e1638aa P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.026s user 0.002s sys 0.004s
I20240430 20:11:28.273478 18988 raft_consensus.cc:348] T c9ff972ade694938888514979e1638aa P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:28.273742 18988 raft_consensus.cc:374] T c9ff972ade694938888514979e1638aa P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:28.273841 18988 raft_consensus.cc:729] T c9ff972ade694938888514979e1638aa P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:28.274114 18988 consensus_queue.cc:260] T c9ff972ade694938888514979e1638aa P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:28.275851 18988 ts_tablet_manager.cc:1398] T c9ff972ade694938888514979e1638aa P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240430 20:11:28.276167 18988 tablet_bootstrap.cc:492] T bbe9a2429a434f3f95d6e7f314fcfe4c P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:28.277531 19342 leader_election.cc:288] T 00000000000000000000000000000000 P 89df84ccf5394c9bb60d4e4ed872fc2e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 88b3f98793354c7aaa5c106e8ec31ed8 (127.17.177.62:36441), b632d6ef4c104a058a6db208024cab33 (127.17.177.60:37225)
I20240430 20:11:28.277786 18983 raft_consensus.cc:348] T 6e826459c3f4437da81c9e6e09f0c22f P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:28.278189 18983 raft_consensus.cc:374] T 6e826459c3f4437da81c9e6e09f0c22f P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:28.278373 18983 raft_consensus.cc:729] T 6e826459c3f4437da81c9e6e09f0c22f P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:28.279142 18983 consensus_queue.cc:260] T 6e826459c3f4437da81c9e6e09f0c22f P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:28.279793 18983 ts_tablet_manager.cc:1398] T 6e826459c3f4437da81c9e6e09f0c22f P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240430 20:11:28.280422 18983 tablet_bootstrap.cc:492] T aec9c09cc0454acab6006bb3e10982b7 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:28.282487 18983 tablet_bootstrap.cc:654] T aec9c09cc0454acab6006bb3e10982b7 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:28.289742 19314 raft_consensus.cc:482] T 372b135dd01f44f98cd713ca8857ba61 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:28.289983 19314 raft_consensus.cc:504] T 372b135dd01f44f98cd713ca8857ba61 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:28.290946 19314 leader_election.cc:288] T 372b135dd01f44f98cd713ca8857ba61 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:28.291896 18623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "372b135dd01f44f98cd713ca8857ba61" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:28.292165 18482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "372b135dd01f44f98cd713ca8857ba61" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:28.292310 18623 raft_consensus.cc:2454] T 372b135dd01f44f98cd713ca8857ba61 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
W20240430 20:11:28.292770 18699 leader_election.cc:341] T 372b135dd01f44f98cd713ca8857ba61 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:28.293015 18696 leader_election.cc:302] T 372b135dd01f44f98cd713ca8857ba61 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a
I20240430 20:11:28.293720 19314 raft_consensus.cc:2781] T 372b135dd01f44f98cd713ca8857ba61 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:28.294963 19314 raft_consensus.cc:482] T 372b135dd01f44f98cd713ca8857ba61 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:28.295821 19185 raft_consensus.cc:686] T cace1398492f4d968dcb6899385d527c P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Becoming Leader. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Running, Role: LEADER
I20240430 20:11:28.296167 18988 tablet_bootstrap.cc:654] T bbe9a2429a434f3f95d6e7f314fcfe4c P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:28.296375 19185 consensus_queue.cc:237] T cace1398492f4d968dcb6899385d527c P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:28.297441 19314 raft_consensus.cc:3037] T 372b135dd01f44f98cd713ca8857ba61 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:28.306679 18987 ts_tablet_manager.cc:1888] T 3f49eda7aeca42308d01b83a19d7d037 P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:28.306699 19314 raft_consensus.cc:504] T 372b135dd01f44f98cd713ca8857ba61 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:28.307062 18987 log.cc:1196] T 3f49eda7aeca42308d01b83a19d7d037 P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/3f49eda7aeca42308d01b83a19d7d037
I20240430 20:11:28.307657 18987 ts_tablet_manager.cc:1909] T 3f49eda7aeca42308d01b83a19d7d037 P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:28.308220 18482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "372b135dd01f44f98cd713ca8857ba61" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:28.308601 18987 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59818 (request call id 162) took 1600 ms. Trace:
I20240430 20:11:28.308760 18987 rpcz_store.cc:276] 0430 20:11:26.708405 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:26.708473 (+ 68us) service_pool.cc:225] Handling call
0430 20:11:28.257465 (+1548992us) ts_tablet_manager.cc:1120] Deleting tablet 3f49eda7aeca42308d01b83a19d7d037
0430 20:11:28.257474 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:28.272126 (+ 14652us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:28.297901 (+ 25775us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:28.308560 (+ 10659us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1548773}
W20240430 20:11:28.309017 18699 leader_election.cc:341] T 372b135dd01f44f98cd713ca8857ba61 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:28.309514 18987 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59818 (request call id 163) took 1532 ms. Trace:
I20240430 20:11:28.309679 18987 rpcz_store.cc:276] 0430 20:11:26.776751 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:26.776812 (+ 61us) service_pool.cc:225] Handling call
0430 20:11:28.309355 (+1532543us) ts_tablet_manager.cc:1120] Deleting tablet 68ca48351cd4412381f5195d81109016
0430 20:11:28.309367 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:28.309477 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1466058}
I20240430 20:11:28.309964 18623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "372b135dd01f44f98cd713ca8857ba61" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:28.310243 18987 tablet_replica.cc:330] T f5236574672b477a9026e91adcaf70ec P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:28.310220 18623 raft_consensus.cc:3037] T 372b135dd01f44f98cd713ca8857ba61 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:28.310526 18987 raft_consensus.cc:2229] T f5236574672b477a9026e91adcaf70ec P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:28.310809 18987 raft_consensus.cc:2258] T f5236574672b477a9026e91adcaf70ec P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:28.311446 18987 ts_tablet_manager.cc:1875] T f5236574672b477a9026e91adcaf70ec P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:28.313133 18623 raft_consensus.cc:2454] T 372b135dd01f44f98cd713ca8857ba61 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:28.313694 18696 leader_election.cc:302] T 372b135dd01f44f98cd713ca8857ba61 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a
I20240430 20:11:28.313987 19261 raft_consensus.cc:2781] T 372b135dd01f44f98cd713ca8857ba61 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:28.314124 19314 leader_election.cc:288] T 372b135dd01f44f98cd713ca8857ba61 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Requested vote from peers 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:28.314752 19261 raft_consensus.cc:686] T 372b135dd01f44f98cd713ca8857ba61 P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Becoming Leader. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Running, Role: LEADER
I20240430 20:11:28.315155 19261 consensus_queue.cc:237] T 372b135dd01f44f98cd713ca8857ba61 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:28.319032 18988 tablet_bootstrap.cc:492] T bbe9a2429a434f3f95d6e7f314fcfe4c P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:28.319229 18988 ts_tablet_manager.cc:1367] T bbe9a2429a434f3f95d6e7f314fcfe4c P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.043s user 0.004s sys 0.008s
I20240430 20:11:28.319213 19341 raft_consensus.cc:482] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 88b3f98793354c7aaa5c106e8ec31ed8)
I20240430 20:11:28.320449 18988 raft_consensus.cc:348] T bbe9a2429a434f3f95d6e7f314fcfe4c P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:28.320708 18988 raft_consensus.cc:374] T bbe9a2429a434f3f95d6e7f314fcfe4c P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:28.320806 18988 raft_consensus.cc:729] T bbe9a2429a434f3f95d6e7f314fcfe4c P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:28.320768 19341 raft_consensus.cc:504] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b3f98793354c7aaa5c106e8ec31ed8" member_type: VOTER last_known_addr { host: "127.17.177.62" port: 36441 } } peers { permanent_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e" member_type: VOTER last_known_addr { host: "127.17.177.61" port: 34777 } } peers { permanent_uuid: "b632d6ef4c104a058a6db208024cab33" member_type: VOTER last_known_addr { host: "127.17.177.60" port: 37225 } }
I20240430 20:11:28.321162 18988 consensus_queue.cc:260] T bbe9a2429a434f3f95d6e7f314fcfe4c P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:28.321800 18988 ts_tablet_manager.cc:1398] T bbe9a2429a434f3f95d6e7f314fcfe4c P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240430 20:11:28.322154 18988 tablet_bootstrap.cc:492] T 13c1107ea4d04eb78a74d3f658c72fb3 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:28.322881 18983 tablet_bootstrap.cc:492] T aec9c09cc0454acab6006bb3e10982b7 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:28.323112 18983 ts_tablet_manager.cc:1367] T aec9c09cc0454acab6006bb3e10982b7 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.043s user 0.004s sys 0.010s
I20240430 20:11:28.323825 18988 tablet_bootstrap.cc:654] T 13c1107ea4d04eb78a74d3f658c72fb3 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:28.324079 18983 raft_consensus.cc:348] T aec9c09cc0454acab6006bb3e10982b7 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:28.324462 18983 raft_consensus.cc:374] T aec9c09cc0454acab6006bb3e10982b7 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:28.324602 18983 raft_consensus.cc:729] T aec9c09cc0454acab6006bb3e10982b7 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:28.325003 18983 consensus_queue.cc:260] T aec9c09cc0454acab6006bb3e10982b7 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:28.325700 18983 ts_tablet_manager.cc:1398] T aec9c09cc0454acab6006bb3e10982b7 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240430 20:11:28.325951 18988 tablet_bootstrap.cc:492] T 13c1107ea4d04eb78a74d3f658c72fb3 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:28.326126 18983 tablet_bootstrap.cc:492] T e2d90ff17e574f88b2061b1c117d4b7d P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:28.326112 18988 ts_tablet_manager.cc:1367] T 13c1107ea4d04eb78a74d3f658c72fb3 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240430 20:11:28.326985 18988 raft_consensus.cc:348] T 13c1107ea4d04eb78a74d3f658c72fb3 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:28.327224 18988 raft_consensus.cc:374] T 13c1107ea4d04eb78a74d3f658c72fb3 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:28.327313 18988 raft_consensus.cc:729] T 13c1107ea4d04eb78a74d3f658c72fb3 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:28.327596 18988 consensus_queue.cc:260] T 13c1107ea4d04eb78a74d3f658c72fb3 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:28.328243 18988 ts_tablet_manager.cc:1398] T 13c1107ea4d04eb78a74d3f658c72fb3 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240430 20:11:28.328231 19341 leader_election.cc:288] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 88b3f98793354c7aaa5c106e8ec31ed8 (127.17.177.62:36441), 89df84ccf5394c9bb60d4e4ed872fc2e (127.17.177.61:34777)
I20240430 20:11:28.328578 18988 tablet_bootstrap.cc:492] T a745334c12c545c1ba825c7552c60359 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:28.329030 18235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b632d6ef4c104a058a6db208024cab33" candidate_term: 5 candidate_status { last_received { term: 4 index: 308 } } ignore_live_leader: false dest_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e" is_pre_election: true
I20240430 20:11:28.329280 18235 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 89df84ccf5394c9bb60d4e4ed872fc2e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b632d6ef4c104a058a6db208024cab33 in term 4.
W20240430 20:11:28.329880 18699 consensus_peers.cc:485] T 0879b02a0bcc4f619fb34cff7b9d7c4c P 3c51c33d5616482cafb68e26688820f4 -> Peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Couldn't send request to peer 2efb542aa55e439b99ccb98d1c5c884a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0879b02a0bcc4f619fb34cff7b9d7c4c. This is attempt 1: this message will repeat every 5th retry.
I20240430 20:11:28.330140 19000 leader_election.cc:302] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89df84ccf5394c9bb60d4e4ed872fc2e, b632d6ef4c104a058a6db208024cab33; no voters:
I20240430 20:11:28.330708 18988 tablet_bootstrap.cc:654] T a745334c12c545c1ba825c7552c60359 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:28.334666 18983 tablet_bootstrap.cc:654] T e2d90ff17e574f88b2061b1c117d4b7d P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:28.335669 19264 raft_consensus.cc:482] T 7bde1d6934784c9aa5911181e8df9eeb P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:28.335847 19264 raft_consensus.cc:504] T 7bde1d6934784c9aa5911181e8df9eeb P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:28.336571 19264 leader_election.cc:288] T 7bde1d6934784c9aa5911181e8df9eeb P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:28.337296 18623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bde1d6934784c9aa5911181e8df9eeb" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:28.337584 18623 raft_consensus.cc:2454] T 7bde1d6934784c9aa5911181e8df9eeb P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:28.338202 18696 leader_election.cc:302] T 7bde1d6934784c9aa5911181e8df9eeb P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:28.338932 19264 raft_consensus.cc:2781] T 7bde1d6934784c9aa5911181e8df9eeb P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:28.338994 19341 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240430 20:11:28.339043 19264 raft_consensus.cc:482] T 7bde1d6934784c9aa5911181e8df9eeb P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:28.338984 18482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bde1d6934784c9aa5911181e8df9eeb" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:28.339126 19341 raft_consensus.cc:482] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 88b3f98793354c7aaa5c106e8ec31ed8)
I20240430 20:11:28.339138 19264 raft_consensus.cc:3037] T 7bde1d6934784c9aa5911181e8df9eeb P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:28.339210 19341 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33 [term 4 FOLLOWER]: Advancing to term 5
I20240430 20:11:28.339234 18482 raft_consensus.cc:2454] T 7bde1d6934784c9aa5911181e8df9eeb P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:28.342777 19264 raft_consensus.cc:504] T 7bde1d6934784c9aa5911181e8df9eeb P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:28.343946 18482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bde1d6934784c9aa5911181e8df9eeb" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:28.344172 18482 raft_consensus.cc:3037] T 7bde1d6934784c9aa5911181e8df9eeb P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:28.344617 18983 tablet_bootstrap.cc:492] T e2d90ff17e574f88b2061b1c117d4b7d P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:28.344857 18983 ts_tablet_manager.cc:1367] T e2d90ff17e574f88b2061b1c117d4b7d P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.001s
I20240430 20:11:28.345144 18987 ts_tablet_manager.cc:1888] T f5236574672b477a9026e91adcaf70ec P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:28.345294 18987 log.cc:1196] T f5236574672b477a9026e91adcaf70ec P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/f5236574672b477a9026e91adcaf70ec
I20240430 20:11:28.345664 18987 ts_tablet_manager.cc:1909] T f5236574672b477a9026e91adcaf70ec P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:28.345710 18983 raft_consensus.cc:348] T e2d90ff17e574f88b2061b1c117d4b7d P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:28.345947 18983 raft_consensus.cc:374] T e2d90ff17e574f88b2061b1c117d4b7d P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:28.346027 18983 raft_consensus.cc:729] T e2d90ff17e574f88b2061b1c117d4b7d P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:28.346184 19341 raft_consensus.cc:504] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b3f98793354c7aaa5c106e8ec31ed8" member_type: VOTER last_known_addr { host: "127.17.177.62" port: 36441 } } peers { permanent_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e" member_type: VOTER last_known_addr { host: "127.17.177.61" port: 34777 } } peers { permanent_uuid: "b632d6ef4c104a058a6db208024cab33" member_type: VOTER last_known_addr { host: "127.17.177.60" port: 37225 } }
I20240430 20:11:28.346323 18983 consensus_queue.cc:260] T e2d90ff17e574f88b2061b1c117d4b7d P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:28.346463 18987 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59818 (request call id 164) took 1566 ms. Trace:
I20240430 20:11:28.346557 18987 rpcz_store.cc:276] 0430 20:11:26.780339 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:26.780450 (+ 111us) service_pool.cc:225] Handling call
0430 20:11:28.310103 (+1529653us) ts_tablet_manager.cc:1120] Deleting tablet f5236574672b477a9026e91adcaf70ec
0430 20:11:28.310187 (+ 84us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:28.331878 (+ 21691us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:28.345002 (+ 13124us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:28.346408 (+ 1406us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1465476}
I20240430 20:11:28.347157 18482 raft_consensus.cc:2454] T 7bde1d6934784c9aa5911181e8df9eeb P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:28.347594 18699 leader_election.cc:302] T 7bde1d6934784c9aa5911181e8df9eeb P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters:
I20240430 20:11:28.347874 19261 raft_consensus.cc:2781] T 7bde1d6934784c9aa5911181e8df9eeb P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:28.348127 18983 ts_tablet_manager.cc:1398] T e2d90ff17e574f88b2061b1c117d4b7d P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240430 20:11:28.348137 19261 raft_consensus.cc:686] T 7bde1d6934784c9aa5911181e8df9eeb P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Becoming Leader. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Running, Role: LEADER
I20240430 20:11:28.348444 18983 tablet_bootstrap.cc:492] T 1398d4d2305b48e08fdb12432bd6c4b0 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:28.348421 19261 consensus_queue.cc:237] T 7bde1d6934784c9aa5911181e8df9eeb P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:28.350220 18983 tablet_bootstrap.cc:654] T 1398d4d2305b48e08fdb12432bd6c4b0 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:28.350665 19264 leader_election.cc:288] T 7bde1d6934784c9aa5911181e8df9eeb P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Requested vote from peers 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:28.351450 18623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bde1d6934784c9aa5911181e8df9eeb" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:28.351734 18623 raft_consensus.cc:3037] T 7bde1d6934784c9aa5911181e8df9eeb P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:28.355023 19074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 308 } } ignore_live_leader: false dest_uuid: "b632d6ef4c104a058a6db208024cab33" is_pre_election: true
I20240430 20:11:28.355634 19074 raft_consensus.cc:2379] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 89df84ccf5394c9bb60d4e4ed872fc2e in current term 5: Already voted for candidate b632d6ef4c104a058a6db208024cab33 in this term.
I20240430 20:11:28.357781 18623 raft_consensus.cc:2454] T 7bde1d6934784c9aa5911181e8df9eeb P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:28.358438 18985 tablet_bootstrap.cc:492] T fae9a35a2ac844b4bc2ebdf214aa7171 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:28.358631 18985 ts_tablet_manager.cc:1367] T fae9a35a2ac844b4bc2ebdf214aa7171 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.113s user 0.015s sys 0.028s
I20240430 20:11:28.359499 18985 raft_consensus.cc:348] T fae9a35a2ac844b4bc2ebdf214aa7171 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:28.359759 18985 raft_consensus.cc:374] T fae9a35a2ac844b4bc2ebdf214aa7171 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:28.359869 18985 raft_consensus.cc:729] T fae9a35a2ac844b4bc2ebdf214aa7171 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:28.360109 18985 consensus_queue.cc:260] T fae9a35a2ac844b4bc2ebdf214aa7171 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:28.360664 18985 ts_tablet_manager.cc:1398] T fae9a35a2ac844b4bc2ebdf214aa7171 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240430 20:11:28.361606 18985 tablet_bootstrap.cc:492] T 372b135dd01f44f98cd713ca8857ba61 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:28.362552 19230 consensus_queue.cc:1035] T 680553d4d0bc4fcb839235c1832757e1 P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:28.363946 18985 tablet_bootstrap.cc:654] T 372b135dd01f44f98cd713ca8857ba61 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:28.364631 18987 tablet_replica.cc:330] T 27f94f2d8f50495dababe417a9f82982 P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:28.364950 18987 raft_consensus.cc:2229] T 27f94f2d8f50495dababe417a9f82982 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:28.365541 19341 leader_election.cc:288] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33 [CANDIDATE]: Term 5 election: Requested vote from peers 88b3f98793354c7aaa5c106e8ec31ed8 (127.17.177.62:36441), 89df84ccf5394c9bb60d4e4ed872fc2e (127.17.177.61:34777)
I20240430 20:11:28.366855 18235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b632d6ef4c104a058a6db208024cab33" candidate_term: 5 candidate_status { last_received { term: 4 index: 308 } } ignore_live_leader: false dest_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e"
I20240430 20:11:28.367179 18235 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 89df84ccf5394c9bb60d4e4ed872fc2e [term 4 FOLLOWER]: Advancing to term 5
W20240430 20:11:28.367197 18696 consensus_peers.cc:485] T 0879b02a0bcc4f619fb34cff7b9d7c4c P 3c51c33d5616482cafb68e26688820f4 -> Peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Couldn't send request to peer 5e1c6397f09747f091534faf93542710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0879b02a0bcc4f619fb34cff7b9d7c4c. This is attempt 1: this message will repeat every 5th retry.
I20240430 20:11:28.365192 18987 raft_consensus.cc:2258] T 27f94f2d8f50495dababe417a9f82982 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:28.372769 18987 ts_tablet_manager.cc:1875] T 27f94f2d8f50495dababe417a9f82982 P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:28.372833 18235 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 89df84ccf5394c9bb60d4e4ed872fc2e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b632d6ef4c104a058a6db208024cab33 in term 5.
I20240430 20:11:28.376683 19000 leader_election.cc:302] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89df84ccf5394c9bb60d4e4ed872fc2e, b632d6ef4c104a058a6db208024cab33; no voters:
I20240430 20:11:28.379943 19341 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33 [term 5 FOLLOWER]: Leader election won for term 5
I20240430 20:11:28.382297 18983 tablet_bootstrap.cc:492] T 1398d4d2305b48e08fdb12432bd6c4b0 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:28.382418 18987 ts_tablet_manager.cc:1888] T 27f94f2d8f50495dababe417a9f82982 P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:28.382495 18983 ts_tablet_manager.cc:1367] T 1398d4d2305b48e08fdb12432bd6c4b0 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.034s user 0.003s sys 0.000s
I20240430 20:11:28.383332 19341 raft_consensus.cc:686] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33 [term 5 LEADER]: Becoming Leader. State: Replica: b632d6ef4c104a058a6db208024cab33, State: Running, Role: LEADER
I20240430 20:11:28.383459 18983 raft_consensus.cc:348] T 1398d4d2305b48e08fdb12432bd6c4b0 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:28.383674 18983 raft_consensus.cc:374] T 1398d4d2305b48e08fdb12432bd6c4b0 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:28.383834 18983 raft_consensus.cc:729] T 1398d4d2305b48e08fdb12432bd6c4b0 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:28.384060 18988 tablet_bootstrap.cc:492] T a745334c12c545c1ba825c7552c60359 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:28.383926 19341 consensus_queue.cc:237] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 308, Committed index: 308, Last appended: 4.308, Last appended by leader: 308, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b3f98793354c7aaa5c106e8ec31ed8" member_type: VOTER last_known_addr { host: "127.17.177.62" port: 36441 } } peers { permanent_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e" member_type: VOTER last_known_addr { host: "127.17.177.61" port: 34777 } } peers { permanent_uuid: "b632d6ef4c104a058a6db208024cab33" member_type: VOTER last_known_addr { host: "127.17.177.60" port: 37225 } }
I20240430 20:11:28.384205 18988 ts_tablet_manager.cc:1367] T a745334c12c545c1ba825c7552c60359 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.056s user 0.013s sys 0.006s
I20240430 20:11:28.384125 18983 consensus_queue.cc:260] T 1398d4d2305b48e08fdb12432bd6c4b0 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:28.385006 18983 ts_tablet_manager.cc:1398] T 1398d4d2305b48e08fdb12432bd6c4b0 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240430 20:11:28.385321 18983 tablet_bootstrap.cc:492] T aacabead3c874e77a47d2ecf19dbfa82 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:28.386734 19180 consensus_queue.cc:1035] T 680553d4d0bc4fcb839235c1832757e1 P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:28.387027 18983 tablet_bootstrap.cc:654] T aacabead3c874e77a47d2ecf19dbfa82 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:28.391204 18988 raft_consensus.cc:348] T a745334c12c545c1ba825c7552c60359 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:28.391526 18988 raft_consensus.cc:374] T a745334c12c545c1ba825c7552c60359 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:28.391646 18988 raft_consensus.cc:729] T a745334c12c545c1ba825c7552c60359 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:28.391942 18988 consensus_queue.cc:260] T a745334c12c545c1ba825c7552c60359 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:28.392660 18988 ts_tablet_manager.cc:1398] T a745334c12c545c1ba825c7552c60359 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20240430 20:11:28.392999 18988 tablet_bootstrap.cc:492] T 6e826459c3f4437da81c9e6e09f0c22f P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:28.394747 18988 tablet_bootstrap.cc:654] T 6e826459c3f4437da81c9e6e09f0c22f P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:28.382560 18987 log.cc:1196] T 27f94f2d8f50495dababe417a9f82982 P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/27f94f2d8f50495dababe417a9f82982
I20240430 20:11:28.395759 18985 tablet_bootstrap.cc:492] T 372b135dd01f44f98cd713ca8857ba61 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:28.395942 18985 ts_tablet_manager.cc:1367] T 372b135dd01f44f98cd713ca8857ba61 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.034s user 0.003s sys 0.006s
I20240430 20:11:28.396246 18987 ts_tablet_manager.cc:1909] T 27f94f2d8f50495dababe417a9f82982 P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:28.396776 18985 raft_consensus.cc:348] T 372b135dd01f44f98cd713ca8857ba61 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:28.397037 18985 raft_consensus.cc:374] T 372b135dd01f44f98cd713ca8857ba61 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:28.397140 18985 raft_consensus.cc:729] T 372b135dd01f44f98cd713ca8857ba61 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:28.397567 18987 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59818 (request call id 165) took 1594 ms. Trace:
I20240430 20:11:28.397408 18985 consensus_queue.cc:260] T 372b135dd01f44f98cd713ca8857ba61 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:28.397768 18987 rpcz_store.cc:276] 0430 20:11:26.803092 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:26.803183 (+ 91us) service_pool.cc:225] Handling call
0430 20:11:28.364509 (+1561326us) ts_tablet_manager.cc:1120] Deleting tablet 27f94f2d8f50495dababe417a9f82982
0430 20:11:28.364526 (+ 17us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:28.379239 (+ 14713us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:28.382268 (+ 3029us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:28.397529 (+ 15261us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1513972}
I20240430 20:11:28.398129 18985 ts_tablet_manager.cc:1398] T 372b135dd01f44f98cd713ca8857ba61 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240430 20:11:28.398415 18987 tablet_replica.cc:330] T 4e97f7315f8b4ac29db16e00fc35bf99 P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:28.398689 18987 raft_consensus.cc:2229] T 4e97f7315f8b4ac29db16e00fc35bf99 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:28.398896 18987 raft_consensus.cc:2258] T 4e97f7315f8b4ac29db16e00fc35bf99 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:28.399685 18987 ts_tablet_manager.cc:1875] T 4e97f7315f8b4ac29db16e00fc35bf99 P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:28.398556 18985 tablet_bootstrap.cc:492] T c5fd0730133c472299c93f4b0830e527 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:28.404106 18985 tablet_bootstrap.cc:654] T c5fd0730133c472299c93f4b0830e527 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
W20240430 20:11:28.408468 18141 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet fcc9698abda4491b98e1e879589913a8 (table default.Table_e431f94220aa44ecba799a54dfb612ae [id=1e785b6c159b4312bd1b064271f55a49], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet fcc9698abda4491b98e1e879589913a8 consensus state has no leader
W20240430 20:11:28.408509 18139 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 76d6e9d547ca4a848eae852944b1d117 (table default.Table_210b7eaff7144b2eb7db1c3b5f7d9505 [id=ff86c0357c164dd0991098c416087773], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 76d6e9d547ca4a848eae852944b1d117 consensus state has no leader
I20240430 20:11:28.411039 18150 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_71075ae6b4d545889273c891648c70d1" } new_table_name: "default.Table_c12ca6938d424fbfa2e1df5cdb00fdbc" modify_external_catalogs: true
I20240430 20:11:28.412730 18154 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54524:
name: "default.Table_a327a8eced314cdf95731f1181f52fb1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:28.413278 18154 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a327a8eced314cdf95731f1181f52fb1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:28.415141 18151 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54524:
name: "default.Table_1a641ff0ab5944a495a73189a6e8499a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240430 20:11:28.415702 18149 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54524:
name: "default.Table_01d62e066e4942d9aa1f582189d60d49"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240430 20:11:28.415843 18148 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54524:
name: "default.Table_aed0b9eeaac74c98944426712706be6b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240430 20:11:28.416216 18985 tablet_bootstrap.cc:492] T c5fd0730133c472299c93f4b0830e527 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
W20240430 20:11:28.416214 18149 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_01d62e066e4942d9aa1f582189d60d49 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:28.416369 18985 ts_tablet_manager.cc:1367] T c5fd0730133c472299c93f4b0830e527 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.000s
W20240430 20:11:28.416496 18148 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_aed0b9eeaac74c98944426712706be6b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:28.417181 18985 raft_consensus.cc:348] T c5fd0730133c472299c93f4b0830e527 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:28.417423 18985 raft_consensus.cc:374] T c5fd0730133c472299c93f4b0830e527 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:28.417511 18985 raft_consensus.cc:729] T c5fd0730133c472299c93f4b0830e527 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:28.417737 18985 consensus_queue.cc:260] T c5fd0730133c472299c93f4b0830e527 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:28.418417 18985 ts_tablet_manager.cc:1398] T c5fd0730133c472299c93f4b0830e527 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240430 20:11:28.418777 18985 tablet_bootstrap.cc:492] T 93d158de64994f3c9fa45a1250a4f533 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:28.420807 18985 tablet_bootstrap.cc:654] T 93d158de64994f3c9fa45a1250a4f533 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:28.422353 18588 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e97f7315f8b4ac29db16e00fc35bf99 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:26 UTC) from {username='slave'} at 127.0.0.1:40202
I20240430 20:11:28.427830 18987 ts_tablet_manager.cc:1888] T 4e97f7315f8b4ac29db16e00fc35bf99 P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:28.428521 18987 log.cc:1196] T 4e97f7315f8b4ac29db16e00fc35bf99 P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/4e97f7315f8b4ac29db16e00fc35bf99
I20240430 20:11:28.430548 18147 master_service.cc:946] ReplaceTablet: received request to replace tablet b688e03ea71940c69bda5520b62c1fc3 from {username='slave'} at 127.0.0.1:54524
I20240430 20:11:28.426683 19361 sys_catalog.cc:455] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b632d6ef4c104a058a6db208024cab33. Latest consensus state: current_term: 5 leader_uuid: "b632d6ef4c104a058a6db208024cab33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b3f98793354c7aaa5c106e8ec31ed8" member_type: VOTER last_known_addr { host: "127.17.177.62" port: 36441 } } peers { permanent_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e" member_type: VOTER last_known_addr { host: "127.17.177.61" port: 34777 } } peers { permanent_uuid: "b632d6ef4c104a058a6db208024cab33" member_type: VOTER last_known_addr { host: "127.17.177.60" port: 37225 } } }
I20240430 20:11:28.430907 19361 sys_catalog.cc:458] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33 [sys.catalog]: This master's current role is: LEADER
I20240430 20:11:28.431202 18156 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_cd719a43207f4bf983e1b8e6d236bb16" } modify_external_catalogs: true
I20240430 20:11:28.431411 18156 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54524:
table { table_name: "default.Table_cd719a43207f4bf983e1b8e6d236bb16" } modify_external_catalogs: true
I20240430 20:11:28.431954 18165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 308 } } ignore_live_leader: false dest_uuid: "88b3f98793354c7aaa5c106e8ec31ed8" is_pre_election: true
I20240430 20:11:28.432592 18211 leader_election.cc:302] T 00000000000000000000000000000000 P 89df84ccf5394c9bb60d4e4ed872fc2e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89df84ccf5394c9bb60d4e4ed872fc2e; no voters: 88b3f98793354c7aaa5c106e8ec31ed8, b632d6ef4c104a058a6db208024cab33
I20240430 20:11:28.432924 19342 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 89df84ccf5394c9bb60d4e4ed872fc2e [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240430 20:11:28.433080 18155 master_service.cc:946] ReplaceTablet: received request to replace tablet 5f56ac2afd444d63a6bfd96988464bbb from {username='slave'} at 127.0.0.1:54524
I20240430 20:11:28.433779 18146 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54524:
name: "default.Table_f33a9e03642c44efbf1c27f1c7e70fb9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240430 20:11:28.434836 19370 tablet_replica.cc:330] T 4e97f7315f8b4ac29db16e00fc35bf99 P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:28.435958 18988 tablet_bootstrap.cc:492] T 6e826459c3f4437da81c9e6e09f0c22f P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:28.436357 18988 ts_tablet_manager.cc:1367] T 6e826459c3f4437da81c9e6e09f0c22f P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.043s user 0.007s sys 0.003s
I20240430 20:11:28.437548 19370 raft_consensus.cc:2229] T 4e97f7315f8b4ac29db16e00fc35bf99 P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Raft consensus shutting down.
I20240430 20:11:28.438417 19370 pending_rounds.cc:62] T 4e97f7315f8b4ac29db16e00fc35bf99 P 5e1c6397f09747f091534faf93542710: Trying to abort 1 pending ops.
I20240430 20:11:28.438239 18988 raft_consensus.cc:348] T 6e826459c3f4437da81c9e6e09f0c22f P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:28.439205 19370 pending_rounds.cc:68] T 4e97f7315f8b4ac29db16e00fc35bf99 P 5e1c6397f09747f091534faf93542710: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7022624309776236544 op_type: NO_OP noop_request { }
I20240430 20:11:28.439327 18987 ts_tablet_manager.cc:1909] T 4e97f7315f8b4ac29db16e00fc35bf99 P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:28.439730 19370 raft_consensus.cc:2866] T 4e97f7315f8b4ac29db16e00fc35bf99 P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240430 20:11:28.440196 19370 raft_consensus.cc:2258] T 4e97f7315f8b4ac29db16e00fc35bf99 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:28.439510 18988 raft_consensus.cc:374] T 6e826459c3f4437da81c9e6e09f0c22f P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:28.440457 18988 raft_consensus.cc:729] T 6e826459c3f4437da81c9e6e09f0c22f P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:28.440780 18988 consensus_queue.cc:260] T 6e826459c3f4437da81c9e6e09f0c22f P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:28.442481 19370 ts_tablet_manager.cc:1875] T 4e97f7315f8b4ac29db16e00fc35bf99 P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:28.443569 18988 ts_tablet_manager.cc:1398] T 6e826459c3f4437da81c9e6e09f0c22f P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20240430 20:11:28.443977 18988 tablet_bootstrap.cc:492] T 93d158de64994f3c9fa45a1250a4f533 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
W20240430 20:11:28.448740 18146 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f33a9e03642c44efbf1c27f1c7e70fb9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240430 20:11:28.449549 18151 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1a641ff0ab5944a495a73189a6e8499a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:28.449865 18987 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59818 (request call id 172) took 1551 ms. Trace:
I20240430 20:11:28.449967 18987 rpcz_store.cc:276] 0430 20:11:26.898670 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:26.900867 (+ 2197us) service_pool.cc:225] Handling call
0430 20:11:28.398342 (+1497475us) ts_tablet_manager.cc:1120] Deleting tablet 4e97f7315f8b4ac29db16e00fc35bf99
0430 20:11:28.398351 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:28.411417 (+ 13066us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:28.414470 (+ 3053us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:28.449822 (+ 35352us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1497186}
I20240430 20:11:28.452147 18987 tablet_replica.cc:330] T fd47db5ca8b24c2b827df0a64e195866 P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:28.452538 18987 raft_consensus.cc:2229] T fd47db5ca8b24c2b827df0a64e195866 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:28.454175 18987 raft_consensus.cc:2258] T fd47db5ca8b24c2b827df0a64e195866 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:28.454829 18987 ts_tablet_manager.cc:1875] T fd47db5ca8b24c2b827df0a64e195866 P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:28.453977 18988 tablet_bootstrap.cc:654] T 93d158de64994f3c9fa45a1250a4f533 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:28.460278 18116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNG3HqZ/build/asan/bin/kudu with pid 18124
I20240430 20:11:28.462468 19371 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240430 20:11:28.463567 19261 raft_consensus.cc:482] T 13c1107ea4d04eb78a74d3f658c72fb3 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:28.463768 19261 raft_consensus.cc:504] T 13c1107ea4d04eb78a74d3f658c72fb3 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:28.464674 18985 tablet_bootstrap.cc:492] T 93d158de64994f3c9fa45a1250a4f533 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:28.464862 18985 ts_tablet_manager.cc:1367] T 93d158de64994f3c9fa45a1250a4f533 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.046s user 0.006s sys 0.002s
I20240430 20:11:28.465842 19261 leader_election.cc:288] T 13c1107ea4d04eb78a74d3f658c72fb3 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:28.465788 18985 raft_consensus.cc:348] T 93d158de64994f3c9fa45a1250a4f533 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:28.466043 18985 raft_consensus.cc:374] T 93d158de64994f3c9fa45a1250a4f533 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:28.466136 18985 raft_consensus.cc:729] T 93d158de64994f3c9fa45a1250a4f533 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
W20240430 20:11:28.466116 18699 leader_election.cc:341] T 13c1107ea4d04eb78a74d3f658c72fb3 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Not found: Tablet not found: 13c1107ea4d04eb78a74d3f658c72fb3
I20240430 20:11:28.466408 18985 consensus_queue.cc:260] T 93d158de64994f3c9fa45a1250a4f533 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:28.467226 18985 ts_tablet_manager.cc:1398] T 93d158de64994f3c9fa45a1250a4f533 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240430 20:11:28.467653 18623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13c1107ea4d04eb78a74d3f658c72fb3" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:28.467916 18623 raft_consensus.cc:2379] T 13c1107ea4d04eb78a74d3f658c72fb3 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c51c33d5616482cafb68e26688820f4 in current term 1: Already voted for candidate 2efb542aa55e439b99ccb98d1c5c884a in this term.
I20240430 20:11:28.468580 19370 ts_tablet_manager.cc:1888] T 4e97f7315f8b4ac29db16e00fc35bf99 P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:28.468640 18696 leader_election.cc:302] T 13c1107ea4d04eb78a74d3f658c72fb3 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710
I20240430 20:11:28.469092 18985 tablet_bootstrap.cc:492] T a745334c12c545c1ba825c7552c60359 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:28.471237 19261 raft_consensus.cc:3037] T 13c1107ea4d04eb78a74d3f658c72fb3 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:28.471987 18985 tablet_bootstrap.cc:654] T a745334c12c545c1ba825c7552c60359 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:28.473101 19370 log.cc:1196] T 4e97f7315f8b4ac29db16e00fc35bf99 P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/4e97f7315f8b4ac29db16e00fc35bf99
I20240430 20:11:28.473474 18987 ts_tablet_manager.cc:1888] T fd47db5ca8b24c2b827df0a64e195866 P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:28.473615 18987 log.cc:1196] T fd47db5ca8b24c2b827df0a64e195866 P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/fd47db5ca8b24c2b827df0a64e195866
I20240430 20:11:28.473783 19370 ts_tablet_manager.cc:1909] T 4e97f7315f8b4ac29db16e00fc35bf99 P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:28.473951 18987 ts_tablet_manager.cc:1909] T fd47db5ca8b24c2b827df0a64e195866 P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:28.474304 19261 raft_consensus.cc:2726] T 13c1107ea4d04eb78a74d3f658c72fb3 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:28.474782 18987 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59818 (request call id 173) took 1562 ms. Trace:
I20240430 20:11:28.480634 18987 rpcz_store.cc:276] 0430 20:11:26.912492 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:26.913644 (+ 1152us) service_pool.cc:225] Handling call
0430 20:11:28.452026 (+1538382us) ts_tablet_manager.cc:1120] Deleting tablet fd47db5ca8b24c2b827df0a64e195866
0430 20:11:28.452042 (+ 16us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:28.467847 (+ 15805us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:28.473317 (+ 5470us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:28.474746 (+ 1429us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1538122}
I20240430 20:11:28.481690 18987 tablet_replica.cc:330] T c9ff972ade694938888514979e1638aa P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:28.482198 18987 raft_consensus.cc:2229] T c9ff972ade694938888514979e1638aa P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:28.482705 18987 raft_consensus.cc:2258] T c9ff972ade694938888514979e1638aa P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:28.484560 18987 ts_tablet_manager.cc:1875] T c9ff972ade694938888514979e1638aa P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:28.489089 18983 tablet_bootstrap.cc:492] T aacabead3c874e77a47d2ecf19dbfa82 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:28.489358 18983 ts_tablet_manager.cc:1367] T aacabead3c874e77a47d2ecf19dbfa82 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.104s user 0.005s sys 0.000s
I20240430 20:11:28.490481 18983 raft_consensus.cc:348] T aacabead3c874e77a47d2ecf19dbfa82 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:28.490882 18983 raft_consensus.cc:374] T aacabead3c874e77a47d2ecf19dbfa82 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:28.490994 18983 raft_consensus.cc:729] T aacabead3c874e77a47d2ecf19dbfa82 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:28.491413 18983 consensus_queue.cc:260] T aacabead3c874e77a47d2ecf19dbfa82 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:28.499256 18983 ts_tablet_manager.cc:1398] T aacabead3c874e77a47d2ecf19dbfa82 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.010s user 0.003s sys 0.000s
I20240430 20:11:28.499629 18988 tablet_bootstrap.cc:492] T 93d158de64994f3c9fa45a1250a4f533 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:28.499656 18983 tablet_bootstrap.cc:492] T bd11c0335a58448da56533c3f2e785a9 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:28.499853 18988 ts_tablet_manager.cc:1367] T 93d158de64994f3c9fa45a1250a4f533 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.056s user 0.004s sys 0.008s
I20240430 20:11:28.500599 18985 tablet_bootstrap.cc:492] T a745334c12c545c1ba825c7552c60359 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:28.500771 18985 ts_tablet_manager.cc:1367] T a745334c12c545c1ba825c7552c60359 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.032s user 0.008s sys 0.015s
I20240430 20:11:28.500654 18988 raft_consensus.cc:348] T 93d158de64994f3c9fa45a1250a4f533 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:28.501104 18988 raft_consensus.cc:374] T 93d158de64994f3c9fa45a1250a4f533 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:28.501225 18988 raft_consensus.cc:729] T 93d158de64994f3c9fa45a1250a4f533 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:28.501504 18988 consensus_queue.cc:260] T 93d158de64994f3c9fa45a1250a4f533 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:28.503609 18988 ts_tablet_manager.cc:1398] T 93d158de64994f3c9fa45a1250a4f533 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240430 20:11:28.503960 18988 tablet_bootstrap.cc:492] T c5fd0730133c472299c93f4b0830e527 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:28.504747 18985 raft_consensus.cc:348] T a745334c12c545c1ba825c7552c60359 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:28.505009 18985 raft_consensus.cc:374] T a745334c12c545c1ba825c7552c60359 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:28.505110 18985 raft_consensus.cc:729] T a745334c12c545c1ba825c7552c60359 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:28.505347 18985 consensus_queue.cc:260] T a745334c12c545c1ba825c7552c60359 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:28.505936 18985 ts_tablet_manager.cc:1398] T a745334c12c545c1ba825c7552c60359 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240430 20:11:28.506318 18985 tablet_bootstrap.cc:492] T aec9c09cc0454acab6006bb3e10982b7 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:28.507324 18983 tablet_bootstrap.cc:654] T bd11c0335a58448da56533c3f2e785a9 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:28.507953 18988 tablet_bootstrap.cc:654] T c5fd0730133c472299c93f4b0830e527 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:28.509717 18985 tablet_bootstrap.cc:654] T aec9c09cc0454acab6006bb3e10982b7 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:28.512612 18987 ts_tablet_manager.cc:1888] T c9ff972ade694938888514979e1638aa P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:28.512804 18987 log.cc:1196] T c9ff972ade694938888514979e1638aa P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/c9ff972ade694938888514979e1638aa
I20240430 20:11:28.513197 18987 ts_tablet_manager.cc:1909] T c9ff972ade694938888514979e1638aa P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:28.514282 18987 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59818 (request call id 176) took 1584 ms. Trace:
I20240430 20:11:28.519057 18987 rpcz_store.cc:276] 0430 20:11:26.929917 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:26.929971 (+ 54us) service_pool.cc:225] Handling call
0430 20:11:28.481612 (+1551641us) ts_tablet_manager.cc:1120] Deleting tablet c9ff972ade694938888514979e1638aa
0430 20:11:28.481626 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:28.508424 (+ 26798us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:28.512490 (+ 4066us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:28.514229 (+ 1739us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1551439}
I20240430 20:11:28.520193 18987 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59818 (request call id 184) took 1523 ms. Trace:
I20240430 20:11:28.520762 18987 rpcz_store.cc:276] 0430 20:11:26.996515 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:26.998658 (+ 2143us) service_pool.cc:225] Handling call
0430 20:11:28.519979 (+1521321us) ts_tablet_manager.cc:1120] Deleting tablet eed6338ad3d64859af087ee15e1ec798
0430 20:11:28.519989 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:28.520126 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1517887}
I20240430 20:11:28.521626 18987 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59818 (request call id 193) took 1258 ms. Trace:
I20240430 20:11:28.522070 18987 rpcz_store.cc:276] 0430 20:11:27.263093 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:27.491122 (+228029us) service_pool.cc:225] Handling call
0430 20:11:28.521460 (+1030338us) ts_tablet_manager.cc:1120] Deleting tablet 255c13ad8fae4ea4aa8e7b7aabb7602e
0430 20:11:28.521470 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:28.521578 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1030087}
I20240430 20:11:28.522840 18987 tablet_replica.cc:330] T 0879b02a0bcc4f619fb34cff7b9d7c4c P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:28.523293 18987 raft_consensus.cc:2229] T 0879b02a0bcc4f619fb34cff7b9d7c4c P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Raft consensus shutting down.
I20240430 20:11:28.523893 18987 pending_rounds.cc:62] T 0879b02a0bcc4f619fb34cff7b9d7c4c P 3c51c33d5616482cafb68e26688820f4: Trying to abort 1 pending ops.
I20240430 20:11:28.524305 18987 pending_rounds.cc:68] T 0879b02a0bcc4f619fb34cff7b9d7c4c P 3c51c33d5616482cafb68e26688820f4: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7022624306429935616 op_type: NO_OP noop_request { }
I20240430 20:11:28.524631 18987 raft_consensus.cc:2866] T 0879b02a0bcc4f619fb34cff7b9d7c4c P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240430 20:11:28.524893 18987 raft_consensus.cc:2258] T 0879b02a0bcc4f619fb34cff7b9d7c4c P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:28.525617 18987 ts_tablet_manager.cc:1875] T 0879b02a0bcc4f619fb34cff7b9d7c4c P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:28.527587 18985 tablet_bootstrap.cc:492] T aec9c09cc0454acab6006bb3e10982b7 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:28.527771 18985 ts_tablet_manager.cc:1367] T aec9c09cc0454acab6006bb3e10982b7 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.010s
I20240430 20:11:28.528679 18985 raft_consensus.cc:348] T aec9c09cc0454acab6006bb3e10982b7 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:28.528928 18985 raft_consensus.cc:374] T aec9c09cc0454acab6006bb3e10982b7 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:28.529009 18985 raft_consensus.cc:729] T aec9c09cc0454acab6006bb3e10982b7 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:28.529273 18985 consensus_queue.cc:260] T aec9c09cc0454acab6006bb3e10982b7 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:28.538170 18985 ts_tablet_manager.cc:1398] T aec9c09cc0454acab6006bb3e10982b7 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20240430 20:11:28.541836 18985 tablet_bootstrap.cc:492] T 6e826459c3f4437da81c9e6e09f0c22f P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:28.542198 19180 raft_consensus.cc:482] T fae9a35a2ac844b4bc2ebdf214aa7171 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:28.542366 19180 raft_consensus.cc:504] T fae9a35a2ac844b4bc2ebdf214aa7171 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:28.543079 19180 leader_election.cc:288] T fae9a35a2ac844b4bc2ebdf214aa7171 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:28.544096 18482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fae9a35a2ac844b4bc2ebdf214aa7171" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:28.544353 18482 raft_consensus.cc:2454] T fae9a35a2ac844b4bc2ebdf214aa7171 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 0.
I20240430 20:11:28.544780 18763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fae9a35a2ac844b4bc2ebdf214aa7171" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:28.545060 18763 raft_consensus.cc:2454] T fae9a35a2ac844b4bc2ebdf214aa7171 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 0.
I20240430 20:11:28.545380 19185 raft_consensus.cc:482] T 15674aeb56b74210a4a2809608fd4457 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:28.545609 18985 tablet_bootstrap.cc:654] T 6e826459c3f4437da81c9e6e09f0c22f P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:28.545544 19185 raft_consensus.cc:504] T 15674aeb56b74210a4a2809608fd4457 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:28.546299 19185 leader_election.cc:288] T 15674aeb56b74210a4a2809608fd4457 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:28.547200 18987 ts_tablet_manager.cc:1888] T 0879b02a0bcc4f619fb34cff7b9d7c4c P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:28.547808 18557 leader_election.cc:302] T fae9a35a2ac844b4bc2ebdf214aa7171 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:28.548151 19180 raft_consensus.cc:2781] T fae9a35a2ac844b4bc2ebdf214aa7171 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:28.548261 19180 raft_consensus.cc:482] T fae9a35a2ac844b4bc2ebdf214aa7171 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:28.548370 19180 raft_consensus.cc:3037] T fae9a35a2ac844b4bc2ebdf214aa7171 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:28.548979 18987 log.cc:1196] T 0879b02a0bcc4f619fb34cff7b9d7c4c P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/0879b02a0bcc4f619fb34cff7b9d7c4c
I20240430 20:11:28.549645 18987 ts_tablet_manager.cc:1909] T 0879b02a0bcc4f619fb34cff7b9d7c4c P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:28.550158 19371 catalog_manager.cc:669] Loaded metadata for table default.Table_f61a45518e134c4496e4ac81fbe40fce [id=02d9fe0426694ee8b5fd70589fb66ead]
W20240430 20:11:28.551463 18415 leader_election.cc:341] T 15674aeb56b74210a4a2809608fd4457 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Not found: Tablet not found: 15674aeb56b74210a4a2809608fd4457
I20240430 20:11:28.552203 18763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15674aeb56b74210a4a2809608fd4457" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:28.552474 18763 raft_consensus.cc:2454] T 15674aeb56b74210a4a2809608fd4457 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 0.
I20240430 20:11:28.552820 18988 tablet_bootstrap.cc:492] T c5fd0730133c472299c93f4b0830e527 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:28.552943 18417 leader_election.cc:302] T 15674aeb56b74210a4a2809608fd4457 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters: 5e1c6397f09747f091534faf93542710
I20240430 20:11:28.553035 18988 ts_tablet_manager.cc:1367] T c5fd0730133c472299c93f4b0830e527 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.049s user 0.008s sys 0.008s
I20240430 20:11:28.553400 19185 raft_consensus.cc:2781] T 15674aeb56b74210a4a2809608fd4457 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:28.553525 19185 raft_consensus.cc:482] T 15674aeb56b74210a4a2809608fd4457 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:28.553622 19185 raft_consensus.cc:3037] T 15674aeb56b74210a4a2809608fd4457 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:28.553719 18983 tablet_bootstrap.cc:492] T bd11c0335a58448da56533c3f2e785a9 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:28.553869 18983 ts_tablet_manager.cc:1367] T bd11c0335a58448da56533c3f2e785a9 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.054s user 0.006s sys 0.000s
I20240430 20:11:28.554177 18988 raft_consensus.cc:348] T c5fd0730133c472299c93f4b0830e527 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:28.554479 18988 raft_consensus.cc:374] T c5fd0730133c472299c93f4b0830e527 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:28.554620 18988 raft_consensus.cc:729] T c5fd0730133c472299c93f4b0830e527 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:28.554561 18983 raft_consensus.cc:348] T bd11c0335a58448da56533c3f2e785a9 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:28.554812 18983 raft_consensus.cc:374] T bd11c0335a58448da56533c3f2e785a9 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:28.554895 18983 raft_consensus.cc:729] T bd11c0335a58448da56533c3f2e785a9 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:28.555068 18988 consensus_queue.cc:260] T c5fd0730133c472299c93f4b0830e527 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:28.555122 18983 consensus_queue.cc:260] T bd11c0335a58448da56533c3f2e785a9 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:28.557303 18988 ts_tablet_manager.cc:1398] T c5fd0730133c472299c93f4b0830e527 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240430 20:11:28.557718 18988 tablet_bootstrap.cc:492] T aec9c09cc0454acab6006bb3e10982b7 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:28.557869 19185 raft_consensus.cc:504] T 15674aeb56b74210a4a2809608fd4457 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:28.558977 19180 raft_consensus.cc:504] T fae9a35a2ac844b4bc2ebdf214aa7171 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:28.560230 18988 tablet_bootstrap.cc:654] T aec9c09cc0454acab6006bb3e10982b7 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:28.561084 18763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fae9a35a2ac844b4bc2ebdf214aa7171" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:28.561213 18987 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59818 (request call id 194) took 1296 ms. Trace:
I20240430 20:11:28.561338 18763 raft_consensus.cc:3037] T fae9a35a2ac844b4bc2ebdf214aa7171 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:28.561311 18987 rpcz_store.cc:276] 0430 20:11:27.264557 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:27.491382 (+226825us) service_pool.cc:225] Handling call
0430 20:11:28.522754 (+1031372us) ts_tablet_manager.cc:1120] Deleting tablet 0879b02a0bcc4f619fb34cff7b9d7c4c
0430 20:11:28.522763 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:28.536895 (+ 14132us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:28.547063 (+ 10168us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:28.561149 (+ 14086us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1031237}
I20240430 20:11:28.563095 18987 tablet_replica.cc:330] T 13c1107ea4d04eb78a74d3f658c72fb3 P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:28.563303 18987 raft_consensus.cc:2229] T 13c1107ea4d04eb78a74d3f658c72fb3 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:28.563498 18987 raft_consensus.cc:2258] T 13c1107ea4d04eb78a74d3f658c72fb3 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:28.564222 19180 leader_election.cc:288] T fae9a35a2ac844b4bc2ebdf214aa7171 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Requested vote from peers 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:28.560222 18482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fae9a35a2ac844b4bc2ebdf214aa7171" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:28.564543 18482 raft_consensus.cc:3037] T fae9a35a2ac844b4bc2ebdf214aa7171 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:28.565280 18763 raft_consensus.cc:2454] T fae9a35a2ac844b4bc2ebdf214aa7171 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
I20240430 20:11:28.565729 18557 leader_election.cc:302] T fae9a35a2ac844b4bc2ebdf214aa7171 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:28.566485 19180 raft_consensus.cc:2781] T fae9a35a2ac844b4bc2ebdf214aa7171 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:28.566673 19180 raft_consensus.cc:686] T fae9a35a2ac844b4bc2ebdf214aa7171 P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Becoming Leader. State: Replica: 5e1c6397f09747f091534faf93542710, State: Running, Role: LEADER
I20240430 20:11:28.566761 19371 catalog_manager.cc:669] Loaded metadata for table default.Table_e63f64a411094f8db2ef641f2257ca79 [id=040dec3a3d804cdfbdada359bbe5123b]
I20240430 20:11:28.566973 19180 consensus_queue.cc:237] T fae9a35a2ac844b4bc2ebdf214aa7171 P 5e1c6397f09747f091534faf93542710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:28.567366 19371 catalog_manager.cc:669] Loaded metadata for table default.Table_0965d6f90dc0474091a9d7a550fcb3cb [id=059a3f2888d74c24bcd708c0e663ccd5]
I20240430 20:11:28.568015 18482 raft_consensus.cc:2454] T fae9a35a2ac844b4bc2ebdf214aa7171 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
I20240430 20:11:28.568049 19371 catalog_manager.cc:669] Loaded metadata for table default.Table_71075ae6b4d545889273c891648c70d1 [id=0fb93f36bc17408ea56bc4cafcfd8580]
I20240430 20:11:28.569432 19371 catalog_manager.cc:669] Loaded metadata for table default.Table_3106434588bd4241a314064d6a8749c3 [id=19f4e6decedf4ee59fccc0a24a05cedd]
I20240430 20:11:28.569986 19371 catalog_manager.cc:669] Loaded metadata for table default.Table_ad1a80db9c774df6add2b60e93577217 [id=19ffa77f540b499482d5dc7b64b12b79]
I20240430 20:11:28.570485 19371 catalog_manager.cc:669] Loaded metadata for table default.Table_b331bf8aeaae4911ac01ed3fbd2627bd [id=1bf5c29b08134670984b6569db298129]
W20240430 20:11:28.570655 18415 leader_election.cc:341] T 15674aeb56b74210a4a2809608fd4457 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Not found: Tablet not found: 15674aeb56b74210a4a2809608fd4457
I20240430 20:11:28.571040 19185 leader_election.cc:288] T 15674aeb56b74210a4a2809608fd4457 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:28.571596 19371 catalog_manager.cc:669] Loaded metadata for table default.Table_3e3885add2344068a5da1614890073e0 [id=1c8e0a3a44ec4a798770c0165ed3dffa]
I20240430 20:11:28.572326 19371 catalog_manager.cc:669] Loaded metadata for table default.Table_208b12b8e9bd4f3e8bc9c6d422ada649 [id=1df4e95702cd46f59ab21b101049a9ed]
I20240430 20:11:28.572685 18983 ts_tablet_manager.cc:1398] T bd11c0335a58448da56533c3f2e785a9 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.019s user 0.000s sys 0.002s
I20240430 20:11:28.572731 18763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15674aeb56b74210a4a2809608fd4457" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:28.573004 19371 catalog_manager.cc:669] Loaded metadata for table default.Table_e431f94220aa44ecba799a54dfb612ae [id=1e785b6c159b4312bd1b064271f55a49]
I20240430 20:11:28.573052 18983 tablet_bootstrap.cc:492] T a1aeee1b02214975a376cfd36957afe8 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:28.573029 18763 raft_consensus.cc:3037] T 15674aeb56b74210a4a2809608fd4457 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:28.573495 19371 catalog_manager.cc:669] Loaded metadata for table default.Table_3b327aece0ea4ec4b882f2264e3f5f0c [id=1fb10bddcfe94ec488ecc71e43ca339d]
I20240430 20:11:28.574033 19371 catalog_manager.cc:669] Loaded metadata for table default.Table_e68dc52988f24f1788517de94d7d2a0e [id=26d4860fb33047b7b812ebd4309f2e28]
I20240430 20:11:28.574749 19371 catalog_manager.cc:669] Loaded metadata for table default.Table_772ce91c27ce4119b6d1b1993e417f21 [id=29b41dafb4334b14a09602eabac2f031]
I20240430 20:11:28.575026 18983 tablet_bootstrap.cc:654] T a1aeee1b02214975a376cfd36957afe8 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:28.576153 19371 catalog_manager.cc:669] Loaded metadata for table default.Table_17b7c8e853bd427db0c3fa2555a9067b [id=2d44ec94131148048021b2ee6366ba30]
I20240430 20:11:28.576727 18763 raft_consensus.cc:2454] T 15674aeb56b74210a4a2809608fd4457 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 1.
I20240430 20:11:28.576980 18985 tablet_bootstrap.cc:492] T 6e826459c3f4437da81c9e6e09f0c22f P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:28.577006 19371 catalog_manager.cc:669] Loaded metadata for table default.Table_08dc70f541b842c78a3a946d2d02e207 [id=38f199a7538f4271b3a59fd93f29efa2]
I20240430 20:11:28.577200 18985 ts_tablet_manager.cc:1367] T 6e826459c3f4437da81c9e6e09f0c22f P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.035s user 0.001s sys 0.002s
I20240430 20:11:28.577347 18417 leader_election.cc:302] T 15674aeb56b74210a4a2809608fd4457 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters: 5e1c6397f09747f091534faf93542710
I20240430 20:11:28.577911 19371 catalog_manager.cc:669] Loaded metadata for table default.Table_27f3d711beeb4387a9088ab291168bc8 [id=434aba8e131342a3a9a1aca3278f9c0d]
I20240430 20:11:28.577966 19185 raft_consensus.cc:2781] T 15674aeb56b74210a4a2809608fd4457 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:28.578143 19185 raft_consensus.cc:686] T 15674aeb56b74210a4a2809608fd4457 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Becoming Leader. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Running, Role: LEADER
I20240430 20:11:28.578270 18985 raft_consensus.cc:348] T 6e826459c3f4437da81c9e6e09f0c22f P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:28.578502 18985 raft_consensus.cc:374] T 6e826459c3f4437da81c9e6e09f0c22f P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:28.578644 18985 raft_consensus.cc:729] T 6e826459c3f4437da81c9e6e09f0c22f P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:28.578508 19185 consensus_queue.cc:237] T 15674aeb56b74210a4a2809608fd4457 P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:28.578933 18985 consensus_queue.cc:260] T 6e826459c3f4437da81c9e6e09f0c22f P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:28.579535 18985 ts_tablet_manager.cc:1398] T 6e826459c3f4437da81c9e6e09f0c22f P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20240430 20:11:28.579592 19371 catalog_manager.cc:669] Loaded metadata for table default.Table_0b4b99c1c03a4e779046063159ce6861 [id=43d7bdc84bab4c54bac58dd1dd0f74ac]
I20240430 20:11:28.579852 18985 tablet_bootstrap.cc:492] T e2d90ff17e574f88b2061b1c117d4b7d P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:28.580183 19371 catalog_manager.cc:669] Loaded metadata for table default.Table_cd719a43207f4bf983e1b8e6d236bb16 [id=47e0b04b17834deb833a8f31c123edc5]
I20240430 20:11:28.581385 19371 catalog_manager.cc:669] Loaded metadata for table default.Table_36adeb442f1245aba00549559bb329f3 [id=4ab2e4b038744db8b2cc94c26dbcc35f]
I20240430 20:11:28.581915 18987 ts_tablet_manager.cc:1875] T 13c1107ea4d04eb78a74d3f658c72fb3 P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:28.581970 18985 tablet_bootstrap.cc:654] T e2d90ff17e574f88b2061b1c117d4b7d P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:28.584960 19371 catalog_manager.cc:669] Loaded metadata for table default.Table_b6ff64dbfc324fb199b8e6eebd51fd50 [id=4b13edffc2a4446dba14079d2b0f2ca2]
I20240430 20:11:28.585593 19371 catalog_manager.cc:669] Loaded metadata for table default.Table_0c5d4686f2ed40fb8d62c31b0166dc05 [id=4dce8a767f0541e1beb5464b2ed95d06]
I20240430 20:11:28.586841 19371 catalog_manager.cc:669] Loaded metadata for table default.Table_e233554a5e1d41389948fecaf498a856 [id=51d177e8498848b38c43e3f8d11f092c]
I20240430 20:11:28.587427 19371 catalog_manager.cc:669] Loaded metadata for table default.Table_a7add9298f1a44c6a2b2a9d73405c834 [id=5227206f44d04fa89b4842b899f33afb]
I20240430 20:11:28.589882 18987 ts_tablet_manager.cc:1888] T 13c1107ea4d04eb78a74d3f658c72fb3 P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:28.591244 18985 tablet_bootstrap.cc:492] T e2d90ff17e574f88b2061b1c117d4b7d P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:28.591459 18985 ts_tablet_manager.cc:1367] T e2d90ff17e574f88b2061b1c117d4b7d P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.003s
I20240430 20:11:28.592851 19371 catalog_manager.cc:669] Loaded metadata for table default.Table_5d3cb1a25c884804b05b07531e1b6965 [id=55e60456c89d441a8b5874a2063d52ef]
I20240430 20:11:28.593420 19371 catalog_manager.cc:669] Loaded metadata for table default.Table_4119d46f13a74c0b950aecb0744d429d [id=5a1c4a7be2fc465fac89a50854eb3322]
I20240430 20:11:28.593340 18985 raft_consensus.cc:348] T e2d90ff17e574f88b2061b1c117d4b7d P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:28.593606 18985 raft_consensus.cc:374] T e2d90ff17e574f88b2061b1c117d4b7d P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:28.593688 18985 raft_consensus.cc:729] T e2d90ff17e574f88b2061b1c117d4b7d P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:28.593950 18985 consensus_queue.cc:260] T e2d90ff17e574f88b2061b1c117d4b7d P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:28.594491 18985 ts_tablet_manager.cc:1398] T e2d90ff17e574f88b2061b1c117d4b7d P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240430 20:11:28.594843 18985 tablet_bootstrap.cc:492] T a1aeee1b02214975a376cfd36957afe8 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:28.595903 19371 catalog_manager.cc:669] Loaded metadata for table default.Table_e8c73c89623d4b49b8d7f0a01d729bdd [id=5cd10682ce52414cb5a103e4433a6a0d]
I20240430 20:11:28.596854 19371 catalog_manager.cc:669] Loaded metadata for table default.Table_6213f980d6a44e099a7a989ceb23c08a [id=608890834da843fa9af0cfc0d548029e]
I20240430 20:11:28.597358 19371 catalog_manager.cc:669] Loaded metadata for table default.Table_999c55e7bef74339b7b3ab1142599447 [id=61c8a5b99ef34a0baeeb343fef7cfd05]
I20240430 20:11:28.597843 19371 catalog_manager.cc:669] Loaded metadata for table default.Table_49e612b137f343d1ac3ec06ded7d42f3 [id=6257559b0e9545c58848a76af0c5a5f7]
I20240430 20:11:28.598917 19371 catalog_manager.cc:669] Loaded metadata for table default.Table_3c5df662747b48bf8beb6ebe4d2ea618 [id=663c8efbd7ac4a148d5f7626f1522033]
I20240430 20:11:28.599599 18987 log.cc:1196] T 13c1107ea4d04eb78a74d3f658c72fb3 P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/13c1107ea4d04eb78a74d3f658c72fb3
I20240430 20:11:28.600314 18987 ts_tablet_manager.cc:1909] T 13c1107ea4d04eb78a74d3f658c72fb3 P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:28.600562 19338 raft_consensus.cc:482] T a745334c12c545c1ba825c7552c60359 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:28.600808 19338 raft_consensus.cc:504] T a745334c12c545c1ba825c7552c60359 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:28.601049 19371 catalog_manager.cc:669] Loaded metadata for table default.Table_19a91a7f17ff456ebb145e8dd89c0940 [id=67283dd3995a44c6a7a7bc8722e4f955]
I20240430 20:11:28.601675 18987 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59818 (request call id 195) took 1318 ms. Trace:
I20240430 20:11:28.601862 19371 catalog_manager.cc:669] Loaded metadata for table default.Table_5924e4520c6c49988ccef21ca04d25d4 [id=68c1eb7252974258aabd9873e6654b10]
I20240430 20:11:28.602833 18482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a745334c12c545c1ba825c7552c60359" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:28.602821 18987 rpcz_store.cc:276] 0430 20:11:27.283022 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:27.491525 (+208503us) service_pool.cc:225] Handling call
0430 20:11:28.562997 (+1071472us) ts_tablet_manager.cc:1120] Deleting tablet 13c1107ea4d04eb78a74d3f658c72fb3
0430 20:11:28.563009 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:28.586283 (+ 23274us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:28.589737 (+ 3454us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:28.601631 (+ 11894us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1071339}
I20240430 20:11:28.603101 18482 raft_consensus.cc:2454] T a745334c12c545c1ba825c7552c60359 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 0.
I20240430 20:11:28.592590 18988 tablet_bootstrap.cc:492] T aec9c09cc0454acab6006bb3e10982b7 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:28.603310 18988 ts_tablet_manager.cc:1367] T aec9c09cc0454acab6006bb3e10982b7 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.046s user 0.008s sys 0.002s
I20240430 20:11:28.603636 19338 leader_election.cc:288] T a745334c12c545c1ba825c7552c60359 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:28.603863 18558 leader_election.cc:302] T a745334c12c545c1ba825c7552c60359 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:28.604171 19338 raft_consensus.cc:2781] T a745334c12c545c1ba825c7552c60359 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:28.604298 19338 raft_consensus.cc:482] T a745334c12c545c1ba825c7552c60359 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:28.604214 18988 raft_consensus.cc:348] T aec9c09cc0454acab6006bb3e10982b7 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:28.604393 19338 raft_consensus.cc:3037] T a745334c12c545c1ba825c7552c60359 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:28.604447 18988 raft_consensus.cc:374] T aec9c09cc0454acab6006bb3e10982b7 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:28.604542 18988 raft_consensus.cc:729] T aec9c09cc0454acab6006bb3e10982b7 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:28.604842 18988 consensus_queue.cc:260] T aec9c09cc0454acab6006bb3e10982b7 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:28.607338 19338 raft_consensus.cc:504] T a745334c12c545c1ba825c7552c60359 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:28.605000 19371 catalog_manager.cc:669] Loaded metadata for table default.Table_972f13be4dca4cd19b9a04534dc59f51 [id=69f37db4de634a3185613e40013e0ed7]
I20240430 20:11:28.608423 18760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a745334c12c545c1ba825c7552c60359" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:28.608649 18760 raft_consensus.cc:3037] T a745334c12c545c1ba825c7552c60359 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:28.608743 19371 catalog_manager.cc:669] Loaded metadata for table default.Table_5fc77d5f60ae44f4bb4ca9f7f183672c [id=6ca223ca6354449c959d85ed2ac6a9c8]
I20240430 20:11:28.608855 18988 ts_tablet_manager.cc:1398] T aec9c09cc0454acab6006bb3e10982b7 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240430 20:11:28.609197 19371 catalog_manager.cc:669] Loaded metadata for table default.Table_3f42877693b04aaa837c89b8983d5a72 [id=6d499a625cee4df2a4f40ee189cc6220]
I20240430 20:11:28.609414 18987 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59818 (request call id 200) took 1237 ms. Trace:
I20240430 20:11:28.609539 18987 rpcz_store.cc:276] 0430 20:11:27.371494 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:27.548364 (+176870us) service_pool.cc:225] Handling call
0430 20:11:28.609192 (+1060828us) ts_tablet_manager.cc:1120] Deleting tablet f5546f9edeea44d7a6c5cb954f5157ba
0430 20:11:28.609206 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:28.609352 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1054780}
I20240430 20:11:28.609848 19371 catalog_manager.cc:669] Loaded metadata for table default.Table_2ec9639eca4a47439b7859929cd705bc [id=6ffe241384bd42b7a22d41154ac59e3a]
I20240430 20:11:28.611511 18482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a745334c12c545c1ba825c7552c60359" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:28.611629 19230 raft_consensus.cc:482] T 1398d4d2305b48e08fdb12432bd6c4b0 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:28.611771 18482 raft_consensus.cc:3037] T a745334c12c545c1ba825c7552c60359 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:28.611776 19230 raft_consensus.cc:504] T 1398d4d2305b48e08fdb12432bd6c4b0 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:28.612526 19230 leader_election.cc:288] T 1398d4d2305b48e08fdb12432bd6c4b0 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:28.613121 18988 tablet_bootstrap.cc:492] T e2d90ff17e574f88b2061b1c117d4b7d P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:28.613752 18985 tablet_bootstrap.cc:654] T a1aeee1b02214975a376cfd36957afe8 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:28.614750 18988 tablet_bootstrap.cc:654] T e2d90ff17e574f88b2061b1c117d4b7d P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:28.615234 18764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1398d4d2305b48e08fdb12432bd6c4b0" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
W20240430 20:11:28.615938 18557 leader_election.cc:341] T 1398d4d2305b48e08fdb12432bd6c4b0 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:28.616066 18987 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59818 (request call id 203) took 1148 ms. Trace:
I20240430 20:11:28.616178 18987 rpcz_store.cc:276] 0430 20:11:27.467693 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:27.625042 (+157349us) service_pool.cc:225] Handling call
0430 20:11:28.615915 (+990873us) ts_tablet_manager.cc:1120] Deleting tablet 02176eee8e044a7db8cac48439e9f594
0430 20:11:28.615927 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:28.616033 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":989629}
I20240430 20:11:28.616740 18483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1398d4d2305b48e08fdb12432bd6c4b0" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:28.617218 18987 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59818 (request call id 204) took 1140 ms. Trace:
W20240430 20:11:28.617513 18558 leader_election.cc:341] T 1398d4d2305b48e08fdb12432bd6c4b0 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:28.617669 18558 leader_election.cc:302] T 1398d4d2305b48e08fdb12432bd6c4b0 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:28.617630 18987 rpcz_store.cc:276] 0430 20:11:27.476842 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:27.626312 (+149470us) service_pool.cc:225] Handling call
0430 20:11:28.617089 (+990777us) ts_tablet_manager.cc:1120] Deleting tablet 34ab40d8d91e401db299ddee7175ba28
0430 20:11:28.617100 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:28.617185 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":990509}
I20240430 20:11:28.618144 19230 raft_consensus.cc:2726] T 1398d4d2305b48e08fdb12432bd6c4b0 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:28.618189 19371 catalog_manager.cc:669] Loaded metadata for table default.Table_88c44b20f2ce4eeca7ed9583a70c7c65 [id=73caec5fb65e4cb090e7ae8c391501fd]
I20240430 20:11:28.618321 18987 tablet_replica.cc:330] T a0e8e07d363f48c698c7e2b17edc1545 P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:28.618541 19338 leader_election.cc:288] T a745334c12c545c1ba825c7552c60359 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:28.618556 18987 raft_consensus.cc:2229] T a0e8e07d363f48c698c7e2b17edc1545 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:28.618752 18987 raft_consensus.cc:2258] T a0e8e07d363f48c698c7e2b17edc1545 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:28.618887 19371 catalog_manager.cc:669] Loaded metadata for table default.Table_95220daaf6d44c08beff4ed6abdbea79 [id=75f0256acc45402d8b6809d92c38a1d8]
I20240430 20:11:28.619365 19371 catalog_manager.cc:669] Loaded metadata for table default.Table_d997b723a0dd48f6bc2d42c9bfd6bc14 [id=7737ec4807974640aa8556cb1c3ca3ef]
I20240430 20:11:28.620137 19371 catalog_manager.cc:669] Loaded metadata for table default.Table_276f62c2338b46a0aab8905430e8b61b [id=789b538b500642d89721e6028b9ac48e]
I20240430 20:11:28.620186 18760 raft_consensus.cc:2454] T a745334c12c545c1ba825c7552c60359 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
I20240430 20:11:28.620568 18987 ts_tablet_manager.cc:1875] T a0e8e07d363f48c698c7e2b17edc1545 P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:28.620761 19371 catalog_manager.cc:669] Loaded metadata for table default.Table_ee86b8fcf2064d8b8c3e327e357d3395 [id=7f3a15a442984827a7521d5d1c98dc3d]
I20240430 20:11:28.621318 18482 raft_consensus.cc:2454] T a745334c12c545c1ba825c7552c60359 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
I20240430 20:11:28.621369 19371 catalog_manager.cc:669] Loaded metadata for table default.Table_c1669746f9114bf7b37a6f6a68fed20e [id=7f602584cef9486eb7d9b4389be3f43e]
I20240430 20:11:28.620802 18557 leader_election.cc:302] T a745334c12c545c1ba825c7552c60359 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:28.622241 19230 raft_consensus.cc:2781] T a745334c12c545c1ba825c7552c60359 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:28.623569 19371 catalog_manager.cc:669] Loaded metadata for table default.Table_327b9bc7f9c64a128209a7a24d2a410a [id=86df5d161d8641c89070e7eed9a39360]
I20240430 20:11:28.624784 19371 catalog_manager.cc:669] Loaded metadata for table default.Table_8af52c0f1d404b42bc53287f64e20c2d [id=8e590e0e404344658b5cee90132d91d1]
I20240430 20:11:28.625010 18763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a745334c12c545c1ba825c7552c60359" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:28.626520 19230 raft_consensus.cc:686] T a745334c12c545c1ba825c7552c60359 P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Becoming Leader. State: Replica: 5e1c6397f09747f091534faf93542710, State: Running, Role: LEADER
I20240430 20:11:28.626713 19371 catalog_manager.cc:669] Loaded metadata for table default.Table_4bce1dae3391497ebea5f1b536700d62 [id=8f0cb0ef65d84fb8964209dd501c0b5c]
I20240430 20:11:28.627014 19230 consensus_queue.cc:237] T a745334c12c545c1ba825c7552c60359 P 5e1c6397f09747f091534faf93542710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:28.627343 19371 catalog_manager.cc:669] Loaded metadata for table default.Table_6f2fd99337344da2ac5a0c83d9b56bb8 [id=8f1ad1f8d0224b3daa5d84dc6916227b]
I20240430 20:11:28.625263 18763 raft_consensus.cc:2362] T a745334c12c545c1ba825c7552c60359 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1. Re-sending same reply.
I20240430 20:11:28.628305 19371 catalog_manager.cc:669] Loaded metadata for table default.Table_0443359dbeff437fb6cffb0db33cbb30 [id=93c44abdf6b147228224388398cd7aeb]
I20240430 20:11:28.628675 18983 tablet_bootstrap.cc:492] T a1aeee1b02214975a376cfd36957afe8 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:28.628840 18983 ts_tablet_manager.cc:1367] T a1aeee1b02214975a376cfd36957afe8 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.056s user 0.008s sys 0.008s
I20240430 20:11:28.628968 19371 catalog_manager.cc:669] Loaded metadata for table default.Table_8ddf4569d2f245469f55cb9fef2f5fb1 [id=955d833ff8f14bf6957abadf9ef669e2]
I20240430 20:11:28.629499 19371 catalog_manager.cc:669] Loaded metadata for table default.Table_e3a0afbced214062a8b026abe2320f3e [id=97d8a36ddfae4bb5b99c04a7a08544af]
I20240430 20:11:28.629565 18983 raft_consensus.cc:348] T a1aeee1b02214975a376cfd36957afe8 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:28.629776 18983 raft_consensus.cc:374] T a1aeee1b02214975a376cfd36957afe8 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:28.629853 18983 raft_consensus.cc:729] T a1aeee1b02214975a376cfd36957afe8 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:28.630110 18983 consensus_queue.cc:260] T a1aeee1b02214975a376cfd36957afe8 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:28.630666 18983 ts_tablet_manager.cc:1398] T a1aeee1b02214975a376cfd36957afe8 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240430 20:11:28.631064 18983 tablet_bootstrap.cc:492] T 77285afcafe04a829e485998bbbe00f0 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:28.637195 18983 tablet_bootstrap.cc:654] T 77285afcafe04a829e485998bbbe00f0 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:28.639016 18985 tablet_bootstrap.cc:492] T a1aeee1b02214975a376cfd36957afe8 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:28.639204 18985 ts_tablet_manager.cc:1367] T a1aeee1b02214975a376cfd36957afe8 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.044s user 0.011s sys 0.000s
I20240430 20:11:28.640023 18985 raft_consensus.cc:348] T a1aeee1b02214975a376cfd36957afe8 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:28.640250 18985 raft_consensus.cc:374] T a1aeee1b02214975a376cfd36957afe8 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:28.640336 18985 raft_consensus.cc:729] T a1aeee1b02214975a376cfd36957afe8 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:28.640566 18985 consensus_queue.cc:260] T a1aeee1b02214975a376cfd36957afe8 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:28.641207 18985 ts_tablet_manager.cc:1398] T a1aeee1b02214975a376cfd36957afe8 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240430 20:11:28.641206 19371 catalog_manager.cc:669] Loaded metadata for table default.Table_a8d5ac73c6d047698d87b3d1ba34e66f [id=9c80193cde2346c982cf84ae9612e2f7]
I20240430 20:11:28.642458 19371 catalog_manager.cc:669] Loaded metadata for table default.Table_0ab5b2f09fa7468bb89ab703f6d3d25c [id=a5d580cffa2447d78fc9f2c8e31264f1]
I20240430 20:11:28.642923 18987 ts_tablet_manager.cc:1888] T a0e8e07d363f48c698c7e2b17edc1545 P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:28.643082 18987 log.cc:1196] T a0e8e07d363f48c698c7e2b17edc1545 P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/a0e8e07d363f48c698c7e2b17edc1545
I20240430 20:11:28.643390 18987 ts_tablet_manager.cc:1909] T a0e8e07d363f48c698c7e2b17edc1545 P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:28.643968 18985 tablet_bootstrap.cc:492] T aacabead3c874e77a47d2ecf19dbfa82 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:28.645165 18987 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59818 (request call id 205) took 1162 ms. Trace:
I20240430 20:11:28.645267 18987 rpcz_store.cc:276] 0430 20:11:27.482570 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:27.626610 (+144040us) service_pool.cc:225] Handling call
0430 20:11:28.618258 (+991648us) ts_tablet_manager.cc:1120] Deleting tablet a0e8e07d363f48c698c7e2b17edc1545
0430 20:11:28.618267 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:28.624232 (+ 5965us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:28.642790 (+ 18558us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:28.645105 (+ 2315us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":990922}
I20240430 20:11:28.645643 18985 tablet_bootstrap.cc:654] T aacabead3c874e77a47d2ecf19dbfa82 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:28.646903 18987 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59818 (request call id 206) took 1159 ms. Trace:
I20240430 20:11:28.646992 18987 rpcz_store.cc:276] 0430 20:11:27.487100 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:27.627356 (+140256us) service_pool.cc:225] Handling call
0430 20:11:28.646729 (+1019373us) ts_tablet_manager.cc:1120] Deleting tablet 75ebf051383e42529dcb5f7dd0770a4b
0430 20:11:28.646740 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:28.646851 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1019152}
I20240430 20:11:28.648849 18987 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59818 (request call id 214) took 1041 ms. Trace:
I20240430 20:11:28.648952 18987 rpcz_store.cc:276] 0430 20:11:27.607306 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:27.698605 (+ 91299us) service_pool.cc:225] Handling call
0430 20:11:28.648665 (+950060us) ts_tablet_manager.cc:1120] Deleting tablet 47c37876694c4a4abb1091c631dab7a5
0430 20:11:28.648677 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:28.648804 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":949801}
I20240430 20:11:28.652732 18987 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59818 (request call id 213) took 1046 ms. Trace:
I20240430 20:11:28.652842 18987 rpcz_store.cc:276] 0430 20:11:27.606056 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:27.695832 (+ 89776us) service_pool.cc:225] Handling call
0430 20:11:28.652561 (+956729us) ts_tablet_manager.cc:1120] Deleting tablet 4e69ca12bd10494fb013614741bc166c
0430 20:11:28.652574 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:28.652680 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":898806}
I20240430 20:11:28.655753 18988 tablet_bootstrap.cc:492] T e2d90ff17e574f88b2061b1c117d4b7d P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:28.655922 18988 ts_tablet_manager.cc:1367] T e2d90ff17e574f88b2061b1c117d4b7d P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.043s user 0.003s sys 0.008s
I20240430 20:11:28.656675 18988 raft_consensus.cc:348] T e2d90ff17e574f88b2061b1c117d4b7d P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:28.656883 18988 raft_consensus.cc:374] T e2d90ff17e574f88b2061b1c117d4b7d P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:28.656940 18988 raft_consensus.cc:729] T e2d90ff17e574f88b2061b1c117d4b7d P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:28.657213 18988 consensus_queue.cc:260] T e2d90ff17e574f88b2061b1c117d4b7d P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:28.661562 18988 ts_tablet_manager.cc:1398] T e2d90ff17e574f88b2061b1c117d4b7d P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240430 20:11:28.664588 18988 tablet_bootstrap.cc:492] T 1398d4d2305b48e08fdb12432bd6c4b0 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:28.665211 19371 catalog_manager.cc:669] Loaded metadata for table default.Table_388d774462294cab8342c6de331f9712 [id=aca48e01301e4f5f9562df8c87829b9d]
I20240430 20:11:28.665342 18985 tablet_bootstrap.cc:492] T aacabead3c874e77a47d2ecf19dbfa82 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:28.665524 18985 ts_tablet_manager.cc:1367] T aacabead3c874e77a47d2ecf19dbfa82 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.022s user 0.006s sys 0.003s
I20240430 20:11:28.665953 19371 catalog_manager.cc:669] Loaded metadata for table default.Table_2e9c7c683df84a41942d31600a265641 [id=aca6fb02776a40bbac312191adbdc136]
I20240430 20:11:28.666327 18988 tablet_bootstrap.cc:654] T 1398d4d2305b48e08fdb12432bd6c4b0 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:28.666462 18985 raft_consensus.cc:348] T aacabead3c874e77a47d2ecf19dbfa82 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:28.666731 18985 raft_consensus.cc:374] T aacabead3c874e77a47d2ecf19dbfa82 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:28.666806 18985 raft_consensus.cc:729] T aacabead3c874e77a47d2ecf19dbfa82 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:28.667130 18985 consensus_queue.cc:260] T aacabead3c874e77a47d2ecf19dbfa82 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:28.667768 18985 ts_tablet_manager.cc:1398] T aacabead3c874e77a47d2ecf19dbfa82 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240430 20:11:28.668164 18985 tablet_bootstrap.cc:492] T bd11c0335a58448da56533c3f2e785a9 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:28.670178 19185 raft_consensus.cc:482] T aec9c09cc0454acab6006bb3e10982b7 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:28.670351 19185 raft_consensus.cc:504] T aec9c09cc0454acab6006bb3e10982b7 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:28.671036 19185 leader_election.cc:288] T aec9c09cc0454acab6006bb3e10982b7 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:28.671705 18763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aec9c09cc0454acab6006bb3e10982b7" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:28.671948 18763 raft_consensus.cc:2454] T aec9c09cc0454acab6006bb3e10982b7 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 0.
I20240430 20:11:28.673355 18985 tablet_bootstrap.cc:654] T bd11c0335a58448da56533c3f2e785a9 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:28.673997 18417 leader_election.cc:302] T aec9c09cc0454acab6006bb3e10982b7 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters:
I20240430 20:11:28.675026 18623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aec9c09cc0454acab6006bb3e10982b7" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:28.675268 18623 raft_consensus.cc:2454] T aec9c09cc0454acab6006bb3e10982b7 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 0.
I20240430 20:11:28.675433 19185 raft_consensus.cc:2781] T aec9c09cc0454acab6006bb3e10982b7 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:28.675539 19185 raft_consensus.cc:482] T aec9c09cc0454acab6006bb3e10982b7 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:28.675678 19185 raft_consensus.cc:3037] T aec9c09cc0454acab6006bb3e10982b7 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:28.678481 19185 raft_consensus.cc:504] T aec9c09cc0454acab6006bb3e10982b7 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:28.683987 18623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aec9c09cc0454acab6006bb3e10982b7" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:28.684312 18623 raft_consensus.cc:3037] T aec9c09cc0454acab6006bb3e10982b7 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:28.684760 18988 tablet_bootstrap.cc:492] T 1398d4d2305b48e08fdb12432bd6c4b0 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:28.685034 18988 ts_tablet_manager.cc:1367] T 1398d4d2305b48e08fdb12432bd6c4b0 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.021s user 0.000s sys 0.007s
I20240430 20:11:28.685946 18988 raft_consensus.cc:348] T 1398d4d2305b48e08fdb12432bd6c4b0 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:28.686178 18988 raft_consensus.cc:374] T 1398d4d2305b48e08fdb12432bd6c4b0 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:28.686269 18988 raft_consensus.cc:729] T 1398d4d2305b48e08fdb12432bd6c4b0 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:28.686584 18988 consensus_queue.cc:260] T 1398d4d2305b48e08fdb12432bd6c4b0 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
W20240430 20:11:28.688093 18417 connection.cc:535] server connection from 127.0.0.1:41678 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240430 20:11:28.688585 18988 ts_tablet_manager.cc:1398] T 1398d4d2305b48e08fdb12432bd6c4b0 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240430 20:11:28.688908 18988 tablet_bootstrap.cc:492] T a1aeee1b02214975a376cfd36957afe8 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:28.690670 18623 raft_consensus.cc:2454] T aec9c09cc0454acab6006bb3e10982b7 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 1.
W20240430 20:11:28.693817 18999 connection.cc:535] client connection to 127.17.177.62:36441 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240430 20:11:28.694130 18999 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240430 20:11:28.690560 18988 tablet_bootstrap.cc:654] T a1aeee1b02214975a376cfd36957afe8 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:28.695320 18415 leader_election.cc:302] T aec9c09cc0454acab6006bb3e10982b7 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:28.695732 19344 raft_consensus.cc:2781] T aec9c09cc0454acab6006bb3e10982b7 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:28.695915 19344 raft_consensus.cc:686] T aec9c09cc0454acab6006bb3e10982b7 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Becoming Leader. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Running, Role: LEADER
I20240430 20:11:28.696331 19344 consensus_queue.cc:237] T aec9c09cc0454acab6006bb3e10982b7 P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
W20240430 20:11:28.698714 18818 connection.cc:535] client connection to 127.17.177.62:36441 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240430 20:11:28.699513 18818 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.17.177.62:36441)
W20240430 20:11:28.699826 18818 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.17.177.62:36441)
W20240430 20:11:28.700443 18818 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.17.177.62:36441)
W20240430 20:11:28.701452 18818 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.17.177.62:36441)
I20240430 20:11:28.701453 18763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aec9c09cc0454acab6006bb3e10982b7" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:28.701709 18763 raft_consensus.cc:3037] T aec9c09cc0454acab6006bb3e10982b7 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:28.704759 18763 raft_consensus.cc:2454] T aec9c09cc0454acab6006bb3e10982b7 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 1.
I20240430 20:11:28.705981 19185 leader_election.cc:288] T aec9c09cc0454acab6006bb3e10982b7 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Requested vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:28.711145 19185 raft_consensus.cc:482] T c5fd0730133c472299c93f4b0830e527 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:28.711582 19185 raft_consensus.cc:504] T c5fd0730133c472299c93f4b0830e527 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:28.713460 18623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd0730133c472299c93f4b0830e527" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:28.713927 18623 raft_consensus.cc:2454] T c5fd0730133c472299c93f4b0830e527 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 0.
I20240430 20:11:28.714629 18415 leader_election.cc:302] T c5fd0730133c472299c93f4b0830e527 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:28.715099 19186 raft_consensus.cc:2781] T c5fd0730133c472299c93f4b0830e527 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:28.715214 19186 raft_consensus.cc:482] T c5fd0730133c472299c93f4b0830e527 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:28.715349 19186 raft_consensus.cc:3037] T c5fd0730133c472299c93f4b0830e527 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
W20240430 20:11:28.719583 18557 connection.cc:535] server connection from 127.0.0.1:40202 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240430 20:11:28.720983 19186 raft_consensus.cc:504] T c5fd0730133c472299c93f4b0830e527 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
W20240430 20:11:28.721611 18697 connection.cc:535] server connection from 127.0.0.1:59818 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240430 20:11:28.723712 19397 negotiation.cc:337] Failed RPC negotiation. Trace:
0430 20:11:28.720213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.177.62:36441 (local address 127.17.177.2:60791)
0430 20:11:28.720895 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0430 20:11:28.720920 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0430 20:11:28.721087 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0430 20:11:28.722453 (+ 1366us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.177.62:36441: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":371,"threads_started":1}
I20240430 20:11:28.724974 18620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd0730133c472299c93f4b0830e527" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:28.725309 18620 raft_consensus.cc:3037] T c5fd0730133c472299c93f4b0830e527 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:28.729390 18620 raft_consensus.cc:2454] T c5fd0730133c472299c93f4b0830e527 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 1.
I20240430 20:11:28.731580 18763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd0730133c472299c93f4b0830e527" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
W20240430 20:11:28.731946 19396 negotiation.cc:337] Failed RPC negotiation. Trace:
0430 20:11:28.712682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.177.62:36441 (local address 127.17.177.3:34757)
0430 20:11:28.731317 (+ 18635us) negotiation.cc:107] Waiting for socket to connect
0430 20:11:28.731538 (+ 221us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.177.62:36441: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":18225,"threads_started":1}
I20240430 20:11:28.732031 18763 raft_consensus.cc:3037] T c5fd0730133c472299c93f4b0830e527 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:28.723738 19186 leader_election.cc:288] T c5fd0730133c472299c93f4b0830e527 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Requested vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
W20240430 20:11:28.737712 19395 negotiation.cc:337] Failed RPC negotiation. Trace:
0430 20:11:28.704704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.177.62:36441 (local address 127.0.0.1:54540)
0430 20:11:28.735443 (+ 30739us) negotiation.cc:107] Waiting for socket to connect
0430 20:11:28.735613 (+ 170us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.177.62:36441: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":30483,"threads_started":1}
I20240430 20:11:28.742947 18760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd0730133c472299c93f4b0830e527" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:28.743000 18988 tablet_bootstrap.cc:492] T a1aeee1b02214975a376cfd36957afe8 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:28.744221 18988 ts_tablet_manager.cc:1367] T a1aeee1b02214975a376cfd36957afe8 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.055s user 0.012s sys 0.006s
I20240430 20:11:28.744493 19185 leader_election.cc:288] T c5fd0730133c472299c93f4b0830e527 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:28.745159 18988 raft_consensus.cc:348] T a1aeee1b02214975a376cfd36957afe8 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:28.745424 18988 raft_consensus.cc:374] T a1aeee1b02214975a376cfd36957afe8 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:28.745515 18988 raft_consensus.cc:729] T a1aeee1b02214975a376cfd36957afe8 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:28.747259 18763 raft_consensus.cc:2454] T c5fd0730133c472299c93f4b0830e527 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 1.
I20240430 20:11:28.749986 18417 leader_election.cc:302] T c5fd0730133c472299c93f4b0830e527 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters:
I20240430 20:11:28.750468 19185 raft_consensus.cc:2781] T c5fd0730133c472299c93f4b0830e527 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:28.750706 19185 raft_consensus.cc:686] T c5fd0730133c472299c93f4b0830e527 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Becoming Leader. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Running, Role: LEADER
I20240430 20:11:28.751075 19185 consensus_queue.cc:237] T c5fd0730133c472299c93f4b0830e527 P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:28.751492 18988 consensus_queue.cc:260] T a1aeee1b02214975a376cfd36957afe8 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:28.751704 19371 catalog_manager.cc:669] Loaded metadata for table default.Table_16b082d140234e1c9a1e70654839b4cd [id=c0d8dc87d1f9426a85f6176f161595a1]
I20240430 20:11:28.752828 18988 ts_tablet_manager.cc:1398] T a1aeee1b02214975a376cfd36957afe8 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240430 20:11:28.753672 18988 tablet_bootstrap.cc:492] T aacabead3c874e77a47d2ecf19dbfa82 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:28.755482 18988 tablet_bootstrap.cc:654] T aacabead3c874e77a47d2ecf19dbfa82 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:28.757314 18983 tablet_bootstrap.cc:492] T 77285afcafe04a829e485998bbbe00f0 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:28.757491 18983 ts_tablet_manager.cc:1367] T 77285afcafe04a829e485998bbbe00f0 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.127s user 0.023s sys 0.009s
I20240430 20:11:28.758425 18983 raft_consensus.cc:348] T 77285afcafe04a829e485998bbbe00f0 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:28.758683 18983 raft_consensus.cc:374] T 77285afcafe04a829e485998bbbe00f0 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:28.758783 18983 raft_consensus.cc:729] T 77285afcafe04a829e485998bbbe00f0 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:28.759083 18983 consensus_queue.cc:260] T 77285afcafe04a829e485998bbbe00f0 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:28.759719 18983 ts_tablet_manager.cc:1398] T 77285afcafe04a829e485998bbbe00f0 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240430 20:11:28.761234 18983 tablet_bootstrap.cc:492] T 0b4d9e860d7c45fd91d7a566605c9283 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
W20240430 20:11:28.763043 18999 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.177.62:36441, user_credentials={real_user=slave}} blocked reactor thread for 68948.4us
I20240430 20:11:28.763579 18983 tablet_bootstrap.cc:654] T 0b4d9e860d7c45fd91d7a566605c9283 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:28.777048 19344 raft_consensus.cc:482] T e2d90ff17e574f88b2061b1c117d4b7d P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:28.777348 19344 raft_consensus.cc:504] T e2d90ff17e574f88b2061b1c117d4b7d P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:28.778225 19371 catalog_manager.cc:669] Loaded metadata for table default.Table_db56d63aef494592b69246a0c13006f2 [id=cde04507ee1d464f9fffc44176da8d16]
I20240430 20:11:28.778930 19344 leader_election.cc:288] T e2d90ff17e574f88b2061b1c117d4b7d P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:28.779501 18988 tablet_bootstrap.cc:492] T aacabead3c874e77a47d2ecf19dbfa82 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:28.779709 18988 ts_tablet_manager.cc:1367] T aacabead3c874e77a47d2ecf19dbfa82 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.026s user 0.000s sys 0.006s
I20240430 20:11:28.779929 19371 catalog_manager.cc:669] Loaded metadata for table default.Table_8c7f3f5e74f64d409fbecfce64acf648 [id=ce9911e0a91949b1997ee47321800255]
I20240430 20:11:28.780676 18988 raft_consensus.cc:348] T aacabead3c874e77a47d2ecf19dbfa82 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:28.780918 18988 raft_consensus.cc:374] T aacabead3c874e77a47d2ecf19dbfa82 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:28.781021 18988 raft_consensus.cc:729] T aacabead3c874e77a47d2ecf19dbfa82 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:28.779526 18763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2d90ff17e574f88b2061b1c117d4b7d" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:28.781442 18763 raft_consensus.cc:2454] T e2d90ff17e574f88b2061b1c117d4b7d P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 0.
I20240430 20:11:28.781952 18417 leader_election.cc:302] T e2d90ff17e574f88b2061b1c117d4b7d P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters:
I20240430 20:11:28.782297 19344 raft_consensus.cc:2781] T e2d90ff17e574f88b2061b1c117d4b7d P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:28.782393 19344 raft_consensus.cc:482] T e2d90ff17e574f88b2061b1c117d4b7d P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:28.782536 19344 raft_consensus.cc:3037] T e2d90ff17e574f88b2061b1c117d4b7d P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:28.783130 19371 catalog_manager.cc:669] Loaded metadata for table default.Table_7a9f9a8c27bb4392acddd2a7ea3cc3b2 [id=cfad6f932073488b8097ef7381bc9fd1]
I20240430 20:11:28.785135 19371 catalog_manager.cc:669] Loaded metadata for table default.Table_44426457fd7045dfa1c149f0a5ed5129 [id=d5cea90cbc2c48df9d8b80da1eccc949]
I20240430 20:11:28.785708 19371 catalog_manager.cc:669] Loaded metadata for table default.Table_d82824c897a94dd0ba45955f9473b626 [id=d794640d21a843a185038673f7eb2b00]
W20240430 20:11:28.782284 18999 leader_election.cc:334] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 88b3f98793354c7aaa5c106e8ec31ed8 (127.17.177.62:36441): Network error: Client connection negotiation failed: client connection to 127.17.177.62:36441: connect: Connection refused (error 111)
I20240430 20:11:28.785974 19344 raft_consensus.cc:504] T e2d90ff17e574f88b2061b1c117d4b7d P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
W20240430 20:11:28.786502 18999 leader_election.cc:334] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 88b3f98793354c7aaa5c106e8ec31ed8 (127.17.177.62:36441): Network error: Client connection negotiation failed: client connection to 127.17.177.62:36441: connect: Connection refused (error 111)
I20240430 20:11:28.787192 18763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2d90ff17e574f88b2061b1c117d4b7d" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:28.787429 18763 raft_consensus.cc:3037] T e2d90ff17e574f88b2061b1c117d4b7d P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:28.788059 19371 catalog_manager.cc:669] Loaded metadata for table default.Table_873c21f6dffd4d47a742b3699ab85268 [id=d7e2c4300ff94fa2acc65d6d6c932c65]
I20240430 20:11:28.788619 19371 catalog_manager.cc:669] Loaded metadata for table default.Table_fda07fd7279c484db2bee67e8c859855 [id=da9ad991953a40c4b77b31317597ad1d]
I20240430 20:11:28.789248 19371 catalog_manager.cc:669] Loaded metadata for table default.Table_da5216d025844afda5a83ada555bc120 [id=dc55b25c3ca14863a21042e805e3bf9a]
I20240430 20:11:28.789294 18623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2d90ff17e574f88b2061b1c117d4b7d" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:28.789672 18623 raft_consensus.cc:2454] T e2d90ff17e574f88b2061b1c117d4b7d P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 0.
I20240430 20:11:28.790477 18763 raft_consensus.cc:2454] T e2d90ff17e574f88b2061b1c117d4b7d P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 1.
I20240430 20:11:28.791064 19344 leader_election.cc:288] T e2d90ff17e574f88b2061b1c117d4b7d P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Requested vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:28.791285 19371 catalog_manager.cc:669] Loaded metadata for table default.Table_7b72c19b4765456dba543151abaf1d44 [id=ec7831570fea4ff2944bc6304fc17bc9]
I20240430 20:11:28.792213 18983 tablet_bootstrap.cc:492] T 0b4d9e860d7c45fd91d7a566605c9283 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:28.792419 18983 ts_tablet_manager.cc:1367] T 0b4d9e860d7c45fd91d7a566605c9283 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.031s user 0.005s sys 0.000s
I20240430 20:11:28.792315 18623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2d90ff17e574f88b2061b1c117d4b7d" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:28.792673 18623 raft_consensus.cc:3037] T e2d90ff17e574f88b2061b1c117d4b7d P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:28.792676 19371 catalog_manager.cc:669] Loaded metadata for table default.Table_49a012a7c4c8457fadb0a8f0fc9654c6 [id=eecb2f8eeef547c5ab1abe6a2430e592]
I20240430 20:11:28.793334 18983 raft_consensus.cc:348] T 0b4d9e860d7c45fd91d7a566605c9283 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:28.793617 18983 raft_consensus.cc:374] T 0b4d9e860d7c45fd91d7a566605c9283 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:28.793730 18983 raft_consensus.cc:729] T 0b4d9e860d7c45fd91d7a566605c9283 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:28.794029 19371 catalog_manager.cc:669] Loaded metadata for table default.Table_276bbdc7cfe542d1a2936485a2aad1ec [id=f146a15e6e8a4acbb4c734c1c863da57]
I20240430 20:11:28.794019 18983 consensus_queue.cc:260] T 0b4d9e860d7c45fd91d7a566605c9283 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:28.794656 18983 ts_tablet_manager.cc:1398] T 0b4d9e860d7c45fd91d7a566605c9283 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240430 20:11:28.795068 18983 tablet_bootstrap.cc:492] T 5f56ac2afd444d63a6bfd96988464bbb P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:28.795483 19371 catalog_manager.cc:669] Loaded metadata for table default.Table_f5a3451cd2284390be4357d771c03390 [id=f4ba9536e9c141deb68f4f62b7e8fa70]
I20240430 20:11:28.796605 19371 catalog_manager.cc:669] Loaded metadata for table default.Table_7199d7c36d3f45f79cf9e4e2b4145592 [id=f6aef8f6a9164dd78f49e03f723b45a9]
I20240430 20:11:28.796829 19180 raft_consensus.cc:482] T aacabead3c874e77a47d2ecf19dbfa82 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:28.797091 19180 raft_consensus.cc:504] T aacabead3c874e77a47d2ecf19dbfa82 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:28.797537 18417 leader_election.cc:302] T e2d90ff17e574f88b2061b1c117d4b7d P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters:
I20240430 20:11:28.790766 18988 consensus_queue.cc:260] T aacabead3c874e77a47d2ecf19dbfa82 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:28.797948 19180 leader_election.cc:288] T aacabead3c874e77a47d2ecf19dbfa82 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:28.798450 18988 ts_tablet_manager.cc:1398] T aacabead3c874e77a47d2ecf19dbfa82 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.019s user 0.000s sys 0.002s
I20240430 20:11:28.798894 18988 tablet_bootstrap.cc:492] T bd11c0335a58448da56533c3f2e785a9 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:28.798874 18482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aacabead3c874e77a47d2ecf19dbfa82" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:28.799182 18482 raft_consensus.cc:2454] T aacabead3c874e77a47d2ecf19dbfa82 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 0.
I20240430 20:11:28.799335 19344 raft_consensus.cc:2781] T e2d90ff17e574f88b2061b1c117d4b7d P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:28.799511 19344 raft_consensus.cc:686] T e2d90ff17e574f88b2061b1c117d4b7d P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Becoming Leader. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Running, Role: LEADER
I20240430 20:11:28.799733 18558 leader_election.cc:302] T aacabead3c874e77a47d2ecf19dbfa82 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:28.800143 19180 raft_consensus.cc:2781] T aacabead3c874e77a47d2ecf19dbfa82 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:28.800240 19180 raft_consensus.cc:482] T aacabead3c874e77a47d2ecf19dbfa82 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:28.800334 19180 raft_consensus.cc:3037] T aacabead3c874e77a47d2ecf19dbfa82 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:28.799777 19344 consensus_queue.cc:237] T e2d90ff17e574f88b2061b1c117d4b7d P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:28.801479 18760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aacabead3c874e77a47d2ecf19dbfa82" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:28.801702 18760 raft_consensus.cc:2454] T aacabead3c874e77a47d2ecf19dbfa82 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 0.
I20240430 20:11:28.802034 18988 tablet_bootstrap.cc:654] T bd11c0335a58448da56533c3f2e785a9 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:28.796923 18983 tablet_bootstrap.cc:654] T 5f56ac2afd444d63a6bfd96988464bbb P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:28.797645 19371 catalog_manager.cc:669] Loaded metadata for table default.Table_fe1773cf77654e2a86e95444437b7201 [id=fe2222c2b62f4360a5ade622ae8b23e5]
I20240430 20:11:28.803689 19180 raft_consensus.cc:504] T aacabead3c874e77a47d2ecf19dbfa82 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:28.805522 18482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aacabead3c874e77a47d2ecf19dbfa82" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:28.805809 18482 raft_consensus.cc:3037] T aacabead3c874e77a47d2ecf19dbfa82 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:28.806134 18763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aacabead3c874e77a47d2ecf19dbfa82" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:28.806771 18763 raft_consensus.cc:3037] T aacabead3c874e77a47d2ecf19dbfa82 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:28.807233 19371 catalog_manager.cc:669] Loaded metadata for table default.Table_210b7eaff7144b2eb7db1c3b5f7d9505 [id=ff86c0357c164dd0991098c416087773]
I20240430 20:11:28.812606 19180 leader_election.cc:288] T aacabead3c874e77a47d2ecf19dbfa82 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:28.813203 18482 raft_consensus.cc:2454] T aacabead3c874e77a47d2ecf19dbfa82 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
I20240430 20:11:28.812675 18623 raft_consensus.cc:2454] T e2d90ff17e574f88b2061b1c117d4b7d P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 1.
I20240430 20:11:28.814385 18558 leader_election.cc:302] T aacabead3c874e77a47d2ecf19dbfa82 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:28.816785 19180 raft_consensus.cc:2781] T aacabead3c874e77a47d2ecf19dbfa82 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:28.816994 19180 raft_consensus.cc:686] T aacabead3c874e77a47d2ecf19dbfa82 P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Becoming Leader. State: Replica: 5e1c6397f09747f091534faf93542710, State: Running, Role: LEADER
I20240430 20:11:28.817332 19180 consensus_queue.cc:237] T aacabead3c874e77a47d2ecf19dbfa82 P 5e1c6397f09747f091534faf93542710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:28.820302 18763 raft_consensus.cc:2454] T aacabead3c874e77a47d2ecf19dbfa82 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
I20240430 20:11:28.821486 18985 tablet_bootstrap.cc:492] T bd11c0335a58448da56533c3f2e785a9 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:28.821617 19344 raft_consensus.cc:482] T 372b135dd01f44f98cd713ca8857ba61 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:28.821657 18985 ts_tablet_manager.cc:1367] T bd11c0335a58448da56533c3f2e785a9 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.154s user 0.023s sys 0.055s
I20240430 20:11:28.821781 19344 raft_consensus.cc:504] T 372b135dd01f44f98cd713ca8857ba61 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:28.822861 18760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "372b135dd01f44f98cd713ca8857ba61" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:28.822877 18985 raft_consensus.cc:348] T bd11c0335a58448da56533c3f2e785a9 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:28.823113 18985 raft_consensus.cc:374] T bd11c0335a58448da56533c3f2e785a9 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:28.823206 18985 raft_consensus.cc:729] T bd11c0335a58448da56533c3f2e785a9 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:28.814850 18983 tablet_bootstrap.cc:492] T 5f56ac2afd444d63a6bfd96988464bbb P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:28.823573 18985 consensus_queue.cc:260] T bd11c0335a58448da56533c3f2e785a9 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:28.824468 18985 ts_tablet_manager.cc:1398] T bd11c0335a58448da56533c3f2e785a9 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240430 20:11:28.824767 18985 tablet_bootstrap.cc:492] T 02176eee8e044a7db8cac48439e9f594 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:28.827736 18985 tablet_bootstrap.cc:654] T 02176eee8e044a7db8cac48439e9f594 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:28.830374 18983 ts_tablet_manager.cc:1367] T 5f56ac2afd444d63a6bfd96988464bbb P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.029s user 0.004s sys 0.001s
I20240430 20:11:28.830757 19344 leader_election.cc:288] T 372b135dd01f44f98cd713ca8857ba61 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:28.831902 18983 raft_consensus.cc:348] T 5f56ac2afd444d63a6bfd96988464bbb P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:28.832322 18983 raft_consensus.cc:374] T 5f56ac2afd444d63a6bfd96988464bbb P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:28.832480 18983 raft_consensus.cc:729] T 5f56ac2afd444d63a6bfd96988464bbb P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:28.832782 18983 consensus_queue.cc:260] T 5f56ac2afd444d63a6bfd96988464bbb P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:28.833082 19371 tablet_loader.cc:96] loaded metadata for tablet 01436e5b6232494babbb414342f03090 (table default.Table_a8d5ac73c6d047698d87b3d1ba34e66f [id=9c80193cde2346c982cf84ae9612e2f7])
I20240430 20:11:28.834410 18623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "372b135dd01f44f98cd713ca8857ba61" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:28.834767 18623 raft_consensus.cc:2379] T 372b135dd01f44f98cd713ca8857ba61 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2efb542aa55e439b99ccb98d1c5c884a in current term 1: Already voted for candidate 3c51c33d5616482cafb68e26688820f4 in this term.
I20240430 20:11:28.835311 18415 leader_election.cc:302] T 372b135dd01f44f98cd713ca8857ba61 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a; no voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710
I20240430 20:11:28.835443 19371 tablet_loader.cc:96] loaded metadata for tablet 09d92a4a6ebe44cbb3553ed660f143d0 (table default.Table_3b327aece0ea4ec4b882f2264e3f5f0c [id=1fb10bddcfe94ec488ecc71e43ca339d])
I20240430 20:11:28.836220 19371 tablet_loader.cc:96] loaded metadata for tablet 0a9066d895744c5bb7d9532b2940603c (table default.Table_7b72c19b4765456dba543151abaf1d44 [id=ec7831570fea4ff2944bc6304fc17bc9])
I20240430 20:11:28.836881 19371 tablet_loader.cc:96] loaded metadata for tablet 0b4d9e860d7c45fd91d7a566605c9283 (table default.Table_44426457fd7045dfa1c149f0a5ed5129 [id=d5cea90cbc2c48df9d8b80da1eccc949])
I20240430 20:11:28.838832 19371 tablet_loader.cc:96] loaded metadata for tablet 0e503a898406412ab977d3e048ea4b5d (table default.Table_fe1773cf77654e2a86e95444437b7201 [id=fe2222c2b62f4360a5ade622ae8b23e5])
I20240430 20:11:28.839570 19371 tablet_loader.cc:96] loaded metadata for tablet 0eb4f3e7ece6496293e01b63620d644b (table default.Table_b6ff64dbfc324fb199b8e6eebd51fd50 [id=4b13edffc2a4446dba14079d2b0f2ca2])
I20240430 20:11:28.840016 19344 raft_consensus.cc:3037] T 372b135dd01f44f98cd713ca8857ba61 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:28.840204 19185 raft_consensus.cc:482] T f5236574672b477a9026e91adcaf70ec P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:28.840745 19185 raft_consensus.cc:504] T f5236574672b477a9026e91adcaf70ec P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:28.841168 19371 tablet_loader.cc:96] loaded metadata for tablet 12c6e131fa904d23865b8673b58a7e33 (table default.Table_3106434588bd4241a314064d6a8749c3 [id=19f4e6decedf4ee59fccc0a24a05cedd])
W20240430 20:11:28.842500 18417 leader_election.cc:341] T f5236574672b477a9026e91adcaf70ec P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Not found: Tablet not found: f5236574672b477a9026e91adcaf70ec
I20240430 20:11:28.843173 19371 tablet_loader.cc:96] loaded metadata for tablet 1462ecf1176c4e66be42abddbe36b00c (table default.Table_49e612b137f343d1ac3ec06ded7d42f3 [id=6257559b0e9545c58848a76af0c5a5f7])
I20240430 20:11:28.843304 19185 leader_election.cc:288] T f5236574672b477a9026e91adcaf70ec P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:28.843434 18623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5236574672b477a9026e91adcaf70ec" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:28.843736 18623 raft_consensus.cc:2454] T f5236574672b477a9026e91adcaf70ec P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 0.
I20240430 20:11:28.843995 19371 tablet_loader.cc:96] loaded metadata for tablet 1555924b5f1444398f12ee82c3cb5b13 (table default.Table_388d774462294cab8342c6de331f9712 [id=aca48e01301e4f5f9562df8c87829b9d])
I20240430 20:11:28.844257 18415 leader_election.cc:302] T f5236574672b477a9026e91adcaf70ec P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters: 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:28.845719 18985 tablet_bootstrap.cc:492] T 02176eee8e044a7db8cac48439e9f594 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:28.845911 18985 ts_tablet_manager.cc:1367] T 02176eee8e044a7db8cac48439e9f594 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.003s
I20240430 20:11:28.846494 19344 raft_consensus.cc:2726] T 372b135dd01f44f98cd713ca8857ba61 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:28.846513 19371 tablet_loader.cc:96] loaded metadata for tablet 20d166756819406ba85044f4fee9c4ce (table default.Table_b331bf8aeaae4911ac01ed3fbd2627bd [id=1bf5c29b08134670984b6569db298129])
I20240430 20:11:28.846722 19185 raft_consensus.cc:2781] T f5236574672b477a9026e91adcaf70ec P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:28.846832 19185 raft_consensus.cc:482] T f5236574672b477a9026e91adcaf70ec P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:28.846719 18985 raft_consensus.cc:348] T 02176eee8e044a7db8cac48439e9f594 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:28.846957 19185 raft_consensus.cc:3037] T f5236574672b477a9026e91adcaf70ec P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:28.847019 18985 raft_consensus.cc:374] T 02176eee8e044a7db8cac48439e9f594 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:28.847095 18985 raft_consensus.cc:729] T 02176eee8e044a7db8cac48439e9f594 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:28.847371 18985 consensus_queue.cc:260] T 02176eee8e044a7db8cac48439e9f594 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:28.848698 19371 tablet_loader.cc:96] loaded metadata for tablet 24dcc20832804229a2664873bf6ddaa7 (table default.Table_17b7c8e853bd427db0c3fa2555a9067b [id=2d44ec94131148048021b2ee6366ba30])
I20240430 20:11:28.850127 19185 raft_consensus.cc:504] T f5236574672b477a9026e91adcaf70ec P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:28.850628 19371 tablet_loader.cc:96] loaded metadata for tablet 26ed12f498644d15bf45e884dcad172e (table default.Table_3f42877693b04aaa837c89b8983d5a72 [id=6d499a625cee4df2a4f40ee189cc6220])
I20240430 20:11:28.850984 18983 ts_tablet_manager.cc:1398] T 5f56ac2afd444d63a6bfd96988464bbb P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.020s user 0.002s sys 0.000s
I20240430 20:11:28.852154 19371 tablet_loader.cc:96] loaded metadata for tablet 294644d7a16b4ca2bbb6d106bcb8c7ff (table default.Table_e3a0afbced214062a8b026abe2320f3e [id=97d8a36ddfae4bb5b99c04a7a08544af])
I20240430 20:11:28.852255 18985 ts_tablet_manager.cc:1398] T 02176eee8e044a7db8cac48439e9f594 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20240430 20:11:28.852887 18988 tablet_bootstrap.cc:492] T bd11c0335a58448da56533c3f2e785a9 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:28.853224 18983 tablet_bootstrap.cc:492] T d9e578c99058424a9b86390641f9c34f P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:28.854009 19371 tablet_loader.cc:96] loaded metadata for tablet 2a1e50f99c5d4b44bdbb51b58a92bc75 (table default.Table_71075ae6b4d545889273c891648c70d1 [id=0fb93f36bc17408ea56bc4cafcfd8580])
I20240430 20:11:28.854866 18985 tablet_bootstrap.cc:492] T 0b4d9e860d7c45fd91d7a566605c9283 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:28.855180 19371 tablet_loader.cc:96] loaded metadata for tablet 2f61e3f88a6448dfa323204bd8d14cc5 (table default.Table_db56d63aef494592b69246a0c13006f2 [id=cde04507ee1d464f9fffc44176da8d16])
I20240430 20:11:28.855827 19371 tablet_loader.cc:96] loaded metadata for tablet 32c823d1700949ab8b22adcd83b7bfd1 (table default.Table_e8c73c89623d4b49b8d7f0a01d729bdd [id=5cd10682ce52414cb5a103e4433a6a0d])
I20240430 20:11:28.856451 18983 tablet_bootstrap.cc:654] T d9e578c99058424a9b86390641f9c34f P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:28.856487 19371 tablet_loader.cc:96] loaded metadata for tablet 34130f2820064fa0898ef1f9d7de1c6d (table default.Table_4119d46f13a74c0b950aecb0744d429d [id=5a1c4a7be2fc465fac89a50854eb3322])
I20240430 20:11:28.856693 19180 raft_consensus.cc:482] T a1aeee1b02214975a376cfd36957afe8 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:28.856880 19180 raft_consensus.cc:504] T a1aeee1b02214975a376cfd36957afe8 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:28.857635 19180 leader_election.cc:288] T a1aeee1b02214975a376cfd36957afe8 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:28.858100 18985 tablet_bootstrap.cc:654] T 0b4d9e860d7c45fd91d7a566605c9283 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:28.858510 18482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1aeee1b02214975a376cfd36957afe8" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:28.858773 18482 raft_consensus.cc:2454] T a1aeee1b02214975a376cfd36957afe8 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 0.
I20240430 20:11:28.858773 19371 tablet_loader.cc:96] loaded metadata for tablet 372b135dd01f44f98cd713ca8857ba61 (table default.Table_ee86b8fcf2064d8b8c3e327e357d3395 [id=7f3a15a442984827a7521d5d1c98dc3d])
I20240430 20:11:28.859066 18763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1aeee1b02214975a376cfd36957afe8" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:28.859249 18558 leader_election.cc:302] T a1aeee1b02214975a376cfd36957afe8 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:28.859352 18763 raft_consensus.cc:2454] T a1aeee1b02214975a376cfd36957afe8 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 0.
I20240430 20:11:28.859642 19180 raft_consensus.cc:2781] T a1aeee1b02214975a376cfd36957afe8 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:28.859761 19180 raft_consensus.cc:482] T a1aeee1b02214975a376cfd36957afe8 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:28.859849 19180 raft_consensus.cc:3037] T a1aeee1b02214975a376cfd36957afe8 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:28.860571 19371 tablet_loader.cc:96] loaded metadata for tablet 43b5e2c005a34df08cc4335c6e014cb5 (table default.Table_08dc70f541b842c78a3a946d2d02e207 [id=38f199a7538f4271b3a59fd93f29efa2])
W20240430 20:11:28.861008 18417 leader_election.cc:341] T f5236574672b477a9026e91adcaf70ec P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Not found: Tablet not found: f5236574672b477a9026e91adcaf70ec
I20240430 20:11:28.861696 19371 tablet_loader.cc:96] loaded metadata for tablet 44fa6917a85c46f3aaa363df46ea586b (table default.Table_5924e4520c6c49988ccef21ca04d25d4 [id=68c1eb7252974258aabd9873e6654b10])
I20240430 20:11:28.861721 19185 leader_election.cc:288] T f5236574672b477a9026e91adcaf70ec P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:28.861824 18623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5236574672b477a9026e91adcaf70ec" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:28.862013 18623 raft_consensus.cc:3037] T f5236574672b477a9026e91adcaf70ec P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:28.865187 18623 raft_consensus.cc:2454] T f5236574672b477a9026e91adcaf70ec P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 1.
I20240430 20:11:28.865119 19371 tablet_loader.cc:96] loaded metadata for tablet 53a899fa5a424a0d99f1b47925e8c7ad (table default.Table_6213f980d6a44e099a7a989ceb23c08a [id=608890834da843fa9af0cfc0d548029e])
I20240430 20:11:28.865953 19371 tablet_loader.cc:96] loaded metadata for tablet 5809e7c9c2204ba293426c24129fb16e (table default.Table_d82824c897a94dd0ba45955f9473b626 [id=d794640d21a843a185038673f7eb2b00])
I20240430 20:11:28.866044 18415 leader_election.cc:302] T f5236574672b477a9026e91adcaf70ec P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters: 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:28.866372 19185 raft_consensus.cc:2781] T f5236574672b477a9026e91adcaf70ec P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:28.866628 19185 raft_consensus.cc:686] T f5236574672b477a9026e91adcaf70ec P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Becoming Leader. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Running, Role: LEADER
I20240430 20:11:28.867754 19371 tablet_loader.cc:96] loaded metadata for tablet 5bde72e2791a497db7dd08686ca36d80 (table default.Table_a7add9298f1a44c6a2b2a9d73405c834 [id=5227206f44d04fa89b4842b899f33afb])
I20240430 20:11:28.867978 19185 consensus_queue.cc:237] T f5236574672b477a9026e91adcaf70ec P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:28.869406 18985 tablet_bootstrap.cc:492] T 0b4d9e860d7c45fd91d7a566605c9283 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:28.869313 19180 raft_consensus.cc:504] T a1aeee1b02214975a376cfd36957afe8 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:28.869588 18985 ts_tablet_manager.cc:1367] T 0b4d9e860d7c45fd91d7a566605c9283 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.000s
I20240430 20:11:28.870584 18763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1aeee1b02214975a376cfd36957afe8" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:28.870877 18763 raft_consensus.cc:3037] T a1aeee1b02214975a376cfd36957afe8 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:28.870577 18985 raft_consensus.cc:348] T 0b4d9e860d7c45fd91d7a566605c9283 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:28.871047 18985 raft_consensus.cc:374] T 0b4d9e860d7c45fd91d7a566605c9283 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:28.871148 18985 raft_consensus.cc:729] T 0b4d9e860d7c45fd91d7a566605c9283 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:28.871392 18985 consensus_queue.cc:260] T 0b4d9e860d7c45fd91d7a566605c9283 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:28.872020 18985 ts_tablet_manager.cc:1398] T 0b4d9e860d7c45fd91d7a566605c9283 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240430 20:11:28.872383 18985 tablet_bootstrap.cc:492] T 1398d4d2305b48e08fdb12432bd6c4b0 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:28.873842 18763 raft_consensus.cc:2454] T a1aeee1b02214975a376cfd36957afe8 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
I20240430 20:11:28.874835 18557 leader_election.cc:302] T a1aeee1b02214975a376cfd36957afe8 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:28.875191 19230 raft_consensus.cc:2781] T a1aeee1b02214975a376cfd36957afe8 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:28.875373 19230 raft_consensus.cc:686] T a1aeee1b02214975a376cfd36957afe8 P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Becoming Leader. State: Replica: 5e1c6397f09747f091534faf93542710, State: Running, Role: LEADER
I20240430 20:11:28.875931 18985 tablet_bootstrap.cc:654] T 1398d4d2305b48e08fdb12432bd6c4b0 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:28.875751 19230 consensus_queue.cc:237] T a1aeee1b02214975a376cfd36957afe8 P 5e1c6397f09747f091534faf93542710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:28.875924 18235 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 89df84ccf5394c9bb60d4e4ed872fc2e [term 5 FOLLOWER]: Refusing update from remote peer b632d6ef4c104a058a6db208024cab33: Log matching property violated. Preceding OpId in replica: term: 4 index: 308. Preceding OpId from leader: term: 5 index: 309. (index mismatch)
I20240430 20:11:28.877210 19361 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e" member_type: VOTER last_known_addr { host: "127.17.177.61" port: 34777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 309, Last known committed idx: 308, Time since last communication: 0.000s
W20240430 20:11:28.877822 18999 consensus_peers.cc:485] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33 -> Peer 88b3f98793354c7aaa5c106e8ec31ed8 (127.17.177.62:36441): Couldn't send request to peer 88b3f98793354c7aaa5c106e8ec31ed8. Status: Network error: Client connection negotiation failed: client connection to 127.17.177.62:36441: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240430 20:11:28.878221 19371 tablet_loader.cc:96] loaded metadata for tablet 5d09baed576e4136bf6d756ca39d08b2 (table default.Table_772ce91c27ce4119b6d1b1993e417f21 [id=29b41dafb4334b14a09602eabac2f031])
I20240430 20:11:28.878911 19371 tablet_loader.cc:96] loaded metadata for tablet 5f02d8862e1f4ccfa9dbda73dbe18615 (table default.Table_0b4b99c1c03a4e779046063159ce6861 [id=43d7bdc84bab4c54bac58dd1dd0f74ac])
I20240430 20:11:28.880596 19371 tablet_loader.cc:96] loaded metadata for tablet 5f56ac2afd444d63a6bfd96988464bbb (table default.Table_19a91a7f17ff456ebb145e8dd89c0940 [id=67283dd3995a44c6a7a7bc8722e4f955])
I20240430 20:11:28.881736 18482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1aeee1b02214975a376cfd36957afe8" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:28.882030 18482 raft_consensus.cc:3037] T a1aeee1b02214975a376cfd36957afe8 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:28.884050 19371 tablet_loader.cc:96] loaded metadata for tablet 680553d4d0bc4fcb839235c1832757e1 (table default.Table_49a012a7c4c8457fadb0a8f0fc9654c6 [id=eecb2f8eeef547c5ab1abe6a2430e592])
I20240430 20:11:28.886631 19386 raft_consensus.cc:482] T 93d158de64994f3c9fa45a1250a4f533 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:28.886735 19344 raft_consensus.cc:482] T 6e826459c3f4437da81c9e6e09f0c22f P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:28.886816 19386 raft_consensus.cc:504] T 93d158de64994f3c9fa45a1250a4f533 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:28.886940 19344 raft_consensus.cc:504] T 6e826459c3f4437da81c9e6e09f0c22f P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:28.887850 19386 leader_election.cc:288] T 93d158de64994f3c9fa45a1250a4f533 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:28.888427 19344 leader_election.cc:288] T 6e826459c3f4437da81c9e6e09f0c22f P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:28.888228 19371 tablet_loader.cc:96] loaded metadata for tablet 6e754af64cb64fd2a082adae7d5f24da (table default.Table_208b12b8e9bd4f3e8bc9c6d422ada649 [id=1df4e95702cd46f59ab21b101049a9ed])
I20240430 20:11:28.888821 18760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93d158de64994f3c9fa45a1250a4f533" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:28.889092 18760 raft_consensus.cc:2454] T 93d158de64994f3c9fa45a1250a4f533 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 0.
I20240430 20:11:28.889621 18557 leader_election.cc:302] T 93d158de64994f3c9fa45a1250a4f533 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:28.890182 19230 raft_consensus.cc:2781] T 93d158de64994f3c9fa45a1250a4f533 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:28.890311 19230 raft_consensus.cc:482] T 93d158de64994f3c9fa45a1250a4f533 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:28.890221 19371 tablet_loader.cc:96] loaded metadata for tablet 6e826459c3f4437da81c9e6e09f0c22f (table default.Table_873c21f6dffd4d47a742b3699ab85268 [id=d7e2c4300ff94fa2acc65d6d6c932c65])
I20240430 20:11:28.890431 19230 raft_consensus.cc:3037] T 93d158de64994f3c9fa45a1250a4f533 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:28.890411 18483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93d158de64994f3c9fa45a1250a4f533" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:28.890703 18483 raft_consensus.cc:2454] T 93d158de64994f3c9fa45a1250a4f533 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 0.
I20240430 20:11:28.893096 19371 tablet_loader.cc:96] loaded metadata for tablet 724e51ff05ec4baebc2bb55e05c8eddc (table default.Table_0c5d4686f2ed40fb8d62c31b0166dc05 [id=4dce8a767f0541e1beb5464b2ed95d06])
I20240430 20:11:28.894109 19230 raft_consensus.cc:504] T 93d158de64994f3c9fa45a1250a4f533 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:28.894502 18623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e826459c3f4437da81c9e6e09f0c22f" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:28.894976 18623 raft_consensus.cc:2454] T 6e826459c3f4437da81c9e6e09f0c22f P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 0.
I20240430 20:11:28.894863 19342 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89df84ccf5394c9bb60d4e4ed872fc2e [sys.catalog]: SysCatalogTable state changed. Reason: New leader b632d6ef4c104a058a6db208024cab33. Latest consensus state: current_term: 5 leader_uuid: "b632d6ef4c104a058a6db208024cab33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b3f98793354c7aaa5c106e8ec31ed8" member_type: VOTER last_known_addr { host: "127.17.177.62" port: 36441 } } peers { permanent_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e" member_type: VOTER last_known_addr { host: "127.17.177.61" port: 34777 } } peers { permanent_uuid: "b632d6ef4c104a058a6db208024cab33" member_type: VOTER last_known_addr { host: "127.17.177.60" port: 37225 } } }
I20240430 20:11:28.895180 19342 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89df84ccf5394c9bb60d4e4ed872fc2e [sys.catalog]: This master's current role is: FOLLOWER
I20240430 20:11:28.895603 18415 leader_election.cc:302] T 6e826459c3f4437da81c9e6e09f0c22f P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:28.896155 19371 tablet_loader.cc:96] loaded metadata for tablet 73453a7fa30342e29db552b17b62f0c1 (table default.Table_7a9f9a8c27bb4392acddd2a7ea3cc3b2 [id=cfad6f932073488b8097ef7381bc9fd1])
I20240430 20:11:28.896258 18985 tablet_bootstrap.cc:492] T 1398d4d2305b48e08fdb12432bd6c4b0 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:28.896433 18985 ts_tablet_manager.cc:1367] T 1398d4d2305b48e08fdb12432bd6c4b0 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.024s user 0.011s sys 0.000s
I20240430 20:11:28.896502 19371 tablet_loader.cc:96] loaded metadata for tablet 7542c7fe6cad452ab718f9ef54dbbbbe (table default.Table_8af52c0f1d404b42bc53287f64e20c2d [id=8e590e0e404344658b5cee90132d91d1])
I20240430 20:11:28.896914 19344 raft_consensus.cc:2781] T 6e826459c3f4437da81c9e6e09f0c22f P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:28.897106 19344 raft_consensus.cc:482] T 6e826459c3f4437da81c9e6e09f0c22f P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:28.897192 18763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e826459c3f4437da81c9e6e09f0c22f" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:28.897322 19344 raft_consensus.cc:3037] T 6e826459c3f4437da81c9e6e09f0c22f P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:28.897441 18763 raft_consensus.cc:2454] T 6e826459c3f4437da81c9e6e09f0c22f P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 0.
I20240430 20:11:28.898648 18763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93d158de64994f3c9fa45a1250a4f533" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:28.898944 18482 raft_consensus.cc:2454] T a1aeee1b02214975a376cfd36957afe8 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
I20240430 20:11:28.898938 18763 raft_consensus.cc:3037] T 93d158de64994f3c9fa45a1250a4f533 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:28.900053 19180 leader_election.cc:288] T a1aeee1b02214975a376cfd36957afe8 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:28.900718 19344 raft_consensus.cc:504] T 6e826459c3f4437da81c9e6e09f0c22f P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:28.902098 18988 ts_tablet_manager.cc:1367] T bd11c0335a58448da56533c3f2e785a9 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.103s user 0.005s sys 0.001s
I20240430 20:11:28.902052 18760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e826459c3f4437da81c9e6e09f0c22f" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:28.902333 18760 raft_consensus.cc:3037] T 6e826459c3f4437da81c9e6e09f0c22f P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:28.903311 18763 raft_consensus.cc:2454] T 93d158de64994f3c9fa45a1250a4f533 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
I20240430 20:11:28.903878 18557 leader_election.cc:302] T 93d158de64994f3c9fa45a1250a4f533 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:28.903841 18988 raft_consensus.cc:348] T bd11c0335a58448da56533c3f2e785a9 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:28.904071 18988 raft_consensus.cc:374] T bd11c0335a58448da56533c3f2e785a9 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:28.904163 18988 raft_consensus.cc:729] T bd11c0335a58448da56533c3f2e785a9 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:28.904436 19180 raft_consensus.cc:2781] T 93d158de64994f3c9fa45a1250a4f533 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:28.905252 18760 raft_consensus.cc:2454] T 6e826459c3f4437da81c9e6e09f0c22f P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 1.
I20240430 20:11:28.905246 18482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93d158de64994f3c9fa45a1250a4f533" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:28.905565 18482 raft_consensus.cc:3037] T 93d158de64994f3c9fa45a1250a4f533 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:28.906157 18417 leader_election.cc:302] T 6e826459c3f4437da81c9e6e09f0c22f P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters:
I20240430 20:11:28.906469 19186 raft_consensus.cc:2781] T 6e826459c3f4437da81c9e6e09f0c22f P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:28.907254 18623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e826459c3f4437da81c9e6e09f0c22f" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:28.907418 19371 tablet_loader.cc:96] loaded metadata for tablet 76d6e9d547ca4a848eae852944b1d117 (table default.Table_210b7eaff7144b2eb7db1c3b5f7d9505 [id=ff86c0357c164dd0991098c416087773])
I20240430 20:11:28.907645 18623 raft_consensus.cc:3037] T 6e826459c3f4437da81c9e6e09f0c22f P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:28.907898 19371 tablet_loader.cc:96] loaded metadata for tablet 77285afcafe04a829e485998bbbe00f0 (table default.Table_e63f64a411094f8db2ef641f2257ca79 [id=040dec3a3d804cdfbdada359bbe5123b])
I20240430 20:11:28.908600 19371 tablet_loader.cc:96] loaded metadata for tablet 7bde1d6934784c9aa5911181e8df9eeb (table default.Table_7199d7c36d3f45f79cf9e4e2b4145592 [id=f6aef8f6a9164dd78f49e03f723b45a9])
I20240430 20:11:28.909034 19371 tablet_loader.cc:96] loaded metadata for tablet 7dfdecba959543a795dfcd3eeda29e58 (table default.Table_da5216d025844afda5a83ada555bc120 [id=dc55b25c3ca14863a21042e805e3bf9a])
I20240430 20:11:28.909499 19371 tablet_loader.cc:96] loaded metadata for tablet 802336602d8f4ac4bd7e0872d6f3dae9 (table default.Table_d997b723a0dd48f6bc2d42c9bfd6bc14 [id=7737ec4807974640aa8556cb1c3ca3ef])
I20240430 20:11:28.910126 18482 raft_consensus.cc:2454] T 93d158de64994f3c9fa45a1250a4f533 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
I20240430 20:11:28.910272 19371 tablet_loader.cc:96] loaded metadata for tablet 81787eccca6c42c1b421377a3bf9ebe6 (table default.Table_c1669746f9114bf7b37a6f6a68fed20e [id=7f602584cef9486eb7d9b4389be3f43e])
I20240430 20:11:28.910676 19371 tablet_loader.cc:96] loaded metadata for tablet 87ac32a0ddb84676adc83038ef664591 (table default.Table_999c55e7bef74339b7b3ab1142599447 [id=61c8a5b99ef34a0baeeb343fef7cfd05])
I20240430 20:11:28.911082 18623 raft_consensus.cc:2454] T 6e826459c3f4437da81c9e6e09f0c22f P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 1.
I20240430 20:11:28.911321 19230 leader_election.cc:288] T 93d158de64994f3c9fa45a1250a4f533 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:28.911677 19344 leader_election.cc:288] T 6e826459c3f4437da81c9e6e09f0c22f P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:28.911981 19371 tablet_loader.cc:96] loaded metadata for tablet a05e32a33aa7488eb287cc6314c09c44 (table default.Table_8ddf4569d2f245469f55cb9fef2f5fb1 [id=955d833ff8f14bf6957abadf9ef669e2])
I20240430 20:11:28.906289 18985 raft_consensus.cc:348] T 1398d4d2305b48e08fdb12432bd6c4b0 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:28.912202 18985 raft_consensus.cc:374] T 1398d4d2305b48e08fdb12432bd6c4b0 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:28.911754 19180 raft_consensus.cc:686] T 93d158de64994f3c9fa45a1250a4f533 P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Becoming Leader. State: Replica: 5e1c6397f09747f091534faf93542710, State: Running, Role: LEADER
I20240430 20:11:28.912317 18985 raft_consensus.cc:729] T 1398d4d2305b48e08fdb12432bd6c4b0 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:28.912847 19371 tablet_loader.cc:96] loaded metadata for tablet a59ef6c545dd41a28265a4c1b8ce10e0 (table default.Table_327b9bc7f9c64a128209a7a24d2a410a [id=86df5d161d8641c89070e7eed9a39360])
I20240430 20:11:28.913232 19371 tablet_loader.cc:96] loaded metadata for tablet a745334c12c545c1ba825c7552c60359 (table default.Table_36adeb442f1245aba00549559bb329f3 [id=4ab2e4b038744db8b2cc94c26dbcc35f])
I20240430 20:11:28.913714 19371 tablet_loader.cc:96] loaded metadata for tablet a9e0659564894b0e8cb7f600bb07a085 (table default.Table_3c5df662747b48bf8beb6ebe4d2ea618 [id=663c8efbd7ac4a148d5f7626f1522033])
I20240430 20:11:28.913486 18985 consensus_queue.cc:260] T 1398d4d2305b48e08fdb12432bd6c4b0 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:28.914320 18985 ts_tablet_manager.cc:1398] T 1398d4d2305b48e08fdb12432bd6c4b0 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.018s user 0.002s sys 0.000s
I20240430 20:11:28.912736 19180 consensus_queue.cc:237] T 93d158de64994f3c9fa45a1250a4f533 P 5e1c6397f09747f091534faf93542710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:28.914947 19371 tablet_loader.cc:96] loaded metadata for tablet acd23f0a63d5448aaa7ed78422e67bad (table default.Table_972f13be4dca4cd19b9a04534dc59f51 [id=69f37db4de634a3185613e40013e0ed7])
I20240430 20:11:28.914978 18985 tablet_bootstrap.cc:492] T ec060e847f024f9fa369cff320e71558 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:28.915395 19371 tablet_loader.cc:96] loaded metadata for tablet acf3b6fead41483a82c43dc0286fab7c (table default.Table_8c7f3f5e74f64d409fbecfce64acf648 [id=ce9911e0a91949b1997ee47321800255])
I20240430 20:11:28.916038 19371 tablet_loader.cc:96] loaded metadata for tablet ae87f5e92a934e2182164c231bc76df0 (table default.Table_27f3d711beeb4387a9088ab291168bc8 [id=434aba8e131342a3a9a1aca3278f9c0d])
I20240430 20:11:28.916472 19371 tablet_loader.cc:96] loaded metadata for tablet aec9c09cc0454acab6006bb3e10982b7 (table default.Table_e68dc52988f24f1788517de94d7d2a0e [id=26d4860fb33047b7b812ebd4309f2e28])
I20240430 20:11:28.919238 19186 raft_consensus.cc:686] T 6e826459c3f4437da81c9e6e09f0c22f P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Becoming Leader. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Running, Role: LEADER
I20240430 20:11:28.919596 18985 tablet_bootstrap.cc:654] T ec060e847f024f9fa369cff320e71558 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:28.919624 19186 consensus_queue.cc:237] T 6e826459c3f4437da81c9e6e09f0c22f P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:28.920408 19371 tablet_loader.cc:96] loaded metadata for tablet b688e03ea71940c69bda5520b62c1fc3 (table default.Table_2ec9639eca4a47439b7859929cd705bc [id=6ffe241384bd42b7a22d41154ac59e3a])
I20240430 20:11:28.921643 19371 tablet_loader.cc:96] loaded metadata for tablet b72fe1f0713b4076a81a3fc1c74f7fa7 (table default.Table_16b082d140234e1c9a1e70654839b4cd [id=c0d8dc87d1f9426a85f6176f161595a1])
I20240430 20:11:28.923341 19371 tablet_loader.cc:96] loaded metadata for tablet b76a4105a2974fe280e759349037d027 (table default.Table_95220daaf6d44c08beff4ed6abdbea79 [id=75f0256acc45402d8b6809d92c38a1d8])
I20240430 20:11:28.924405 18985 tablet_bootstrap.cc:492] T ec060e847f024f9fa369cff320e71558 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:28.924580 18985 ts_tablet_manager.cc:1367] T ec060e847f024f9fa369cff320e71558 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.002s
I20240430 20:11:28.924722 19371 tablet_loader.cc:96] loaded metadata for tablet bd11c0335a58448da56533c3f2e785a9 (table default.Table_88c44b20f2ce4eeca7ed9583a70c7c65 [id=73caec5fb65e4cb090e7ae8c391501fd])
I20240430 20:11:28.925467 18985 raft_consensus.cc:348] T ec060e847f024f9fa369cff320e71558 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:28.925688 18985 raft_consensus.cc:374] T ec060e847f024f9fa369cff320e71558 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:28.925791 18985 raft_consensus.cc:729] T ec060e847f024f9fa369cff320e71558 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:28.926007 18985 consensus_queue.cc:260] T ec060e847f024f9fa369cff320e71558 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:28.925853 18988 consensus_queue.cc:260] T bd11c0335a58448da56533c3f2e785a9 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:28.926793 18985 ts_tablet_manager.cc:1398] T ec060e847f024f9fa369cff320e71558 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240430 20:11:28.927114 18985 tablet_bootstrap.cc:492] T 0e503a898406412ab977d3e048ea4b5d P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:28.927623 19371 tablet_loader.cc:96] loaded metadata for tablet c3d9aa85e2bf4c838fdc062cc135f638 (table default.Table_0965d6f90dc0474091a9d7a550fcb3cb [id=059a3f2888d74c24bcd708c0e663ccd5])
I20240430 20:11:28.928884 18985 tablet_bootstrap.cc:654] T 0e503a898406412ab977d3e048ea4b5d P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:28.929644 19371 tablet_loader.cc:96] loaded metadata for tablet c66c142419ca4e1399ecd1c8d1e62a59 (table default.Table_0443359dbeff437fb6cffb0db33cbb30 [id=93c44abdf6b147228224388398cd7aeb])
I20240430 20:11:28.931108 19371 tablet_loader.cc:96] loaded metadata for tablet c9e7660eccc448dbb99d135f82bfd261 (table default.Table_e233554a5e1d41389948fecaf498a856 [id=51d177e8498848b38c43e3f8d11f092c])
I20240430 20:11:28.931743 18988 ts_tablet_manager.cc:1398] T bd11c0335a58448da56533c3f2e785a9 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.029s user 0.002s sys 0.000s
I20240430 20:11:28.932173 18988 tablet_bootstrap.cc:492] T 0b4d9e860d7c45fd91d7a566605c9283 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:28.932453 19371 tablet_loader.cc:96] loaded metadata for tablet ca64de16d40f4a44b8b411f5526ba11a (table default.Table_f5a3451cd2284390be4357d771c03390 [id=f4ba9536e9c141deb68f4f62b7e8fa70])
I20240430 20:11:28.933843 19417 raft_consensus.cc:482] T ec060e847f024f9fa369cff320e71558 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:28.933985 19371 tablet_loader.cc:96] loaded metadata for tablet d4508e4f6dbd42109d951c648b3dc7df (table default.Table_276bbdc7cfe542d1a2936485a2aad1ec [id=f146a15e6e8a4acbb4c734c1c863da57])
I20240430 20:11:28.934074 19417 raft_consensus.cc:504] T ec060e847f024f9fa369cff320e71558 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:28.935758 19417 leader_election.cc:288] T ec060e847f024f9fa369cff320e71558 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:28.935720 18623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec060e847f024f9fa369cff320e71558" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
W20240430 20:11:28.936477 18415 leader_election.cc:341] T ec060e847f024f9fa369cff320e71558 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:28.936882 19371 tablet_loader.cc:96] loaded metadata for tablet dd6083f12c7e4d1e8a2006b3ddf9aa3a (table default.Table_276f62c2338b46a0aab8905430e8b61b [id=789b538b500642d89721e6028b9ac48e])
I20240430 20:11:28.937768 19371 tablet_loader.cc:96] loaded metadata for tablet dd9b893d48684cfea7c242b93ecba01f (table default.Table_4bce1dae3391497ebea5f1b536700d62 [id=8f0cb0ef65d84fb8964209dd501c0b5c])
I20240430 20:11:28.935719 18760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec060e847f024f9fa369cff320e71558" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
W20240430 20:11:28.938498 18417 leader_election.cc:341] T ec060e847f024f9fa369cff320e71558 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:28.938702 18417 leader_election.cc:302] T ec060e847f024f9fa369cff320e71558 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a; no voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710
I20240430 20:11:28.939388 19417 raft_consensus.cc:2726] T ec060e847f024f9fa369cff320e71558 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:28.941171 19371 tablet_loader.cc:96] loaded metadata for tablet e2d90ff17e574f88b2061b1c117d4b7d (table default.Table_6f2fd99337344da2ac5a0c83d9b56bb8 [id=8f1ad1f8d0224b3daa5d84dc6916227b])
I20240430 20:11:28.941836 18985 tablet_bootstrap.cc:492] T 0e503a898406412ab977d3e048ea4b5d P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:28.942019 18985 ts_tablet_manager.cc:1367] T 0e503a898406412ab977d3e048ea4b5d P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.000s
I20240430 20:11:28.904445 18983 tablet_bootstrap.cc:492] T d9e578c99058424a9b86390641f9c34f P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:28.942878 18983 ts_tablet_manager.cc:1367] T d9e578c99058424a9b86390641f9c34f P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.090s user 0.007s sys 0.003s
I20240430 20:11:28.944023 18983 raft_consensus.cc:348] T d9e578c99058424a9b86390641f9c34f P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:28.944272 18985 raft_consensus.cc:348] T 0e503a898406412ab977d3e048ea4b5d P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:28.944514 18985 raft_consensus.cc:374] T 0e503a898406412ab977d3e048ea4b5d P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:28.944590 18985 raft_consensus.cc:729] T 0e503a898406412ab977d3e048ea4b5d P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:28.945355 18985 consensus_queue.cc:260] T 0e503a898406412ab977d3e048ea4b5d P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:28.946185 18988 tablet_bootstrap.cc:654] T 0b4d9e860d7c45fd91d7a566605c9283 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:28.946684 18985 ts_tablet_manager.cc:1398] T 0e503a898406412ab977d3e048ea4b5d P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240430 20:11:28.948320 19371 tablet_loader.cc:96] loaded metadata for tablet e3e74e76ab2b4fe492ffcc1838656fd1 (table default.Table_5d3cb1a25c884804b05b07531e1b6965 [id=55e60456c89d441a8b5874a2063d52ef])
I20240430 20:11:28.952198 19371 tablet_loader.cc:96] loaded metadata for tablet e5111d7607dd42188a0176cf5be2ec05 (table default.Table_f61a45518e134c4496e4ac81fbe40fce [id=02d9fe0426694ee8b5fd70589fb66ead])
I20240430 20:11:28.952509 18985 tablet_bootstrap.cc:492] T 34ab40d8d91e401db299ddee7175ba28 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:28.953620 19371 tablet_loader.cc:96] loaded metadata for tablet e6c0c808bfc94be3837ab835292b3c1b (table default.Table_5fc77d5f60ae44f4bb4ca9f7f183672c [id=6ca223ca6354449c959d85ed2ac6a9c8])
I20240430 20:11:28.954339 19371 tablet_loader.cc:96] loaded metadata for tablet e6eac92dfa5f4456bc49571878f0f8dd (table default.Table_ad1a80db9c774df6add2b60e93577217 [id=19ffa77f540b499482d5dc7b64b12b79])
I20240430 20:11:28.955078 18985 tablet_bootstrap.cc:654] T 34ab40d8d91e401db299ddee7175ba28 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:28.955302 19371 tablet_loader.cc:96] loaded metadata for tablet ebf74809b52f47f3b0ed21b6ac0f541a (table default.Table_3e3885add2344068a5da1614890073e0 [id=1c8e0a3a44ec4a798770c0165ed3dffa])
I20240430 20:11:28.958431 19371 tablet_loader.cc:96] loaded metadata for tablet ec060e847f024f9fa369cff320e71558 (table default.Table_2e9c7c683df84a41942d31600a265641 [id=aca6fb02776a40bbac312191adbdc136])
I20240430 20:11:28.958509 18985 tablet_bootstrap.cc:492] T 34ab40d8d91e401db299ddee7175ba28 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:28.958904 18985 ts_tablet_manager.cc:1367] T 34ab40d8d91e401db299ddee7175ba28 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.000s
I20240430 20:11:28.960510 19371 tablet_loader.cc:96] loaded metadata for tablet f77086a5675f44e99e8a867b694df4b8 (table default.Table_cd719a43207f4bf983e1b8e6d236bb16 [id=47e0b04b17834deb833a8f31c123edc5])
I20240430 20:11:28.961114 19371 tablet_loader.cc:96] loaded metadata for tablet fae9a35a2ac844b4bc2ebdf214aa7171 (table default.Table_0ab5b2f09fa7468bb89ab703f6d3d25c [id=a5d580cffa2447d78fc9f2c8e31264f1])
I20240430 20:11:28.960943 18985 raft_consensus.cc:348] T 34ab40d8d91e401db299ddee7175ba28 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:28.961442 18985 raft_consensus.cc:374] T 34ab40d8d91e401db299ddee7175ba28 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:28.961653 19371 tablet_loader.cc:96] loaded metadata for tablet fcc9698abda4491b98e1e879589913a8 (table default.Table_e431f94220aa44ecba799a54dfb612ae [id=1e785b6c159b4312bd1b064271f55a49])
I20240430 20:11:28.961663 18985 raft_consensus.cc:729] T 34ab40d8d91e401db299ddee7175ba28 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:28.962057 19371 tablet_loader.cc:96] loaded metadata for tablet fd40967ab4a04001819e16c06a6c4300 (table default.Table_fda07fd7279c484db2bee67e8c859855 [id=da9ad991953a40c4b77b31317597ad1d])
I20240430 20:11:28.962239 18985 consensus_queue.cc:260] T 34ab40d8d91e401db299ddee7175ba28 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:28.964028 19371 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240430 20:11:28.964866 18985 ts_tablet_manager.cc:1398] T 34ab40d8d91e401db299ddee7175ba28 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20240430 20:11:28.965531 18985 tablet_bootstrap.cc:492] T 29974dc2c7dd45b99b6cc0712378fc30 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:28.968060 18985 tablet_bootstrap.cc:654] T 29974dc2c7dd45b99b6cc0712378fc30 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:28.970268 19371 catalog_manager.cc:1259] Loaded cluster ID: 889e0faede1f41d1ab74633d6826c0d4
I20240430 20:11:28.972628 18983 raft_consensus.cc:374] T d9e578c99058424a9b86390641f9c34f P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:28.972752 18983 raft_consensus.cc:729] T d9e578c99058424a9b86390641f9c34f P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:28.974840 19371 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240430 20:11:28.974874 18983 consensus_queue.cc:260] T d9e578c99058424a9b86390641f9c34f P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:28.975880 18983 ts_tablet_manager.cc:1398] T d9e578c99058424a9b86390641f9c34f P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.033s user 0.002s sys 0.000s
I20240430 20:11:28.977459 18983 tablet_bootstrap.cc:492] T 02176eee8e044a7db8cac48439e9f594 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:28.979310 18983 tablet_bootstrap.cc:654] T 02176eee8e044a7db8cac48439e9f594 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:28.990694 19342 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89df84ccf5394c9bb60d4e4ed872fc2e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 5 leader_uuid: "b632d6ef4c104a058a6db208024cab33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b3f98793354c7aaa5c106e8ec31ed8" member_type: VOTER last_known_addr { host: "127.17.177.62" port: 36441 } } peers { permanent_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e" member_type: VOTER last_known_addr { host: "127.17.177.61" port: 34777 } } peers { permanent_uuid: "b632d6ef4c104a058a6db208024cab33" member_type: VOTER last_known_addr { host: "127.17.177.60" port: 37225 } } }
I20240430 20:11:28.990965 19342 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89df84ccf5394c9bb60d4e4ed872fc2e [sys.catalog]: This master's current role is: FOLLOWER
I20240430 20:11:28.994457 19261 raft_consensus.cc:482] T bd11c0335a58448da56533c3f2e785a9 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:28.994735 19261 raft_consensus.cc:504] T bd11c0335a58448da56533c3f2e785a9 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:28.995424 19261 leader_election.cc:288] T bd11c0335a58448da56533c3f2e785a9 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:28.996023 18482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd11c0335a58448da56533c3f2e785a9" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:28.996258 18482 raft_consensus.cc:2454] T bd11c0335a58448da56533c3f2e785a9 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:28.996718 18699 leader_election.cc:302] T bd11c0335a58448da56533c3f2e785a9 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters:
I20240430 20:11:28.998351 19261 raft_consensus.cc:2781] T bd11c0335a58448da56533c3f2e785a9 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:28.998476 19261 raft_consensus.cc:482] T bd11c0335a58448da56533c3f2e785a9 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:28.998569 19261 raft_consensus.cc:3037] T bd11c0335a58448da56533c3f2e785a9 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:29.001627 19261 raft_consensus.cc:504] T bd11c0335a58448da56533c3f2e785a9 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:29.001441 19416 sys_catalog.cc:455] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 5 leader_uuid: "b632d6ef4c104a058a6db208024cab33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b3f98793354c7aaa5c106e8ec31ed8" member_type: VOTER last_known_addr { host: "127.17.177.62" port: 36441 } } peers { permanent_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e" member_type: VOTER last_known_addr { host: "127.17.177.61" port: 34777 } } peers { permanent_uuid: "b632d6ef4c104a058a6db208024cab33" member_type: VOTER last_known_addr { host: "127.17.177.60" port: 37225 } } }
I20240430 20:11:29.001854 19416 sys_catalog.cc:458] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33 [sys.catalog]: This master's current role is: LEADER
I20240430 20:11:29.008687 19371 catalog_manager.cc:1504] Loading token signing keys...
I20240430 20:11:29.013478 18482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd11c0335a58448da56533c3f2e785a9" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:29.013795 18482 raft_consensus.cc:3037] T bd11c0335a58448da56533c3f2e785a9 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:29.019073 19371 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: Loaded TSK: 0
I20240430 20:11:29.020696 19371 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240430 20:11:29.020723 18482 raft_consensus.cc:2454] T bd11c0335a58448da56533c3f2e785a9 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:29.021661 18699 leader_election.cc:302] T bd11c0335a58448da56533c3f2e785a9 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters:
I20240430 20:11:29.022202 19417 raft_consensus.cc:482] T c9ff972ade694938888514979e1638aa P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:29.022361 19417 raft_consensus.cc:504] T c9ff972ade694938888514979e1638aa P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:29.023043 19417 leader_election.cc:288] T c9ff972ade694938888514979e1638aa P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:29.023169 19417 raft_consensus.cc:482] T 02176eee8e044a7db8cac48439e9f594 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:29.023303 19417 raft_consensus.cc:504] T 02176eee8e044a7db8cac48439e9f594 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:29.023826 19417 leader_election.cc:288] T 02176eee8e044a7db8cac48439e9f594 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:29.023977 19261 leader_election.cc:288] T bd11c0335a58448da56533c3f2e785a9 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Requested vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:29.024752 19261 raft_consensus.cc:2781] T bd11c0335a58448da56533c3f2e785a9 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:29.024914 19261 raft_consensus.cc:686] T bd11c0335a58448da56533c3f2e785a9 P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Becoming Leader. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Running, Role: LEADER
I20240430 20:11:29.025194 19261 consensus_queue.cc:237] T bd11c0335a58448da56533c3f2e785a9 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
W20240430 20:11:29.025660 18415 leader_election.cc:341] T c9ff972ade694938888514979e1638aa P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Not found: Tablet not found: c9ff972ade694938888514979e1638aa
I20240430 20:11:29.027397 18760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02176eee8e044a7db8cac48439e9f594" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
W20240430 20:11:29.028534 18417 leader_election.cc:341] T 02176eee8e044a7db8cac48439e9f594 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
W20240430 20:11:29.028815 18417 leader_election.cc:341] T c9ff972ade694938888514979e1638aa P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Not found: Tablet not found: c9ff972ade694938888514979e1638aa
I20240430 20:11:29.028909 18417 leader_election.cc:302] T c9ff972ade694938888514979e1638aa P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a; no voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710
I20240430 20:11:29.029345 19186 raft_consensus.cc:2726] T c9ff972ade694938888514979e1638aa P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:28.997519 18623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd11c0335a58448da56533c3f2e785a9" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:29.030167 18623 raft_consensus.cc:2454] T bd11c0335a58448da56533c3f2e785a9 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:29.003597 18620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd11c0335a58448da56533c3f2e785a9" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:29.030467 18620 raft_consensus.cc:3037] T bd11c0335a58448da56533c3f2e785a9 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:29.033372 18620 raft_consensus.cc:2454] T bd11c0335a58448da56533c3f2e785a9 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:29.025265 18622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02176eee8e044a7db8cac48439e9f594" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:29.027541 19230 consensus_queue.cc:1035] T 0fe1247349344863a6273a769ce8ddf6 P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240430 20:11:29.034010 18415 leader_election.cc:341] T 02176eee8e044a7db8cac48439e9f594 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:29.034137 18415 leader_election.cc:302] T 02176eee8e044a7db8cac48439e9f594 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a; no voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710
I20240430 20:11:29.034570 19186 raft_consensus.cc:2726] T 02176eee8e044a7db8cac48439e9f594 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:29.042157 18988 tablet_bootstrap.cc:492] T 0b4d9e860d7c45fd91d7a566605c9283 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:29.043114 18988 ts_tablet_manager.cc:1367] T 0b4d9e860d7c45fd91d7a566605c9283 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.111s user 0.004s sys 0.000s
I20240430 20:11:29.044281 18988 raft_consensus.cc:348] T 0b4d9e860d7c45fd91d7a566605c9283 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:29.045596 18988 raft_consensus.cc:374] T 0b4d9e860d7c45fd91d7a566605c9283 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:29.045874 18988 raft_consensus.cc:729] T 0b4d9e860d7c45fd91d7a566605c9283 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:29.045704 19033 catalog_manager.cc:5484] T bd11c0335a58448da56533c3f2e785a9 P 3c51c33d5616482cafb68e26688820f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c51c33d5616482cafb68e26688820f4 (127.17.177.3). New cstate: current_term: 1 leader_uuid: "3c51c33d5616482cafb68e26688820f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } health_report { overall_health: UNKNOWN } } }
I20240430 20:11:29.046340 18988 consensus_queue.cc:260] T 0b4d9e860d7c45fd91d7a566605c9283 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:29.047509 18988 ts_tablet_manager.cc:1398] T 0b4d9e860d7c45fd91d7a566605c9283 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240430 20:11:29.050944 18988 tablet_bootstrap.cc:492] T 02176eee8e044a7db8cac48439e9f594 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:29.051846 19033 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet bd11c0335a58448da56533c3f2e785a9 (table default.Table_88c44b20f2ce4eeca7ed9583a70c7c65 [id=73caec5fb65e4cb090e7ae8c391501fd]). Expected version 1 got 0
I20240430 20:11:29.056641 18988 tablet_bootstrap.cc:654] T 02176eee8e044a7db8cac48439e9f594 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:29.062186 19230 consensus_queue.cc:1035] T 0fe1247349344863a6273a769ce8ddf6 P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:29.071514 18985 tablet_bootstrap.cc:492] T 29974dc2c7dd45b99b6cc0712378fc30 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:29.073035 18983 tablet_bootstrap.cc:492] T 02176eee8e044a7db8cac48439e9f594 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:29.073303 18983 ts_tablet_manager.cc:1367] T 02176eee8e044a7db8cac48439e9f594 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.096s user 0.006s sys 0.002s
I20240430 20:11:29.074494 18983 raft_consensus.cc:348] T 02176eee8e044a7db8cac48439e9f594 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:29.074851 18983 raft_consensus.cc:374] T 02176eee8e044a7db8cac48439e9f594 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:29.075374 18983 raft_consensus.cc:729] T 02176eee8e044a7db8cac48439e9f594 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:29.075879 18988 tablet_bootstrap.cc:492] T 02176eee8e044a7db8cac48439e9f594 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:29.076118 18988 ts_tablet_manager.cc:1367] T 02176eee8e044a7db8cac48439e9f594 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.025s user 0.006s sys 0.000s
I20240430 20:11:29.076279 18983 consensus_queue.cc:260] T 02176eee8e044a7db8cac48439e9f594 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:29.077244 18988 raft_consensus.cc:348] T 02176eee8e044a7db8cac48439e9f594 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:29.077486 18985 ts_tablet_manager.cc:1367] T 29974dc2c7dd45b99b6cc0712378fc30 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.112s user 0.021s sys 0.029s
I20240430 20:11:29.077522 18988 raft_consensus.cc:374] T 02176eee8e044a7db8cac48439e9f594 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:29.077685 18988 raft_consensus.cc:729] T 02176eee8e044a7db8cac48439e9f594 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:29.078011 18988 consensus_queue.cc:260] T 02176eee8e044a7db8cac48439e9f594 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:29.078302 18985 raft_consensus.cc:348] T 29974dc2c7dd45b99b6cc0712378fc30 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:29.078647 18988 ts_tablet_manager.cc:1398] T 02176eee8e044a7db8cac48439e9f594 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240430 20:11:29.079026 18988 tablet_bootstrap.cc:492] T 0e503a898406412ab977d3e048ea4b5d P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:29.079699 18985 raft_consensus.cc:374] T 29974dc2c7dd45b99b6cc0712378fc30 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:29.079806 18985 raft_consensus.cc:729] T 29974dc2c7dd45b99b6cc0712378fc30 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:29.080092 18985 consensus_queue.cc:260] T 29974dc2c7dd45b99b6cc0712378fc30 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:29.081781 18985 ts_tablet_manager.cc:1398] T 29974dc2c7dd45b99b6cc0712378fc30 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240430 20:11:29.082129 18985 tablet_bootstrap.cc:492] T 5f56ac2afd444d63a6bfd96988464bbb P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:29.082661 18988 tablet_bootstrap.cc:654] T 0e503a898406412ab977d3e048ea4b5d P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:29.083910 18985 tablet_bootstrap.cc:654] T 5f56ac2afd444d63a6bfd96988464bbb P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:29.056130 19049 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_d5b955117a0a4c1eb8ed8151e155b32e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240430 20:11:29.087237 18983 ts_tablet_manager.cc:1398] T 02176eee8e044a7db8cac48439e9f594 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
W20240430 20:11:29.087508 19049 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d5b955117a0a4c1eb8ed8151e155b32e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:29.056905 19051 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_02d2b1f3815549c481a305c622c809ff"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:29.090323 19051 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_02d2b1f3815549c481a305c622c809ff in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:29.057893 19029 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_0965d6f90dc0474091a9d7a550fcb3cb" } modify_external_catalogs: true
I20240430 20:11:29.091540 19029 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_0965d6f90dc0474091a9d7a550fcb3cb" } modify_external_catalogs: true
I20240430 20:11:29.058508 19028 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_ad1a80db9c774df6add2b60e93577217" } modify_external_catalogs: true
I20240430 20:11:29.093183 19028 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_ad1a80db9c774df6add2b60e93577217" } modify_external_catalogs: true
I20240430 20:11:29.093881 18983 tablet_bootstrap.cc:492] T ec060e847f024f9fa369cff320e71558 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:29.059321 19027 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_0034641cfa4d42a68ff4341e87fcb396"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:29.094990 19027 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0034641cfa4d42a68ff4341e87fcb396 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:29.060106 19026 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_8028554bc993419e80dbf7d526b63bf4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240430 20:11:29.096138 18983 tablet_bootstrap.cc:654] T ec060e847f024f9fa369cff320e71558 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
W20240430 20:11:29.096454 19026 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8028554bc993419e80dbf7d526b63bf4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:29.061591 19024 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_8ddf4569d2f245469f55cb9fef2f5fb1" } new_table_name: "default.Table_6551367971354d3195e4f7d47364ce7c" modify_external_catalogs: true
I20240430 20:11:29.062827 19023 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_7b72c19b4765456dba543151abaf1d44" } modify_external_catalogs: true
I20240430 20:11:29.099067 19023 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_7b72c19b4765456dba543151abaf1d44" } modify_external_catalogs: true
I20240430 20:11:29.102326 18988 tablet_bootstrap.cc:492] T 0e503a898406412ab977d3e048ea4b5d P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:29.102497 18988 ts_tablet_manager.cc:1367] T 0e503a898406412ab977d3e048ea4b5d P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.024s user 0.005s sys 0.000s
I20240430 20:11:29.103217 18983 tablet_bootstrap.cc:492] T ec060e847f024f9fa369cff320e71558 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:29.103433 18983 ts_tablet_manager.cc:1367] T ec060e847f024f9fa369cff320e71558 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20240430 20:11:29.103356 18988 raft_consensus.cc:348] T 0e503a898406412ab977d3e048ea4b5d P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:29.103591 18988 raft_consensus.cc:374] T 0e503a898406412ab977d3e048ea4b5d P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:29.103730 18988 raft_consensus.cc:729] T 0e503a898406412ab977d3e048ea4b5d P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:29.061026 19048 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_4119d46f13a74c0b950aecb0744d429d" } new_table_name: "default.Table_4537f891b01048ceb51202abc3600911" modify_external_catalogs: true
I20240430 20:11:29.104027 18988 consensus_queue.cc:260] T 0e503a898406412ab977d3e048ea4b5d P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:29.104593 18988 ts_tablet_manager.cc:1398] T 0e503a898406412ab977d3e048ea4b5d P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240430 20:11:29.104398 18983 raft_consensus.cc:348] T ec060e847f024f9fa369cff320e71558 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:29.104791 18983 raft_consensus.cc:374] T ec060e847f024f9fa369cff320e71558 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:29.104898 18988 tablet_bootstrap.cc:492] T 34ab40d8d91e401db299ddee7175ba28 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:29.104903 18983 raft_consensus.cc:729] T ec060e847f024f9fa369cff320e71558 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:29.105142 18983 consensus_queue.cc:260] T ec060e847f024f9fa369cff320e71558 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:29.105803 18983 ts_tablet_manager.cc:1398] T ec060e847f024f9fa369cff320e71558 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240430 20:11:29.106340 18983 tablet_bootstrap.cc:492] T 29974dc2c7dd45b99b6cc0712378fc30 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:29.106854 18988 tablet_bootstrap.cc:654] T 34ab40d8d91e401db299ddee7175ba28 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:29.108141 18983 tablet_bootstrap.cc:654] T 29974dc2c7dd45b99b6cc0712378fc30 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:29.116981 19230 raft_consensus.cc:482] T 77285afcafe04a829e485998bbbe00f0 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:29.117273 19230 raft_consensus.cc:504] T 77285afcafe04a829e485998bbbe00f0 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:29.118170 19230 leader_election.cc:288] T 77285afcafe04a829e485998bbbe00f0 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:29.119072 18763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77285afcafe04a829e485998bbbe00f0" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:29.120551 18482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77285afcafe04a829e485998bbbe00f0" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
W20240430 20:11:29.121592 18557 leader_election.cc:341] T 77285afcafe04a829e485998bbbe00f0 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
W20240430 20:11:29.122563 18558 leader_election.cc:341] T 77285afcafe04a829e485998bbbe00f0 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:29.122743 18558 leader_election.cc:302] T 77285afcafe04a829e485998bbbe00f0 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:29.123579 19230 raft_consensus.cc:2726] T 77285afcafe04a829e485998bbbe00f0 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:29.132519 18983 tablet_bootstrap.cc:492] T 29974dc2c7dd45b99b6cc0712378fc30 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:29.132787 18983 ts_tablet_manager.cc:1367] T 29974dc2c7dd45b99b6cc0712378fc30 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.027s user 0.018s sys 0.008s
I20240430 20:11:29.133033 18985 tablet_bootstrap.cc:492] T 5f56ac2afd444d63a6bfd96988464bbb P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:29.133200 18985 ts_tablet_manager.cc:1367] T 5f56ac2afd444d63a6bfd96988464bbb P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.051s user 0.010s sys 0.000s
I20240430 20:11:29.133841 18983 raft_consensus.cc:348] T 29974dc2c7dd45b99b6cc0712378fc30 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:29.133977 18985 raft_consensus.cc:348] T 5f56ac2afd444d63a6bfd96988464bbb P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:29.134178 18985 raft_consensus.cc:374] T 5f56ac2afd444d63a6bfd96988464bbb P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:29.134209 18983 raft_consensus.cc:374] T 29974dc2c7dd45b99b6cc0712378fc30 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:29.134294 18985 raft_consensus.cc:729] T 5f56ac2afd444d63a6bfd96988464bbb P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:29.134493 18983 raft_consensus.cc:729] T 29974dc2c7dd45b99b6cc0712378fc30 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:29.134619 18985 consensus_queue.cc:260] T 5f56ac2afd444d63a6bfd96988464bbb P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:29.135627 18985 ts_tablet_manager.cc:1398] T 5f56ac2afd444d63a6bfd96988464bbb P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240430 20:11:29.135002 18983 consensus_queue.cc:260] T 29974dc2c7dd45b99b6cc0712378fc30 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:29.137800 19028 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: Sending DeleteTablet for 3 replicas of tablet e6eac92dfa5f4456bc49571878f0f8dd
I20240430 20:11:29.138845 18983 ts_tablet_manager.cc:1398] T 29974dc2c7dd45b99b6cc0712378fc30 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240430 20:11:29.139770 18983 tablet_bootstrap.cc:492] T 0e503a898406412ab977d3e048ea4b5d P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:29.146163 18988 tablet_bootstrap.cc:492] T 34ab40d8d91e401db299ddee7175ba28 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:29.146386 18988 ts_tablet_manager.cc:1367] T 34ab40d8d91e401db299ddee7175ba28 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.042s user 0.001s sys 0.008s
I20240430 20:11:29.147210 18988 raft_consensus.cc:348] T 34ab40d8d91e401db299ddee7175ba28 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:29.147495 18988 raft_consensus.cc:374] T 34ab40d8d91e401db299ddee7175ba28 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:29.147601 18988 raft_consensus.cc:729] T 34ab40d8d91e401db299ddee7175ba28 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:29.147917 18988 consensus_queue.cc:260] T 34ab40d8d91e401db299ddee7175ba28 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:29.148619 18985 tablet_bootstrap.cc:492] T 77285afcafe04a829e485998bbbe00f0 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:29.149434 18983 tablet_bootstrap.cc:654] T 0e503a898406412ab977d3e048ea4b5d P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:29.150663 18985 tablet_bootstrap.cc:654] T 77285afcafe04a829e485998bbbe00f0 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:29.155489 18988 ts_tablet_manager.cc:1398] T 34ab40d8d91e401db299ddee7175ba28 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.009s user 0.001s sys 0.002s
I20240430 20:11:29.156929 18988 tablet_bootstrap.cc:492] T ec060e847f024f9fa369cff320e71558 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:29.159896 18988 tablet_bootstrap.cc:654] T ec060e847f024f9fa369cff320e71558 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:29.161116 18983 tablet_bootstrap.cc:492] T 0e503a898406412ab977d3e048ea4b5d P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:29.161343 18983 ts_tablet_manager.cc:1367] T 0e503a898406412ab977d3e048ea4b5d P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.004s
I20240430 20:11:29.161537 18985 tablet_bootstrap.cc:492] T 77285afcafe04a829e485998bbbe00f0 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:29.161724 18985 ts_tablet_manager.cc:1367] T 77285afcafe04a829e485998bbbe00f0 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.026s user 0.004s sys 0.000s
I20240430 20:11:29.162169 18983 raft_consensus.cc:348] T 0e503a898406412ab977d3e048ea4b5d P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:29.162446 18983 raft_consensus.cc:374] T 0e503a898406412ab977d3e048ea4b5d P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:29.162537 18983 raft_consensus.cc:729] T 0e503a898406412ab977d3e048ea4b5d P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:29.162465 18985 raft_consensus.cc:348] T 77285afcafe04a829e485998bbbe00f0 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:29.162709 18985 raft_consensus.cc:374] T 77285afcafe04a829e485998bbbe00f0 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:29.162787 18985 raft_consensus.cc:729] T 77285afcafe04a829e485998bbbe00f0 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:29.162940 18983 consensus_queue.cc:260] T 0e503a898406412ab977d3e048ea4b5d P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:29.165115 18985 consensus_queue.cc:260] T 77285afcafe04a829e485998bbbe00f0 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:29.167826 18985 ts_tablet_manager.cc:1398] T 77285afcafe04a829e485998bbbe00f0 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240430 20:11:29.168035 18983 ts_tablet_manager.cc:1398] T 0e503a898406412ab977d3e048ea4b5d P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20240430 20:11:29.169309 18988 tablet_bootstrap.cc:492] T ec060e847f024f9fa369cff320e71558 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:29.169481 18988 ts_tablet_manager.cc:1367] T ec060e847f024f9fa369cff320e71558 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.000s
I20240430 20:11:29.170275 18988 raft_consensus.cc:348] T ec060e847f024f9fa369cff320e71558 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:29.170476 18988 raft_consensus.cc:374] T ec060e847f024f9fa369cff320e71558 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:29.170563 18988 raft_consensus.cc:729] T ec060e847f024f9fa369cff320e71558 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:29.170805 18988 consensus_queue.cc:260] T ec060e847f024f9fa369cff320e71558 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:29.172478 18985 tablet_bootstrap.cc:492] T d9e578c99058424a9b86390641f9c34f P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:29.175601 18983 tablet_bootstrap.cc:492] T 34ab40d8d91e401db299ddee7175ba28 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:29.174180 18985 tablet_bootstrap.cc:654] T d9e578c99058424a9b86390641f9c34f P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:29.177460 18983 tablet_bootstrap.cc:654] T 34ab40d8d91e401db299ddee7175ba28 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:29.177764 18988 ts_tablet_manager.cc:1398] T ec060e847f024f9fa369cff320e71558 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240430 20:11:29.179009 18988 tablet_bootstrap.cc:492] T 5f56ac2afd444d63a6bfd96988464bbb P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:29.181435 18988 tablet_bootstrap.cc:654] T 5f56ac2afd444d63a6bfd96988464bbb P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:29.187556 18983 tablet_bootstrap.cc:492] T 34ab40d8d91e401db299ddee7175ba28 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:29.187639 18985 tablet_bootstrap.cc:492] T d9e578c99058424a9b86390641f9c34f P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:29.187801 18985 ts_tablet_manager.cc:1367] T d9e578c99058424a9b86390641f9c34f P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.001s
I20240430 20:11:29.187896 18983 ts_tablet_manager.cc:1367] T 34ab40d8d91e401db299ddee7175ba28 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.005s
I20240430 20:11:29.188678 18985 raft_consensus.cc:348] T d9e578c99058424a9b86390641f9c34f P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:29.188899 18985 raft_consensus.cc:374] T d9e578c99058424a9b86390641f9c34f P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:29.189013 18985 raft_consensus.cc:729] T d9e578c99058424a9b86390641f9c34f P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:29.188853 18983 raft_consensus.cc:348] T 34ab40d8d91e401db299ddee7175ba28 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:29.189342 18983 raft_consensus.cc:374] T 34ab40d8d91e401db299ddee7175ba28 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:29.189483 18983 raft_consensus.cc:729] T 34ab40d8d91e401db299ddee7175ba28 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:29.189832 18983 consensus_queue.cc:260] T 34ab40d8d91e401db299ddee7175ba28 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:29.192848 18988 tablet_bootstrap.cc:492] T 5f56ac2afd444d63a6bfd96988464bbb P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:29.193094 18988 ts_tablet_manager.cc:1367] T 5f56ac2afd444d63a6bfd96988464bbb P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.001s
I20240430 20:11:29.194062 18988 raft_consensus.cc:348] T 5f56ac2afd444d63a6bfd96988464bbb P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:29.194316 18988 raft_consensus.cc:374] T 5f56ac2afd444d63a6bfd96988464bbb P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:29.194416 18988 raft_consensus.cc:729] T 5f56ac2afd444d63a6bfd96988464bbb P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:29.194667 18988 consensus_queue.cc:260] T 5f56ac2afd444d63a6bfd96988464bbb P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:29.195734 18985 consensus_queue.cc:260] T d9e578c99058424a9b86390641f9c34f P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:29.196060 18983 ts_tablet_manager.cc:1398] T 34ab40d8d91e401db299ddee7175ba28 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240430 20:11:29.196686 18983 tablet_bootstrap.cc:492] T 7dfdecba959543a795dfcd3eeda29e58 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:29.199821 18985 ts_tablet_manager.cc:1398] T d9e578c99058424a9b86390641f9c34f P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.012s user 0.002s sys 0.001s
I20240430 20:11:29.201475 18985 tablet_bootstrap.cc:492] T 5bde72e2791a497db7dd08686ca36d80 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:29.202955 18983 tablet_bootstrap.cc:654] T 7dfdecba959543a795dfcd3eeda29e58 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:29.203101 18985 tablet_bootstrap.cc:654] T 5bde72e2791a497db7dd08686ca36d80 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:29.205677 18988 ts_tablet_manager.cc:1398] T 5f56ac2afd444d63a6bfd96988464bbb P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.012s user 0.003s sys 0.000s
I20240430 20:11:29.206048 18988 tablet_bootstrap.cc:492] T d9e578c99058424a9b86390641f9c34f P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:29.209513 18988 tablet_bootstrap.cc:654] T d9e578c99058424a9b86390641f9c34f P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:29.212275 18985 tablet_bootstrap.cc:492] T 5bde72e2791a497db7dd08686ca36d80 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:29.212554 18985 ts_tablet_manager.cc:1367] T 5bde72e2791a497db7dd08686ca36d80 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.000s
I20240430 20:11:29.213624 18985 raft_consensus.cc:348] T 5bde72e2791a497db7dd08686ca36d80 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:29.213928 18985 raft_consensus.cc:374] T 5bde72e2791a497db7dd08686ca36d80 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:29.214031 18985 raft_consensus.cc:729] T 5bde72e2791a497db7dd08686ca36d80 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:29.220717 18988 tablet_bootstrap.cc:492] T d9e578c99058424a9b86390641f9c34f P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:29.214385 18985 consensus_queue.cc:260] T 5bde72e2791a497db7dd08686ca36d80 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:29.220929 18988 ts_tablet_manager.cc:1367] T d9e578c99058424a9b86390641f9c34f P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.001s
I20240430 20:11:29.221871 18985 ts_tablet_manager.cc:1398] T 5bde72e2791a497db7dd08686ca36d80 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.009s user 0.003s sys 0.000s
I20240430 20:11:29.222229 18985 tablet_bootstrap.cc:492] T d4508e4f6dbd42109d951c648b3dc7df P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:29.224323 18985 tablet_bootstrap.cc:654] T d4508e4f6dbd42109d951c648b3dc7df P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:29.232599 19186 raft_consensus.cc:482] T 29974dc2c7dd45b99b6cc0712378fc30 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:29.232728 18988 raft_consensus.cc:348] T d9e578c99058424a9b86390641f9c34f P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:29.232841 19186 raft_consensus.cc:504] T 29974dc2c7dd45b99b6cc0712378fc30 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:29.233052 18988 raft_consensus.cc:374] T d9e578c99058424a9b86390641f9c34f P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:29.233160 18988 raft_consensus.cc:729] T d9e578c99058424a9b86390641f9c34f P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:29.233433 18988 consensus_queue.cc:260] T d9e578c99058424a9b86390641f9c34f P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:29.233772 19186 leader_election.cc:288] T 29974dc2c7dd45b99b6cc0712378fc30 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:29.234880 18622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29974dc2c7dd45b99b6cc0712378fc30" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:29.235174 18622 raft_consensus.cc:2454] T 29974dc2c7dd45b99b6cc0712378fc30 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 0.
I20240430 20:11:29.235338 19261 raft_consensus.cc:482] T bbe9a2429a434f3f95d6e7f314fcfe4c P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:29.235567 19261 raft_consensus.cc:504] T bbe9a2429a434f3f95d6e7f314fcfe4c P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:29.236071 18415 leader_election.cc:302] T 29974dc2c7dd45b99b6cc0712378fc30 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:29.236169 19261 leader_election.cc:288] T bbe9a2429a434f3f95d6e7f314fcfe4c P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:29.236450 19186 raft_consensus.cc:2781] T 29974dc2c7dd45b99b6cc0712378fc30 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:29.236619 19186 raft_consensus.cc:482] T 29974dc2c7dd45b99b6cc0712378fc30 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:29.236745 19186 raft_consensus.cc:3037] T 29974dc2c7dd45b99b6cc0712378fc30 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
W20240430 20:11:29.237141 18699 leader_election.cc:341] T bbe9a2429a434f3f95d6e7f314fcfe4c P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Not found: Tablet not found: bbe9a2429a434f3f95d6e7f314fcfe4c
I20240430 20:11:29.243105 19186 raft_consensus.cc:504] T 29974dc2c7dd45b99b6cc0712378fc30 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:29.243933 18988 ts_tablet_manager.cc:1398] T d9e578c99058424a9b86390641f9c34f P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s
I20240430 20:11:29.244778 18622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29974dc2c7dd45b99b6cc0712378fc30" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:29.244961 19186 leader_election.cc:288] T 29974dc2c7dd45b99b6cc0712378fc30 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Requested vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:29.245110 18622 raft_consensus.cc:3037] T 29974dc2c7dd45b99b6cc0712378fc30 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:29.247016 18988 tablet_bootstrap.cc:492] T 77285afcafe04a829e485998bbbe00f0 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:29.247169 18763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29974dc2c7dd45b99b6cc0712378fc30" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:29.247619 18760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29974dc2c7dd45b99b6cc0712378fc30" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
W20240430 20:11:29.248200 18417 leader_election.cc:341] T 29974dc2c7dd45b99b6cc0712378fc30 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:29.248378 18622 raft_consensus.cc:2454] T 29974dc2c7dd45b99b6cc0712378fc30 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 1.
W20240430 20:11:29.248535 18417 leader_election.cc:341] T 29974dc2c7dd45b99b6cc0712378fc30 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:29.248839 18988 tablet_bootstrap.cc:654] T 77285afcafe04a829e485998bbbe00f0 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:29.248847 18415 leader_election.cc:302] T 29974dc2c7dd45b99b6cc0712378fc30 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters: 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:29.250433 19186 raft_consensus.cc:2781] T 29974dc2c7dd45b99b6cc0712378fc30 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:29.253105 19261 raft_consensus.cc:482] T 34ab40d8d91e401db299ddee7175ba28 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:29.253294 19261 raft_consensus.cc:504] T 34ab40d8d91e401db299ddee7175ba28 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:29.254073 19261 leader_election.cc:288] T 34ab40d8d91e401db299ddee7175ba28 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:29.256075 18482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34ab40d8d91e401db299ddee7175ba28" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:29.262115 18482 raft_consensus.cc:2454] T 34ab40d8d91e401db299ddee7175ba28 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:29.262935 18988 tablet_bootstrap.cc:492] T 77285afcafe04a829e485998bbbe00f0 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:29.263085 18988 ts_tablet_manager.cc:1367] T 77285afcafe04a829e485998bbbe00f0 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.001s
I20240430 20:11:29.263814 18988 raft_consensus.cc:348] T 77285afcafe04a829e485998bbbe00f0 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:29.264019 18988 raft_consensus.cc:374] T 77285afcafe04a829e485998bbbe00f0 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:29.264082 18988 raft_consensus.cc:729] T 77285afcafe04a829e485998bbbe00f0 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:29.264941 18699 leader_election.cc:302] T 34ab40d8d91e401db299ddee7175ba28 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters:
I20240430 20:11:29.265575 19261 raft_consensus.cc:2781] T 34ab40d8d91e401db299ddee7175ba28 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:29.265726 19261 raft_consensus.cc:482] T 34ab40d8d91e401db299ddee7175ba28 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:29.265828 19261 raft_consensus.cc:3037] T 34ab40d8d91e401db299ddee7175ba28 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:29.269500 18988 consensus_queue.cc:260] T 77285afcafe04a829e485998bbbe00f0 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:29.270092 18988 ts_tablet_manager.cc:1398] T 77285afcafe04a829e485998bbbe00f0 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240430 20:11:29.270133 19261 raft_consensus.cc:504] T 34ab40d8d91e401db299ddee7175ba28 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:29.271951 18482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34ab40d8d91e401db299ddee7175ba28" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:29.272270 18482 raft_consensus.cc:3037] T 34ab40d8d91e401db299ddee7175ba28 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:29.272320 18988 tablet_bootstrap.cc:492] T 29974dc2c7dd45b99b6cc0712378fc30 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:29.274631 18988 tablet_bootstrap.cc:654] T 29974dc2c7dd45b99b6cc0712378fc30 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:29.277316 18482 raft_consensus.cc:2454] T 34ab40d8d91e401db299ddee7175ba28 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:29.278091 18699 leader_election.cc:302] T 34ab40d8d91e401db299ddee7175ba28 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters:
I20240430 20:11:29.278543 19427 raft_consensus.cc:2781] T 34ab40d8d91e401db299ddee7175ba28 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:29.279804 19427 raft_consensus.cc:686] T 34ab40d8d91e401db299ddee7175ba28 P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Becoming Leader. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Running, Role: LEADER
I20240430 20:11:29.280164 19427 consensus_queue.cc:237] T 34ab40d8d91e401db299ddee7175ba28 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
W20240430 20:11:29.282049 19024 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a05e32a33aa7488eb287cc6314c09c44 consensus state has no leader
I20240430 20:11:29.282560 19261 leader_election.cc:288] T 34ab40d8d91e401db299ddee7175ba28 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Requested vote from peers 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:29.285390 19186 raft_consensus.cc:686] T 29974dc2c7dd45b99b6cc0712378fc30 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Becoming Leader. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Running, Role: LEADER
I20240430 20:11:29.285826 19186 consensus_queue.cc:237] T 29974dc2c7dd45b99b6cc0712378fc30 P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:29.286132 19180 raft_consensus.cc:482] T ec060e847f024f9fa369cff320e71558 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:29.286346 19180 raft_consensus.cc:504] T ec060e847f024f9fa369cff320e71558 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:29.287035 19180 leader_election.cc:288] T ec060e847f024f9fa369cff320e71558 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:29.287937 18760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec060e847f024f9fa369cff320e71558" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:29.288192 18760 raft_consensus.cc:2454] T ec060e847f024f9fa369cff320e71558 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 0.
I20240430 20:11:29.288173 18622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbe9a2429a434f3f95d6e7f314fcfe4c" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:29.288376 18622 raft_consensus.cc:2454] T bbe9a2429a434f3f95d6e7f314fcfe4c P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:29.288641 18557 leader_election.cc:302] T ec060e847f024f9fa369cff320e71558 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:29.288822 18622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34ab40d8d91e401db299ddee7175ba28" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:29.289013 18622 raft_consensus.cc:2454] T 34ab40d8d91e401db299ddee7175ba28 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:29.289067 19180 raft_consensus.cc:2781] T ec060e847f024f9fa369cff320e71558 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:29.289142 19180 raft_consensus.cc:482] T ec060e847f024f9fa369cff320e71558 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:29.289263 19180 raft_consensus.cc:3037] T ec060e847f024f9fa369cff320e71558 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:29.289480 18622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34ab40d8d91e401db299ddee7175ba28" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:29.289659 18622 raft_consensus.cc:3037] T 34ab40d8d91e401db299ddee7175ba28 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:29.292477 18622 raft_consensus.cc:2454] T 34ab40d8d91e401db299ddee7175ba28 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:29.292546 19180 raft_consensus.cc:504] T ec060e847f024f9fa369cff320e71558 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:29.294090 18760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec060e847f024f9fa369cff320e71558" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:29.294327 18760 raft_consensus.cc:3037] T ec060e847f024f9fa369cff320e71558 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:29.297304 19027 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_17b7c8e853bd427db0c3fa2555a9067b" } new_table_name: "default.Table_be2ebe77f859489f9cd3e912cb1edc8b" modify_external_catalogs: true
I20240430 20:11:29.297529 19023 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: Sending DeleteTablet for 3 replicas of tablet 0a9066d895744c5bb7d9532b2940603c
I20240430 20:11:29.297833 18760 raft_consensus.cc:2454] T ec060e847f024f9fa369cff320e71558 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
I20240430 20:11:29.298393 18557 leader_election.cc:302] T ec060e847f024f9fa369cff320e71558 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:29.298475 19023 master_service.cc:946] ReplaceTablet: received request to replace tablet 0e503a898406412ab977d3e048ea4b5d from {username='slave'} at 127.0.0.1:51664
I20240430 20:11:29.299142 19026 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_3b327aece0ea4ec4b882f2264e3f5f0c" } modify_external_catalogs: true
I20240430 20:11:29.299379 19026 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_3b327aece0ea4ec4b882f2264e3f5f0c" } modify_external_catalogs: true
I20240430 20:11:29.300057 19386 raft_consensus.cc:2781] T ec060e847f024f9fa369cff320e71558 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:29.300249 19386 raft_consensus.cc:686] T ec060e847f024f9fa369cff320e71558 P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Becoming Leader. State: Replica: 5e1c6397f09747f091534faf93542710, State: Running, Role: LEADER
I20240430 20:11:29.300554 19386 consensus_queue.cc:237] T ec060e847f024f9fa369cff320e71558 P 5e1c6397f09747f091534faf93542710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:29.301229 19028 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_07993f6164104b888058e1b35c9f4ed7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240430 20:11:29.304752 18983 tablet_bootstrap.cc:492] T 7dfdecba959543a795dfcd3eeda29e58 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:29.301970 19024 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_e3a0afbced214062a8b026abe2320f3e" } new_table_name: "default.Table_190b2d86515a44898fa38a49517b9df1" modify_external_catalogs: true
I20240430 20:11:29.304962 18983 ts_tablet_manager.cc:1367] T 7dfdecba959543a795dfcd3eeda29e58 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.108s user 0.032s sys 0.037s
I20240430 20:11:29.305063 19180 leader_election.cc:288] T ec060e847f024f9fa369cff320e71558 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Requested vote from peers 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:29.305838 18983 raft_consensus.cc:348] T 7dfdecba959543a795dfcd3eeda29e58 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:29.306079 18983 raft_consensus.cc:374] T 7dfdecba959543a795dfcd3eeda29e58 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:29.306159 18983 raft_consensus.cc:729] T 7dfdecba959543a795dfcd3eeda29e58 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:29.306491 18983 consensus_queue.cc:260] T 7dfdecba959543a795dfcd3eeda29e58 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:29.307098 19051 master_service.cc:946] ReplaceTablet: received request to replace tablet d4508e4f6dbd42109d951c648b3dc7df from {username='slave'} at 127.0.0.1:51664
I20240430 20:11:29.307284 18983 ts_tablet_manager.cc:1398] T 7dfdecba959543a795dfcd3eeda29e58 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
W20240430 20:11:29.307305 19028 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_07993f6164104b888058e1b35c9f4ed7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:29.307646 18983 tablet_bootstrap.cc:492] T f77086a5675f44e99e8a867b694df4b8 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:29.308694 18529 heartbeater.cc:503] Master 127.17.177.60:37225 was elected leader, sending a full tablet report...
I20240430 20:11:29.309973 18983 tablet_bootstrap.cc:654] T f77086a5675f44e99e8a867b694df4b8 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:29.310026 19049 master_service.cc:946] ReplaceTablet: received request to replace tablet f77086a5675f44e99e8a867b694df4b8 from {username='slave'} at 127.0.0.1:51664
I20240430 20:11:29.311033 18669 heartbeater.cc:503] Master 127.17.177.60:37225 was elected leader, sending a full tablet report...
I20240430 20:11:29.314006 18696 leader_election.cc:302] T bbe9a2429a434f3f95d6e7f314fcfe4c P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a
W20240430 20:11:29.314750 19000 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet a05e32a33aa7488eb287cc6314c09c44 (table default.Table_6551367971354d3195e4f7d47364ce7c [id=955d833ff8f14bf6957abadf9ef669e2], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet a05e32a33aa7488eb287cc6314c09c44 consensus state has no leader
I20240430 20:11:29.315886 18482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec060e847f024f9fa369cff320e71558" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:29.316322 18482 raft_consensus.cc:2454] T ec060e847f024f9fa369cff320e71558 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 0.
I20240430 20:11:29.321465 19029 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet c3d9aa85e2bf4c838fdc062cc135f638
I20240430 20:11:29.315237 19427 consensus_queue.cc:1035] T 372b135dd01f44f98cd713ca8857ba61 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:29.321710 19029 master_service.cc:946] ReplaceTablet: received request to replace tablet 5f56ac2afd444d63a6bfd96988464bbb from {username='slave'} at 127.0.0.1:51664
I20240430 20:11:29.323225 19427 raft_consensus.cc:2781] T bbe9a2429a434f3f95d6e7f314fcfe4c P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:29.323341 19427 raft_consensus.cc:482] T bbe9a2429a434f3f95d6e7f314fcfe4c P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:29.323446 19427 raft_consensus.cc:3037] T bbe9a2429a434f3f95d6e7f314fcfe4c P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:29.326189 19427 raft_consensus.cc:504] T bbe9a2429a434f3f95d6e7f314fcfe4c P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:29.329257 19461 consensus_queue.cc:1035] T 7bde1d6934784c9aa5911181e8df9eeb P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240430 20:11:29.329794 18699 leader_election.cc:341] T bbe9a2429a434f3f95d6e7f314fcfe4c P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Not found: Tablet not found: bbe9a2429a434f3f95d6e7f314fcfe4c
I20240430 20:11:29.330298 19427 leader_election.cc:288] T bbe9a2429a434f3f95d6e7f314fcfe4c P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Requested vote from peers 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
W20240430 20:11:29.332509 19048 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 34130f2820064fa0898ef1f9d7de1c6d has no consensus state
I20240430 20:11:29.332674 18482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec060e847f024f9fa369cff320e71558" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:29.332828 19048 master_service.cc:946] ReplaceTablet: received request to replace tablet b688e03ea71940c69bda5520b62c1fc3 from {username='slave'} at 127.0.0.1:51664
I20240430 20:11:29.332895 18482 raft_consensus.cc:3037] T ec060e847f024f9fa369cff320e71558 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
W20240430 20:11:29.333381 19027 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 24dcc20832804229a2664873bf6ddaa7 consensus state has no leader
I20240430 20:11:29.334733 19023 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: Sending DeleteTablet for 3 replicas of tablet 0e503a898406412ab977d3e048ea4b5d
I20240430 20:11:29.335361 19023 catalog_manager.cc:6554] ReplaceTablet: tablet 0e503a898406412ab977d3e048ea4b5d deleted and replaced by tablet 5bd0362ff41b4faca0205d136d596737
I20240430 20:11:29.336299 18482 raft_consensus.cc:2454] T ec060e847f024f9fa369cff320e71558 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
I20240430 20:11:29.319864 19261 raft_consensus.cc:482] T 5f56ac2afd444d63a6bfd96988464bbb P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:29.336202 18810 heartbeater.cc:503] Master 127.17.177.60:37225 was elected leader, sending a full tablet report...
I20240430 20:11:29.340212 19261 raft_consensus.cc:504] T 5f56ac2afd444d63a6bfd96988464bbb P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:29.343473 19180 raft_consensus.cc:482] T d9e578c99058424a9b86390641f9c34f P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:29.343847 19180 raft_consensus.cc:504] T d9e578c99058424a9b86390641f9c34f P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:29.344492 18620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbe9a2429a434f3f95d6e7f314fcfe4c" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:29.344710 18620 raft_consensus.cc:3037] T bbe9a2429a434f3f95d6e7f314fcfe4c P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:29.345703 19261 leader_election.cc:288] T 5f56ac2afd444d63a6bfd96988464bbb P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:29.346737 18760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9e578c99058424a9b86390641f9c34f" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:29.347018 18588 tablet_service.cc:1514] Processing DeleteTablet for tablet e6eac92dfa5f4456bc49571878f0f8dd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:29 UTC) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:29.346968 18760 raft_consensus.cc:2454] T d9e578c99058424a9b86390641f9c34f P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 0.
I20240430 20:11:29.347894 19186 consensus_queue.cc:1035] T cace1398492f4d968dcb6899385d527c P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:29.348810 18988 tablet_bootstrap.cc:492] T 29974dc2c7dd45b99b6cc0712378fc30 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:29.348973 18988 ts_tablet_manager.cc:1367] T 29974dc2c7dd45b99b6cc0712378fc30 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.077s user 0.005s sys 0.012s
I20240430 20:11:29.349866 18557 leader_election.cc:302] T d9e578c99058424a9b86390641f9c34f P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:29.350554 18593 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a9066d895744c5bb7d9532b2940603c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:29 UTC) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:29.350736 19230 raft_consensus.cc:2781] T d9e578c99058424a9b86390641f9c34f P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:29.351401 19230 raft_consensus.cc:482] T d9e578c99058424a9b86390641f9c34f P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:29.351548 19230 raft_consensus.cc:3037] T d9e578c99058424a9b86390641f9c34f P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:29.355459 19230 raft_consensus.cc:504] T d9e578c99058424a9b86390641f9c34f P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:29.356674 18988 raft_consensus.cc:348] T 29974dc2c7dd45b99b6cc0712378fc30 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:29.356935 18988 raft_consensus.cc:374] T 29974dc2c7dd45b99b6cc0712378fc30 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:29.355818 18620 raft_consensus.cc:2454] T bbe9a2429a434f3f95d6e7f314fcfe4c P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:29.357017 18988 raft_consensus.cc:729] T 29974dc2c7dd45b99b6cc0712378fc30 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:29.357100 19186 raft_consensus.cc:482] T 0e503a898406412ab977d3e048ea4b5d P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:29.357293 19186 raft_consensus.cc:504] T 0e503a898406412ab977d3e048ea4b5d P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:29.358117 19230 leader_election.cc:288] T d9e578c99058424a9b86390641f9c34f P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:29.358508 18763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e503a898406412ab977d3e048ea4b5d" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:29.358794 18763 raft_consensus.cc:2454] T 0e503a898406412ab977d3e048ea4b5d P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 0.
I20240430 20:11:29.358523 18620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e503a898406412ab977d3e048ea4b5d" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:29.359215 18417 leader_election.cc:302] T 0e503a898406412ab977d3e048ea4b5d P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters:
I20240430 20:11:29.359266 18620 raft_consensus.cc:2454] T 0e503a898406412ab977d3e048ea4b5d P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 0.
I20240430 20:11:29.359553 19417 raft_consensus.cc:2781] T 0e503a898406412ab977d3e048ea4b5d P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:29.359653 19417 raft_consensus.cc:482] T 0e503a898406412ab977d3e048ea4b5d P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:29.359758 19417 raft_consensus.cc:3037] T 0e503a898406412ab977d3e048ea4b5d P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:29.360366 19180 leader_election.cc:288] T d9e578c99058424a9b86390641f9c34f P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:29.363111 18763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9e578c99058424a9b86390641f9c34f" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:29.363320 18763 raft_consensus.cc:3037] T d9e578c99058424a9b86390641f9c34f P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:29.366063 19261 raft_consensus.cc:482] T 0b4d9e860d7c45fd91d7a566605c9283 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:29.366233 19261 raft_consensus.cc:504] T 0b4d9e860d7c45fd91d7a566605c9283 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:29.366693 18763 raft_consensus.cc:2454] T d9e578c99058424a9b86390641f9c34f P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
I20240430 20:11:29.366688 18588 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fe1247349344863a6273a769ce8ddf6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:24 UTC) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:29.366909 19261 leader_election.cc:288] T 0b4d9e860d7c45fd91d7a566605c9283 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:29.367146 19471 tablet_replica.cc:330] T 0fe1247349344863a6273a769ce8ddf6 P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:29.367414 19471 raft_consensus.cc:2229] T 0fe1247349344863a6273a769ce8ddf6 P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Raft consensus shutting down.
I20240430 20:11:29.367898 19417 raft_consensus.cc:504] T 0e503a898406412ab977d3e048ea4b5d P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:29.368024 19471 raft_consensus.cc:2258] T 0fe1247349344863a6273a769ce8ddf6 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:29.368729 19185 consensus_queue.cc:1035] T cace1398492f4d968dcb6899385d527c P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:29.368707 19417 leader_election.cc:288] T 0e503a898406412ab977d3e048ea4b5d P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Requested vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:29.369309 19471 ts_tablet_manager.cc:1875] T 0fe1247349344863a6273a769ce8ddf6 P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:29.369405 18622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e503a898406412ab977d3e048ea4b5d" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:29.369653 18622 raft_consensus.cc:3037] T 0e503a898406412ab977d3e048ea4b5d P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:29.369997 18557 leader_election.cc:302] T d9e578c99058424a9b86390641f9c34f P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:29.370407 18483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b4d9e860d7c45fd91d7a566605c9283" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:29.370637 18483 raft_consensus.cc:2454] T 0b4d9e860d7c45fd91d7a566605c9283 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:29.371101 18763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e503a898406412ab977d3e048ea4b5d" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:29.371367 18763 raft_consensus.cc:3037] T 0e503a898406412ab977d3e048ea4b5d P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:29.372423 19230 raft_consensus.cc:2781] T d9e578c99058424a9b86390641f9c34f P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:29.372555 18622 raft_consensus.cc:2454] T 0e503a898406412ab977d3e048ea4b5d P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 1.
I20240430 20:11:29.373023 19261 raft_consensus.cc:482] T 02176eee8e044a7db8cac48439e9f594 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:29.373174 19261 raft_consensus.cc:504] T 02176eee8e044a7db8cac48439e9f594 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:29.373781 19261 leader_election.cc:288] T 02176eee8e044a7db8cac48439e9f594 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:29.374137 18988 consensus_queue.cc:260] T 29974dc2c7dd45b99b6cc0712378fc30 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:29.374645 18415 leader_election.cc:302] T 0e503a898406412ab977d3e048ea4b5d P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:29.374773 18988 ts_tablet_manager.cc:1398] T 29974dc2c7dd45b99b6cc0712378fc30 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.026s user 0.001s sys 0.001s
I20240430 20:11:29.374925 19185 raft_consensus.cc:482] T 1398d4d2305b48e08fdb12432bd6c4b0 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:29.375082 19185 raft_consensus.cc:504] T 1398d4d2305b48e08fdb12432bd6c4b0 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:29.375777 19185 leader_election.cc:288] T 1398d4d2305b48e08fdb12432bd6c4b0 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:29.375978 19185 raft_consensus.cc:2781] T 0e503a898406412ab977d3e048ea4b5d P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:29.376119 19185 raft_consensus.cc:686] T 0e503a898406412ab977d3e048ea4b5d P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Becoming Leader. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Running, Role: LEADER
I20240430 20:11:29.376403 19185 consensus_queue.cc:237] T 0e503a898406412ab977d3e048ea4b5d P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:29.378563 18988 tablet_bootstrap.cc:492] T d4508e4f6dbd42109d951c648b3dc7df P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:29.379189 18622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1398d4d2305b48e08fdb12432bd6c4b0" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:29.379436 18622 raft_consensus.cc:2454] T 1398d4d2305b48e08fdb12432bd6c4b0 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 0.
I20240430 20:11:29.380131 18415 leader_election.cc:302] T 1398d4d2305b48e08fdb12432bd6c4b0 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:29.380355 18988 tablet_bootstrap.cc:654] T d4508e4f6dbd42109d951c648b3dc7df P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:29.380447 19344 raft_consensus.cc:2781] T 1398d4d2305b48e08fdb12432bd6c4b0 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:29.380550 19344 raft_consensus.cc:482] T 1398d4d2305b48e08fdb12432bd6c4b0 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:29.380643 19344 raft_consensus.cc:3037] T 1398d4d2305b48e08fdb12432bd6c4b0 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:29.383340 19344 raft_consensus.cc:504] T 1398d4d2305b48e08fdb12432bd6c4b0 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:29.384053 19344 leader_election.cc:288] T 1398d4d2305b48e08fdb12432bd6c4b0 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Requested vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:29.384968 18622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1398d4d2305b48e08fdb12432bd6c4b0" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:29.385182 18622 raft_consensus.cc:3037] T 1398d4d2305b48e08fdb12432bd6c4b0 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:29.387663 19230 raft_consensus.cc:686] T d9e578c99058424a9b86390641f9c34f P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Becoming Leader. State: Replica: 5e1c6397f09747f091534faf93542710, State: Running, Role: LEADER
I20240430 20:11:29.388332 18622 raft_consensus.cc:2454] T 1398d4d2305b48e08fdb12432bd6c4b0 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 1.
I20240430 20:11:29.389019 18588 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e503a898406412ab977d3e048ea4b5d with delete_type TABLET_DATA_DELETED (replaced by tablet 5bd0362ff41b4faca0205d136d596737) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:29.389122 18415 leader_election.cc:302] T 1398d4d2305b48e08fdb12432bd6c4b0 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:29.389463 19344 raft_consensus.cc:2781] T 1398d4d2305b48e08fdb12432bd6c4b0 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:29.389433 18696 leader_election.cc:302] T bbe9a2429a434f3f95d6e7f314fcfe4c P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a
I20240430 20:11:29.389555 18983 tablet_bootstrap.cc:492] T f77086a5675f44e99e8a867b694df4b8 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:29.389518 18481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9e578c99058424a9b86390641f9c34f" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:29.389705 18983 ts_tablet_manager.cc:1367] T f77086a5675f44e99e8a867b694df4b8 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.082s user 0.019s sys 0.034s
I20240430 20:11:29.389766 18481 raft_consensus.cc:3037] T d9e578c99058424a9b86390641f9c34f P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:29.388737 19230 consensus_queue.cc:237] T d9e578c99058424a9b86390641f9c34f P 5e1c6397f09747f091534faf93542710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:29.390203 19261 raft_consensus.cc:2781] T bbe9a2429a434f3f95d6e7f314fcfe4c P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:29.390275 18764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1398d4d2305b48e08fdb12432bd6c4b0" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:29.390503 18764 raft_consensus.cc:2454] T 1398d4d2305b48e08fdb12432bd6c4b0 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 0.
I20240430 20:11:29.390666 19261 raft_consensus.cc:686] T bbe9a2429a434f3f95d6e7f314fcfe4c P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Becoming Leader. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Running, Role: LEADER
I20240430 20:11:29.390959 19261 consensus_queue.cc:237] T bbe9a2429a434f3f95d6e7f314fcfe4c P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:29.391444 18764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1398d4d2305b48e08fdb12432bd6c4b0" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:29.391716 18764 raft_consensus.cc:3037] T 1398d4d2305b48e08fdb12432bd6c4b0 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:29.392624 18481 raft_consensus.cc:2454] T d9e578c99058424a9b86390641f9c34f P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
I20240430 20:11:29.389619 19344 raft_consensus.cc:686] T 1398d4d2305b48e08fdb12432bd6c4b0 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Becoming Leader. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Running, Role: LEADER
I20240430 20:11:29.394443 19344 consensus_queue.cc:237] T 1398d4d2305b48e08fdb12432bd6c4b0 P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:29.394762 18699 leader_election.cc:302] T 0b4d9e860d7c45fd91d7a566605c9283 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters:
I20240430 20:11:29.395036 19427 raft_consensus.cc:2781] T 0b4d9e860d7c45fd91d7a566605c9283 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:29.395162 19427 raft_consensus.cc:482] T 0b4d9e860d7c45fd91d7a566605c9283 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:29.395272 19427 raft_consensus.cc:3037] T 0b4d9e860d7c45fd91d7a566605c9283 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:29.395841 18481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02176eee8e044a7db8cac48439e9f594" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:29.396045 18481 raft_consensus.cc:2454] T 02176eee8e044a7db8cac48439e9f594 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:29.397399 18764 raft_consensus.cc:2454] T 1398d4d2305b48e08fdb12432bd6c4b0 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 1.
I20240430 20:11:29.398839 19427 raft_consensus.cc:504] T 0b4d9e860d7c45fd91d7a566605c9283 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:29.399633 19427 leader_election.cc:288] T 0b4d9e860d7c45fd91d7a566605c9283 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Requested vote from peers 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:29.402123 18699 leader_election.cc:302] T 02176eee8e044a7db8cac48439e9f594 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters:
I20240430 20:11:29.374258 18763 raft_consensus.cc:2454] T 0e503a898406412ab977d3e048ea4b5d P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 1.
I20240430 20:11:29.411684 19484 consensus_queue.cc:1035] T 372b135dd01f44f98cd713ca8857ba61 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:29.412609 19484 raft_consensus.cc:2781] T 02176eee8e044a7db8cac48439e9f594 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:29.412811 19484 raft_consensus.cc:482] T 02176eee8e044a7db8cac48439e9f594 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:29.412973 19484 raft_consensus.cc:3037] T 02176eee8e044a7db8cac48439e9f594 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:29.390781 18983 raft_consensus.cc:348] T f77086a5675f44e99e8a867b694df4b8 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:29.413872 18983 raft_consensus.cc:374] T f77086a5675f44e99e8a867b694df4b8 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:29.414011 18983 raft_consensus.cc:729] T f77086a5675f44e99e8a867b694df4b8 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:29.414336 18983 consensus_queue.cc:260] T f77086a5675f44e99e8a867b694df4b8 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:29.414984 18983 ts_tablet_manager.cc:1398] T f77086a5675f44e99e8a867b694df4b8 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.025s user 0.002s sys 0.000s
I20240430 20:11:29.415356 18983 tablet_bootstrap.cc:492] T 5bde72e2791a497db7dd08686ca36d80 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:29.415850 19344 raft_consensus.cc:482] T 77285afcafe04a829e485998bbbe00f0 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:29.416199 19344 raft_consensus.cc:504] T 77285afcafe04a829e485998bbbe00f0 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:29.417059 18983 tablet_bootstrap.cc:654] T 5bde72e2791a497db7dd08686ca36d80 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:29.417093 19344 leader_election.cc:288] T 77285afcafe04a829e485998bbbe00f0 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:29.419574 18763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77285afcafe04a829e485998bbbe00f0" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:29.419924 18763 raft_consensus.cc:2454] T 77285afcafe04a829e485998bbbe00f0 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 0.
I20240430 20:11:29.420639 18417 leader_election.cc:302] T 77285afcafe04a829e485998bbbe00f0 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters:
W20240430 20:11:29.420756 18999 catalog_manager.cc:4810] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): delete failed for tablet e6eac92dfa5f4456bc49571878f0f8dd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e6eac92dfa5f4456bc49571878f0f8dd already in progress: creating tablet
W20240430 20:11:29.421612 18999 catalog_manager.cc:4810] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): delete failed for tablet 0a9066d895744c5bb7d9532b2940603c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0a9066d895744c5bb7d9532b2940603c already in progress: creating tablet
I20240430 20:11:29.422744 18623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02176eee8e044a7db8cac48439e9f594" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:29.423050 18623 raft_consensus.cc:2454] T 02176eee8e044a7db8cac48439e9f594 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:29.423682 18620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f56ac2afd444d63a6bfd96988464bbb" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:29.424006 18620 raft_consensus.cc:2454] T 5f56ac2afd444d63a6bfd96988464bbb P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:29.426724 18622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b4d9e860d7c45fd91d7a566605c9283" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:29.426941 18622 raft_consensus.cc:2454] T 0b4d9e860d7c45fd91d7a566605c9283 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:29.428056 18482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f56ac2afd444d63a6bfd96988464bbb" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:29.428344 18482 raft_consensus.cc:2454] T 5f56ac2afd444d63a6bfd96988464bbb P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:29.431423 19478 raft_consensus.cc:2781] T 77285afcafe04a829e485998bbbe00f0 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:29.431537 19478 raft_consensus.cc:482] T 77285afcafe04a829e485998bbbe00f0 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:29.431639 19478 raft_consensus.cc:3037] T 77285afcafe04a829e485998bbbe00f0 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:29.432478 19186 leader_election.cc:288] T 0e503a898406412ab977d3e048ea4b5d P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:29.433264 18730 tablet_service.cc:1514] Processing DeleteTablet for tablet e6eac92dfa5f4456bc49571878f0f8dd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:29 UTC) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:29.433703 18736 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a9066d895744c5bb7d9532b2940603c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:29 UTC) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:29.433285 18735 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e503a898406412ab977d3e048ea4b5d with delete_type TABLET_DATA_DELETED (replaced by tablet 5bd0362ff41b4faca0205d136d596737) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:29.436749 18622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77285afcafe04a829e485998bbbe00f0" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:29.437165 18622 raft_consensus.cc:2454] T 77285afcafe04a829e485998bbbe00f0 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 0.
I20240430 20:11:29.442792 18699 leader_election.cc:302] T 5f56ac2afd444d63a6bfd96988464bbb P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters:
I20240430 20:11:29.443212 19461 raft_consensus.cc:2781] T 5f56ac2afd444d63a6bfd96988464bbb P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:29.443387 19461 raft_consensus.cc:482] T 5f56ac2afd444d63a6bfd96988464bbb P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:29.443580 19461 raft_consensus.cc:3037] T 5f56ac2afd444d63a6bfd96988464bbb P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:29.445060 18482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9e578c99058424a9b86390641f9c34f" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:29.445473 18482 raft_consensus.cc:2362] T d9e578c99058424a9b86390641f9c34f P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1. Re-sending same reply.
I20240430 20:11:29.445794 18983 tablet_bootstrap.cc:492] T 5bde72e2791a497db7dd08686ca36d80 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:29.446064 18983 ts_tablet_manager.cc:1367] T 5bde72e2791a497db7dd08686ca36d80 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.031s user 0.005s sys 0.004s
I20240430 20:11:29.447206 18983 raft_consensus.cc:348] T 5bde72e2791a497db7dd08686ca36d80 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:29.447556 18983 raft_consensus.cc:374] T 5bde72e2791a497db7dd08686ca36d80 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:29.447610 19471 ts_tablet_manager.cc:1888] T 0fe1247349344863a6273a769ce8ddf6 P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:29.447847 18481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b4d9e860d7c45fd91d7a566605c9283" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:29.448113 18481 raft_consensus.cc:3037] T 0b4d9e860d7c45fd91d7a566605c9283 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:29.454027 18483 raft_consensus.cc:1261] T bd11c0335a58448da56533c3f2e785a9 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Refusing update from remote peer 3c51c33d5616482cafb68e26688820f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240430 20:11:29.455427 19261 consensus_queue.cc:1035] T bd11c0335a58448da56533c3f2e785a9 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:29.459084 18481 raft_consensus.cc:2454] T 0b4d9e860d7c45fd91d7a566605c9283 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:29.461531 18462 tablet_service.cc:1514] Processing DeleteTablet for tablet e6eac92dfa5f4456bc49571878f0f8dd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:29 UTC) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:29.461917 18455 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a9066d895744c5bb7d9532b2940603c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:29 UTC) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:29.462095 18457 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fe1247349344863a6273a769ce8ddf6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:24 UTC) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:29.462314 18456 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e503a898406412ab977d3e048ea4b5d with delete_type TABLET_DATA_DELETED (replaced by tablet 5bd0362ff41b4faca0205d136d596737) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:29.465127 19489 tablet_replica.cc:330] T 0e503a898406412ab977d3e048ea4b5d P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
W20240430 20:11:29.465096 19001 catalog_manager.cc:4810] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): delete failed for tablet e6eac92dfa5f4456bc49571878f0f8dd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e6eac92dfa5f4456bc49571878f0f8dd already in progress: creating tablet
I20240430 20:11:29.465449 19489 raft_consensus.cc:2229] T 0e503a898406412ab977d3e048ea4b5d P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:29.465744 19489 raft_consensus.cc:2258] T 0e503a898406412ab977d3e048ea4b5d P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240430 20:11:29.465728 19001 catalog_manager.cc:4810] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): delete failed for tablet 0a9066d895744c5bb7d9532b2940603c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0a9066d895744c5bb7d9532b2940603c already in progress: creating tablet
I20240430 20:11:29.473551 19478 raft_consensus.cc:504] T 77285afcafe04a829e485998bbbe00f0 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:29.447669 18983 raft_consensus.cc:729] T 5bde72e2791a497db7dd08686ca36d80 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:29.474769 19471 log.cc:1196] T 0fe1247349344863a6273a769ce8ddf6 P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/0fe1247349344863a6273a769ce8ddf6
I20240430 20:11:29.474910 18983 consensus_queue.cc:260] T 5bde72e2791a497db7dd08686ca36d80 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:29.475648 18983 ts_tablet_manager.cc:1398] T 5bde72e2791a497db7dd08686ca36d80 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.029s user 0.003s sys 0.000s
I20240430 20:11:29.479615 18988 tablet_bootstrap.cc:492] T d4508e4f6dbd42109d951c648b3dc7df P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:29.479854 18988 ts_tablet_manager.cc:1367] T d4508e4f6dbd42109d951c648b3dc7df P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.101s user 0.016s sys 0.025s
I20240430 20:11:29.480809 18988 raft_consensus.cc:348] T d4508e4f6dbd42109d951c648b3dc7df P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:29.481070 18988 raft_consensus.cc:374] T d4508e4f6dbd42109d951c648b3dc7df P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:29.481153 18988 raft_consensus.cc:729] T d4508e4f6dbd42109d951c648b3dc7df P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:29.481462 18988 consensus_queue.cc:260] T d4508e4f6dbd42109d951c648b3dc7df P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:29.483846 19478 leader_election.cc:288] T 77285afcafe04a829e485998bbbe00f0 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Requested vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:29.485721 18988 ts_tablet_manager.cc:1398] T d4508e4f6dbd42109d951c648b3dc7df P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240430 20:11:29.486099 18988 tablet_bootstrap.cc:492] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:29.488104 18988 tablet_bootstrap.cc:654] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:29.490077 18622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77285afcafe04a829e485998bbbe00f0" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:29.493273 18763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77285afcafe04a829e485998bbbe00f0" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:29.493505 18763 raft_consensus.cc:3037] T 77285afcafe04a829e485998bbbe00f0 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:29.496747 18763 raft_consensus.cc:2454] T 77285afcafe04a829e485998bbbe00f0 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 1.
I20240430 20:11:29.496990 18983 tablet_bootstrap.cc:492] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:29.497776 18622 raft_consensus.cc:3037] T 77285afcafe04a829e485998bbbe00f0 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:29.498366 18623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b4d9e860d7c45fd91d7a566605c9283" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:29.498661 18623 raft_consensus.cc:3037] T 0b4d9e860d7c45fd91d7a566605c9283 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:29.500768 18622 raft_consensus.cc:2454] T 77285afcafe04a829e485998bbbe00f0 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 1.
I20240430 20:11:29.501518 18623 raft_consensus.cc:2454] T 0b4d9e860d7c45fd91d7a566605c9283 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:29.501961 18415 leader_election.cc:302] T 77285afcafe04a829e485998bbbe00f0 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:29.496872 18699 leader_election.cc:302] T 0b4d9e860d7c45fd91d7a566605c9283 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters:
I20240430 20:11:29.502364 19261 raft_consensus.cc:2781] T 0b4d9e860d7c45fd91d7a566605c9283 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:29.502926 19261 raft_consensus.cc:686] T 0b4d9e860d7c45fd91d7a566605c9283 P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Becoming Leader. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Running, Role: LEADER
I20240430 20:11:29.503340 19261 consensus_queue.cc:237] T 0b4d9e860d7c45fd91d7a566605c9283 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:29.504268 19344 raft_consensus.cc:2781] T 77285afcafe04a829e485998bbbe00f0 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:29.504635 19344 raft_consensus.cc:686] T 77285afcafe04a829e485998bbbe00f0 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Becoming Leader. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Running, Role: LEADER
I20240430 20:11:29.505048 19344 consensus_queue.cc:237] T 77285afcafe04a829e485998bbbe00f0 P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:29.508747 19489 ts_tablet_manager.cc:1875] T 0e503a898406412ab977d3e048ea4b5d P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:29.512266 19471 ts_tablet_manager.cc:1909] T 0fe1247349344863a6273a769ce8ddf6 P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:29.513068 19471 tablet_replica.cc:330] T 0e503a898406412ab977d3e048ea4b5d P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:29.513422 19471 raft_consensus.cc:2229] T 0e503a898406412ab977d3e048ea4b5d P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:29.513600 18983 tablet_bootstrap.cc:654] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:29.513788 19471 raft_consensus.cc:2258] T 0e503a898406412ab977d3e048ea4b5d P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:29.521955 19471 ts_tablet_manager.cc:1875] T 0e503a898406412ab977d3e048ea4b5d P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:29.524044 18999 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet 0fe1247349344863a6273a769ce8ddf6 (table default.Table_7bfb47ba65114665943b7edccabd5a8b [id=c487c255c03741199e57b71670bc84e0]) successfully deleted
I20240430 20:11:29.524442 18988 tablet_bootstrap.cc:492] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:29.524636 18988 ts_tablet_manager.cc:1367] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.039s user 0.005s sys 0.004s
I20240430 20:11:29.525519 18988 raft_consensus.cc:348] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:29.525732 18988 raft_consensus.cc:374] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:29.525816 18988 raft_consensus.cc:729] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:29.526248 18988 consensus_queue.cc:260] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:29.522522 18623 raft_consensus.cc:1261] T bd11c0335a58448da56533c3f2e785a9 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Refusing update from remote peer 3c51c33d5616482cafb68e26688820f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240430 20:11:29.523715 19386 consensus_queue.cc:1035] T a745334c12c545c1ba825c7552c60359 P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:29.526950 18988 ts_tablet_manager.cc:1398] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240430 20:11:29.527287 18988 tablet_bootstrap.cc:492] T f77086a5675f44e99e8a867b694df4b8 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:29.529227 18988 tablet_bootstrap.cc:654] T f77086a5675f44e99e8a867b694df4b8 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:29.514792 19461 raft_consensus.cc:504] T 5f56ac2afd444d63a6bfd96988464bbb P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:29.531821 18483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f56ac2afd444d63a6bfd96988464bbb" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:29.532058 18483 raft_consensus.cc:3037] T 5f56ac2afd444d63a6bfd96988464bbb P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:29.534076 18983 tablet_bootstrap.cc:492] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:29.534242 18983 ts_tablet_manager.cc:1367] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.058s user 0.009s sys 0.004s
I20240430 20:11:29.535415 18483 raft_consensus.cc:2454] T 5f56ac2afd444d63a6bfd96988464bbb P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:29.536626 18699 leader_election.cc:302] T 5f56ac2afd444d63a6bfd96988464bbb P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters:
I20240430 20:11:29.537365 19386 consensus_queue.cc:1035] T fae9a35a2ac844b4bc2ebdf214aa7171 P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:29.538162 19490 raft_consensus.cc:2781] T 5f56ac2afd444d63a6bfd96988464bbb P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:29.538388 19490 raft_consensus.cc:686] T 5f56ac2afd444d63a6bfd96988464bbb P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Becoming Leader. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Running, Role: LEADER
I20240430 20:11:29.538843 19490 consensus_queue.cc:237] T 5f56ac2afd444d63a6bfd96988464bbb P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:29.544541 18988 tablet_bootstrap.cc:492] T f77086a5675f44e99e8a867b694df4b8 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:29.545639 19027 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_f33a9e03642c44efbf1c27f1c7e70fb9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240430 20:11:29.546324 19495 tablet_replica.cc:330] T 0fe1247349344863a6273a769ce8ddf6 P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
W20240430 20:11:29.546321 19027 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f33a9e03642c44efbf1c27f1c7e70fb9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:29.546545 19495 raft_consensus.cc:2229] T 0fe1247349344863a6273a769ce8ddf6 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:29.546814 19495 raft_consensus.cc:2258] T 0fe1247349344863a6273a769ce8ddf6 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:29.547663 19495 ts_tablet_manager.cc:1875] T 0fe1247349344863a6273a769ce8ddf6 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:29.547713 19033 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_cd719a43207f4bf983e1b8e6d236bb16" } modify_external_catalogs: true
I20240430 20:11:29.547906 19033 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_cd719a43207f4bf983e1b8e6d236bb16" } modify_external_catalogs: true
I20240430 20:11:29.548599 19023 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_aed0b9eeaac74c98944426712706be6b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:29.549144 19023 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_aed0b9eeaac74c98944426712706be6b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240430 20:11:29.545792 19004 catalog_manager.cc:4810] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): delete failed for tablet e6eac92dfa5f4456bc49571878f0f8dd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e6eac92dfa5f4456bc49571878f0f8dd already in progress: creating tablet
W20240430 20:11:29.554744 19004 catalog_manager.cc:4810] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): delete failed for tablet 0a9066d895744c5bb7d9532b2940603c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0a9066d895744c5bb7d9532b2940603c already in progress: creating tablet
I20240430 20:11:29.555174 19461 leader_election.cc:288] T 5f56ac2afd444d63a6bfd96988464bbb P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Requested vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:29.555306 18988 ts_tablet_manager.cc:1367] T f77086a5675f44e99e8a867b694df4b8 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.028s user 0.006s sys 0.000s
I20240430 20:11:29.535115 18983 raft_consensus.cc:348] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:29.559878 18983 raft_consensus.cc:374] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:29.560040 18983 raft_consensus.cc:729] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:29.560328 18988 raft_consensus.cc:348] T f77086a5675f44e99e8a867b694df4b8 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:29.560727 18988 raft_consensus.cc:374] T f77086a5675f44e99e8a867b694df4b8 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:29.560850 18988 raft_consensus.cc:729] T f77086a5675f44e99e8a867b694df4b8 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:29.561561 19026 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet 09d92a4a6ebe44cbb3553ed660f143d0
I20240430 20:11:29.561566 18988 consensus_queue.cc:260] T f77086a5675f44e99e8a867b694df4b8 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:29.562172 18988 ts_tablet_manager.cc:1398] T f77086a5675f44e99e8a867b694df4b8 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240430 20:11:29.566188 19478 consensus_queue.cc:1035] T 15674aeb56b74210a4a2809608fd4457 P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:29.568085 18988 tablet_bootstrap.cc:492] T 5bde72e2791a497db7dd08686ca36d80 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
W20240430 20:11:29.576125 19024 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 294644d7a16b4ca2bbb6d106bcb8c7ff consensus state has no leader
I20240430 20:11:29.577517 19051 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: Sending DeleteTablet for 3 replicas of tablet d4508e4f6dbd42109d951c648b3dc7df
I20240430 20:11:29.578131 19051 catalog_manager.cc:6554] ReplaceTablet: tablet d4508e4f6dbd42109d951c648b3dc7df deleted and replaced by tablet 2e79df8744e74b5991a79ce978ff0202
I20240430 20:11:29.580017 18623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f56ac2afd444d63a6bfd96988464bbb" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:29.580300 18623 raft_consensus.cc:3037] T 5f56ac2afd444d63a6bfd96988464bbb P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:29.581313 18462 tablet_service.cc:1514] Processing DeleteTablet for tablet d4508e4f6dbd42109d951c648b3dc7df with delete_type TABLET_DATA_DELETED (replaced by tablet 2e79df8744e74b5991a79ce978ff0202) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:29.582811 19049 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: Sending DeleteTablet for 3 replicas of tablet f77086a5675f44e99e8a867b694df4b8
I20240430 20:11:29.583354 19049 catalog_manager.cc:6554] ReplaceTablet: tablet f77086a5675f44e99e8a867b694df4b8 deleted and replaced by tablet 115c4a12440647a4bb46eb6824f7ea98
I20240430 20:11:29.583549 18623 raft_consensus.cc:2454] T 5f56ac2afd444d63a6bfd96988464bbb P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:29.584736 18588 tablet_service.cc:1514] Processing DeleteTablet for tablet d4508e4f6dbd42109d951c648b3dc7df with delete_type TABLET_DATA_DELETED (replaced by tablet 2e79df8744e74b5991a79ce978ff0202) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:29.585471 18588 tablet_service.cc:1514] Processing DeleteTablet for tablet f77086a5675f44e99e8a867b694df4b8 with delete_type TABLET_DATA_DELETED (replaced by tablet 115c4a12440647a4bb46eb6824f7ea98) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:29.586061 18462 tablet_service.cc:1514] Processing DeleteTablet for tablet f77086a5675f44e99e8a867b694df4b8 with delete_type TABLET_DATA_DELETED (replaced by tablet 115c4a12440647a4bb46eb6824f7ea98) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:29.586942 18983 consensus_queue.cc:260] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:29.590958 18983 ts_tablet_manager.cc:1398] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.057s user 0.003s sys 0.000s
I20240430 20:11:29.593514 19510 tablet.cc:1722] T bd11c0335a58448da56533c3f2e785a9 P 2efb542aa55e439b99ccb98d1c5c884a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240430 20:11:29.602845 19028 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_1a641ff0ab5944a495a73189a6e8499a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:29.603718 19028 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1a641ff0ab5944a495a73189a6e8499a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:29.604773 18735 tablet_service.cc:1514] Processing DeleteTablet for tablet f77086a5675f44e99e8a867b694df4b8 with delete_type TABLET_DATA_DELETED (replaced by tablet 115c4a12440647a4bb46eb6824f7ea98) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:29.604851 19049 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_a327a8eced314cdf95731f1181f52fb1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:29.605419 19049 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a327a8eced314cdf95731f1181f52fb1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:29.605945 19479 consensus_queue.cc:1035] T 7bde1d6934784c9aa5911181e8df9eeb P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.026s
I20240430 20:11:29.616508 19344 consensus_queue.cc:1035] T e2d90ff17e574f88b2061b1c117d4b7d P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:29.619268 19417 consensus_queue.cc:1035] T aec9c09cc0454acab6006bb3e10982b7 P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:29.623205 19230 consensus_queue.cc:1035] T a745334c12c545c1ba825c7552c60359 P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:29.624241 19461 consensus_queue.cc:1035] T bd11c0335a58448da56533c3f2e785a9 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.045s
I20240430 20:11:29.631688 19478 consensus_queue.cc:1035] T aec9c09cc0454acab6006bb3e10982b7 P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:29.632357 18116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNG3HqZ/build/asan/bin/kudu
/tmp/dist-test-taskNG3HqZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.177.62:36441
--webserver_interface=127.17.177.62
--webserver_port=33023
--builtin_ntp_servers=127.17.177.20:42079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.177.62:36441,127.17.177.61:34777,127.17.177.60:37225
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20240430 20:11:29.638774 18988 tablet_bootstrap.cc:654] T 5bde72e2791a497db7dd08686ca36d80 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:29.643218 18730 tablet_service.cc:1514] Processing DeleteTablet for tablet d4508e4f6dbd42109d951c648b3dc7df with delete_type TABLET_DATA_DELETED (replaced by tablet 2e79df8744e74b5991a79ce978ff0202) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:29.643575 19508 tablet.cc:1722] T bd11c0335a58448da56533c3f2e785a9 P 3c51c33d5616482cafb68e26688820f4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240430 20:11:29.644798 19230 consensus_queue.cc:1035] T aacabead3c874e77a47d2ecf19dbfa82 P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:29.645633 19489 ts_tablet_manager.cc:1888] T 0e503a898406412ab977d3e048ea4b5d P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:29.645773 19489 log.cc:1196] T 0e503a898406412ab977d3e048ea4b5d P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/0e503a898406412ab977d3e048ea4b5d
I20240430 20:11:29.645504 19484 raft_consensus.cc:504] T 02176eee8e044a7db8cac48439e9f594 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:29.646477 19495 ts_tablet_manager.cc:1888] T 0fe1247349344863a6273a769ce8ddf6 P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:29.646842 19489 ts_tablet_manager.cc:1909] T 0e503a898406412ab977d3e048ea4b5d P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:29.649524 19495 log.cc:1196] T 0fe1247349344863a6273a769ce8ddf6 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/0fe1247349344863a6273a769ce8ddf6
I20240430 20:11:29.650452 19001 catalog_manager.cc:4830] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): tablet 0e503a898406412ab977d3e048ea4b5d (table default.Table_fe1773cf77654e2a86e95444437b7201 [id=fe2222c2b62f4360a5ade622ae8b23e5]) successfully deleted
I20240430 20:11:29.650681 19495 ts_tablet_manager.cc:1909] T 0fe1247349344863a6273a769ce8ddf6 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:29.659269 18983 tablet_bootstrap.cc:492] T d4508e4f6dbd42109d951c648b3dc7df P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:29.661094 18983 tablet_bootstrap.cc:654] T d4508e4f6dbd42109d951c648b3dc7df P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:29.661554 19489 tablet_replica.cc:330] T f77086a5675f44e99e8a867b694df4b8 P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:29.663635 19004 catalog_manager.cc:4830] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): tablet 0fe1247349344863a6273a769ce8ddf6 (table default.Table_7bfb47ba65114665943b7edccabd5a8b [id=c487c255c03741199e57b71670bc84e0]) successfully deleted
I20240430 20:11:29.665330 19489 raft_consensus.cc:2229] T f77086a5675f44e99e8a867b694df4b8 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:29.665462 19029 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: Sending DeleteTablet for 3 replicas of tablet 5f56ac2afd444d63a6bfd96988464bbb
I20240430 20:11:29.665544 19489 raft_consensus.cc:2258] T f77086a5675f44e99e8a867b694df4b8 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:29.665731 19495 tablet_replica.cc:330] T 0e503a898406412ab977d3e048ea4b5d P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:29.666005 19029 catalog_manager.cc:6554] ReplaceTablet: tablet 5f56ac2afd444d63a6bfd96988464bbb deleted and replaced by tablet 2da900b6bbfa4fa99fe8494fd2e68da8
I20240430 20:11:29.665999 19495 raft_consensus.cc:2229] T 0e503a898406412ab977d3e048ea4b5d P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Raft consensus shutting down.
I20240430 20:11:29.666272 19495 pending_rounds.cc:62] T 0e503a898406412ab977d3e048ea4b5d P 2efb542aa55e439b99ccb98d1c5c884a: Trying to abort 1 pending ops.
I20240430 20:11:29.666347 19495 pending_rounds.cc:68] T 0e503a898406412ab977d3e048ea4b5d P 2efb542aa55e439b99ccb98d1c5c884a: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7022624314888204288 op_type: NO_OP noop_request { }
I20240430 20:11:29.666476 19495 raft_consensus.cc:2866] T 0e503a898406412ab977d3e048ea4b5d P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240430 20:11:29.667562 19495 raft_consensus.cc:2258] T 0e503a898406412ab977d3e048ea4b5d P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:29.670257 18588 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f56ac2afd444d63a6bfd96988464bbb with delete_type TABLET_DATA_DELETED (replaced by tablet 2da900b6bbfa4fa99fe8494fd2e68da8) from {username='slave'} at 127.0.0.1:40218
W20240430 20:11:29.671968 18415 consensus_peers.cc:485] T 15674aeb56b74210a4a2809608fd4457 P 2efb542aa55e439b99ccb98d1c5c884a -> Peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Couldn't send request to peer 5e1c6397f09747f091534faf93542710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 15674aeb56b74210a4a2809608fd4457. This is attempt 1: this message will repeat every 5th retry.
I20240430 20:11:29.672997 18462 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f56ac2afd444d63a6bfd96988464bbb with delete_type TABLET_DATA_DELETED (replaced by tablet 2da900b6bbfa4fa99fe8494fd2e68da8) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:29.680347 19048 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: Sending DeleteTablet for 3 replicas of tablet b688e03ea71940c69bda5520b62c1fc3
I20240430 20:11:29.681043 19048 catalog_manager.cc:6554] ReplaceTablet: tablet b688e03ea71940c69bda5520b62c1fc3 deleted and replaced by tablet 7683caba80594627900a0185f772c131
I20240430 20:11:29.683223 19230 raft_consensus.cc:482] T 5bde72e2791a497db7dd08686ca36d80 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:29.683739 19230 raft_consensus.cc:504] T 5bde72e2791a497db7dd08686ca36d80 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:29.684923 18588 tablet_service.cc:1514] Processing DeleteTablet for tablet b688e03ea71940c69bda5520b62c1fc3 with delete_type TABLET_DATA_DELETED (replaced by tablet 7683caba80594627900a0185f772c131) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:29.685781 18462 tablet_service.cc:1514] Processing DeleteTablet for tablet b688e03ea71940c69bda5520b62c1fc3 with delete_type TABLET_DATA_DELETED (replaced by tablet 7683caba80594627900a0185f772c131) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:29.686219 18481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bde72e2791a497db7dd08686ca36d80" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:29.686549 18481 raft_consensus.cc:2454] T 5bde72e2791a497db7dd08686ca36d80 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 0.
I20240430 20:11:29.687283 18558 leader_election.cc:302] T 5bde72e2791a497db7dd08686ca36d80 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:29.687693 19474 raft_consensus.cc:2781] T 5bde72e2791a497db7dd08686ca36d80 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:29.687661 19027 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_71075ae6b4d545889273c891648c70d1" } new_table_name: "default.Table_c12ca6938d424fbfa2e1df5cdb00fdbc" modify_external_catalogs: true
I20240430 20:11:29.687798 19474 raft_consensus.cc:482] T 5bde72e2791a497db7dd08686ca36d80 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:29.687942 19474 raft_consensus.cc:3037] T 5bde72e2791a497db7dd08686ca36d80 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:29.689642 19029 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_01d62e066e4942d9aa1f582189d60d49"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:29.690375 19029 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_01d62e066e4942d9aa1f582189d60d49 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:29.692124 19023 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_9579e986b42043f18f8e4aa8d0f6f754"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240430 20:11:29.692314 19474 raft_consensus.cc:504] T 5bde72e2791a497db7dd08686ca36d80 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
W20240430 20:11:29.692760 19023 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9579e986b42043f18f8e4aa8d0f6f754 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:29.694381 19048 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_8e9e6e39df854972804dfa7c260af4e4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240430 20:11:29.695008 18481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bde72e2791a497db7dd08686ca36d80" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
W20240430 20:11:29.695113 19048 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8e9e6e39df854972804dfa7c260af4e4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:29.695327 18481 raft_consensus.cc:3037] T 5bde72e2791a497db7dd08686ca36d80 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:29.698884 19474 leader_election.cc:288] T 5bde72e2791a497db7dd08686ca36d80 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:29.700206 18481 raft_consensus.cc:2454] T 5bde72e2791a497db7dd08686ca36d80 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
I20240430 20:11:29.701022 18558 leader_election.cc:302] T 5bde72e2791a497db7dd08686ca36d80 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:29.701939 19474 raft_consensus.cc:2781] T 5bde72e2791a497db7dd08686ca36d80 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:29.703964 19474 raft_consensus.cc:686] T 5bde72e2791a497db7dd08686ca36d80 P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Becoming Leader. State: Replica: 5e1c6397f09747f091534faf93542710, State: Running, Role: LEADER
I20240430 20:11:29.704504 19474 consensus_queue.cc:237] T 5bde72e2791a497db7dd08686ca36d80 P 5e1c6397f09747f091534faf93542710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:29.705082 19386 raft_consensus.cc:482] T 7dfdecba959543a795dfcd3eeda29e58 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:29.705392 19386 raft_consensus.cc:504] T 7dfdecba959543a795dfcd3eeda29e58 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:29.706378 19386 leader_election.cc:288] T 7dfdecba959543a795dfcd3eeda29e58 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:29.708621 18985 tablet_bootstrap.cc:492] T d4508e4f6dbd42109d951c648b3dc7df P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:29.708889 18985 ts_tablet_manager.cc:1367] T d4508e4f6dbd42109d951c648b3dc7df P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.487s user 0.011s sys 0.054s
I20240430 20:11:29.709312 19230 leader_election.cc:288] T 5bde72e2791a497db7dd08686ca36d80 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:29.709623 18481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dfdecba959543a795dfcd3eeda29e58" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:29.710172 18985 raft_consensus.cc:348] T d4508e4f6dbd42109d951c648b3dc7df P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:29.710508 18985 raft_consensus.cc:374] T d4508e4f6dbd42109d951c648b3dc7df P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:29.710654 18985 raft_consensus.cc:729] T d4508e4f6dbd42109d951c648b3dc7df P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
W20240430 20:11:29.711465 18558 leader_election.cc:341] T 7dfdecba959543a795dfcd3eeda29e58 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:29.721252 19028 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_3c5df662747b48bf8beb6ebe4d2ea618" } modify_external_catalogs: true
I20240430 20:11:29.721624 19028 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_3c5df662747b48bf8beb6ebe4d2ea618" } modify_external_catalogs: true
I20240430 20:11:29.722687 19049 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_ca129cca71d34045824d5f945570c3bd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:29.724326 19049 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ca129cca71d34045824d5f945570c3bd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:29.734884 19495 ts_tablet_manager.cc:1875] T 0e503a898406412ab977d3e048ea4b5d P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:29.736304 19471 ts_tablet_manager.cc:1888] T 0e503a898406412ab977d3e048ea4b5d P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:29.736460 19471 log.cc:1196] T 0e503a898406412ab977d3e048ea4b5d P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/0e503a898406412ab977d3e048ea4b5d
I20240430 20:11:29.737928 18623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02176eee8e044a7db8cac48439e9f594" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:29.738255 18623 raft_consensus.cc:3037] T 02176eee8e044a7db8cac48439e9f594 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:29.738854 19471 ts_tablet_manager.cc:1909] T 0e503a898406412ab977d3e048ea4b5d P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:29.740213 19471 tablet_replica.cc:330] T f77086a5675f44e99e8a867b694df4b8 P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:29.740061 18999 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet 0e503a898406412ab977d3e048ea4b5d (table default.Table_fe1773cf77654e2a86e95444437b7201 [id=fe2222c2b62f4360a5ade622ae8b23e5]) successfully deleted
I20240430 20:11:29.740532 19471 raft_consensus.cc:2229] T f77086a5675f44e99e8a867b694df4b8 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Raft consensus shutting down.
W20240430 20:11:29.740561 18999 catalog_manager.cc:4810] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): delete failed for tablet d4508e4f6dbd42109d951c648b3dc7df because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d4508e4f6dbd42109d951c648b3dc7df already in progress: creating tablet
I20240430 20:11:29.739711 19230 consensus_queue.cc:1035] T fae9a35a2ac844b4bc2ebdf214aa7171 P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240430 20:11:29.740957 19471 raft_consensus.cc:2258] T f77086a5675f44e99e8a867b694df4b8 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:29.743114 18983 tablet_bootstrap.cc:492] T d4508e4f6dbd42109d951c648b3dc7df P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:29.743337 18983 ts_tablet_manager.cc:1367] T d4508e4f6dbd42109d951c648b3dc7df P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.149s user 0.003s sys 0.000s
I20240430 20:11:29.743800 18985 consensus_queue.cc:260] T d4508e4f6dbd42109d951c648b3dc7df P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:29.744657 18985 ts_tablet_manager.cc:1398] T d4508e4f6dbd42109d951c648b3dc7df P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.036s user 0.000s sys 0.002s
I20240430 20:11:29.745026 18985 tablet_bootstrap.cc:492] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:29.745230 19523 tablet.cc:1722] T bd11c0335a58448da56533c3f2e785a9 P 5e1c6397f09747f091534faf93542710: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240430 20:11:29.747335 19471 ts_tablet_manager.cc:1875] T f77086a5675f44e99e8a867b694df4b8 P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:29.753072 19029 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_42c1cf2f2e9542feaaa80106ced02347"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:29.753868 19029 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_42c1cf2f2e9542feaaa80106ced02347 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:29.755766 19471 ts_tablet_manager.cc:1888] T f77086a5675f44e99e8a867b694df4b8 P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:29.755991 19471 log.cc:1196] T f77086a5675f44e99e8a867b694df4b8 P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/f77086a5675f44e99e8a867b694df4b8
I20240430 20:11:29.756237 19484 leader_election.cc:288] T 02176eee8e044a7db8cac48439e9f594 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Requested vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:29.756445 19489 ts_tablet_manager.cc:1875] T f77086a5675f44e99e8a867b694df4b8 P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:29.756675 19471 ts_tablet_manager.cc:1909] T f77086a5675f44e99e8a867b694df4b8 P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:29.757134 18730 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f56ac2afd444d63a6bfd96988464bbb with delete_type TABLET_DATA_DELETED (replaced by tablet 2da900b6bbfa4fa99fe8494fd2e68da8) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:29.759263 19261 raft_consensus.cc:482] T 29974dc2c7dd45b99b6cc0712378fc30 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:29.761087 19261 raft_consensus.cc:504] T 29974dc2c7dd45b99b6cc0712378fc30 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:29.762164 19427 raft_consensus.cc:482] T d4508e4f6dbd42109d951c648b3dc7df P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:29.763458 18623 raft_consensus.cc:2454] T 02176eee8e044a7db8cac48439e9f594 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:29.763756 18735 tablet_service.cc:1514] Processing DeleteTablet for tablet b688e03ea71940c69bda5520b62c1fc3 with delete_type TABLET_DATA_DELETED (replaced by tablet 7683caba80594627900a0185f772c131) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:29.763856 18760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bde72e2791a497db7dd08686ca36d80" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:29.765565 19261 leader_election.cc:288] T 29974dc2c7dd45b99b6cc0712378fc30 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:29.765834 18763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dfdecba959543a795dfcd3eeda29e58" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:29.766614 19427 raft_consensus.cc:504] T d4508e4f6dbd42109d951c648b3dc7df P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:29.767714 18983 raft_consensus.cc:348] T d4508e4f6dbd42109d951c648b3dc7df P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:29.767994 18983 raft_consensus.cc:374] T d4508e4f6dbd42109d951c648b3dc7df P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:29.768086 18983 raft_consensus.cc:729] T d4508e4f6dbd42109d951c648b3dc7df P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:29.768357 18983 consensus_queue.cc:260] T d4508e4f6dbd42109d951c648b3dc7df P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
W20240430 20:11:29.764623 18557 leader_election.cc:341] T 5bde72e2791a497db7dd08686ca36d80 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
W20240430 20:11:29.769165 18557 leader_election.cc:341] T 7dfdecba959543a795dfcd3eeda29e58 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:29.769299 18557 leader_election.cc:302] T 7dfdecba959543a795dfcd3eeda29e58 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:29.770236 19386 raft_consensus.cc:2726] T 7dfdecba959543a795dfcd3eeda29e58 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:29.774190 18481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29974dc2c7dd45b99b6cc0712378fc30" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:29.776239 19471 tablet_replica.cc:330] T 5f56ac2afd444d63a6bfd96988464bbb P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:29.776499 19471 raft_consensus.cc:2229] T 5f56ac2afd444d63a6bfd96988464bbb P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:29.776683 19471 raft_consensus.cc:2258] T 5f56ac2afd444d63a6bfd96988464bbb P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:29.777300 19471 ts_tablet_manager.cc:1875] T 5f56ac2afd444d63a6bfd96988464bbb P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:29.779256 18620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29974dc2c7dd45b99b6cc0712378fc30" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:29.780062 18620 raft_consensus.cc:2379] T 29974dc2c7dd45b99b6cc0712378fc30 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c51c33d5616482cafb68e26688820f4 in current term 1: Already voted for candidate 2efb542aa55e439b99ccb98d1c5c884a in this term.
I20240430 20:11:29.781214 18999 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet f77086a5675f44e99e8a867b694df4b8 (table default.Table_cd719a43207f4bf983e1b8e6d236bb16 [id=47e0b04b17834deb833a8f31c123edc5]) successfully deleted
I20240430 20:11:29.790459 19023 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_a7add9298f1a44c6a2b2a9d73405c834" } new_table_name: "default.Table_9c1e6182953140bb9cd594f08458c7d9" modify_external_catalogs: true
I20240430 20:11:29.793216 19471 ts_tablet_manager.cc:1888] T 5f56ac2afd444d63a6bfd96988464bbb P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:29.794229 19048 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_f61a45518e134c4496e4ac81fbe40fce" } modify_external_catalogs: true
I20240430 20:11:29.794517 19048 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_f61a45518e134c4496e4ac81fbe40fce" } modify_external_catalogs: true
I20240430 20:11:29.796440 18985 tablet_bootstrap.cc:654] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:29.798440 19482 consensus_queue.cc:1035] T e2d90ff17e574f88b2061b1c117d4b7d P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:29.801141 18482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02176eee8e044a7db8cac48439e9f594" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:29.806121 18482 raft_consensus.cc:3037] T 02176eee8e044a7db8cac48439e9f594 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:29.806466 19482 consensus_queue.cc:1035] T c5fd0730133c472299c93f4b0830e527 P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:29.809900 19049 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_8af52c0f1d404b42bc53287f64e20c2d" } modify_external_catalogs: true
I20240430 20:11:29.810223 19049 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_8af52c0f1d404b42bc53287f64e20c2d" } modify_external_catalogs: true
I20240430 20:11:29.813743 19029 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_44426457fd7045dfa1c149f0a5ed5129" } new_table_name: "default.Table_421069f84cb844368066c93102a8faf8" modify_external_catalogs: true
I20240430 20:11:29.819734 19471 log.cc:1196] T 5f56ac2afd444d63a6bfd96988464bbb P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/5f56ac2afd444d63a6bfd96988464bbb
I20240430 20:11:29.766515 18758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bde72e2791a497db7dd08686ca36d80" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:29.820227 19471 ts_tablet_manager.cc:1909] T 5f56ac2afd444d63a6bfd96988464bbb P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
W20240430 20:11:29.820642 18557 leader_election.cc:341] T 5bde72e2791a497db7dd08686ca36d80 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:29.769161 18983 ts_tablet_manager.cc:1398] T d4508e4f6dbd42109d951c648b3dc7df P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.026s user 0.001s sys 0.000s
I20240430 20:11:29.811035 19474 consensus_queue.cc:1035] T a1aeee1b02214975a376cfd36957afe8 P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:29.822506 18623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4508e4f6dbd42109d951c648b3dc7df" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:29.822788 18623 raft_consensus.cc:2454] T d4508e4f6dbd42109d951c648b3dc7df P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:29.823343 18983 tablet_bootstrap.cc:492] T 68ca48351cd4412381f5195d81109016 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:29.825695 18482 raft_consensus.cc:2454] T 02176eee8e044a7db8cac48439e9f594 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:29.826242 19386 consensus_queue.cc:1035] T aacabead3c874e77a47d2ecf19dbfa82 P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240430 20:11:29.827064 19427 leader_election.cc:288] T d4508e4f6dbd42109d951c648b3dc7df P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:29.827797 18983 tablet_bootstrap.cc:654] T 68ca48351cd4412381f5195d81109016 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:29.827807 18696 leader_election.cc:302] T 02176eee8e044a7db8cac48439e9f594 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:29.828182 18696 leader_election.cc:302] T 29974dc2c7dd45b99b6cc0712378fc30 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710
I20240430 20:11:29.828619 18696 leader_election.cc:302] T d4508e4f6dbd42109d951c648b3dc7df P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:29.829202 19492 raft_consensus.cc:2781] T 02176eee8e044a7db8cac48439e9f594 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:29.829319 19492 raft_consensus.cc:686] T 02176eee8e044a7db8cac48439e9f594 P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Becoming Leader. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Running, Role: LEADER
I20240430 20:11:29.829604 19492 consensus_queue.cc:237] T 02176eee8e044a7db8cac48439e9f594 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:29.830706 19484 raft_consensus.cc:3037] T 29974dc2c7dd45b99b6cc0712378fc30 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:29.832556 19386 raft_consensus.cc:482] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:29.833885 19484 raft_consensus.cc:2726] T 29974dc2c7dd45b99b6cc0712378fc30 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:29.834165 19484 raft_consensus.cc:2781] T d4508e4f6dbd42109d951c648b3dc7df P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:29.834081 19386 raft_consensus.cc:504] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:29.834260 19484 raft_consensus.cc:482] T d4508e4f6dbd42109d951c648b3dc7df P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:29.834378 19484 raft_consensus.cc:3037] T d4508e4f6dbd42109d951c648b3dc7df P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:29.834851 19386 leader_election.cc:288] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:29.835651 18483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd6083f12c7e4d1e8a2006b3ddf9aa3a" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
W20240430 20:11:29.836217 18558 leader_election.cc:341] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:29.836505 18999 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet 5f56ac2afd444d63a6bfd96988464bbb (table default.Table_19a91a7f17ff456ebb145e8dd89c0940 [id=67283dd3995a44c6a7a7bc8722e4f955]) successfully deleted
W20240430 20:11:29.836777 18999 catalog_manager.cc:4810] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): delete failed for tablet b688e03ea71940c69bda5520b62c1fc3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b688e03ea71940c69bda5520b62c1fc3 already in progress: creating tablet
I20240430 20:11:29.837234 18988 tablet_bootstrap.cc:492] T 5bde72e2791a497db7dd08686ca36d80 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:29.837385 18988 ts_tablet_manager.cc:1367] T 5bde72e2791a497db7dd08686ca36d80 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.269s user 0.013s sys 0.013s
I20240430 20:11:29.838248 18988 raft_consensus.cc:348] T 5bde72e2791a497db7dd08686ca36d80 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:29.838485 18988 raft_consensus.cc:374] T 5bde72e2791a497db7dd08686ca36d80 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:29.838577 18988 raft_consensus.cc:729] T 5bde72e2791a497db7dd08686ca36d80 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:29.838819 18988 consensus_queue.cc:260] T 5bde72e2791a497db7dd08686ca36d80 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:29.839391 18988 ts_tablet_manager.cc:1398] T 5bde72e2791a497db7dd08686ca36d80 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240430 20:11:29.839686 18988 tablet_bootstrap.cc:492] T 7dfdecba959543a795dfcd3eeda29e58 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:29.841385 18988 tablet_bootstrap.cc:654] T 7dfdecba959543a795dfcd3eeda29e58 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:29.843822 18483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4508e4f6dbd42109d951c648b3dc7df" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:29.844125 18483 raft_consensus.cc:2454] T d4508e4f6dbd42109d951c648b3dc7df P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:29.844386 18758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd6083f12c7e4d1e8a2006b3ddf9aa3a" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:29.844638 18758 raft_consensus.cc:2454] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 0.
I20240430 20:11:29.845281 18557 leader_election.cc:302] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a
I20240430 20:11:29.846109 19180 raft_consensus.cc:2781] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:29.846242 19180 raft_consensus.cc:482] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:29.846386 19180 raft_consensus.cc:3037] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:29.850104 19180 raft_consensus.cc:504] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:29.853394 19484 raft_consensus.cc:504] T d4508e4f6dbd42109d951c648b3dc7df P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:29.854719 19484 leader_election.cc:288] T d4508e4f6dbd42109d951c648b3dc7df P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Requested vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:29.856745 18483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4508e4f6dbd42109d951c648b3dc7df" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:29.857048 18483 raft_consensus.cc:3037] T d4508e4f6dbd42109d951c648b3dc7df P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:29.860322 19489 ts_tablet_manager.cc:1888] T f77086a5675f44e99e8a867b694df4b8 P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:29.853608 18988 tablet_bootstrap.cc:492] T 7dfdecba959543a795dfcd3eeda29e58 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:29.860662 19489 log.cc:1196] T f77086a5675f44e99e8a867b694df4b8 P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/f77086a5675f44e99e8a867b694df4b8
I20240430 20:11:29.860936 18983 tablet_bootstrap.cc:492] T 68ca48351cd4412381f5195d81109016 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:29.861097 18983 ts_tablet_manager.cc:1367] T 68ca48351cd4412381f5195d81109016 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.038s user 0.005s sys 0.001s
I20240430 20:11:29.861205 19489 ts_tablet_manager.cc:1909] T f77086a5675f44e99e8a867b694df4b8 P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:29.861850 19001 catalog_manager.cc:4830] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): tablet f77086a5675f44e99e8a867b694df4b8 (table default.Table_cd719a43207f4bf983e1b8e6d236bb16 [id=47e0b04b17834deb833a8f31c123edc5]) successfully deleted
I20240430 20:11:29.861959 18983 raft_consensus.cc:348] T 68ca48351cd4412381f5195d81109016 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:29.862162 18983 raft_consensus.cc:374] T 68ca48351cd4412381f5195d81109016 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:29.862249 18983 raft_consensus.cc:729] T 68ca48351cd4412381f5195d81109016 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:29.862342 18988 ts_tablet_manager.cc:1367] T 7dfdecba959543a795dfcd3eeda29e58 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.003s
I20240430 20:11:29.862576 18983 consensus_queue.cc:260] T 68ca48351cd4412381f5195d81109016 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:29.863303 18983 ts_tablet_manager.cc:1398] T 68ca48351cd4412381f5195d81109016 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20240430 20:11:29.863258 18483 raft_consensus.cc:2454] T d4508e4f6dbd42109d951c648b3dc7df P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:29.863256 18988 raft_consensus.cc:348] T 7dfdecba959543a795dfcd3eeda29e58 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:29.863528 19489 tablet_replica.cc:330] T d4508e4f6dbd42109d951c648b3dc7df P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:29.863615 18983 tablet_bootstrap.cc:492] T c66c142419ca4e1399ecd1c8d1e62a59 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:29.863729 19489 raft_consensus.cc:2229] T d4508e4f6dbd42109d951c648b3dc7df P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:29.863991 19489 raft_consensus.cc:2258] T d4508e4f6dbd42109d951c648b3dc7df P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:29.864233 18988 raft_consensus.cc:374] T 7dfdecba959543a795dfcd3eeda29e58 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:29.866068 18620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4508e4f6dbd42109d951c648b3dc7df" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:29.866333 18620 raft_consensus.cc:3037] T d4508e4f6dbd42109d951c648b3dc7df P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:29.866096 18983 tablet_bootstrap.cc:654] T c66c142419ca4e1399ecd1c8d1e62a59 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:29.867905 19180 leader_election.cc:288] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:29.869257 18620 raft_consensus.cc:2454] T d4508e4f6dbd42109d951c648b3dc7df P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:29.870530 19495 ts_tablet_manager.cc:1888] T 0e503a898406412ab977d3e048ea4b5d P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:29.870831 18483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd6083f12c7e4d1e8a2006b3ddf9aa3a" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
W20240430 20:11:29.898692 18558 leader_election.cc:341] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:29.899505 19495 log.cc:1196] T 0e503a898406412ab977d3e048ea4b5d P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/0e503a898406412ab977d3e048ea4b5d
I20240430 20:11:29.900053 19495 ts_tablet_manager.cc:1909] T 0e503a898406412ab977d3e048ea4b5d P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:29.902164 18985 tablet_bootstrap.cc:492] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:29.902362 18985 ts_tablet_manager.cc:1367] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.157s user 0.000s sys 0.004s
I20240430 20:11:29.903306 18985 raft_consensus.cc:348] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:29.903654 19004 catalog_manager.cc:4830] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): tablet 0e503a898406412ab977d3e048ea4b5d (table default.Table_fe1773cf77654e2a86e95444437b7201 [id=fe2222c2b62f4360a5ade622ae8b23e5]) successfully deleted
I20240430 20:11:29.903769 18985 raft_consensus.cc:374] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:29.903841 18985 raft_consensus.cc:729] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:29.904129 18985 consensus_queue.cc:260] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:29.904732 18985 ts_tablet_manager.cc:1398] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240430 20:11:29.905083 18985 tablet_bootstrap.cc:492] T f77086a5675f44e99e8a867b694df4b8 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:29.907044 18985 tablet_bootstrap.cc:654] T f77086a5675f44e99e8a867b694df4b8 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:29.909413 18588 tablet_service.cc:1467] Processing CreateTablet for tablet 3fd0e8c360734dcbb75dc2958b70c2b8 (DEFAULT_TABLE table=default.Table_02d2b1f3815549c481a305c622c809ff [id=7e61baf429e4487fab0ba2761a2af49d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:29.909948 18588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fd0e8c360734dcbb75dc2958b70c2b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:29.911361 19495 tablet_replica.cc:330] T d4508e4f6dbd42109d951c648b3dc7df P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:29.911610 19495 raft_consensus.cc:2229] T d4508e4f6dbd42109d951c648b3dc7df P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:29.911780 19495 raft_consensus.cc:2258] T d4508e4f6dbd42109d951c648b3dc7df P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:29.913429 18983 tablet_bootstrap.cc:492] T c66c142419ca4e1399ecd1c8d1e62a59 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:29.913674 18983 ts_tablet_manager.cc:1367] T c66c142419ca4e1399ecd1c8d1e62a59 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.050s user 0.007s sys 0.011s
I20240430 20:11:29.915185 19495 ts_tablet_manager.cc:1875] T d4508e4f6dbd42109d951c648b3dc7df P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:29.916066 18983 raft_consensus.cc:348] T c66c142419ca4e1399ecd1c8d1e62a59 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:29.916409 18983 raft_consensus.cc:374] T c66c142419ca4e1399ecd1c8d1e62a59 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:29.916525 18983 raft_consensus.cc:729] T c66c142419ca4e1399ecd1c8d1e62a59 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:29.916898 18983 consensus_queue.cc:260] T c66c142419ca4e1399ecd1c8d1e62a59 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:29.922046 19495 ts_tablet_manager.cc:1888] T d4508e4f6dbd42109d951c648b3dc7df P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:29.922320 18988 raft_consensus.cc:729] T 7dfdecba959543a795dfcd3eeda29e58 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:29.922816 18699 leader_election.cc:302] T d4508e4f6dbd42109d951c648b3dc7df P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters:
I20240430 20:11:29.924870 18983 ts_tablet_manager.cc:1398] T c66c142419ca4e1399ecd1c8d1e62a59 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.011s user 0.002s sys 0.002s
I20240430 20:11:29.925542 18988 consensus_queue.cc:260] T 7dfdecba959543a795dfcd3eeda29e58 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:29.926182 18988 ts_tablet_manager.cc:1398] T 7dfdecba959543a795dfcd3eeda29e58 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.064s user 0.002s sys 0.000s
I20240430 20:11:29.926752 18988 tablet_bootstrap.cc:492] T e5111d7607dd42188a0176cf5be2ec05 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:29.928364 18983 tablet_bootstrap.cc:492] T 724e51ff05ec4baebc2bb55e05c8eddc P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:29.928532 18988 tablet_bootstrap.cc:654] T e5111d7607dd42188a0176cf5be2ec05 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:29.922746 19489 ts_tablet_manager.cc:1875] T d4508e4f6dbd42109d951c648b3dc7df P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:29.930670 18983 tablet_bootstrap.cc:654] T 724e51ff05ec4baebc2bb55e05c8eddc P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
W20240430 20:11:29.931721 18699 raft_consensus.cc:2670] T d4508e4f6dbd42109d951c648b3dc7df P 3c51c33d5616482cafb68e26688820f4: unable to run election callback: Service unavailable: Thread pool token was shut down
W20240430 20:11:29.931886 18699 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.177.1:41217, user_credentials={real_user=slave}} blocked reactor thread for 67579.4us
I20240430 20:11:29.932806 18735 tablet_service.cc:1467] Processing CreateTablet for tablet 3fd0e8c360734dcbb75dc2958b70c2b8 (DEFAULT_TABLE table=default.Table_02d2b1f3815549c481a305c622c809ff [id=7e61baf429e4487fab0ba2761a2af49d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:29.934000 18735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fd0e8c360734dcbb75dc2958b70c2b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:29.936484 19230 consensus_queue.cc:1035] T a1aeee1b02214975a376cfd36957afe8 P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240430 20:11:29.942699 18696 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.177.2:46027, user_credentials={real_user=slave}} blocked reactor thread for 67995.4us
I20240430 20:11:29.944880 18764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd6083f12c7e4d1e8a2006b3ddf9aa3a" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:29.945132 18764 raft_consensus.cc:3037] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:29.923364 18462 tablet_service.cc:1467] Processing CreateTablet for tablet 3fd0e8c360734dcbb75dc2958b70c2b8 (DEFAULT_TABLE table=default.Table_02d2b1f3815549c481a305c622c809ff [id=7e61baf429e4487fab0ba2761a2af49d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:29.949891 18462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fd0e8c360734dcbb75dc2958b70c2b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:29.952682 18764 raft_consensus.cc:2454] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
I20240430 20:11:29.953462 18557 leader_election.cc:302] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a
I20240430 20:11:29.953998 19230 raft_consensus.cc:2781] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:29.954144 19230 raft_consensus.cc:686] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Becoming Leader. State: Replica: 5e1c6397f09747f091534faf93542710, State: Running, Role: LEADER
I20240430 20:11:29.954444 19230 consensus_queue.cc:237] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 5e1c6397f09747f091534faf93542710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:29.941469 19478 consensus_queue.cc:1035] T c5fd0730133c472299c93f4b0830e527 P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.013s
I20240430 20:11:29.981151 19185 consensus_queue.cc:1035] T 6e826459c3f4437da81c9e6e09f0c22f P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:29.922226 19495 log.cc:1196] T d4508e4f6dbd42109d951c648b3dc7df P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/d4508e4f6dbd42109d951c648b3dc7df
I20240430 20:11:29.991680 19495 ts_tablet_manager.cc:1909] T d4508e4f6dbd42109d951c648b3dc7df P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:29.992954 19495 tablet_replica.cc:330] T 5f56ac2afd444d63a6bfd96988464bbb P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:29.993191 19495 raft_consensus.cc:2229] T 5f56ac2afd444d63a6bfd96988464bbb P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:29.993358 19004 catalog_manager.cc:4830] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): tablet d4508e4f6dbd42109d951c648b3dc7df (table default.Table_276bbdc7cfe542d1a2936485a2aad1ec [id=f146a15e6e8a4acbb4c734c1c863da57]) successfully deleted
W20240430 20:11:29.993716 19004 catalog_manager.cc:4810] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): delete failed for tablet f77086a5675f44e99e8a867b694df4b8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f77086a5675f44e99e8a867b694df4b8 already in progress: creating tablet
I20240430 20:11:29.994360 18588 tablet_service.cc:1467] Processing CreateTablet for tablet ca64de16d40f4a44b8b411f5526ba11a (DEFAULT_TABLE table=default.Table_f5a3451cd2284390be4357d771c03390 [id=f4ba9536e9c141deb68f4f62b7e8fa70]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:29.994987 18588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca64de16d40f4a44b8b411f5526ba11a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:29.997797 18730 tablet_service.cc:1467] Processing CreateTablet for tablet ca64de16d40f4a44b8b411f5526ba11a (DEFAULT_TABLE table=default.Table_f5a3451cd2284390be4357d771c03390 [id=f4ba9536e9c141deb68f4f62b7e8fa70]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:29.998250 18730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca64de16d40f4a44b8b411f5526ba11a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:30.000978 19386 raft_consensus.cc:482] T 68ca48351cd4412381f5195d81109016 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:30.001159 19386 raft_consensus.cc:504] T 68ca48351cd4412381f5195d81109016 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:30.001909 19386 leader_election.cc:288] T 68ca48351cd4412381f5195d81109016 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:30.004426 18593 tablet_service.cc:1467] Processing CreateTablet for tablet 2f61e3f88a6448dfa323204bd8d14cc5 (DEFAULT_TABLE table=default.Table_db56d63aef494592b69246a0c13006f2 [id=cde04507ee1d464f9fffc44176da8d16]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:30.004932 18593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f61e3f88a6448dfa323204bd8d14cc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:30.007329 18599 tablet_service.cc:1467] Processing CreateTablet for tablet 7542c7fe6cad452ab718f9ef54dbbbbe (DEFAULT_TABLE table=default.Table_8af52c0f1d404b42bc53287f64e20c2d [id=8e590e0e404344658b5cee90132d91d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:30.007783 18599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7542c7fe6cad452ab718f9ef54dbbbbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:29.995297 19482 consensus_queue.cc:1035] T f5236574672b477a9026e91adcaf70ec P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:29.998836 18462 tablet_service.cc:1467] Processing CreateTablet for tablet ca64de16d40f4a44b8b411f5526ba11a (DEFAULT_TABLE table=default.Table_f5a3451cd2284390be4357d771c03390 [id=f4ba9536e9c141deb68f4f62b7e8fa70]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:30.012528 18462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca64de16d40f4a44b8b411f5526ba11a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:30.002427 18483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68ca48351cd4412381f5195d81109016" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:30.004496 18456 tablet_service.cc:1467] Processing CreateTablet for tablet 2f61e3f88a6448dfa323204bd8d14cc5 (DEFAULT_TABLE table=default.Table_db56d63aef494592b69246a0c13006f2 [id=cde04507ee1d464f9fffc44176da8d16]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:30.013372 18456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f61e3f88a6448dfa323204bd8d14cc5. 1 dirs total, 0 dirs full, 0 dirs failed
W20240430 20:11:30.017117 18558 leader_election.cc:341] T 68ca48351cd4412381f5195d81109016 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:30.021287 19495 raft_consensus.cc:2258] T 5f56ac2afd444d63a6bfd96988464bbb P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:30.021955 19495 ts_tablet_manager.cc:1875] T 5f56ac2afd444d63a6bfd96988464bbb P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:30.027966 18760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68ca48351cd4412381f5195d81109016" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:30.028851 18736 tablet_service.cc:1467] Processing CreateTablet for tablet 7542c7fe6cad452ab718f9ef54dbbbbe (DEFAULT_TABLE table=default.Table_8af52c0f1d404b42bc53287f64e20c2d [id=8e590e0e404344658b5cee90132d91d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:30.029282 18736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7542c7fe6cad452ab718f9ef54dbbbbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:30.030639 18741 tablet_service.cc:1467] Processing CreateTablet for tablet 2f61e3f88a6448dfa323204bd8d14cc5 (DEFAULT_TABLE table=default.Table_db56d63aef494592b69246a0c13006f2 [id=cde04507ee1d464f9fffc44176da8d16]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:30.031375 18741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f61e3f88a6448dfa323204bd8d14cc5. 1 dirs total, 0 dirs full, 0 dirs failed
W20240430 20:11:30.042886 18557 leader_election.cc:341] T 68ca48351cd4412381f5195d81109016 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:30.043248 18557 leader_election.cc:302] T 68ca48351cd4412381f5195d81109016 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:30.043888 19386 raft_consensus.cc:2726] T 68ca48351cd4412381f5195d81109016 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:30.029232 18456 tablet_service.cc:1467] Processing CreateTablet for tablet 7542c7fe6cad452ab718f9ef54dbbbbe (DEFAULT_TABLE table=default.Table_8af52c0f1d404b42bc53287f64e20c2d [id=8e590e0e404344658b5cee90132d91d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:30.045260 18456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7542c7fe6cad452ab718f9ef54dbbbbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:30.043582 19495 ts_tablet_manager.cc:1888] T 5f56ac2afd444d63a6bfd96988464bbb P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:30.051028 19495 log.cc:1196] T 5f56ac2afd444d63a6bfd96988464bbb P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/5f56ac2afd444d63a6bfd96988464bbb
I20240430 20:11:30.051715 19495 ts_tablet_manager.cc:1909] T 5f56ac2afd444d63a6bfd96988464bbb P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
W20240430 20:11:30.054541 18417 consensus_peers.cc:485] T f5236574672b477a9026e91adcaf70ec P 2efb542aa55e439b99ccb98d1c5c884a -> Peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Couldn't send request to peer 3c51c33d5616482cafb68e26688820f4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f5236574672b477a9026e91adcaf70ec. This is attempt 1: this message will repeat every 5th retry.
I20240430 20:11:30.054831 18462 tablet_service.cc:1467] Processing CreateTablet for tablet 73453a7fa30342e29db552b17b62f0c1 (DEFAULT_TABLE table=default.Table_7a9f9a8c27bb4392acddd2a7ea3cc3b2 [id=cfad6f932073488b8097ef7381bc9fd1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:30.055250 19004 catalog_manager.cc:4830] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): tablet 5f56ac2afd444d63a6bfd96988464bbb (table default.Table_19a91a7f17ff456ebb145e8dd89c0940 [id=67283dd3995a44c6a7a7bc8722e4f955]) successfully deleted
I20240430 20:11:30.055596 18462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73453a7fa30342e29db552b17b62f0c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:30.058223 18593 tablet_service.cc:1467] Processing CreateTablet for tablet 73453a7fa30342e29db552b17b62f0c1 (DEFAULT_TABLE table=default.Table_7a9f9a8c27bb4392acddd2a7ea3cc3b2 [id=cfad6f932073488b8097ef7381bc9fd1]), partition=RANGE (key) PARTITION UNBOUNDED
W20240430 20:11:30.058415 19004 catalog_manager.cc:4810] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): delete failed for tablet b688e03ea71940c69bda5520b62c1fc3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b688e03ea71940c69bda5520b62c1fc3 already in progress: creating tablet
I20240430 20:11:30.058930 18593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73453a7fa30342e29db552b17b62f0c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:30.060899 18597 tablet_service.cc:1467] Processing CreateTablet for tablet ae87f5e92a934e2182164c231bc76df0 (DEFAULT_TABLE table=default.Table_27f3d711beeb4387a9088ab291168bc8 [id=434aba8e131342a3a9a1aca3278f9c0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:30.060951 18457 tablet_service.cc:1467] Processing CreateTablet for tablet 5f02d8862e1f4ccfa9dbda73dbe18615 (DEFAULT_TABLE table=default.Table_0b4b99c1c03a4e779046063159ce6861 [id=43d7bdc84bab4c54bac58dd1dd0f74ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:30.061321 18597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae87f5e92a934e2182164c231bc76df0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:30.061419 18457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f02d8862e1f4ccfa9dbda73dbe18615. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:30.066610 18454 tablet_service.cc:1467] Processing CreateTablet for tablet 34130f2820064fa0898ef1f9d7de1c6d (DEFAULT_TABLE table=default.Table_4537f891b01048ceb51202abc3600911 [id=5a1c4a7be2fc465fac89a50854eb3322]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:30.067107 18455 tablet_service.cc:1467] Processing CreateTablet for tablet ae87f5e92a934e2182164c231bc76df0 (DEFAULT_TABLE table=default.Table_27f3d711beeb4387a9088ab291168bc8 [id=434aba8e131342a3a9a1aca3278f9c0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:30.067107 18454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34130f2820064fa0898ef1f9d7de1c6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:30.067881 18591 tablet_service.cc:1467] Processing CreateTablet for tablet 34130f2820064fa0898ef1f9d7de1c6d (DEFAULT_TABLE table=default.Table_4537f891b01048ceb51202abc3600911 [id=5a1c4a7be2fc465fac89a50854eb3322]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:30.068277 18591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34130f2820064fa0898ef1f9d7de1c6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:30.069380 18450 tablet_service.cc:1467] Processing CreateTablet for tablet c2b019f529a54228b804437bb025e31d (DEFAULT_TABLE table=default.Table_8028554bc993419e80dbf7d526b63bf4 [id=27ef5b3429fb4353afe80dc3089e9914]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:30.069782 18450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2b019f529a54228b804437bb025e31d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:30.070760 18455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae87f5e92a934e2182164c231bc76df0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:30.075526 18586 tablet_service.cc:1467] Processing CreateTablet for tablet c2b019f529a54228b804437bb025e31d (DEFAULT_TABLE table=default.Table_8028554bc993419e80dbf7d526b63bf4 [id=27ef5b3429fb4353afe80dc3089e9914]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:30.076021 18586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2b019f529a54228b804437bb025e31d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:30.080359 18450 tablet_service.cc:1467] Processing CreateTablet for tablet 46251c08277e498da04a41d009310c62 (DEFAULT_TABLE table=default.Table_d5b955117a0a4c1eb8ed8151e155b32e [id=b1312a6cd1a54003b1db727a2a9d125d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:30.080624 18724 tablet_service.cc:1467] Processing CreateTablet for tablet 34130f2820064fa0898ef1f9d7de1c6d (DEFAULT_TABLE table=default.Table_4537f891b01048ceb51202abc3600911 [id=5a1c4a7be2fc465fac89a50854eb3322]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:30.080875 18447 tablet_service.cc:1467] Processing CreateTablet for tablet 87ac32a0ddb84676adc83038ef664591 (DEFAULT_TABLE table=default.Table_999c55e7bef74339b7b3ab1142599447 [id=61c8a5b99ef34a0baeeb343fef7cfd05]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:30.081002 18724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34130f2820064fa0898ef1f9d7de1c6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:30.081084 18462 tablet_service.cc:1467] Processing CreateTablet for tablet ebf74809b52f47f3b0ed21b6ac0f541a (DEFAULT_TABLE table=default.Table_3e3885add2344068a5da1614890073e0 [id=1c8e0a3a44ec4a798770c0165ed3dffa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:30.080873 18450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46251c08277e498da04a41d009310c62. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:30.081576 18447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87ac32a0ddb84676adc83038ef664591. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:30.081790 18462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebf74809b52f47f3b0ed21b6ac0f541a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:30.083158 18737 tablet_service.cc:1467] Processing CreateTablet for tablet ebf74809b52f47f3b0ed21b6ac0f541a (DEFAULT_TABLE table=default.Table_3e3885add2344068a5da1614890073e0 [id=1c8e0a3a44ec4a798770c0165ed3dffa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:30.083663 18737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebf74809b52f47f3b0ed21b6ac0f541a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:30.083168 18738 tablet_service.cc:1467] Processing CreateTablet for tablet c2b019f529a54228b804437bb025e31d (DEFAULT_TABLE table=default.Table_8028554bc993419e80dbf7d526b63bf4 [id=27ef5b3429fb4353afe80dc3089e9914]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:30.087503 18738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2b019f529a54228b804437bb025e31d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:30.091265 18734 tablet_service.cc:1467] Processing CreateTablet for tablet 46251c08277e498da04a41d009310c62 (DEFAULT_TABLE table=default.Table_d5b955117a0a4c1eb8ed8151e155b32e [id=b1312a6cd1a54003b1db727a2a9d125d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:30.091761 18734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46251c08277e498da04a41d009310c62. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:30.095003 18739 tablet_service.cc:1467] Processing CreateTablet for tablet 87ac32a0ddb84676adc83038ef664591 (DEFAULT_TABLE table=default.Table_999c55e7bef74339b7b3ab1142599447 [id=61c8a5b99ef34a0baeeb343fef7cfd05]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:30.095396 18739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87ac32a0ddb84676adc83038ef664591. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:30.097198 18455 tablet_service.cc:1467] Processing CreateTablet for tablet 43b5e2c005a34df08cc4335c6e014cb5 (DEFAULT_TABLE table=default.Table_08dc70f541b842c78a3a946d2d02e207 [id=38f199a7538f4271b3a59fd93f29efa2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:30.098381 18455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43b5e2c005a34df08cc4335c6e014cb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:30.099689 18583 tablet_service.cc:1467] Processing CreateTablet for tablet 46251c08277e498da04a41d009310c62 (DEFAULT_TABLE table=default.Table_d5b955117a0a4c1eb8ed8151e155b32e [id=b1312a6cd1a54003b1db727a2a9d125d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:30.100119 18583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46251c08277e498da04a41d009310c62. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:30.097819 18447 tablet_service.cc:1467] Processing CreateTablet for tablet 84568f90b9c54cf292099808d4cbf5dc (DEFAULT_TABLE table=default.Table_0034641cfa4d42a68ff4341e87fcb396 [id=53c496365b43473fa7e61ca706cdd984]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:30.101161 18447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84568f90b9c54cf292099808d4cbf5dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:30.103300 18590 tablet_service.cc:1467] Processing CreateTablet for tablet 87ac32a0ddb84676adc83038ef664591 (DEFAULT_TABLE table=default.Table_999c55e7bef74339b7b3ab1142599447 [id=61c8a5b99ef34a0baeeb343fef7cfd05]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:30.103950 18590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87ac32a0ddb84676adc83038ef664591. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:30.103332 18592 tablet_service.cc:1467] Processing CreateTablet for tablet a59ef6c545dd41a28265a4c1b8ce10e0 (DEFAULT_TABLE table=default.Table_327b9bc7f9c64a128209a7a24d2a410a [id=86df5d161d8641c89070e7eed9a39360]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:30.107234 18592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a59ef6c545dd41a28265a4c1b8ce10e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:30.115329 18603 tablet_service.cc:1467] Processing CreateTablet for tablet 1462ecf1176c4e66be42abddbe36b00c (DEFAULT_TABLE table=default.Table_49e612b137f343d1ac3ec06ded7d42f3 [id=6257559b0e9545c58848a76af0c5a5f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:30.115909 18603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1462ecf1176c4e66be42abddbe36b00c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:30.119381 18587 tablet_service.cc:1467] Processing CreateTablet for tablet 43b5e2c005a34df08cc4335c6e014cb5 (DEFAULT_TABLE table=default.Table_08dc70f541b842c78a3a946d2d02e207 [id=38f199a7538f4271b3a59fd93f29efa2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:30.119840 18587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43b5e2c005a34df08cc4335c6e014cb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:30.122192 18454 tablet_service.cc:1467] Processing CreateTablet for tablet 1462ecf1176c4e66be42abddbe36b00c (DEFAULT_TABLE table=default.Table_49e612b137f343d1ac3ec06ded7d42f3 [id=6257559b0e9545c58848a76af0c5a5f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:30.122818 18454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1462ecf1176c4e66be42abddbe36b00c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:30.124063 18589 tablet_service.cc:1467] Processing CreateTablet for tablet 84568f90b9c54cf292099808d4cbf5dc (DEFAULT_TABLE table=default.Table_0034641cfa4d42a68ff4341e87fcb396 [id=53c496365b43473fa7e61ca706cdd984]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:30.124611 18589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84568f90b9c54cf292099808d4cbf5dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:30.128957 19186 consensus_queue.cc:1035] T 6e826459c3f4437da81c9e6e09f0c22f P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:30.133252 19479 consensus_queue.cc:1035] T 34ab40d8d91e401db299ddee7175ba28 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:30.137290 18739 tablet_service.cc:1467] Processing CreateTablet for tablet 1462ecf1176c4e66be42abddbe36b00c (DEFAULT_TABLE table=default.Table_49e612b137f343d1ac3ec06ded7d42f3 [id=6257559b0e9545c58848a76af0c5a5f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:30.137825 18739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1462ecf1176c4e66be42abddbe36b00c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:30.140647 18744 tablet_service.cc:1467] Processing CreateTablet for tablet 84568f90b9c54cf292099808d4cbf5dc (DEFAULT_TABLE table=default.Table_0034641cfa4d42a68ff4341e87fcb396 [id=53c496365b43473fa7e61ca706cdd984]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:30.141186 18744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84568f90b9c54cf292099808d4cbf5dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:30.141950 19386 consensus_queue.cc:1035] T 93d158de64994f3c9fa45a1250a4f533 P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:30.142961 18734 tablet_service.cc:1467] Processing CreateTablet for tablet 43b5e2c005a34df08cc4335c6e014cb5 (DEFAULT_TABLE table=default.Table_08dc70f541b842c78a3a946d2d02e207 [id=38f199a7538f4271b3a59fd93f29efa2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:30.143388 18734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43b5e2c005a34df08cc4335c6e014cb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:30.152382 19386 consensus_queue.cc:1035] T 93d158de64994f3c9fa45a1250a4f533 P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:30.164532 19386 consensus_queue.cc:1035] T ec060e847f024f9fa369cff320e71558 P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:30.171072 19386 consensus_queue.cc:1035] T ec060e847f024f9fa369cff320e71558 P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:30.175263 18738 tablet_service.cc:1467] Processing CreateTablet for tablet a59ef6c545dd41a28265a4c1b8ce10e0 (DEFAULT_TABLE table=default.Table_327b9bc7f9c64a128209a7a24d2a410a [id=86df5d161d8641c89070e7eed9a39360]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:30.175818 18738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a59ef6c545dd41a28265a4c1b8ce10e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:30.177632 19489 ts_tablet_manager.cc:1888] T d4508e4f6dbd42109d951c648b3dc7df P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:30.177861 19489 log.cc:1196] T d4508e4f6dbd42109d951c648b3dc7df P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/d4508e4f6dbd42109d951c648b3dc7df
I20240430 20:11:30.178546 19489 ts_tablet_manager.cc:1909] T d4508e4f6dbd42109d951c648b3dc7df P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:30.182719 18588 tablet_service.cc:1467] Processing CreateTablet for tablet ebf74809b52f47f3b0ed21b6ac0f541a (DEFAULT_TABLE table=default.Table_3e3885add2344068a5da1614890073e0 [id=1c8e0a3a44ec4a798770c0165ed3dffa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:30.183377 18588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebf74809b52f47f3b0ed21b6ac0f541a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:30.188071 18599 tablet_service.cc:1467] Processing CreateTablet for tablet 5f02d8862e1f4ccfa9dbda73dbe18615 (DEFAULT_TABLE table=default.Table_0b4b99c1c03a4e779046063159ce6861 [id=43d7bdc84bab4c54bac58dd1dd0f74ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:30.188730 18599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f02d8862e1f4ccfa9dbda73dbe18615. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:30.191613 18456 tablet_service.cc:1467] Processing CreateTablet for tablet a59ef6c545dd41a28265a4c1b8ce10e0 (DEFAULT_TABLE table=default.Table_327b9bc7f9c64a128209a7a24d2a410a [id=86df5d161d8641c89070e7eed9a39360]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:30.192391 18456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a59ef6c545dd41a28265a4c1b8ce10e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:30.195207 18736 tablet_service.cc:1467] Processing CreateTablet for tablet ae87f5e92a934e2182164c231bc76df0 (DEFAULT_TABLE table=default.Table_27f3d711beeb4387a9088ab291168bc8 [id=434aba8e131342a3a9a1aca3278f9c0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:30.199162 18741 tablet_service.cc:1467] Processing CreateTablet for tablet 73453a7fa30342e29db552b17b62f0c1 (DEFAULT_TABLE table=default.Table_7a9f9a8c27bb4392acddd2a7ea3cc3b2 [id=cfad6f932073488b8097ef7381bc9fd1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:30.201104 18730 tablet_service.cc:1467] Processing CreateTablet for tablet 5f02d8862e1f4ccfa9dbda73dbe18615 (DEFAULT_TABLE table=default.Table_0b4b99c1c03a4e779046063159ce6861 [id=43d7bdc84bab4c54bac58dd1dd0f74ac]), partition=RANGE (key) PARTITION UNBOUNDED
W20240430 20:11:30.211059 19512 log.cc:924] Time spent T e2d90ff17e574f88b2061b1c117d4b7d P 3c51c33d5616482cafb68e26688820f4: Append to log took a long time: real 0.216s user 0.000s sys 0.000s
I20240430 20:11:30.217696 19001 catalog_manager.cc:4830] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): tablet d4508e4f6dbd42109d951c648b3dc7df (table default.Table_276bbdc7cfe542d1a2936485a2aad1ec [id=f146a15e6e8a4acbb4c734c1c863da57]) successfully deleted
I20240430 20:11:30.222837 18736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae87f5e92a934e2182164c231bc76df0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:30.227208 18741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73453a7fa30342e29db552b17b62f0c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:30.231150 18730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f02d8862e1f4ccfa9dbda73dbe18615. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:30.234901 19489 tablet_replica.cc:330] T 5f56ac2afd444d63a6bfd96988464bbb P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:30.235293 19489 raft_consensus.cc:2229] T 5f56ac2afd444d63a6bfd96988464bbb P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Raft consensus shutting down.
I20240430 20:11:30.235703 19489 pending_rounds.cc:62] T 5f56ac2afd444d63a6bfd96988464bbb P 3c51c33d5616482cafb68e26688820f4: Trying to abort 1 pending ops.
I20240430 20:11:30.235796 19489 pending_rounds.cc:68] T 5f56ac2afd444d63a6bfd96988464bbb P 3c51c33d5616482cafb68e26688820f4: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7022624315555557376 op_type: NO_OP noop_request { }
I20240430 20:11:30.235950 19489 raft_consensus.cc:2866] T 5f56ac2afd444d63a6bfd96988464bbb P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240430 20:11:30.236100 19489 raft_consensus.cc:2258] T 5f56ac2afd444d63a6bfd96988464bbb P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:30.237299 19489 ts_tablet_manager.cc:1875] T 5f56ac2afd444d63a6bfd96988464bbb P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:30.244946 19049 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: Sending DeleteTablet for 3 replicas of tablet 7542c7fe6cad452ab718f9ef54dbbbbe
I20240430 20:11:30.246091 19230 consensus_queue.cc:1035] T d9e578c99058424a9b86390641f9c34f P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:30.247548 18735 tablet_service.cc:1514] Processing DeleteTablet for tablet 7542c7fe6cad452ab718f9ef54dbbbbe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:29 UTC) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:30.252664 19479 raft_consensus.cc:482] T 5bde72e2791a497db7dd08686ca36d80 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:30.252821 19479 raft_consensus.cc:504] T 5bde72e2791a497db7dd08686ca36d80 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:30.253602 19479 leader_election.cc:288] T 5bde72e2791a497db7dd08686ca36d80 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:30.254460 18455 tablet_service.cc:1514] Processing DeleteTablet for tablet 7542c7fe6cad452ab718f9ef54dbbbbe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:29 UTC) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:30.254673 18480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bde72e2791a497db7dd08686ca36d80" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:30.255048 18480 raft_consensus.cc:2379] T 5bde72e2791a497db7dd08686ca36d80 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c51c33d5616482cafb68e26688820f4 in current term 1: Already voted for candidate 5e1c6397f09747f091534faf93542710 in this term.
W20240430 20:11:30.256201 19004 catalog_manager.cc:4810] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): delete failed for tablet 7542c7fe6cad452ab718f9ef54dbbbbe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7542c7fe6cad452ab718f9ef54dbbbbe already in progress: creating tablet
I20240430 20:11:30.256861 19049 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_6213f980d6a44e099a7a989ceb23c08a" } new_table_name: "default.Table_deadf728d74a462fb79641891dfedd9c" modify_external_catalogs: true
I20240430 20:11:30.259366 18623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bde72e2791a497db7dd08686ca36d80" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:30.265345 19024 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 24dcc20832804229a2664873bf6ddaa7 (table default.Table_be2ebe77f859489f9cd3e912cb1edc8b [id=2d44ec94131148048021b2ee6366ba30]). Expected version 2 got 0
I20240430 20:11:30.265751 19024 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet acd23f0a63d5448aaa7ed78422e67bad (table default.Table_972f13be4dca4cd19b9a04534dc59f51 [id=69f37db4de634a3185613e40013e0ed7]). Expected version 1 got 0
I20240430 20:11:30.266081 19024 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 294644d7a16b4ca2bbb6d106bcb8c7ff (table default.Table_190b2d86515a44898fa38a49517b9df1 [id=97d8a36ddfae4bb5b99c04a7a08544af]). Expected version 2 got 0
I20240430 20:11:30.266474 19024 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet c66c142419ca4e1399ecd1c8d1e62a59 (table default.Table_0443359dbeff437fb6cffb0db33cbb30 [id=93c44abdf6b147228224388398cd7aeb]). Expected version 2 got 0
I20240430 20:11:30.266749 19024 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 724e51ff05ec4baebc2bb55e05c8eddc (table default.Table_0c5d4686f2ed40fb8d62c31b0166dc05 [id=4dce8a767f0541e1beb5464b2ed95d06]). Expected version 1 got 0
I20240430 20:11:30.267190 19024 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 77285afcafe04a829e485998bbbe00f0 (table default.Table_e63f64a411094f8db2ef641f2257ca79 [id=040dec3a3d804cdfbdada359bbe5123b]). Expected version 1 got 0
I20240430 20:11:30.267618 19024 catalog_manager.cc:5484] T e2d90ff17e574f88b2061b1c117d4b7d P 2efb542aa55e439b99ccb98d1c5c884a reported cstate change: term changed from 0 to 1, leader changed from <none> to 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1). New cstate: current_term: 1 leader_uuid: "2efb542aa55e439b99ccb98d1c5c884a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } health_report { overall_health: HEALTHY } } }
I20240430 20:11:30.268299 19024 catalog_manager.cc:5484] T aec9c09cc0454acab6006bb3e10982b7 P 2efb542aa55e439b99ccb98d1c5c884a reported cstate change: term changed from 0 to 1, leader changed from <none> to 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1). New cstate: current_term: 1 leader_uuid: "2efb542aa55e439b99ccb98d1c5c884a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } health_report { overall_health: HEALTHY } } }
I20240430 20:11:30.268942 19024 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet aec9c09cc0454acab6006bb3e10982b7 (table default.Table_e68dc52988f24f1788517de94d7d2a0e [id=26d4860fb33047b7b812ebd4309f2e28]). Expected version 1 got 0
I20240430 20:11:30.273872 19024 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 372b135dd01f44f98cd713ca8857ba61 (table default.Table_ee86b8fcf2064d8b8c3e327e357d3395 [id=7f3a15a442984827a7521d5d1c98dc3d]). Expected version 2 got 0
I20240430 20:11:30.276014 18696 leader_election.cc:302] T 5bde72e2791a497db7dd08686ca36d80 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710
I20240430 20:11:30.276084 19482 consensus_queue.cc:1035] T 1398d4d2305b48e08fdb12432bd6c4b0 P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:30.276427 19024 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet fd40967ab4a04001819e16c06a6c4300 (table default.Table_fda07fd7279c484db2bee67e8c859855 [id=da9ad991953a40c4b77b31317597ad1d]). Expected version 3 got 0
I20240430 20:11:30.276599 19024 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet fae9a35a2ac844b4bc2ebdf214aa7171 (table default.Table_0ab5b2f09fa7468bb89ab703f6d3d25c [id=a5d580cffa2447d78fc9f2c8e31264f1]). Expected version 4 got 0
I20240430 20:11:30.276777 19024 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet bd11c0335a58448da56533c3f2e785a9 (table default.Table_88c44b20f2ce4eeca7ed9583a70c7c65 [id=73caec5fb65e4cb090e7ae8c391501fd]). Expected version 1 got 0
I20240430 20:11:30.277065 19024 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 76d6e9d547ca4a848eae852944b1d117 (table default.Table_210b7eaff7144b2eb7db1c3b5f7d9505 [id=ff86c0357c164dd0991098c416087773]). Expected version 1 got 0
I20240430 20:11:30.284094 19482 consensus_queue.cc:1035] T 29974dc2c7dd45b99b6cc0712378fc30 P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:30.285310 19479 raft_consensus.cc:3037] T 5bde72e2791a497db7dd08686ca36d80 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:30.286063 19492 consensus_queue.cc:1035] T 34ab40d8d91e401db299ddee7175ba28 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:30.287431 18985 tablet_bootstrap.cc:492] T f77086a5675f44e99e8a867b694df4b8 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:30.287652 18985 ts_tablet_manager.cc:1367] T f77086a5675f44e99e8a867b694df4b8 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.383s user 0.018s sys 0.030s
I20240430 20:11:30.288790 18985 raft_consensus.cc:348] T f77086a5675f44e99e8a867b694df4b8 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:30.289201 18985 raft_consensus.cc:374] T f77086a5675f44e99e8a867b694df4b8 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:30.289368 18985 raft_consensus.cc:729] T f77086a5675f44e99e8a867b694df4b8 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:30.289682 19479 raft_consensus.cc:2726] T 5bde72e2791a497db7dd08686ca36d80 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:30.291090 18988 tablet_bootstrap.cc:492] T e5111d7607dd42188a0176cf5be2ec05 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:30.291424 18988 ts_tablet_manager.cc:1367] T e5111d7607dd42188a0176cf5be2ec05 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.365s user 0.029s sys 0.024s
I20240430 20:11:30.292549 18988 raft_consensus.cc:348] T e5111d7607dd42188a0176cf5be2ec05 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:30.292872 18988 raft_consensus.cc:374] T e5111d7607dd42188a0176cf5be2ec05 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:30.292984 18988 raft_consensus.cc:729] T e5111d7607dd42188a0176cf5be2ec05 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:30.297134 18985 consensus_queue.cc:260] T f77086a5675f44e99e8a867b694df4b8 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:30.298038 18988 consensus_queue.cc:260] T e5111d7607dd42188a0176cf5be2ec05 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:30.298770 18592 tablet_service.cc:1514] Processing DeleteTablet for tablet 7542c7fe6cad452ab718f9ef54dbbbbe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:29 UTC) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:30.303572 18985 ts_tablet_manager.cc:1398] T f77086a5675f44e99e8a867b694df4b8 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.016s user 0.002s sys 0.002s
I20240430 20:11:30.306324 19024 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 680553d4d0bc4fcb839235c1832757e1 (table default.Table_49a012a7c4c8457fadb0a8f0fc9654c6 [id=eecb2f8eeef547c5ab1abe6a2430e592]). Expected version 1 got 0
I20240430 20:11:30.310889 19024 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 6e754af64cb64fd2a082adae7d5f24da (table default.Table_208b12b8e9bd4f3e8bc9c6d422ada649 [id=1df4e95702cd46f59ab21b101049a9ed]). Expected version 1 got 0
I20240430 20:11:30.315122 18985 tablet_bootstrap.cc:492] T 7dfdecba959543a795dfcd3eeda29e58 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:30.317054 18985 tablet_bootstrap.cc:654] T 7dfdecba959543a795dfcd3eeda29e58 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:30.318749 19186 raft_consensus.cc:482] T f77086a5675f44e99e8a867b694df4b8 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:30.318943 19186 raft_consensus.cc:504] T f77086a5675f44e99e8a867b694df4b8 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:30.319746 19186 leader_election.cc:288] T f77086a5675f44e99e8a867b694df4b8 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:30.321583 19024 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet fcc9698abda4491b98e1e879589913a8 (table default.Table_e431f94220aa44ecba799a54dfb612ae [id=1e785b6c159b4312bd1b064271f55a49]). Expected version 1 got 0
W20240430 20:11:30.321755 18415 leader_election.cc:341] T f77086a5675f44e99e8a867b694df4b8 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Not found: Tablet not found: f77086a5675f44e99e8a867b694df4b8
I20240430 20:11:30.336004 19024 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 32c823d1700949ab8b22adcd83b7bfd1 (table default.Table_e8c73c89623d4b49b8d7f0a01d729bdd [id=5cd10682ce52414cb5a103e4433a6a0d]). Expected version 1 got 0
I20240430 20:11:30.336721 19024 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet a05e32a33aa7488eb287cc6314c09c44 (table default.Table_6551367971354d3195e4f7d47364ce7c [id=955d833ff8f14bf6957abadf9ef669e2]). Expected version 1 got 0
I20240430 20:11:30.337488 19024 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 0b4d9e860d7c45fd91d7a566605c9283 (table default.Table_44426457fd7045dfa1c149f0a5ed5129 [id=d5cea90cbc2c48df9d8b80da1eccc949]). Expected version 1 got 0
I20240430 20:11:30.337996 19024 catalog_manager.cc:5484] T 7bde1d6934784c9aa5911181e8df9eeb P 2efb542aa55e439b99ccb98d1c5c884a reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c51c33d5616482cafb68e26688820f4 (127.17.177.3). New cstate: current_term: 1 leader_uuid: "3c51c33d5616482cafb68e26688820f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } }
W20240430 20:11:30.335379 18999 catalog_manager.cc:4810] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): delete failed for tablet 7542c7fe6cad452ab718f9ef54dbbbbe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7542c7fe6cad452ab718f9ef54dbbbbe already in progress: creating tablet
I20240430 20:11:30.338804 18983 tablet_bootstrap.cc:492] T 724e51ff05ec4baebc2bb55e05c8eddc P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:30.338989 18983 ts_tablet_manager.cc:1367] T 724e51ff05ec4baebc2bb55e05c8eddc P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.411s user 0.066s sys 0.092s
I20240430 20:11:30.339830 18985 tablet_bootstrap.cc:492] T 7dfdecba959543a795dfcd3eeda29e58 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:30.339850 18983 raft_consensus.cc:348] T 724e51ff05ec4baebc2bb55e05c8eddc P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:30.340050 18985 ts_tablet_manager.cc:1367] T 7dfdecba959543a795dfcd3eeda29e58 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.025s user 0.001s sys 0.002s
I20240430 20:11:30.340107 18983 raft_consensus.cc:374] T 724e51ff05ec4baebc2bb55e05c8eddc P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:30.340195 18983 raft_consensus.cc:729] T 724e51ff05ec4baebc2bb55e05c8eddc P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:30.340438 18983 consensus_queue.cc:260] T 724e51ff05ec4baebc2bb55e05c8eddc P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:30.340840 18985 raft_consensus.cc:348] T 7dfdecba959543a795dfcd3eeda29e58 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:30.341065 18985 raft_consensus.cc:374] T 7dfdecba959543a795dfcd3eeda29e58 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:30.341131 18985 raft_consensus.cc:729] T 7dfdecba959543a795dfcd3eeda29e58 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:30.341482 18983 ts_tablet_manager.cc:1398] T 724e51ff05ec4baebc2bb55e05c8eddc P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240430 20:11:30.341617 18988 ts_tablet_manager.cc:1398] T e5111d7607dd42188a0176cf5be2ec05 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.044s user 0.000s sys 0.003s
I20240430 20:11:30.341931 18988 tablet_bootstrap.cc:492] T 44fa6917a85c46f3aaa363df46ea586b P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:30.343955 18988 tablet_bootstrap.cc:654] T 44fa6917a85c46f3aaa363df46ea586b P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:30.343984 18985 consensus_queue.cc:260] T 7dfdecba959543a795dfcd3eeda29e58 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:30.347955 19186 consensus_queue.cc:1035] T 1398d4d2305b48e08fdb12432bd6c4b0 P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240430 20:11:30.349844 18417 leader_election.cc:341] T f77086a5675f44e99e8a867b694df4b8 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Not found: Tablet not found: f77086a5675f44e99e8a867b694df4b8
I20240430 20:11:30.350005 18417 leader_election.cc:302] T f77086a5675f44e99e8a867b694df4b8 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a; no voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710
I20240430 20:11:30.350564 19186 raft_consensus.cc:2726] T f77086a5675f44e99e8a867b694df4b8 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:30.353363 18985 ts_tablet_manager.cc:1398] T 7dfdecba959543a795dfcd3eeda29e58 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.013s user 0.001s sys 0.001s
I20240430 20:11:30.356113 18983 tablet_bootstrap.cc:492] T e5111d7607dd42188a0176cf5be2ec05 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:30.358304 18983 tablet_bootstrap.cc:654] T e5111d7607dd42188a0176cf5be2ec05 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:30.347589 19024 catalog_manager.cc:5484] T 6e826459c3f4437da81c9e6e09f0c22f P 2efb542aa55e439b99ccb98d1c5c884a reported cstate change: term changed from 0 to 1, leader changed from <none> to 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1). New cstate: current_term: 1 leader_uuid: "2efb542aa55e439b99ccb98d1c5c884a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } health_report { overall_health: UNKNOWN } } }
I20240430 20:11:30.360172 19024 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 6e826459c3f4437da81c9e6e09f0c22f (table default.Table_873c21f6dffd4d47a742b3699ab85268 [id=d7e2c4300ff94fa2acc65d6d6c932c65]). Expected version 1 got 0
I20240430 20:11:30.360559 19024 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet b72fe1f0713b4076a81a3fc1c74f7fa7 (table default.Table_16b082d140234e1c9a1e70654839b4cd [id=c0d8dc87d1f9426a85f6176f161595a1]). Expected version 3 got 0
I20240430 20:11:30.363649 18985 tablet_bootstrap.cc:492] T 724e51ff05ec4baebc2bb55e05c8eddc P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:30.367391 18529 heartbeater.cc:503] Master 127.17.177.60:37225 was elected leader, sending a full tablet report...
I20240430 20:11:30.375334 18985 tablet_bootstrap.cc:654] T 724e51ff05ec4baebc2bb55e05c8eddc P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:30.385175 19386 consensus_queue.cc:1035] T d9e578c99058424a9b86390641f9c34f P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20240430 20:11:30.385527 19051 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet acd23f0a63d5448aaa7ed78422e67bad (table default.Table_972f13be4dca4cd19b9a04534dc59f51 [id=69f37db4de634a3185613e40013e0ed7]). Expected version 1 got 0
I20240430 20:11:30.386432 19186 consensus_queue.cc:1035] T 77285afcafe04a829e485998bbbe00f0 P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:30.392081 19051 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 24dcc20832804229a2664873bf6ddaa7 (table default.Table_be2ebe77f859489f9cd3e912cb1edc8b [id=2d44ec94131148048021b2ee6366ba30]). Expected version 2 got 0
I20240430 20:11:30.393656 18983 tablet_bootstrap.cc:492] T e5111d7607dd42188a0176cf5be2ec05 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:30.394119 19051 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 294644d7a16b4ca2bbb6d106bcb8c7ff (table default.Table_190b2d86515a44898fa38a49517b9df1 [id=97d8a36ddfae4bb5b99c04a7a08544af]). Expected version 2 got 0
I20240430 20:11:30.395044 19051 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 724e51ff05ec4baebc2bb55e05c8eddc (table default.Table_0c5d4686f2ed40fb8d62c31b0166dc05 [id=4dce8a767f0541e1beb5464b2ed95d06]). Expected version 1 got 0
W20240430 20:11:30.395972 19000 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet a05e32a33aa7488eb287cc6314c09c44 (table default.Table_6551367971354d3195e4f7d47364ce7c [id=955d833ff8f14bf6957abadf9ef669e2], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet a05e32a33aa7488eb287cc6314c09c44 consensus state has no leader
I20240430 20:11:30.396495 18983 ts_tablet_manager.cc:1367] T e5111d7607dd42188a0176cf5be2ec05 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.040s user 0.004s sys 0.000s
I20240430 20:11:30.397714 18983 raft_consensus.cc:348] T e5111d7607dd42188a0176cf5be2ec05 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:30.400710 19051 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet c66c142419ca4e1399ecd1c8d1e62a59 (table default.Table_0443359dbeff437fb6cffb0db33cbb30 [id=93c44abdf6b147228224388398cd7aeb]). Expected version 2 got 0
I20240430 20:11:30.401398 19051 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 77285afcafe04a829e485998bbbe00f0 (table default.Table_e63f64a411094f8db2ef641f2257ca79 [id=040dec3a3d804cdfbdada359bbe5123b]). Expected version 1 got 0
I20240430 20:11:30.401752 19051 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet bd11c0335a58448da56533c3f2e785a9 (table default.Table_88c44b20f2ce4eeca7ed9583a70c7c65 [id=73caec5fb65e4cb090e7ae8c391501fd]). Expected version 1 got 0
I20240430 20:11:30.402182 19051 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet aec9c09cc0454acab6006bb3e10982b7 (table default.Table_e68dc52988f24f1788517de94d7d2a0e [id=26d4860fb33047b7b812ebd4309f2e28]). Expected version 1 got 0
I20240430 20:11:30.402894 18983 raft_consensus.cc:374] T e5111d7607dd42188a0176cf5be2ec05 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:30.403409 19051 catalog_manager.cc:5484] T a745334c12c545c1ba825c7552c60359 P 5e1c6397f09747f091534faf93542710 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e1c6397f09747f091534faf93542710 (127.17.177.2). New cstate: current_term: 1 leader_uuid: "5e1c6397f09747f091534faf93542710" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } health_report { overall_health: UNKNOWN } } }
I20240430 20:11:30.404146 19051 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 372b135dd01f44f98cd713ca8857ba61 (table default.Table_ee86b8fcf2064d8b8c3e327e357d3395 [id=7f3a15a442984827a7521d5d1c98dc3d]). Expected version 2 got 0
I20240430 20:11:30.404487 19051 catalog_manager.cc:5484] T fae9a35a2ac844b4bc2ebdf214aa7171 P 5e1c6397f09747f091534faf93542710 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e1c6397f09747f091534faf93542710 (127.17.177.2). New cstate: current_term: 1 leader_uuid: "5e1c6397f09747f091534faf93542710" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } health_report { overall_health: UNKNOWN } } }
I20240430 20:11:30.405022 19051 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet fae9a35a2ac844b4bc2ebdf214aa7171 (table default.Table_0ab5b2f09fa7468bb89ab703f6d3d25c [id=a5d580cffa2447d78fc9f2c8e31264f1]). Expected version 4 got 0
I20240430 20:11:30.405220 19051 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 6e754af64cb64fd2a082adae7d5f24da (table default.Table_208b12b8e9bd4f3e8bc9c6d422ada649 [id=1df4e95702cd46f59ab21b101049a9ed]). Expected version 1 got 0
I20240430 20:11:30.405890 19051 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 0b4d9e860d7c45fd91d7a566605c9283 (table default.Table_44426457fd7045dfa1c149f0a5ed5129 [id=d5cea90cbc2c48df9d8b80da1eccc949]). Expected version 1 got 0
I20240430 20:11:30.406134 19051 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 32c823d1700949ab8b22adcd83b7bfd1 (table default.Table_e8c73c89623d4b49b8d7f0a01d729bdd [id=5cd10682ce52414cb5a103e4433a6a0d]). Expected version 1 got 0
I20240430 20:11:30.406489 19051 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet fd40967ab4a04001819e16c06a6c4300 (table default.Table_fda07fd7279c484db2bee67e8c859855 [id=da9ad991953a40c4b77b31317597ad1d]). Expected version 3 got 0
I20240430 20:11:30.406821 19051 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 76d6e9d547ca4a848eae852944b1d117 (table default.Table_210b7eaff7144b2eb7db1c3b5f7d9505 [id=ff86c0357c164dd0991098c416087773]). Expected version 1 got 0
I20240430 20:11:30.407079 19051 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 680553d4d0bc4fcb839235c1832757e1 (table default.Table_49a012a7c4c8457fadb0a8f0fc9654c6 [id=eecb2f8eeef547c5ab1abe6a2430e592]). Expected version 1 got 0
I20240430 20:11:30.407450 19051 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet fcc9698abda4491b98e1e879589913a8 (table default.Table_e431f94220aa44ecba799a54dfb612ae [id=1e785b6c159b4312bd1b064271f55a49]). Expected version 1 got 0
I20240430 20:11:30.407842 19051 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet a05e32a33aa7488eb287cc6314c09c44 (table default.Table_6551367971354d3195e4f7d47364ce7c [id=955d833ff8f14bf6957abadf9ef669e2]). Expected version 1 got 0
I20240430 20:11:30.408157 19051 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 6e826459c3f4437da81c9e6e09f0c22f (table default.Table_873c21f6dffd4d47a742b3699ab85268 [id=d7e2c4300ff94fa2acc65d6d6c932c65]). Expected version 1 got 0
I20240430 20:11:30.408687 19051 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet b72fe1f0713b4076a81a3fc1c74f7fa7 (table default.Table_16b082d140234e1c9a1e70654839b4cd [id=c0d8dc87d1f9426a85f6176f161595a1]). Expected version 3 got 0
I20240430 20:11:30.408954 19051 catalog_manager.cc:5484] T ec060e847f024f9fa369cff320e71558 P 5e1c6397f09747f091534faf93542710 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e1c6397f09747f091534faf93542710 (127.17.177.2). New cstate: current_term: 1 leader_uuid: "5e1c6397f09747f091534faf93542710" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } health_report { overall_health: UNKNOWN } } }
I20240430 20:11:30.409235 19489 ts_tablet_manager.cc:1888] T 5f56ac2afd444d63a6bfd96988464bbb P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:30.409441 19489 log.cc:1196] T 5f56ac2afd444d63a6bfd96988464bbb P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/5f56ac2afd444d63a6bfd96988464bbb
I20240430 20:11:30.410341 19489 ts_tablet_manager.cc:1909] T 5f56ac2afd444d63a6bfd96988464bbb P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:30.411620 19001 catalog_manager.cc:4830] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): tablet 5f56ac2afd444d63a6bfd96988464bbb (table default.Table_19a91a7f17ff456ebb145e8dd89c0940 [id=67283dd3995a44c6a7a7bc8722e4f955]) successfully deleted
W20240430 20:11:30.412698 19001 catalog_manager.cc:4810] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): delete failed for tablet b688e03ea71940c69bda5520b62c1fc3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b688e03ea71940c69bda5520b62c1fc3 already in progress: creating tablet
I20240430 20:11:30.413606 18983 raft_consensus.cc:729] T e5111d7607dd42188a0176cf5be2ec05 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:30.413978 18983 consensus_queue.cc:260] T e5111d7607dd42188a0176cf5be2ec05 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:30.414569 18810 heartbeater.cc:503] Master 127.17.177.60:37225 was elected leader, sending a full tablet report...
I20240430 20:11:30.414570 18983 ts_tablet_manager.cc:1398] T e5111d7607dd42188a0176cf5be2ec05 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.018s user 0.001s sys 0.001s
I20240430 20:11:30.417644 18983 tablet_bootstrap.cc:492] T 44fa6917a85c46f3aaa363df46ea586b P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:30.419591 18983 tablet_bootstrap.cc:654] T 44fa6917a85c46f3aaa363df46ea586b P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
W20240430 20:11:30.424177 19001 catalog_manager.cc:4810] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): delete failed for tablet 7542c7fe6cad452ab718f9ef54dbbbbe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7542c7fe6cad452ab718f9ef54dbbbbe already in progress: creating tablet
I20240430 20:11:30.424918 18988 tablet_bootstrap.cc:492] T 44fa6917a85c46f3aaa363df46ea586b P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:30.425119 18988 ts_tablet_manager.cc:1367] T 44fa6917a85c46f3aaa363df46ea586b P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.083s user 0.004s sys 0.004s
I20240430 20:11:30.426365 18985 tablet_bootstrap.cc:492] T 724e51ff05ec4baebc2bb55e05c8eddc P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:30.426559 18985 ts_tablet_manager.cc:1367] T 724e51ff05ec4baebc2bb55e05c8eddc P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.063s user 0.003s sys 0.003s
I20240430 20:11:30.427525 18985 raft_consensus.cc:348] T 724e51ff05ec4baebc2bb55e05c8eddc P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:30.427774 18985 raft_consensus.cc:374] T 724e51ff05ec4baebc2bb55e05c8eddc P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:30.427909 18985 raft_consensus.cc:729] T 724e51ff05ec4baebc2bb55e05c8eddc P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:30.428759 19482 consensus_queue.cc:1035] T 77285afcafe04a829e485998bbbe00f0 P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:30.428352 18985 consensus_queue.cc:260] T 724e51ff05ec4baebc2bb55e05c8eddc P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:30.429435 18985 ts_tablet_manager.cc:1398] T 724e51ff05ec4baebc2bb55e05c8eddc P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240430 20:11:30.434995 18985 tablet_bootstrap.cc:492] T 44fa6917a85c46f3aaa363df46ea586b P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:30.437407 18985 tablet_bootstrap.cc:654] T 44fa6917a85c46f3aaa363df46ea586b P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:30.440492 19026 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet acd23f0a63d5448aaa7ed78422e67bad (table default.Table_972f13be4dca4cd19b9a04534dc59f51 [id=69f37db4de634a3185613e40013e0ed7]). Expected version 1 got 0
I20240430 20:11:30.441035 19026 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 24dcc20832804229a2664873bf6ddaa7 (table default.Table_be2ebe77f859489f9cd3e912cb1edc8b [id=2d44ec94131148048021b2ee6366ba30]). Expected version 2 got 0
I20240430 20:11:30.441432 19026 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 294644d7a16b4ca2bbb6d106bcb8c7ff (table default.Table_190b2d86515a44898fa38a49517b9df1 [id=97d8a36ddfae4bb5b99c04a7a08544af]). Expected version 2 got 0
I20240430 20:11:30.442045 19026 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet c66c142419ca4e1399ecd1c8d1e62a59 (table default.Table_0443359dbeff437fb6cffb0db33cbb30 [id=93c44abdf6b147228224388398cd7aeb]). Expected version 2 got 0
I20240430 20:11:30.443171 19026 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 724e51ff05ec4baebc2bb55e05c8eddc (table default.Table_0c5d4686f2ed40fb8d62c31b0166dc05 [id=4dce8a767f0541e1beb5464b2ed95d06]). Expected version 1 got 0
I20240430 20:11:30.443619 19026 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 77285afcafe04a829e485998bbbe00f0 (table default.Table_e63f64a411094f8db2ef641f2257ca79 [id=040dec3a3d804cdfbdada359bbe5123b]). Expected version 1 got 0
I20240430 20:11:30.443974 19026 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet bd11c0335a58448da56533c3f2e785a9 (table default.Table_88c44b20f2ce4eeca7ed9583a70c7c65 [id=73caec5fb65e4cb090e7ae8c391501fd]). Expected version 1 got 0
I20240430 20:11:30.445282 19026 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet aec9c09cc0454acab6006bb3e10982b7 (table default.Table_e68dc52988f24f1788517de94d7d2a0e [id=26d4860fb33047b7b812ebd4309f2e28]). Expected version 1 got 0
I20240430 20:11:30.445688 19026 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet fae9a35a2ac844b4bc2ebdf214aa7171 (table default.Table_0ab5b2f09fa7468bb89ab703f6d3d25c [id=a5d580cffa2447d78fc9f2c8e31264f1]). Expected version 4 got 0
I20240430 20:11:30.448642 19026 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 76d6e9d547ca4a848eae852944b1d117 (table default.Table_210b7eaff7144b2eb7db1c3b5f7d9505 [id=ff86c0357c164dd0991098c416087773]). Expected version 1 got 0
I20240430 20:11:30.448930 19026 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 680553d4d0bc4fcb839235c1832757e1 (table default.Table_49a012a7c4c8457fadb0a8f0fc9654c6 [id=eecb2f8eeef547c5ab1abe6a2430e592]). Expected version 1 got 0
I20240430 20:11:30.449349 19026 catalog_manager.cc:5484] T 372b135dd01f44f98cd713ca8857ba61 P 3c51c33d5616482cafb68e26688820f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c51c33d5616482cafb68e26688820f4 (127.17.177.3). New cstate: current_term: 1 leader_uuid: "3c51c33d5616482cafb68e26688820f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } health_report { overall_health: UNKNOWN } } }
I20240430 20:11:30.449862 19026 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 372b135dd01f44f98cd713ca8857ba61 (table default.Table_ee86b8fcf2064d8b8c3e327e357d3395 [id=7f3a15a442984827a7521d5d1c98dc3d]). Expected version 2 got 0
I20240430 20:11:30.450199 19026 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 32c823d1700949ab8b22adcd83b7bfd1 (table default.Table_e8c73c89623d4b49b8d7f0a01d729bdd [id=5cd10682ce52414cb5a103e4433a6a0d]). Expected version 1 got 0
I20240430 20:11:30.450392 19026 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet fd40967ab4a04001819e16c06a6c4300 (table default.Table_fda07fd7279c484db2bee67e8c859855 [id=da9ad991953a40c4b77b31317597ad1d]). Expected version 3 got 0
I20240430 20:11:30.450659 19026 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 6e754af64cb64fd2a082adae7d5f24da (table default.Table_208b12b8e9bd4f3e8bc9c6d422ada649 [id=1df4e95702cd46f59ab21b101049a9ed]). Expected version 1 got 0
I20240430 20:11:30.451262 19026 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet fcc9698abda4491b98e1e879589913a8 (table default.Table_e431f94220aa44ecba799a54dfb612ae [id=1e785b6c159b4312bd1b064271f55a49]). Expected version 1 got 0
I20240430 20:11:30.451643 19026 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 0b4d9e860d7c45fd91d7a566605c9283 (table default.Table_44426457fd7045dfa1c149f0a5ed5129 [id=d5cea90cbc2c48df9d8b80da1eccc949]). Expected version 1 got 0
I20240430 20:11:30.452073 19026 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet a05e32a33aa7488eb287cc6314c09c44 (table default.Table_6551367971354d3195e4f7d47364ce7c [id=955d833ff8f14bf6957abadf9ef669e2]). Expected version 1 got 0
I20240430 20:11:30.452495 19026 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 6e826459c3f4437da81c9e6e09f0c22f (table default.Table_873c21f6dffd4d47a742b3699ab85268 [id=d7e2c4300ff94fa2acc65d6d6c932c65]). Expected version 1 got 0
I20240430 20:11:30.452747 19026 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet b72fe1f0713b4076a81a3fc1c74f7fa7 (table default.Table_16b082d140234e1c9a1e70654839b4cd [id=c0d8dc87d1f9426a85f6176f161595a1]). Expected version 3 got 0
I20240430 20:11:30.453112 19048 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: Sending DeleteTablet for 3 replicas of tablet e5111d7607dd42188a0176cf5be2ec05
I20240430 20:11:30.459826 18988 raft_consensus.cc:348] T 44fa6917a85c46f3aaa363df46ea586b P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:30.460192 18988 raft_consensus.cc:374] T 44fa6917a85c46f3aaa363df46ea586b P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:30.460280 18988 raft_consensus.cc:729] T 44fa6917a85c46f3aaa363df46ea586b P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:30.463323 18730 tablet_service.cc:1514] Processing DeleteTablet for tablet e5111d7607dd42188a0176cf5be2ec05 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:29 UTC) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:30.463658 19489 tablet_replica.cc:330] T e5111d7607dd42188a0176cf5be2ec05 P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:30.463938 19489 raft_consensus.cc:2229] T e5111d7607dd42188a0176cf5be2ec05 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:30.464143 19489 raft_consensus.cc:2258] T e5111d7607dd42188a0176cf5be2ec05 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:30.465374 18588 tablet_service.cc:1514] Processing DeleteTablet for tablet e5111d7607dd42188a0176cf5be2ec05 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:29 UTC) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:30.465690 19489 ts_tablet_manager.cc:1875] T e5111d7607dd42188a0176cf5be2ec05 P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:30.465767 19471 tablet_replica.cc:330] T e5111d7607dd42188a0176cf5be2ec05 P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:30.465960 19471 raft_consensus.cc:2229] T e5111d7607dd42188a0176cf5be2ec05 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:30.466153 19471 raft_consensus.cc:2258] T e5111d7607dd42188a0176cf5be2ec05 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:30.467020 19471 ts_tablet_manager.cc:1875] T e5111d7607dd42188a0176cf5be2ec05 P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:30.472457 18988 consensus_queue.cc:260] T 44fa6917a85c46f3aaa363df46ea586b P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:30.473120 18988 ts_tablet_manager.cc:1398] T 44fa6917a85c46f3aaa363df46ea586b P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.048s user 0.001s sys 0.001s
I20240430 20:11:30.476002 19482 consensus_queue.cc:1035] T 29974dc2c7dd45b99b6cc0712378fc30 P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.028s
I20240430 20:11:30.477169 18988 tablet_bootstrap.cc:492] T 724e51ff05ec4baebc2bb55e05c8eddc P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:30.478884 18988 tablet_bootstrap.cc:654] T 724e51ff05ec4baebc2bb55e05c8eddc P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:30.479770 19489 ts_tablet_manager.cc:1888] T e5111d7607dd42188a0176cf5be2ec05 P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:30.480031 19489 log.cc:1196] T e5111d7607dd42188a0176cf5be2ec05 P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/e5111d7607dd42188a0176cf5be2ec05
I20240430 20:11:30.480521 19489 ts_tablet_manager.cc:1909] T e5111d7607dd42188a0176cf5be2ec05 P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:30.484858 19001 catalog_manager.cc:4830] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): tablet e5111d7607dd42188a0176cf5be2ec05 (table default.Table_f61a45518e134c4496e4ac81fbe40fce [id=02d9fe0426694ee8b5fd70589fb66ead]) successfully deleted
I20240430 20:11:30.493125 19492 raft_consensus.cc:482] T 7dfdecba959543a795dfcd3eeda29e58 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:30.493345 19492 raft_consensus.cc:504] T 7dfdecba959543a795dfcd3eeda29e58 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:30.494238 19492 leader_election.cc:288] T 7dfdecba959543a795dfcd3eeda29e58 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:30.495420 18620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dfdecba959543a795dfcd3eeda29e58" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:30.495724 18620 raft_consensus.cc:2454] T 7dfdecba959543a795dfcd3eeda29e58 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:30.496445 18696 leader_election.cc:302] T 7dfdecba959543a795dfcd3eeda29e58 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:30.496956 19492 raft_consensus.cc:2781] T 7dfdecba959543a795dfcd3eeda29e58 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:30.497078 19492 raft_consensus.cc:482] T 7dfdecba959543a795dfcd3eeda29e58 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:30.497166 19492 raft_consensus.cc:3037] T 7dfdecba959543a795dfcd3eeda29e58 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:30.500166 19492 raft_consensus.cc:504] T 7dfdecba959543a795dfcd3eeda29e58 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:30.501533 19492 leader_election.cc:288] T 7dfdecba959543a795dfcd3eeda29e58 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Requested vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:30.502941 18620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dfdecba959543a795dfcd3eeda29e58" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:30.503227 18620 raft_consensus.cc:3037] T 7dfdecba959543a795dfcd3eeda29e58 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:30.506371 18620 raft_consensus.cc:2454] T 7dfdecba959543a795dfcd3eeda29e58 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:30.507153 18696 leader_election.cc:302] T 7dfdecba959543a795dfcd3eeda29e58 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:30.507567 19492 raft_consensus.cc:2781] T 7dfdecba959543a795dfcd3eeda29e58 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:30.508030 19492 raft_consensus.cc:686] T 7dfdecba959543a795dfcd3eeda29e58 P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Becoming Leader. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Running, Role: LEADER
I20240430 20:11:30.508378 19492 consensus_queue.cc:237] T 7dfdecba959543a795dfcd3eeda29e58 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:30.511720 19471 ts_tablet_manager.cc:1888] T e5111d7607dd42188a0176cf5be2ec05 P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:30.512176 18988 tablet_bootstrap.cc:492] T 724e51ff05ec4baebc2bb55e05c8eddc P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:30.512382 18988 ts_tablet_manager.cc:1367] T 724e51ff05ec4baebc2bb55e05c8eddc P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.035s user 0.003s sys 0.000s
I20240430 20:11:30.512410 19471 log.cc:1196] T e5111d7607dd42188a0176cf5be2ec05 P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/e5111d7607dd42188a0176cf5be2ec05
I20240430 20:11:30.513458 18988 raft_consensus.cc:348] T 724e51ff05ec4baebc2bb55e05c8eddc P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:30.513763 18988 raft_consensus.cc:374] T 724e51ff05ec4baebc2bb55e05c8eddc P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:30.513887 18988 raft_consensus.cc:729] T 724e51ff05ec4baebc2bb55e05c8eddc P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:30.514408 19471 ts_tablet_manager.cc:1909] T e5111d7607dd42188a0176cf5be2ec05 P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:30.514732 18456 tablet_service.cc:1514] Processing DeleteTablet for tablet e5111d7607dd42188a0176cf5be2ec05 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:29 UTC) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:30.516520 18483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dfdecba959543a795dfcd3eeda29e58" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:30.516841 18483 raft_consensus.cc:2454] T 7dfdecba959543a795dfcd3eeda29e58 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
W20240430 20:11:30.516935 19004 catalog_manager.cc:4810] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): delete failed for tablet e5111d7607dd42188a0176cf5be2ec05 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e5111d7607dd42188a0176cf5be2ec05 already in progress: creating tablet
I20240430 20:11:30.516923 18999 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet e5111d7607dd42188a0176cf5be2ec05 (table default.Table_f61a45518e134c4496e4ac81fbe40fce [id=02d9fe0426694ee8b5fd70589fb66ead]) successfully deleted
I20240430 20:11:30.520097 18483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dfdecba959543a795dfcd3eeda29e58" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:30.520345 18483 raft_consensus.cc:3037] T 7dfdecba959543a795dfcd3eeda29e58 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:30.524041 18483 raft_consensus.cc:2454] T 7dfdecba959543a795dfcd3eeda29e58 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:30.532244 18669 heartbeater.cc:503] Master 127.17.177.60:37225 was elected leader, sending a full tablet report...
I20240430 20:11:30.532723 19479 consensus_queue.cc:1035] T bbe9a2429a434f3f95d6e7f314fcfe4c P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:30.540753 18985 tablet_bootstrap.cc:492] T 44fa6917a85c46f3aaa363df46ea586b P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:30.540957 18985 ts_tablet_manager.cc:1367] T 44fa6917a85c46f3aaa363df46ea586b P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.106s user 0.015s sys 0.023s
I20240430 20:11:30.541905 18985 raft_consensus.cc:348] T 44fa6917a85c46f3aaa363df46ea586b P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:30.542136 18985 raft_consensus.cc:374] T 44fa6917a85c46f3aaa363df46ea586b P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:30.542212 18985 raft_consensus.cc:729] T 44fa6917a85c46f3aaa363df46ea586b P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:30.542457 18985 consensus_queue.cc:260] T 44fa6917a85c46f3aaa363df46ea586b P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:30.543067 18985 ts_tablet_manager.cc:1398] T 44fa6917a85c46f3aaa363df46ea586b P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240430 20:11:30.543447 18985 tablet_bootstrap.cc:492] T c66c142419ca4e1399ecd1c8d1e62a59 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:30.544157 19386 consensus_queue.cc:1035] T 5bde72e2791a497db7dd08686ca36d80 P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:30.545248 18985 tablet_bootstrap.cc:654] T c66c142419ca4e1399ecd1c8d1e62a59 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
W20240430 20:11:30.558384 18699 consensus_peers.cc:485] T bbe9a2429a434f3f95d6e7f314fcfe4c P 3c51c33d5616482cafb68e26688820f4 -> Peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Couldn't send request to peer 2efb542aa55e439b99ccb98d1c5c884a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bbe9a2429a434f3f95d6e7f314fcfe4c. This is attempt 1: this message will repeat every 5th retry.
I20240430 20:11:30.567044 18988 consensus_queue.cc:260] T 724e51ff05ec4baebc2bb55e05c8eddc P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:30.569330 18983 tablet_bootstrap.cc:492] T 44fa6917a85c46f3aaa363df46ea586b P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:30.569530 18983 ts_tablet_manager.cc:1367] T 44fa6917a85c46f3aaa363df46ea586b P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.152s user 0.026s sys 0.040s
I20240430 20:11:30.569772 18988 ts_tablet_manager.cc:1398] T 724e51ff05ec4baebc2bb55e05c8eddc P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.057s user 0.002s sys 0.001s
I20240430 20:11:30.570418 18983 raft_consensus.cc:348] T 44fa6917a85c46f3aaa363df46ea586b P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:30.570724 18983 raft_consensus.cc:374] T 44fa6917a85c46f3aaa363df46ea586b P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:30.570822 18983 raft_consensus.cc:729] T 44fa6917a85c46f3aaa363df46ea586b P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:30.571087 18983 consensus_queue.cc:260] T 44fa6917a85c46f3aaa363df46ea586b P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:30.571483 18988 tablet_bootstrap.cc:492] T c66c142419ca4e1399ecd1c8d1e62a59 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:30.572194 18983 ts_tablet_manager.cc:1398] T 44fa6917a85c46f3aaa363df46ea586b P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240430 20:11:30.572510 18983 tablet_bootstrap.cc:492] T b688e03ea71940c69bda5520b62c1fc3 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:30.575564 18983 tablet_bootstrap.cc:654] T b688e03ea71940c69bda5520b62c1fc3 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:30.589566 18988 tablet_bootstrap.cc:654] T c66c142419ca4e1399ecd1c8d1e62a59 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:30.592294 19492 consensus_queue.cc:1035] T 02176eee8e044a7db8cac48439e9f594 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240430 20:11:30.602923 18985 tablet_bootstrap.cc:492] T c66c142419ca4e1399ecd1c8d1e62a59 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:30.603127 18985 ts_tablet_manager.cc:1367] T c66c142419ca4e1399ecd1c8d1e62a59 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.060s user 0.015s sys 0.010s
I20240430 20:11:30.604143 18985 raft_consensus.cc:348] T c66c142419ca4e1399ecd1c8d1e62a59 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:30.604391 18985 raft_consensus.cc:374] T c66c142419ca4e1399ecd1c8d1e62a59 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:30.604504 18985 raft_consensus.cc:729] T c66c142419ca4e1399ecd1c8d1e62a59 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:30.604789 18985 consensus_queue.cc:260] T c66c142419ca4e1399ecd1c8d1e62a59 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:30.605576 18985 ts_tablet_manager.cc:1398] T c66c142419ca4e1399ecd1c8d1e62a59 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240430 20:11:30.605929 18985 tablet_bootstrap.cc:492] T e5111d7607dd42188a0176cf5be2ec05 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:30.607848 18985 tablet_bootstrap.cc:654] T e5111d7607dd42188a0176cf5be2ec05 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:30.615511 18988 tablet_bootstrap.cc:492] T c66c142419ca4e1399ecd1c8d1e62a59 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:30.615746 18988 ts_tablet_manager.cc:1367] T c66c142419ca4e1399ecd1c8d1e62a59 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.044s user 0.002s sys 0.002s
I20240430 20:11:30.616525 18988 raft_consensus.cc:348] T c66c142419ca4e1399ecd1c8d1e62a59 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:30.616762 18988 raft_consensus.cc:374] T c66c142419ca4e1399ecd1c8d1e62a59 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:30.616827 18988 raft_consensus.cc:729] T c66c142419ca4e1399ecd1c8d1e62a59 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:30.618507 18985 tablet_bootstrap.cc:492] T e5111d7607dd42188a0176cf5be2ec05 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:30.618695 18985 ts_tablet_manager.cc:1367] T e5111d7607dd42188a0176cf5be2ec05 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.000s
I20240430 20:11:30.619400 18985 raft_consensus.cc:348] T e5111d7607dd42188a0176cf5be2ec05 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:30.619638 18985 raft_consensus.cc:374] T e5111d7607dd42188a0176cf5be2ec05 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:30.619720 18985 raft_consensus.cc:729] T e5111d7607dd42188a0176cf5be2ec05 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:30.619549 18988 consensus_queue.cc:260] T c66c142419ca4e1399ecd1c8d1e62a59 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:30.619977 18985 consensus_queue.cc:260] T e5111d7607dd42188a0176cf5be2ec05 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:30.620510 18985 ts_tablet_manager.cc:1398] T e5111d7607dd42188a0176cf5be2ec05 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240430 20:11:30.621141 18985 tablet_bootstrap.cc:492] T 68ca48351cd4412381f5195d81109016 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:30.620642 19479 consensus_queue.cc:1035] T 02176eee8e044a7db8cac48439e9f594 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240430 20:11:30.622664 19479 consensus_queue.cc:1035] T 0b4d9e860d7c45fd91d7a566605c9283 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:30.623252 18985 tablet_bootstrap.cc:654] T 68ca48351cd4412381f5195d81109016 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:30.626981 18988 ts_tablet_manager.cc:1398] T c66c142419ca4e1399ecd1c8d1e62a59 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.011s user 0.001s sys 0.001s
I20240430 20:11:30.627346 18988 tablet_bootstrap.cc:492] T 68ca48351cd4412381f5195d81109016 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:30.629016 18988 tablet_bootstrap.cc:654] T 68ca48351cd4412381f5195d81109016 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:30.640655 19492 consensus_queue.cc:1035] T 0b4d9e860d7c45fd91d7a566605c9283 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:30.646519 19180 raft_consensus.cc:482] T c66c142419ca4e1399ecd1c8d1e62a59 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:30.646785 19180 raft_consensus.cc:504] T c66c142419ca4e1399ecd1c8d1e62a59 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:30.648023 19180 leader_election.cc:288] T c66c142419ca4e1399ecd1c8d1e62a59 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:30.649643 18481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66c142419ca4e1399ecd1c8d1e62a59" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:30.649917 18481 raft_consensus.cc:2454] T c66c142419ca4e1399ecd1c8d1e62a59 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 0.
I20240430 20:11:30.650074 18758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66c142419ca4e1399ecd1c8d1e62a59" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:30.650323 18758 raft_consensus.cc:2454] T c66c142419ca4e1399ecd1c8d1e62a59 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 0.
I20240430 20:11:30.650568 18558 leader_election.cc:302] T c66c142419ca4e1399ecd1c8d1e62a59 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:30.651667 19180 raft_consensus.cc:2781] T c66c142419ca4e1399ecd1c8d1e62a59 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:30.651795 19180 raft_consensus.cc:482] T c66c142419ca4e1399ecd1c8d1e62a59 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:30.651949 19180 raft_consensus.cc:3037] T c66c142419ca4e1399ecd1c8d1e62a59 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:30.655673 19180 raft_consensus.cc:504] T c66c142419ca4e1399ecd1c8d1e62a59 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:30.656899 18758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66c142419ca4e1399ecd1c8d1e62a59" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:30.657137 18758 raft_consensus.cc:3037] T c66c142419ca4e1399ecd1c8d1e62a59 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:30.660513 18758 raft_consensus.cc:2454] T c66c142419ca4e1399ecd1c8d1e62a59 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
I20240430 20:11:30.661249 18557 leader_election.cc:302] T c66c142419ca4e1399ecd1c8d1e62a59 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:30.661676 19474 raft_consensus.cc:2781] T c66c142419ca4e1399ecd1c8d1e62a59 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:30.661877 19474 raft_consensus.cc:686] T c66c142419ca4e1399ecd1c8d1e62a59 P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Becoming Leader. State: Replica: 5e1c6397f09747f091534faf93542710, State: Running, Role: LEADER
I20240430 20:11:30.662297 19474 consensus_queue.cc:237] T c66c142419ca4e1399ecd1c8d1e62a59 P 5e1c6397f09747f091534faf93542710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:30.668174 18988 tablet_bootstrap.cc:492] T 68ca48351cd4412381f5195d81109016 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:30.668423 18988 ts_tablet_manager.cc:1367] T 68ca48351cd4412381f5195d81109016 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.041s user 0.001s sys 0.002s
I20240430 20:11:30.669320 18988 raft_consensus.cc:348] T 68ca48351cd4412381f5195d81109016 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:30.669545 18988 raft_consensus.cc:374] T 68ca48351cd4412381f5195d81109016 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:30.669648 18988 raft_consensus.cc:729] T 68ca48351cd4412381f5195d81109016 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:30.669873 18988 consensus_queue.cc:260] T 68ca48351cd4412381f5195d81109016 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:30.670094 18481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66c142419ca4e1399ecd1c8d1e62a59" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:30.670322 18481 raft_consensus.cc:3037] T c66c142419ca4e1399ecd1c8d1e62a59 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:30.674366 18481 raft_consensus.cc:2454] T c66c142419ca4e1399ecd1c8d1e62a59 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
I20240430 20:11:30.678570 19180 leader_election.cc:288] T c66c142419ca4e1399ecd1c8d1e62a59 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:30.689363 18985 tablet_bootstrap.cc:492] T 68ca48351cd4412381f5195d81109016 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:30.689574 18985 ts_tablet_manager.cc:1367] T 68ca48351cd4412381f5195d81109016 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.069s user 0.014s sys 0.012s
I20240430 20:11:30.690450 18985 raft_consensus.cc:348] T 68ca48351cd4412381f5195d81109016 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:30.690718 18985 raft_consensus.cc:374] T 68ca48351cd4412381f5195d81109016 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:30.690810 18985 raft_consensus.cc:729] T 68ca48351cd4412381f5195d81109016 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:30.691061 18985 consensus_queue.cc:260] T 68ca48351cd4412381f5195d81109016 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:30.691648 18985 ts_tablet_manager.cc:1398] T 68ca48351cd4412381f5195d81109016 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240430 20:11:30.692356 18985 tablet_bootstrap.cc:492] T eed6338ad3d64859af087ee15e1ec798 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:30.695143 18985 tablet_bootstrap.cc:654] T eed6338ad3d64859af087ee15e1ec798 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:30.703744 18988 ts_tablet_manager.cc:1398] T 68ca48351cd4412381f5195d81109016 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.035s user 0.000s sys 0.004s
I20240430 20:11:30.704257 18988 tablet_bootstrap.cc:492] T 12c6e131fa904d23865b8673b58a7e33 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:30.715606 19386 consensus_queue.cc:1035] T 5bde72e2791a497db7dd08686ca36d80 P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:30.719357 18985 tablet_bootstrap.cc:492] T eed6338ad3d64859af087ee15e1ec798 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:30.719573 18985 ts_tablet_manager.cc:1367] T eed6338ad3d64859af087ee15e1ec798 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.027s user 0.005s sys 0.004s
I20240430 20:11:30.720809 18985 raft_consensus.cc:348] T eed6338ad3d64859af087ee15e1ec798 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:30.721057 18985 raft_consensus.cc:374] T eed6338ad3d64859af087ee15e1ec798 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:30.721169 18985 raft_consensus.cc:729] T eed6338ad3d64859af087ee15e1ec798 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:30.721513 18985 consensus_queue.cc:260] T eed6338ad3d64859af087ee15e1ec798 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:30.722378 18985 ts_tablet_manager.cc:1398] T eed6338ad3d64859af087ee15e1ec798 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20240430 20:11:30.722973 18985 tablet_bootstrap.cc:492] T 20d166756819406ba85044f4fee9c4ce P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:30.725539 18985 tablet_bootstrap.cc:654] T 20d166756819406ba85044f4fee9c4ce P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:30.742806 18988 tablet_bootstrap.cc:654] T 12c6e131fa904d23865b8673b58a7e33 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:30.761847 18985 tablet_bootstrap.cc:492] T 20d166756819406ba85044f4fee9c4ce P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:30.762032 18985 ts_tablet_manager.cc:1367] T 20d166756819406ba85044f4fee9c4ce P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.039s user 0.010s sys 0.011s
I20240430 20:11:30.762917 18985 raft_consensus.cc:348] T 20d166756819406ba85044f4fee9c4ce P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:30.763151 18985 raft_consensus.cc:374] T 20d166756819406ba85044f4fee9c4ce P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:30.763242 18985 raft_consensus.cc:729] T 20d166756819406ba85044f4fee9c4ce P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:30.763460 18983 tablet_bootstrap.cc:492] T b688e03ea71940c69bda5520b62c1fc3 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:30.763540 18985 consensus_queue.cc:260] T 20d166756819406ba85044f4fee9c4ce P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:30.765208 18985 ts_tablet_manager.cc:1398] T 20d166756819406ba85044f4fee9c4ce P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240430 20:11:30.765555 18985 tablet_bootstrap.cc:492] T a05e32a33aa7488eb287cc6314c09c44 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:30.772428 19482 raft_consensus.cc:482] T e5111d7607dd42188a0176cf5be2ec05 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:30.772625 19482 raft_consensus.cc:504] T e5111d7607dd42188a0176cf5be2ec05 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:30.773278 19482 leader_election.cc:288] T e5111d7607dd42188a0176cf5be2ec05 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
W20240430 20:11:30.774942 18417 leader_election.cc:341] T e5111d7607dd42188a0176cf5be2ec05 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Not found: Tablet not found: e5111d7607dd42188a0176cf5be2ec05
I20240430 20:11:30.775557 18985 tablet_bootstrap.cc:654] T a05e32a33aa7488eb287cc6314c09c44 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
W20240430 20:11:30.777150 18415 leader_election.cc:341] T e5111d7607dd42188a0176cf5be2ec05 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Not found: Tablet not found: e5111d7607dd42188a0176cf5be2ec05
I20240430 20:11:30.777320 18415 leader_election.cc:302] T e5111d7607dd42188a0176cf5be2ec05 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a; no voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710
I20240430 20:11:30.777954 19482 raft_consensus.cc:2726] T e5111d7607dd42188a0176cf5be2ec05 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:30.782760 18730 tablet_service.cc:1467] Processing CreateTablet for tablet e87fb2f0361d4aa3ada27bbdfa88e94c (DEFAULT_TABLE table=default.Table_42c1cf2f2e9542feaaa80106ced02347 [id=02a19d9d05264236b35244d4e9998d36]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:30.783013 18588 tablet_service.cc:1467] Processing CreateTablet for tablet e87fb2f0361d4aa3ada27bbdfa88e94c (DEFAULT_TABLE table=default.Table_42c1cf2f2e9542feaaa80106ced02347 [id=02a19d9d05264236b35244d4e9998d36]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:30.783244 18730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e87fb2f0361d4aa3ada27bbdfa88e94c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:30.783640 18588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e87fb2f0361d4aa3ada27bbdfa88e94c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:30.784557 19482 raft_consensus.cc:482] T 724e51ff05ec4baebc2bb55e05c8eddc P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:30.784750 19482 raft_consensus.cc:504] T 724e51ff05ec4baebc2bb55e05c8eddc P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:30.785212 19048 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 24dcc20832804229a2664873bf6ddaa7 (table default.Table_be2ebe77f859489f9cd3e912cb1edc8b [id=2d44ec94131148048021b2ee6366ba30]). Expected version 2 got 0
I20240430 20:11:30.785471 19482 leader_election.cc:288] T 724e51ff05ec4baebc2bb55e05c8eddc P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:30.785545 19048 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet acd23f0a63d5448aaa7ed78422e67bad (table default.Table_972f13be4dca4cd19b9a04534dc59f51 [id=69f37db4de634a3185613e40013e0ed7]). Expected version 1 got 0
I20240430 20:11:30.785701 19048 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 294644d7a16b4ca2bbb6d106bcb8c7ff (table default.Table_190b2d86515a44898fa38a49517b9df1 [id=97d8a36ddfae4bb5b99c04a7a08544af]). Expected version 2 got 0
I20240430 20:11:30.786136 18763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "724e51ff05ec4baebc2bb55e05c8eddc" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:30.786377 18763 raft_consensus.cc:2454] T 724e51ff05ec4baebc2bb55e05c8eddc P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 0.
I20240430 20:11:30.786856 18983 ts_tablet_manager.cc:1367] T b688e03ea71940c69bda5520b62c1fc3 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.214s user 0.040s sys 0.039s
I20240430 20:11:30.787184 18417 leader_election.cc:302] T 724e51ff05ec4baebc2bb55e05c8eddc P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters:
I20240430 20:11:30.787473 19185 raft_consensus.cc:482] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:30.787604 19185 raft_consensus.cc:504] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:30.788242 19185 leader_election.cc:288] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:30.788386 19185 raft_consensus.cc:2781] T 724e51ff05ec4baebc2bb55e05c8eddc P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:30.788471 19185 raft_consensus.cc:482] T 724e51ff05ec4baebc2bb55e05c8eddc P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:30.788548 19185 raft_consensus.cc:3037] T 724e51ff05ec4baebc2bb55e05c8eddc P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:30.788703 19048 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 0b4d9e860d7c45fd91d7a566605c9283 (table default.Table_44426457fd7045dfa1c149f0a5ed5129 [id=d5cea90cbc2c48df9d8b80da1eccc949]). Expected version 1 got 0
I20240430 20:11:30.787775 18983 raft_consensus.cc:348] T b688e03ea71940c69bda5520b62c1fc3 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:30.789837 18983 raft_consensus.cc:374] T b688e03ea71940c69bda5520b62c1fc3 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:30.789935 18983 raft_consensus.cc:729] T b688e03ea71940c69bda5520b62c1fc3 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:30.790177 18983 consensus_queue.cc:260] T b688e03ea71940c69bda5520b62c1fc3 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:30.791286 19048 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet a05e32a33aa7488eb287cc6314c09c44 (table default.Table_6551367971354d3195e4f7d47364ce7c [id=955d833ff8f14bf6957abadf9ef669e2]). Expected version 1 got 0
I20240430 20:11:30.791396 18588 tablet_service.cc:1467] Processing CreateTablet for tablet 681cdd85ac6847a0a1aaded310456582 (DEFAULT_TABLE table=default.Table_8e9e6e39df854972804dfa7c260af4e4 [id=e36e9fabccf14feeb0c643cb41b4a4f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:30.791648 19048 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 680553d4d0bc4fcb839235c1832757e1 (table default.Table_49a012a7c4c8457fadb0a8f0fc9654c6 [id=eecb2f8eeef547c5ab1abe6a2430e592]). Expected version 1 got 0
I20240430 20:11:30.791636 18741 tablet_service.cc:1467] Processing CreateTablet for tablet 681cdd85ac6847a0a1aaded310456582 (DEFAULT_TABLE table=default.Table_8e9e6e39df854972804dfa7c260af4e4 [id=e36e9fabccf14feeb0c643cb41b4a4f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:30.791857 18588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 681cdd85ac6847a0a1aaded310456582. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:30.791846 19048 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 76d6e9d547ca4a848eae852944b1d117 (table default.Table_210b7eaff7144b2eb7db1c3b5f7d9505 [id=ff86c0357c164dd0991098c416087773]). Expected version 1 got 0
I20240430 20:11:30.792084 18741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 681cdd85ac6847a0a1aaded310456582. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:30.792167 19048 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 6e754af64cb64fd2a082adae7d5f24da (table default.Table_208b12b8e9bd4f3e8bc9c6d422ada649 [id=1df4e95702cd46f59ab21b101049a9ed]). Expected version 1 got 0
I20240430 20:11:30.792356 19048 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 34130f2820064fa0898ef1f9d7de1c6d (table default.Table_4537f891b01048ceb51202abc3600911 [id=5a1c4a7be2fc465fac89a50854eb3322]). Expected version 1 got 0
I20240430 20:11:30.792426 18985 tablet_bootstrap.cc:492] T a05e32a33aa7488eb287cc6314c09c44 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:30.792997 18985 ts_tablet_manager.cc:1367] T a05e32a33aa7488eb287cc6314c09c44 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.027s user 0.006s sys 0.001s
I20240430 20:11:30.793520 18730 tablet_service.cc:1467] Processing CreateTablet for tablet abdaf243e5be4ad58036b23151b14fe4 (DEFAULT_TABLE table=default.Table_1a641ff0ab5944a495a73189a6e8499a [id=c2bcc99b99d14bd480881a9c8dc37dbb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:30.793948 18730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abdaf243e5be4ad58036b23151b14fe4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:30.793835 18985 raft_consensus.cc:348] T a05e32a33aa7488eb287cc6314c09c44 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:30.794061 18985 raft_consensus.cc:374] T a05e32a33aa7488eb287cc6314c09c44 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:30.794129 18985 raft_consensus.cc:729] T a05e32a33aa7488eb287cc6314c09c44 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:30.794409 18985 consensus_queue.cc:260] T a05e32a33aa7488eb287cc6314c09c44 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:30.795136 18985 ts_tablet_manager.cc:1398] T a05e32a33aa7488eb287cc6314c09c44 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240430 20:11:30.795509 18985 tablet_bootstrap.cc:492] T 47c37876694c4a4abb1091c631dab7a5 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:30.796072 18456 tablet_service.cc:1467] Processing CreateTablet for tablet e87fb2f0361d4aa3ada27bbdfa88e94c (DEFAULT_TABLE table=default.Table_42c1cf2f2e9542feaaa80106ced02347 [id=02a19d9d05264236b35244d4e9998d36]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:30.796490 18456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e87fb2f0361d4aa3ada27bbdfa88e94c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:30.797199 18985 tablet_bootstrap.cc:654] T 47c37876694c4a4abb1091c631dab7a5 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:30.798902 18455 tablet_service.cc:1467] Processing CreateTablet for tablet 681cdd85ac6847a0a1aaded310456582 (DEFAULT_TABLE table=default.Table_8e9e6e39df854972804dfa7c260af4e4 [id=e36e9fabccf14feeb0c643cb41b4a4f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:30.799259 18455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 681cdd85ac6847a0a1aaded310456582. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:30.800963 18983 ts_tablet_manager.cc:1398] T b688e03ea71940c69bda5520b62c1fc3 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.014s user 0.000s sys 0.002s
I20240430 20:11:30.801306 18983 tablet_bootstrap.cc:492] T 01436e5b6232494babbb414342f03090 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:30.803130 18983 tablet_bootstrap.cc:654] T 01436e5b6232494babbb414342f03090 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:30.804025 18763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd6083f12c7e4d1e8a2006b3ddf9aa3a" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:30.804354 18763 raft_consensus.cc:2379] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2efb542aa55e439b99ccb98d1c5c884a in current term 1: Already voted for candidate 5e1c6397f09747f091534faf93542710 in this term.
I20240430 20:11:30.805413 18599 tablet_service.cc:1467] Processing CreateTablet for tablet abdaf243e5be4ad58036b23151b14fe4 (DEFAULT_TABLE table=default.Table_1a641ff0ab5944a495a73189a6e8499a [id=c2bcc99b99d14bd480881a9c8dc37dbb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:30.805794 19048 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 32c823d1700949ab8b22adcd83b7bfd1 (table default.Table_e8c73c89623d4b49b8d7f0a01d729bdd [id=5cd10682ce52414cb5a103e4433a6a0d]). Expected version 1 got 0
I20240430 20:11:30.805994 18599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abdaf243e5be4ad58036b23151b14fe4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:30.807286 18736 tablet_service.cc:1467] Processing CreateTablet for tablet 2cdccf4be7094ca5b863dd5c3432d55d (DEFAULT_TABLE table=default.Table_f33a9e03642c44efbf1c27f1c7e70fb9 [id=c64cf189816b46cdb25574e5fddd9684]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:30.807680 18736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cdccf4be7094ca5b863dd5c3432d55d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:30.807781 18985 tablet_bootstrap.cc:492] T 47c37876694c4a4abb1091c631dab7a5 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:30.807926 18985 ts_tablet_manager.cc:1367] T 47c37876694c4a4abb1091c631dab7a5 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240430 20:11:30.808789 18985 raft_consensus.cc:348] T 47c37876694c4a4abb1091c631dab7a5 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:30.809003 18985 raft_consensus.cc:374] T 47c37876694c4a4abb1091c631dab7a5 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:30.809088 18985 raft_consensus.cc:729] T 47c37876694c4a4abb1091c631dab7a5 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:30.809092 18730 tablet_service.cc:1467] Processing CreateTablet for tablet 5bd0362ff41b4faca0205d136d596737 (DEFAULT_TABLE table=default.Table_fe1773cf77654e2a86e95444437b7201 [id=fe2222c2b62f4360a5ade622ae8b23e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:30.809291 18985 consensus_queue.cc:260] T 47c37876694c4a4abb1091c631dab7a5 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:30.809520 18730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bd0362ff41b4faca0205d136d596737. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:30.809993 18985 ts_tablet_manager.cc:1398] T 47c37876694c4a4abb1091c631dab7a5 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240430 20:11:30.810269 18985 tablet_bootstrap.cc:492] T b76a4105a2974fe280e759349037d027 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:30.811298 19048 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 6e826459c3f4437da81c9e6e09f0c22f (table default.Table_873c21f6dffd4d47a742b3699ab85268 [id=d7e2c4300ff94fa2acc65d6d6c932c65]). Expected version 1 got 0
I20240430 20:11:30.811913 18454 tablet_service.cc:1467] Processing CreateTablet for tablet abdaf243e5be4ad58036b23151b14fe4 (DEFAULT_TABLE table=default.Table_1a641ff0ab5944a495a73189a6e8499a [id=c2bcc99b99d14bd480881a9c8dc37dbb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:30.811954 18593 tablet_service.cc:1467] Processing CreateTablet for tablet 2cdccf4be7094ca5b863dd5c3432d55d (DEFAULT_TABLE table=default.Table_f33a9e03642c44efbf1c27f1c7e70fb9 [id=c64cf189816b46cdb25574e5fddd9684]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:30.812310 18454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abdaf243e5be4ad58036b23151b14fe4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:30.812358 18985 tablet_bootstrap.cc:654] T b76a4105a2974fe280e759349037d027 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:30.812352 18593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cdccf4be7094ca5b863dd5c3432d55d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:30.813064 18738 tablet_service.cc:1467] Processing CreateTablet for tablet 7ce09fcbe1ac457a9c05efaf000c3263 (DEFAULT_TABLE table=default.Table_ca129cca71d34045824d5f945570c3bd [id=78fd3f516083489ab3b42d90dcef7e4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:30.813418 18738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ce09fcbe1ac457a9c05efaf000c3263. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:30.813629 19048 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet b72fe1f0713b4076a81a3fc1c74f7fa7 (table default.Table_16b082d140234e1c9a1e70654839b4cd [id=c0d8dc87d1f9426a85f6176f161595a1]). Expected version 3 got 0
I20240430 20:11:30.814105 19048 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet fcc9698abda4491b98e1e879589913a8 (table default.Table_e431f94220aa44ecba799a54dfb612ae [id=1e785b6c159b4312bd1b064271f55a49]). Expected version 1 got 0
I20240430 20:11:30.814260 19048 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet fae9a35a2ac844b4bc2ebdf214aa7171 (table default.Table_0ab5b2f09fa7468bb89ab703f6d3d25c [id=a5d580cffa2447d78fc9f2c8e31264f1]). Expected version 4 got 0
I20240430 20:11:30.814488 19048 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet fd40967ab4a04001819e16c06a6c4300 (table default.Table_fda07fd7279c484db2bee67e8c859855 [id=da9ad991953a40c4b77b31317597ad1d]). Expected version 3 got 0
I20240430 20:11:30.814687 19048 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 372b135dd01f44f98cd713ca8857ba61 (table default.Table_ee86b8fcf2064d8b8c3e327e357d3395 [id=7f3a15a442984827a7521d5d1c98dc3d]). Expected version 2 got 0
I20240430 20:11:30.814915 18447 tablet_service.cc:1467] Processing CreateTablet for tablet 2cdccf4be7094ca5b863dd5c3432d55d (DEFAULT_TABLE table=default.Table_f33a9e03642c44efbf1c27f1c7e70fb9 [id=c64cf189816b46cdb25574e5fddd9684]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:30.814942 18592 tablet_service.cc:1467] Processing CreateTablet for tablet 5bd0362ff41b4faca0205d136d596737 (DEFAULT_TABLE table=default.Table_fe1773cf77654e2a86e95444437b7201 [id=fe2222c2b62f4360a5ade622ae8b23e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:30.815074 19048 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet aec9c09cc0454acab6006bb3e10982b7 (table default.Table_e68dc52988f24f1788517de94d7d2a0e [id=26d4860fb33047b7b812ebd4309f2e28]). Expected version 1 got 0
I20240430 20:11:30.815336 18447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cdccf4be7094ca5b863dd5c3432d55d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:30.815342 18592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bd0362ff41b4faca0205d136d596737. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:30.815558 19048 catalog_manager.cc:5484] T 77285afcafe04a829e485998bbbe00f0 P 2efb542aa55e439b99ccb98d1c5c884a reported cstate change: term changed from 0 to 1, leader changed from <none> to 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1). New cstate: current_term: 1 leader_uuid: "2efb542aa55e439b99ccb98d1c5c884a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } health_report { overall_health: HEALTHY } } }
I20240430 20:11:30.816056 19048 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 77285afcafe04a829e485998bbbe00f0 (table default.Table_e63f64a411094f8db2ef641f2257ca79 [id=040dec3a3d804cdfbdada359bbe5123b]). Expected version 1 got 0
I20240430 20:11:30.816316 19048 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 724e51ff05ec4baebc2bb55e05c8eddc (table default.Table_0c5d4686f2ed40fb8d62c31b0166dc05 [id=4dce8a767f0541e1beb5464b2ed95d06]). Expected version 1 got 0
I20240430 20:11:30.816462 19048 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet c66c142419ca4e1399ecd1c8d1e62a59 (table default.Table_0443359dbeff437fb6cffb0db33cbb30 [id=93c44abdf6b147228224388398cd7aeb]). Expected version 2 got 0
W20240430 20:11:30.818868 19027 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2a1e50f99c5d4b44bdbb51b58a92bc75 consensus state has no leader
I20240430 20:11:30.819067 19027 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:51664 (request call id 29) took 1752 ms. Trace:
W20240430 20:11:30.820044 19049 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 53a899fa5a424a0d99f1b47925e8c7ad consensus state has no leader
I20240430 20:11:30.820240 19049 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:51664 (request call id 39) took 1283 ms. Trace:
I20240430 20:11:30.820561 18737 tablet_service.cc:1467] Processing CreateTablet for tablet 50486644ec234933b9a0b45ee39a6685 (DEFAULT_TABLE table=default.Table_aed0b9eeaac74c98944426712706be6b [id=86d795a82e674d859104741ebd04f4e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:30.819185 19027 rpcz_store.cc:276] 0430 20:11:29.066389 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:29.674769 (+608380us) service_pool.cc:225] Handling call
0430 20:11:29.687561 (+ 12792us) spinlock_profiling.cc:243] Waited 12.5 ms on lock 0x616000040654. stack: 00007feb16c74756 00007feb16c74510 00007feb15ca748d 0000000000955804 00000000009557cd 00007feb257b315d 00007feb257b3253 00007feb224e5d41 00007feb2a0d134d 00007feb2a25ff46 00007feb21049c26 00007feb21049955 00007feb1d97c04e 00007feb1d97af2e 00007feb1d981405 00007feb1d981d2f
0430 20:11:29.687941 (+ 380us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0430 20:11:29.688208 (+ 267us) catalog_manager.cc:3613] Apply alter schema
0430 20:11:29.688474 (+ 266us) catalog_manager.cc:3636] Acquired catalog manager lock
0430 20:11:29.688605 (+ 131us) catalog_manager.cc:3783] Updating metadata on disk
0430 20:11:29.747889 (+ 59284us) catalog_manager.cc:3814] Committing alterations to in-memory state
0430 20:11:30.819014 (+1071125us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:29.707149 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:29.707322 (+ 173us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:29.707332 (+ 10us) write_op.cc:427] Acquired schema lock
0430 20:11:29.707336 (+ 4us) tablet.cc:660] Decoding operations
0430 20:11:29.707621 (+ 285us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:29.707642 (+ 21us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:29.707651 (+ 9us) tablet.cc:683] Acquiring locks for 1 operations
0430 20:11:29.707764 (+ 113us) tablet.cc:699] Row locks acquired
0430 20:11:29.707769 (+ 5us) write_op.cc:260] PREPARE: finished
0430 20:11:29.707824 (+ 55us) write_op.cc:270] Start()
0430 20:11:29.707872 (+ 48us) write_op.cc:276] Timestamp: P: 1714507889707775 usec, L: 0
0430 20:11:29.707879 (+ 7us) op_driver.cc:331] REPLICATION: starting
0430 20:11:29.726972 (+ 19093us) log.cc:842] Serialized 405 byte log entry
0430 20:11:29.731334 (+ 4362us) spinlock_profiling.cc:243] Waited 18.8 ms on lock 0x616000040654. stack: 00007feb16c74756 00007feb16c74510 00007feb15ca748d 0000000000955804 00000000009557cd 000000000095519a 00007feb224b9162 00007feb23821b02 00007feb23820629 00007feb2382853b 00007feb238283b8 0000000000a60bcc 00007feb16cbab74 00007feb16cbe19b 00007feb16cbe018 0000000000a60bcc
0430 20:11:29.746962 (+ 15628us) op_driver.cc:444] REPLICATION: finished
0430 20:11:29.747099 (+ 137us) write_op.cc:301] APPLY: starting
0430 20:11:29.747125 (+ 26us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:29.747197 (+ 72us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:29.747202 (+ 5us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:29.747272 (+ 70us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:29.747323 (+ 51us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0430 20:11:29.747336 (+ 13us) write_op.cc:312] APPLY: finished
0430 20:11:29.747442 (+ 106us) log.cc:842] Serialized 23 byte log entry
0430 20:11:29.747746 (+ 304us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:29.747775 (+ 29us) write_op.cc:446] Released schema lock
0430 20:11:29.747785 (+ 10us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:29.748015 (+ 230us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:29.748022 (+ 7us) write_op.cc:446] Released schema lock
Metrics: {"spinlock_wait_cycles":28756352,"child_traces":[["op",{"apply.queue_time_us":41,"apply.run_cpu_time_us":619,"apply.run_wall_time_us":986,"num_ops":1,"prepare.queue_time_us":17881,"prepare.run_cpu_time_us":1564,"prepare.run_wall_time_us":24290,"raft.queue_time_us":3869,"raft.run_cpu_time_us":435,"raft.run_wall_time_us":951,"replication_time_us":39029,"spinlock_wait_cycles":43276672}]]}
I20240430 20:11:30.820335 19049 rpcz_store.cc:276] 0430 20:11:29.536263 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:30.245556 (+709293us) service_pool.cc:225] Handling call
0430 20:11:30.256781 (+ 11225us) spinlock_profiling.cc:243] Waited 10.9 ms on lock 0x616000040654. stack: 00007feb16c74756 00007feb16c74510 00007feb15ca748d 0000000000955804 00000000009557cd 00007feb257b315d 00007feb257b3253 00007feb224e5d41 00007feb2a0d134d 00007feb2a25ff46 00007feb21049c26 00007feb21049955 00007feb1d97c04e 00007feb1d97af2e 00007feb1d981405 00007feb1d981d2f
0430 20:11:30.257083 (+ 302us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0430 20:11:30.257305 (+ 222us) catalog_manager.cc:3613] Apply alter schema
0430 20:11:30.257515 (+ 210us) catalog_manager.cc:3636] Acquired catalog manager lock
0430 20:11:30.257635 (+ 120us) catalog_manager.cc:3783] Updating metadata on disk
0430 20:11:30.329668 (+ 72033us) catalog_manager.cc:3814] Committing alterations to in-memory state
0430 20:11:30.820175 (+490507us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:30.307334 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:30.307487 (+ 153us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:30.307497 (+ 10us) write_op.cc:427] Acquired schema lock
0430 20:11:30.307501 (+ 4us) tablet.cc:660] Decoding operations
0430 20:11:30.307639 (+ 138us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:30.307657 (+ 18us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:30.307663 (+ 6us) tablet.cc:683] Acquiring locks for 1 operations
0430 20:11:30.307735 (+ 72us) tablet.cc:699] Row locks acquired
0430 20:11:30.307737 (+ 2us) write_op.cc:260] PREPARE: finished
0430 20:11:30.307779 (+ 42us) write_op.cc:270] Start()
0430 20:11:30.321947 (+ 14168us) write_op.cc:276] Timestamp: P: 1714507890307733 usec, L: 0
0430 20:11:30.321955 (+ 8us) op_driver.cc:331] REPLICATION: starting
0430 20:11:30.322182 (+ 227us) log.cc:842] Serialized 405 byte log entry
0430 20:11:30.326403 (+ 4221us) op_driver.cc:444] REPLICATION: finished
0430 20:11:30.329118 (+ 2715us) write_op.cc:301] APPLY: starting
0430 20:11:30.329140 (+ 22us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:30.329191 (+ 51us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:30.329225 (+ 34us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:30.329299 (+ 74us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:30.329344 (+ 45us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0430 20:11:30.329356 (+ 12us) write_op.cc:312] APPLY: finished
0430 20:11:30.329468 (+ 112us) log.cc:842] Serialized 23 byte log entry
0430 20:11:30.329536 (+ 68us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:30.329561 (+ 25us) write_op.cc:446] Released schema lock
0430 20:11:30.329570 (+ 9us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:30.329775 (+ 205us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:30.329781 (+ 6us) write_op.cc:446] Released schema lock
Metrics: {"spinlock_wait_cycles":25152896,"child_traces":[["op",{"apply.queue_time_us":2642,"apply.run_cpu_time_us":597,"apply.run_wall_time_us":730,"num_ops":1,"prepare.queue_time_us":49145,"prepare.run_cpu_time_us":972,"prepare.run_wall_time_us":27978,"raft.queue_time_us":40,"raft.run_cpu_time_us":246,"raft.run_wall_time_us":244,"replication_time_us":4421}]]}
I20240430 20:11:30.821020 18737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50486644ec234933b9a0b45ee39a6685. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:30.822759 19033 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: Sending DeleteTablet for 3 replicas of tablet 115c4a12440647a4bb46eb6824f7ea98
I20240430 20:11:30.823278 19033 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:51664 (request call id 23) took 1759 ms. Trace:
I20240430 20:11:30.823374 19033 rpcz_store.cc:276] 0430 20:11:29.063868 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:29.332971 (+269103us) service_pool.cc:225] Handling call
0430 20:11:29.547656 (+214685us) spinlock_profiling.cc:243] Waited 214 ms on lock 0x616000040654. stack: 00007feb16c74756 00007feb16c74510 00007feb15ca748d 0000000000955804 00000000009557cd 00007feb257b315d 00007feb257b3253 00007feb224e5d41 00007feb2a0d134d 00007feb2a25eca6 00007feb21047f66 00007feb21047c95 00007feb1d97c04e 00007feb1d97af2e 00007feb1d981405 00007feb1d981d2f
0430 20:11:29.548015 (+ 359us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0430 20:11:29.548146 (+ 131us) catalog_manager.cc:2749] Modifying in-memory table state
0430 20:11:29.548201 (+ 55us) catalog_manager.cc:2758] Locking tablets
0430 20:11:29.584240 (+ 36039us) catalog_manager.cc:2772] Removing table and tablets from system table
0430 20:11:29.697811 (+113571us) catalog_manager.cc:2790] Removing table from by-name map
0430 20:11:29.697944 (+ 133us) catalog_manager.cc:2806] Committing in-memory state
0430 20:11:30.822684 (+1124740us) catalog_manager.cc:2810] Aborting table tasks
0430 20:11:30.823220 (+ 536us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:29.594791 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:29.595136 (+ 345us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:29.595146 (+ 10us) write_op.cc:427] Acquired schema lock
0430 20:11:29.595151 (+ 5us) tablet.cc:660] Decoding operations
0430 20:11:29.595329 (+ 178us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:29.595346 (+ 17us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:29.595354 (+ 8us) tablet.cc:683] Acquiring locks for 2 operations
0430 20:11:29.595515 (+ 161us) tablet.cc:699] Row locks acquired
0430 20:11:29.595519 (+ 4us) write_op.cc:260] PREPARE: finished
0430 20:11:29.595558 (+ 39us) write_op.cc:270] Start()
0430 20:11:29.595598 (+ 40us) write_op.cc:276] Timestamp: P: 1714507889595525 usec, L: 0
0430 20:11:29.595603 (+ 5us) op_driver.cc:331] REPLICATION: starting
0430 20:11:29.625398 (+ 29795us) log.cc:842] Serialized 728 byte log entry
0430 20:11:29.648452 (+ 23054us) spinlock_profiling.cc:243] Waited 29.2 ms on lock 0x616000040654. stack: 00007feb16c74756 00007feb16c74510 00007feb15ca748d 0000000000955804 00000000009557cd 000000000095519a 00007feb224b9162 00007feb23821b02 00007feb23820629 00007feb2382853b 00007feb238283b8 0000000000a60bcc 00007feb16cbab74 00007feb16cbe19b 00007feb16cbe018 0000000000a60bcc
0430 20:11:29.696645 (+ 48193us) op_driver.cc:444] REPLICATION: finished
0430 20:11:29.696958 (+ 313us) write_op.cc:301] APPLY: starting
0430 20:11:29.696986 (+ 28us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:29.697054 (+ 68us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:29.697059 (+ 5us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:29.697155 (+ 96us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:29.697206 (+ 51us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0430 20:11:29.697218 (+ 12us) write_op.cc:312] APPLY: finished
0430 20:11:29.697330 (+ 112us) log.cc:842] Serialized 29 byte log entry
0430 20:11:29.697670 (+ 340us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:29.697701 (+ 31us) write_op.cc:446] Released schema lock
0430 20:11:29.697712 (+ 11us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:29.706696 (+ 8984us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:29.706710 (+ 14us) write_op.cc:446] Released schema lock
Metrics: {"spinlock_wait_cycles":493157248,"child_traces":[["op",{"apply.queue_time_us":232,"apply.run_cpu_time_us":724,"apply.run_wall_time_us":9835,"num_ops":2,"prepare.queue_time_us":9797,"prepare.run_cpu_time_us":1397,"prepare.run_wall_time_us":53755,"raft.queue_time_us":1508,"raft.run_cpu_time_us":4,"raft.run_wall_time_us":4,"replication_time_us":101016,"spinlock_wait_cycles":67230080}]]}
I20240430 20:11:30.824168 19033 master_service.cc:946] ReplaceTablet: received request to replace tablet d4508e4f6dbd42109d951c648b3dc7df from {username='slave'} at 127.0.0.1:51664
I20240430 20:11:30.824610 19033 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:51664 (request call id 40) took 1275 ms. Trace:
I20240430 20:11:30.824709 19033 rpcz_store.cc:276] 0430 20:11:29.549533 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:30.824075 (+1274542us) service_pool.cc:225] Handling call
0430 20:11:30.824574 (+ 499us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240430 20:11:30.825063 19033 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_f33a9e03642c44efbf1c27f1c7e70fb9" } modify_external_catalogs: true
I20240430 20:11:30.825263 19033 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_f33a9e03642c44efbf1c27f1c7e70fb9" } modify_external_catalogs: true
W20240430 20:11:30.826217 19023 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5bde72e2791a497db7dd08686ca36d80 consensus state has no leader
I20240430 20:11:30.826390 19023 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:51664 (request call id 35) took 1341 ms. Trace:
I20240430 20:11:30.826489 19023 rpcz_store.cc:276] 0430 20:11:29.484511 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:29.784541 (+300030us) service_pool.cc:225] Handling call
0430 20:11:29.790350 (+ 5809us) spinlock_profiling.cc:243] Waited 5.46 ms on lock 0x616000040654. stack: 00007feb16c74756 00007feb16c74510 00007feb15ca748d 0000000000955804 00000000009557cd 00007feb257b315d 00007feb257b3253 00007feb224e5d41 00007feb2a0d134d 00007feb2a25ff46 00007feb21049c26 00007feb21049955 00007feb1d97c04e 00007feb1d97af2e 00007feb1d981405 00007feb1d981d2f
0430 20:11:29.790839 (+ 489us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0430 20:11:29.791120 (+ 281us) catalog_manager.cc:3613] Apply alter schema
0430 20:11:29.791358 (+ 238us) catalog_manager.cc:3636] Acquired catalog manager lock
0430 20:11:29.791463 (+ 105us) catalog_manager.cc:3783] Updating metadata on disk
0430 20:11:29.927419 (+135956us) catalog_manager.cc:3814] Committing alterations to in-memory state
0430 20:11:30.826341 (+898922us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:29.896865 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:29.897006 (+ 141us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:29.897013 (+ 7us) write_op.cc:427] Acquired schema lock
0430 20:11:29.897015 (+ 2us) tablet.cc:660] Decoding operations
0430 20:11:29.897148 (+ 133us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:29.897163 (+ 15us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:29.897167 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0430 20:11:29.897240 (+ 73us) tablet.cc:699] Row locks acquired
0430 20:11:29.897243 (+ 3us) write_op.cc:260] PREPARE: finished
0430 20:11:29.897287 (+ 44us) write_op.cc:270] Start()
0430 20:11:29.897326 (+ 39us) write_op.cc:276] Timestamp: P: 1714507889897225 usec, L: 0
0430 20:11:29.897330 (+ 4us) op_driver.cc:331] REPLICATION: starting
0430 20:11:29.907377 (+ 10047us) log.cc:842] Serialized 405 byte log entry
0430 20:11:29.919466 (+ 12089us) spinlock_profiling.cc:243] Waited 9.67 ms on lock 0x616000040654. stack: 00007feb16c74756 00007feb16c74510 00007feb15ca748d 0000000000955804 00000000009557cd 000000000095519a 00007feb224b9162 00007feb23821b02 00007feb23820629 00007feb2382853b 00007feb238283b8 0000000000a60bcc 00007feb16cbab74 00007feb16cbe19b 00007feb16cbe018 0000000000a60bcc
0430 20:11:29.926218 (+ 6752us) op_driver.cc:444] REPLICATION: finished
0430 20:11:29.926628 (+ 410us) write_op.cc:301] APPLY: starting
0430 20:11:29.926656 (+ 28us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:29.926697 (+ 41us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:29.926702 (+ 5us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:29.926757 (+ 55us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:29.926795 (+ 38us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0430 20:11:29.926803 (+ 8us) write_op.cc:312] APPLY: finished
0430 20:11:29.927007 (+ 204us) log.cc:842] Serialized 23 byte log entry
0430 20:11:29.927309 (+ 302us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:29.927328 (+ 19us) write_op.cc:446] Released schema lock
0430 20:11:29.927336 (+ 8us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:29.927521 (+ 185us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:29.927526 (+ 5us) write_op.cc:446] Released schema lock
Metrics: {"spinlock_wait_cycles":12567808,"child_traces":[["op",{"apply.queue_time_us":168,"apply.run_cpu_time_us":558,"apply.run_wall_time_us":1018,"num_ops":1,"prepare.queue_time_us":104788,"prepare.run_cpu_time_us":1181,"prepare.run_wall_time_us":23581,"raft.queue_time_us":902,"raft.run_cpu_time_us":297,"raft.run_wall_time_us":747,"replication_time_us":28857,"spinlock_wait_cycles":22251392}]]}
I20240430 20:11:30.827677 18735 tablet_service.cc:1467] Processing CreateTablet for tablet d61c279d20b0471a967942525ceb4fcb (DEFAULT_TABLE table=default.Table_a327a8eced314cdf95731f1181f52fb1 [id=a95fed1bc3fa4865bf9a35dfc51b659c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:30.827726 19023 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_210b7eaff7144b2eb7db1c3b5f7d9505" } new_table_name: "default.Table_104ae015924a48fd8d27220792f07470" modify_external_catalogs: true
I20240430 20:11:30.828147 18735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d61c279d20b0471a967942525ceb4fcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:30.828305 18724 tablet_service.cc:1467] Processing CreateTablet for tablet 30a521916bda46678be20a066bd1d3f3 (DEFAULT_TABLE table=default.Table_9579e986b42043f18f8e4aa8d0f6f754 [id=21e6703c9d844a37977f0f161c0b8c08]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:30.828612 18724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30a521916bda46678be20a066bd1d3f3. 1 dirs total, 0 dirs full, 0 dirs failed
W20240430 20:11:30.829408 19029 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0b4d9e860d7c45fd91d7a566605c9283 consensus state has no leader
I20240430 20:11:30.829612 19029 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:51664 (request call id 38) took 1294 ms. Trace:
I20240430 20:11:30.822769 19028 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: Sending DeleteTablet for 3 replicas of tablet a9e0659564894b0e8cb7f600bb07a085
I20240430 20:11:30.830953 19028 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:51664 (request call id 32) took 1351 ms. Trace:
I20240430 20:11:30.831068 19028 rpcz_store.cc:276] 0430 20:11:29.479977 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:29.713268 (+233291us) service_pool.cc:225] Handling call
0430 20:11:29.721167 (+ 7899us) spinlock_profiling.cc:243] Waited 7.5 ms on lock 0x616000040654. stack: 00007feb16c74756 00007feb16c74510 00007feb15ca748d 0000000000955804 00000000009557cd 00007feb257b315d 00007feb257b3253 00007feb224e5d41 00007feb2a0d134d 00007feb2a25eca6 00007feb21047f66 00007feb21047c95 00007feb1d97c04e 00007feb1d97af2e 00007feb1d981405 00007feb1d981d2f
0430 20:11:29.721771 (+ 604us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0430 20:11:29.721954 (+ 183us) catalog_manager.cc:2749] Modifying in-memory table state
0430 20:11:29.722064 (+ 110us) catalog_manager.cc:2758] Locking tablets
0430 20:11:29.904387 (+182323us) catalog_manager.cc:2772] Removing table and tablets from system table
0430 20:11:30.248578 (+344191us) catalog_manager.cc:2790] Removing table from by-name map
0430 20:11:30.248704 (+ 126us) catalog_manager.cc:2806] Committing in-memory state
0430 20:11:30.822710 (+574006us) catalog_manager.cc:2810] Aborting table tasks
0430 20:11:30.830892 (+ 8182us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:29.992860 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:29.993021 (+ 161us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:29.993028 (+ 7us) write_op.cc:427] Acquired schema lock
0430 20:11:29.993031 (+ 3us) tablet.cc:660] Decoding operations
0430 20:11:29.993172 (+ 141us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:29.993187 (+ 15us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:29.993193 (+ 6us) tablet.cc:683] Acquiring locks for 2 operations
0430 20:11:29.993313 (+ 120us) tablet.cc:699] Row locks acquired
0430 20:11:29.993316 (+ 3us) write_op.cc:260] PREPARE: finished
0430 20:11:29.993362 (+ 46us) write_op.cc:270] Start()
0430 20:11:29.993402 (+ 40us) write_op.cc:276] Timestamp: P: 1714507889993344 usec, L: 0
0430 20:11:29.993409 (+ 7us) op_driver.cc:331] REPLICATION: starting
0430 20:11:29.993638 (+ 229us) log.cc:842] Serialized 759 byte log entry
0430 20:11:30.247422 (+253784us) op_driver.cc:444] REPLICATION: finished
0430 20:11:30.247533 (+ 111us) write_op.cc:301] APPLY: starting
0430 20:11:30.247552 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:30.247601 (+ 49us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:30.247626 (+ 25us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:30.247704 (+ 78us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:30.247747 (+ 43us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0430 20:11:30.247756 (+ 9us) write_op.cc:312] APPLY: finished
0430 20:11:30.248092 (+ 336us) log.cc:842] Serialized 29 byte log entry
0430 20:11:30.248435 (+ 343us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:30.248464 (+ 29us) write_op.cc:446] Released schema lock
0430 20:11:30.248474 (+ 10us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:30.248761 (+ 287us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:30.248768 (+ 7us) write_op.cc:446] Released schema lock
Metrics: {"spinlock_wait_cycles":17253248,"child_traces":[["op",{"apply.queue_time_us":59,"apply.run_cpu_time_us":599,"apply.run_wall_time_us":1300,"num_ops":2,"prepare.queue_time_us":87708,"prepare.run_cpu_time_us":1014,"prepare.run_wall_time_us":249366,"raft.queue_time_us":63,"raft.run_cpu_time_us":132,"raft.run_wall_time_us":133,"replication_time_us":253990}]]}
I20240430 20:11:30.826782 19024 catalog_manager.cc:6029] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: default.Table_c1669746f9114bf7b37a6f6a68fed20e [id=7f602584cef9486eb7d9b4389be3f43e] alter complete (version 2)
I20240430 20:11:30.832284 19028 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_9f4b05e2470349c7bbc8c4ae15e778b2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:30.833019 19028 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9f4b05e2470349c7bbc8c4ae15e778b2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:30.836959 18590 tablet_service.cc:1467] Processing CreateTablet for tablet 7ce09fcbe1ac457a9c05efaf000c3263 (DEFAULT_TABLE table=default.Table_ca129cca71d34045824d5f945570c3bd [id=78fd3f516083489ab3b42d90dcef7e4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:30.837074 18591 tablet_service.cc:1467] Processing CreateTablet for tablet f940d17b02b541289121463e05b0bb5f (DEFAULT_TABLE table=default.Table_07993f6164104b888058e1b35c9f4ed7 [id=f2e6cac20f6a4261bb60f2e40f210491]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:30.837460 18590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ce09fcbe1ac457a9c05efaf000c3263. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:30.837500 18602 tablet_service.cc:1514] Processing DeleteTablet for tablet a9e0659564894b0e8cb7f600bb07a085 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:29 UTC) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:30.837553 18985 tablet_bootstrap.cc:492] T b76a4105a2974fe280e759349037d027 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:30.837759 18985 ts_tablet_manager.cc:1367] T b76a4105a2974fe280e759349037d027 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.028s user 0.003s sys 0.003s
I20240430 20:11:30.838572 18985 raft_consensus.cc:348] T b76a4105a2974fe280e759349037d027 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:30.838842 18591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f940d17b02b541289121463e05b0bb5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:30.838846 18985 raft_consensus.cc:374] T b76a4105a2974fe280e759349037d027 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:30.838917 18985 raft_consensus.cc:729] T b76a4105a2974fe280e759349037d027 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:30.839174 18985 consensus_queue.cc:260] T b76a4105a2974fe280e759349037d027 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:30.840128 18985 ts_tablet_manager.cc:1398] T b76a4105a2974fe280e759349037d027 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20240430 20:11:30.840441 18985 tablet_bootstrap.cc:492] T 12c6e131fa904d23865b8673b58a7e33 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:30.842242 18985 tablet_bootstrap.cc:654] T 12c6e131fa904d23865b8673b58a7e33 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:30.842583 18583 tablet_service.cc:1467] Processing CreateTablet for tablet d61c279d20b0471a967942525ceb4fcb (DEFAULT_TABLE table=default.Table_a327a8eced314cdf95731f1181f52fb1 [id=a95fed1bc3fa4865bf9a35dfc51b659c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:30.843052 18586 tablet_service.cc:1467] Processing CreateTablet for tablet 50486644ec234933b9a0b45ee39a6685 (DEFAULT_TABLE table=default.Table_aed0b9eeaac74c98944426712706be6b [id=86d795a82e674d859104741ebd04f4e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:30.843528 18603 tablet_service.cc:1467] Processing CreateTablet for tablet 30a521916bda46678be20a066bd1d3f3 (DEFAULT_TABLE table=default.Table_9579e986b42043f18f8e4aa8d0f6f754 [id=21e6703c9d844a37977f0f161c0b8c08]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:30.829715 19029 rpcz_store.cc:276] 0430 20:11:29.535019 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:29.813481 (+278462us) service_pool.cc:225] Handling call
0430 20:11:29.813994 (+ 513us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0430 20:11:29.814299 (+ 305us) catalog_manager.cc:3613] Apply alter schema
0430 20:11:29.814551 (+ 252us) catalog_manager.cc:3636] Acquired catalog manager lock
0430 20:11:29.814651 (+ 100us) catalog_manager.cc:3783] Updating metadata on disk
0430 20:11:29.930439 (+115788us) catalog_manager.cc:3814] Committing alterations to in-memory state
0430 20:11:30.829546 (+899107us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:29.920475 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:29.920992 (+ 517us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:29.920999 (+ 7us) write_op.cc:427] Acquired schema lock
0430 20:11:29.921002 (+ 3us) tablet.cc:660] Decoding operations
0430 20:11:29.921122 (+ 120us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:29.921135 (+ 13us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:29.921141 (+ 6us) tablet.cc:683] Acquiring locks for 1 operations
0430 20:11:29.921209 (+ 68us) tablet.cc:699] Row locks acquired
0430 20:11:29.921211 (+ 2us) write_op.cc:260] PREPARE: finished
0430 20:11:29.921247 (+ 36us) write_op.cc:270] Start()
0430 20:11:29.921278 (+ 31us) write_op.cc:276] Timestamp: P: 1714507889921187 usec, L: 0
0430 20:11:29.921282 (+ 4us) op_driver.cc:331] REPLICATION: starting
0430 20:11:29.921458 (+ 176us) log.cc:842] Serialized 405 byte log entry
0430 20:11:29.928796 (+ 7338us) op_driver.cc:444] REPLICATION: finished
0430 20:11:29.929724 (+ 928us) write_op.cc:301] APPLY: starting
0430 20:11:29.929740 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:29.929787 (+ 47us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:29.929827 (+ 40us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:29.929882 (+ 55us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:29.929920 (+ 38us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0430 20:11:29.929928 (+ 8us) write_op.cc:312] APPLY: finished
0430 20:11:29.930024 (+ 96us) log.cc:842] Serialized 23 byte log entry
0430 20:11:29.930118 (+ 94us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:29.930144 (+ 26us) write_op.cc:446] Released schema lock
0430 20:11:29.930154 (+ 10us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:29.930229 (+ 75us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:29.930235 (+ 6us) write_op.cc:446] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":848,"apply.run_cpu_time_us":578,"apply.run_wall_time_us":579,"num_ops":1,"prepare.queue_time_us":105220,"prepare.run_cpu_time_us":795,"prepare.run_wall_time_us":7215,"raft.queue_time_us":744,"raft.run_cpu_time_us":4,"raft.run_wall_time_us":6,"replication_time_us":7488}]]}
I20240430 20:11:30.843951 18589 tablet_service.cc:1467] Processing CreateTablet for tablet 115c4a12440647a4bb46eb6824f7ea98 (DEFAULT_TABLE table=default.Table_cd719a43207f4bf983e1b8e6d236bb16 [id=47e0b04b17834deb833a8f31c123edc5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:30.844269 18731 tablet_service.cc:1514] Processing DeleteTablet for tablet 115c4a12440647a4bb46eb6824f7ea98 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:29 UTC) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:30.844357 18597 tablet_service.cc:1467] Processing CreateTablet for tablet 2e79df8744e74b5991a79ce978ff0202 (DEFAULT_TABLE table=default.Table_276bbdc7cfe542d1a2936485a2aad1ec [id=f146a15e6e8a4acbb4c734c1c863da57]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:30.844730 18587 tablet_service.cc:1467] Processing CreateTablet for tablet f7b0132c17d44a9ba17288e0321a7f11 (DEFAULT_TABLE table=default.Table_01d62e066e4942d9aa1f582189d60d49 [id=97861b0b1ce94638b62bbd123d774d7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:30.845023 18600 tablet_service.cc:1514] Processing DeleteTablet for tablet 115c4a12440647a4bb46eb6824f7ea98 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:29 UTC) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:30.845345 19029 master_service.cc:946] ReplaceTablet: received request to replace tablet a9e0659564894b0e8cb7f600bb07a085 from {username='slave'} at 127.0.0.1:51664
I20240430 20:11:30.845522 18586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50486644ec234933b9a0b45ee39a6685. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:30.846872 18603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30a521916bda46678be20a066bd1d3f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:30.848420 18462 tablet_service.cc:1467] Processing CreateTablet for tablet 7ce09fcbe1ac457a9c05efaf000c3263 (DEFAULT_TABLE table=default.Table_ca129cca71d34045824d5f945570c3bd [id=78fd3f516083489ab3b42d90dcef7e4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:30.848873 18462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ce09fcbe1ac457a9c05efaf000c3263. 1 dirs total, 0 dirs full, 0 dirs failed
W20240430 20:11:30.848902 19023 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 76d6e9d547ca4a848eae852944b1d117 consensus state has no leader
I20240430 20:11:30.849027 18450 tablet_service.cc:1467] Processing CreateTablet for tablet 5bd0362ff41b4faca0205d136d596737 (DEFAULT_TABLE table=default.Table_fe1773cf77654e2a86e95444437b7201 [id=fe2222c2b62f4360a5ade622ae8b23e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:30.849112 19023 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:51664 (request call id 42) took 1069 ms. Trace:
I20240430 20:11:30.849347 18450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bd0362ff41b4faca0205d136d596737. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:30.849190 19023 rpcz_store.cc:276] 0430 20:11:29.779921 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:30.827510 (+1047589us) service_pool.cc:225] Handling call
0430 20:11:30.827976 (+ 466us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0430 20:11:30.828220 (+ 244us) catalog_manager.cc:3613] Apply alter schema
0430 20:11:30.828442 (+ 222us) catalog_manager.cc:3636] Acquired catalog manager lock
0430 20:11:30.828521 (+ 79us) catalog_manager.cc:3783] Updating metadata on disk
0430 20:11:30.848631 (+ 20110us) catalog_manager.cc:3814] Committing alterations to in-memory state
0430 20:11:30.849058 (+ 427us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:30.830684 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:30.830785 (+ 101us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:30.830793 (+ 8us) write_op.cc:427] Acquired schema lock
0430 20:11:30.830795 (+ 2us) tablet.cc:660] Decoding operations
0430 20:11:30.830908 (+ 113us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:30.830923 (+ 15us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:30.830928 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0430 20:11:30.830996 (+ 68us) tablet.cc:699] Row locks acquired
0430 20:11:30.830998 (+ 2us) write_op.cc:260] PREPARE: finished
0430 20:11:30.831038 (+ 40us) write_op.cc:270] Start()
0430 20:11:30.831080 (+ 42us) write_op.cc:276] Timestamp: P: 1714507890830989 usec, L: 0
0430 20:11:30.831084 (+ 4us) op_driver.cc:331] REPLICATION: starting
0430 20:11:30.831336 (+ 252us) log.cc:842] Serialized 405 byte log entry
0430 20:11:30.835802 (+ 4466us) op_driver.cc:444] REPLICATION: finished
0430 20:11:30.836054 (+ 252us) write_op.cc:301] APPLY: starting
0430 20:11:30.836073 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:30.836122 (+ 49us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:30.836155 (+ 33us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:30.836217 (+ 62us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:30.836256 (+ 39us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0430 20:11:30.836264 (+ 8us) write_op.cc:312] APPLY: finished
0430 20:11:30.836350 (+ 86us) log.cc:842] Serialized 23 byte log entry
0430 20:11:30.836421 (+ 71us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:30.836442 (+ 21us) write_op.cc:446] Released schema lock
0430 20:11:30.836449 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:30.836510 (+ 61us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:30.836514 (+ 4us) write_op.cc:446] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":175,"apply.run_cpu_time_us":515,"apply.run_wall_time_us":516,"num_ops":1,"prepare.queue_time_us":1560,"prepare.run_cpu_time_us":801,"prepare.run_wall_time_us":805,"raft.queue_time_us":980,"raft.run_cpu_time_us":240,"raft.run_wall_time_us":240,"replication_time_us":4695}]]}
I20240430 20:11:30.849859 19023 master_service.cc:946] ReplaceTablet: received request to replace tablet 2a1e50f99c5d4b44bdbb51b58a92bc75 from {username='slave'} at 127.0.0.1:51664
I20240430 20:11:30.851187 18589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 115c4a12440647a4bb46eb6824f7ea98. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:30.852959 18457 tablet_service.cc:1467] Processing CreateTablet for tablet d61c279d20b0471a967942525ceb4fcb (DEFAULT_TABLE table=default.Table_a327a8eced314cdf95731f1181f52fb1 [id=a95fed1bc3fa4865bf9a35dfc51b659c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:30.853044 18449 tablet_service.cc:1467] Processing CreateTablet for tablet 50486644ec234933b9a0b45ee39a6685 (DEFAULT_TABLE table=default.Table_aed0b9eeaac74c98944426712706be6b [id=86d795a82e674d859104741ebd04f4e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:30.853387 18449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50486644ec234933b9a0b45ee39a6685. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:30.854866 18597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e79df8744e74b5991a79ce978ff0202. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:30.857923 18446 tablet_service.cc:1467] Processing CreateTablet for tablet 30a521916bda46678be20a066bd1d3f3 (DEFAULT_TABLE table=default.Table_9579e986b42043f18f8e4aa8d0f6f754 [id=21e6703c9d844a37977f0f161c0b8c08]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:30.858456 18446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30a521916bda46678be20a066bd1d3f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:30.858964 18587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7b0132c17d44a9ba17288e0321a7f11. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:30.862946 18583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d61c279d20b0471a967942525ceb4fcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:30.863111 18445 tablet_service.cc:1467] Processing CreateTablet for tablet 115c4a12440647a4bb46eb6824f7ea98 (DEFAULT_TABLE table=default.Table_cd719a43207f4bf983e1b8e6d236bb16 [id=47e0b04b17834deb833a8f31c123edc5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:30.863631 18445 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 115c4a12440647a4bb46eb6824f7ea98. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:30.867138 18463 tablet_service.cc:1467] Processing CreateTablet for tablet f940d17b02b541289121463e05b0bb5f (DEFAULT_TABLE table=default.Table_07993f6164104b888058e1b35c9f4ed7 [id=f2e6cac20f6a4261bb60f2e40f210491]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:30.867766 18463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f940d17b02b541289121463e05b0bb5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:30.872956 18457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d61c279d20b0471a967942525ceb4fcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:30.874632 18983 tablet_bootstrap.cc:492] T 01436e5b6232494babbb414342f03090 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:30.874850 18983 ts_tablet_manager.cc:1367] T 01436e5b6232494babbb414342f03090 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.074s user 0.004s sys 0.000s
I20240430 20:11:30.875635 18983 raft_consensus.cc:348] T 01436e5b6232494babbb414342f03090 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:30.876085 18983 raft_consensus.cc:374] T 01436e5b6232494babbb414342f03090 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:30.876200 18983 raft_consensus.cc:729] T 01436e5b6232494babbb414342f03090 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:30.876487 18983 consensus_queue.cc:260] T 01436e5b6232494babbb414342f03090 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:30.877343 18983 ts_tablet_manager.cc:1398] T 01436e5b6232494babbb414342f03090 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240430 20:11:30.877666 18983 tablet_bootstrap.cc:492] T 294644d7a16b4ca2bbb6d106bcb8c7ff P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
W20240430 20:11:30.880575 18999 catalog_manager.cc:4810] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): delete failed for tablet a9e0659564894b0e8cb7f600bb07a085 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a9e0659564894b0e8cb7f600bb07a085 already in progress: creating tablet
W20240430 20:11:30.880914 18999 catalog_manager.cc:4797] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): delete failed for tablet 115c4a12440647a4bb46eb6824f7ea98 because the tablet was not found. No further retry: Not found: Tablet not found: 115c4a12440647a4bb46eb6824f7ea98
I20240430 20:11:30.881729 19029 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:51664 (request call id 44) took 1070 ms. Trace:
I20240430 20:11:30.881851 19029 rpcz_store.cc:276] 0430 20:11:29.811316 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:30.844014 (+1032698us) service_pool.cc:225] Handling call
0430 20:11:30.881678 (+ 37664us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240430 20:11:30.884382 18461 tablet_service.cc:1467] Processing CreateTablet for tablet 2e79df8744e74b5991a79ce978ff0202 (DEFAULT_TABLE table=default.Table_276bbdc7cfe542d1a2936485a2aad1ec [id=f146a15e6e8a4acbb4c734c1c863da57]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:30.884938 18461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e79df8744e74b5991a79ce978ff0202. 1 dirs total, 0 dirs full, 0 dirs failed
W20240430 20:11:30.884593 19001 catalog_manager.cc:4797] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): delete failed for tablet 115c4a12440647a4bb46eb6824f7ea98 because the tablet was not found. No further retry: Not found: Tablet not found: 115c4a12440647a4bb46eb6824f7ea98
I20240430 20:11:30.894675 18734 tablet_service.cc:1467] Processing CreateTablet for tablet 115c4a12440647a4bb46eb6824f7ea98 (DEFAULT_TABLE table=default.Table_cd719a43207f4bf983e1b8e6d236bb16 [id=47e0b04b17834deb833a8f31c123edc5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:30.895191 18744 tablet_service.cc:1467] Processing CreateTablet for tablet 2e79df8744e74b5991a79ce978ff0202 (DEFAULT_TABLE table=default.Table_276bbdc7cfe542d1a2936485a2aad1ec [id=f146a15e6e8a4acbb4c734c1c863da57]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:30.896735 19230 consensus_queue.cc:1035] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:30.901990 18732 tablet_service.cc:1467] Processing CreateTablet for tablet f7b0132c17d44a9ba17288e0321a7f11 (DEFAULT_TABLE table=default.Table_01d62e066e4942d9aa1f582189d60d49 [id=97861b0b1ce94638b62bbd123d774d7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:30.902027 18737 tablet_service.cc:1514] Processing DeleteTablet for tablet a9e0659564894b0e8cb7f600bb07a085 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:29 UTC) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:30.902755 19185 raft_consensus.cc:504] T 724e51ff05ec4baebc2bb55e05c8eddc P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:30.904053 19185 leader_election.cc:288] T 724e51ff05ec4baebc2bb55e05c8eddc P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Requested vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:30.907754 18734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 115c4a12440647a4bb46eb6824f7ea98. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:30.910790 18744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e79df8744e74b5991a79ce978ff0202. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:30.911648 18732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7b0132c17d44a9ba17288e0321a7f11. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:30.914898 18462 tablet_service.cc:1514] Processing DeleteTablet for tablet 115c4a12440647a4bb46eb6824f7ea98 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:29 UTC) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:30.915699 18462 tablet_service.cc:1514] Processing DeleteTablet for tablet a9e0659564894b0e8cb7f600bb07a085 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:29 UTC) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:30.916505 18462 tablet_service.cc:1467] Processing CreateTablet for tablet f7b0132c17d44a9ba17288e0321a7f11 (DEFAULT_TABLE table=default.Table_01d62e066e4942d9aa1f582189d60d49 [id=97861b0b1ce94638b62bbd123d774d7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:30.916913 18462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7b0132c17d44a9ba17288e0321a7f11. 1 dirs total, 0 dirs full, 0 dirs failed
W20240430 20:11:30.919981 19004 catalog_manager.cc:4797] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): delete failed for tablet 115c4a12440647a4bb46eb6824f7ea98 because the tablet was not found. No further retry: Not found: Tablet not found: 115c4a12440647a4bb46eb6824f7ea98
W20240430 20:11:30.920388 19004 catalog_manager.cc:4810] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): delete failed for tablet a9e0659564894b0e8cb7f600bb07a085 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a9e0659564894b0e8cb7f600bb07a085 already in progress: creating tablet
I20240430 20:11:30.921561 18761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "724e51ff05ec4baebc2bb55e05c8eddc" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:30.921774 18761 raft_consensus.cc:3037] T 724e51ff05ec4baebc2bb55e05c8eddc P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:30.921342 18739 tablet_service.cc:1467] Processing CreateTablet for tablet f940d17b02b541289121463e05b0bb5f (DEFAULT_TABLE table=default.Table_07993f6164104b888058e1b35c9f4ed7 [id=f2e6cac20f6a4261bb60f2e40f210491]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:30.926354 19492 raft_consensus.cc:482] T 44fa6917a85c46f3aaa363df46ea586b P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240430 20:11:30.926764 19001 catalog_manager.cc:4810] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): delete failed for tablet a9e0659564894b0e8cb7f600bb07a085 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a9e0659564894b0e8cb7f600bb07a085 already in progress: creating tablet
I20240430 20:11:30.927589 19033 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: Sending DeleteTablet for 3 replicas of tablet 2cdccf4be7094ca5b863dd5c3432d55d
I20240430 20:11:30.928508 18739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f940d17b02b541289121463e05b0bb5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:30.928472 19033 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:51664 (request call id 41) took 1153 ms. Trace:
W20240430 20:11:30.928620 19051 catalog_manager.cc:5676] Failed to send AlterTable RPC for tablet acd23f0a63d5448aaa7ed78422e67bad (table default.Table_972f13be4dca4cd19b9a04534dc59f51 [id=69f37db4de634a3185613e40013e0ed7], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet acd23f0a63d5448aaa7ed78422e67bad consensus state has no leader
I20240430 20:11:30.929039 19028 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:51664 (request call id 43) took 1129 ms. Trace:
W20240430 20:11:30.929350 19051 catalog_manager.cc:5676] Failed to send AlterTable RPC for tablet 724e51ff05ec4baebc2bb55e05c8eddc (table default.Table_0c5d4686f2ed40fb8d62c31b0166dc05 [id=4dce8a767f0541e1beb5464b2ed95d06], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 724e51ff05ec4baebc2bb55e05c8eddc consensus state has no leader
I20240430 20:11:30.929620 18600 tablet_service.cc:1514] Processing DeleteTablet for tablet 75ebf051383e42529dcb5f7dd0770a4b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:27 UTC) from {username='slave'} at 127.0.0.1:40218
W20240430 20:11:30.930218 19051 catalog_manager.cc:5676] Failed to send AlterTable RPC for tablet c66c142419ca4e1399ecd1c8d1e62a59 (table default.Table_0443359dbeff437fb6cffb0db33cbb30 [id=93c44abdf6b147228224388398cd7aeb], current schema version=2): Not found: failed to reset TS proxy: No leader found: Tablet c66c142419ca4e1399ecd1c8d1e62a59 consensus state has no leader
I20240430 20:11:30.931159 18602 tablet_service.cc:1514] Processing DeleteTablet for tablet 47c37876694c4a4abb1091c631dab7a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:27 UTC) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:30.931344 19492 raft_consensus.cc:504] T 44fa6917a85c46f3aaa363df46ea586b P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
W20240430 20:11:30.931990 18999 catalog_manager.cc:4810] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): delete failed for tablet 75ebf051383e42529dcb5f7dd0770a4b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 75ebf051383e42529dcb5f7dd0770a4b already in progress: creating tablet
I20240430 20:11:30.932111 19492 leader_election.cc:288] T 44fa6917a85c46f3aaa363df46ea586b P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
W20240430 20:11:30.932368 18999 catalog_manager.cc:4810] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): delete failed for tablet 47c37876694c4a4abb1091c631dab7a5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 47c37876694c4a4abb1091c631dab7a5 already in progress: creating tablet
I20240430 20:11:30.933039 18600 tablet_service.cc:1514] Processing DeleteTablet for tablet d4508e4f6dbd42109d951c648b3dc7df with delete_type TABLET_DATA_DELETED (replaced by tablet 2e79df8744e74b5991a79ce978ff0202) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:30.933131 18601 tablet_service.cc:1514] Processing DeleteTablet for tablet 1398d4d2305b48e08fdb12432bd6c4b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:25 UTC) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:30.933043 18602 tablet_service.cc:1514] Processing DeleteTablet for tablet 68ca48351cd4412381f5195d81109016 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:26 UTC) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:30.933528 19471 tablet_replica.cc:330] T d4508e4f6dbd42109d951c648b3dc7df P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:30.933815 18601 tablet_service.cc:1514] Processing DeleteTablet for tablet f5546f9edeea44d7a6c5cb954f5157ba with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:27 UTC) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:30.933919 18602 tablet_service.cc:1514] Processing DeleteTablet for tablet f77086a5675f44e99e8a867b694df4b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:29 UTC) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:30.934028 19471 raft_consensus.cc:2229] T d4508e4f6dbd42109d951c648b3dc7df P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:30.934239 19471 raft_consensus.cc:2258] T d4508e4f6dbd42109d951c648b3dc7df P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:30.934928 19471 ts_tablet_manager.cc:1875] T d4508e4f6dbd42109d951c648b3dc7df P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:30.935268 18730 tablet_service.cc:1514] Processing DeleteTablet for tablet 2cdccf4be7094ca5b863dd5c3432d55d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:30 UTC) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:30.935678 18603 tablet_service.cc:1514] Processing DeleteTablet for tablet 2cdccf4be7094ca5b863dd5c3432d55d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:30 UTC) from {username='slave'} at 127.0.0.1:40218
W20240430 20:11:30.937770 19001 catalog_manager.cc:4797] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): delete failed for tablet 2cdccf4be7094ca5b863dd5c3432d55d because the tablet was not found. No further retry: Not found: Tablet not found: 2cdccf4be7094ca5b863dd5c3432d55d
I20240430 20:11:30.938524 18621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44fa6917a85c46f3aaa363df46ea586b" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:30.938789 18621 raft_consensus.cc:2454] T 44fa6917a85c46f3aaa363df46ea586b P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:30.942579 18483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44fa6917a85c46f3aaa363df46ea586b" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:30.944464 18483 raft_consensus.cc:2454] T 44fa6917a85c46f3aaa363df46ea586b P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
W20240430 20:11:30.947213 19024 catalog_manager.cc:5676] Failed to send AlterTable RPC for tablet fd40967ab4a04001819e16c06a6c4300 (table default.Table_fda07fd7279c484db2bee67e8c859855 [id=da9ad991953a40c4b77b31317597ad1d], current schema version=3): Not found: failed to reset TS proxy: No leader found: Tablet fd40967ab4a04001819e16c06a6c4300 consensus state has no leader
I20240430 20:11:30.951380 18761 raft_consensus.cc:2454] T 724e51ff05ec4baebc2bb55e05c8eddc P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 1.
W20240430 20:11:30.957063 19051 catalog_manager.cc:5676] Failed to send AlterTable RPC for tablet 6e754af64cb64fd2a082adae7d5f24da (table default.Table_208b12b8e9bd4f3e8bc9c6d422ada649 [id=1df4e95702cd46f59ab21b101049a9ed], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 6e754af64cb64fd2a082adae7d5f24da consensus state has no leader
W20240430 20:11:30.957625 19051 catalog_manager.cc:5676] Failed to send AlterTable RPC for tablet 32c823d1700949ab8b22adcd83b7bfd1 (table default.Table_e8c73c89623d4b49b8d7f0a01d729bdd [id=5cd10682ce52414cb5a103e4433a6a0d], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 32c823d1700949ab8b22adcd83b7bfd1 consensus state has no leader
I20240430 20:11:30.958838 18620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "724e51ff05ec4baebc2bb55e05c8eddc" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:30.959287 18620 raft_consensus.cc:3037] T 724e51ff05ec4baebc2bb55e05c8eddc P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:30.929242 19028 rpcz_store.cc:276] 0430 20:11:29.799395 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:30.831847 (+1032452us) service_pool.cc:225] Handling call
0430 20:11:30.833159 (+ 1312us) catalog_manager.cc:2072] Acquired catalog manager lock
0430 20:11:30.833704 (+ 545us) catalog_manager.cc:2118] Created new table and tablet info
0430 20:11:30.928721 (+ 95017us) catalog_manager.cc:2178] Wrote table and tablets to system table
0430 20:11:30.928861 (+ 140us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0430 20:11:30.928966 (+ 105us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:30.839861 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:30.839981 (+ 120us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:30.839987 (+ 6us) write_op.cc:427] Acquired schema lock
0430 20:11:30.839989 (+ 2us) tablet.cc:660] Decoding operations
0430 20:11:30.840071 (+ 82us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:30.840081 (+ 10us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:30.840086 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0430 20:11:30.840155 (+ 69us) tablet.cc:699] Row locks acquired
0430 20:11:30.840158 (+ 3us) write_op.cc:260] PREPARE: finished
0430 20:11:30.840192 (+ 34us) write_op.cc:270] Start()
0430 20:11:30.840222 (+ 30us) write_op.cc:276] Timestamp: P: 1714507890840144 usec, L: 0
0430 20:11:30.840226 (+ 4us) op_driver.cc:331] REPLICATION: starting
0430 20:11:30.840770 (+ 544us) log.cc:842] Serialized 433 byte log entry
0430 20:11:30.851201 (+ 10431us) op_driver.cc:444] REPLICATION: finished
0430 20:11:30.851875 (+ 674us) write_op.cc:301] APPLY: starting
0430 20:11:30.851894 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:30.851957 (+ 63us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:30.851987 (+ 30us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:30.852112 (+ 125us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:30.852152 (+ 40us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0430 20:11:30.852160 (+ 8us) write_op.cc:312] APPLY: finished
0430 20:11:30.852244 (+ 84us) log.cc:842] Serialized 29 byte log entry
0430 20:11:30.852310 (+ 66us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:30.852331 (+ 21us) write_op.cc:446] Released schema lock
0430 20:11:30.852338 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:30.852397 (+ 59us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:30.852401 (+ 4us) write_op.cc:446] Released schema lock
Metrics: {"spinlock_wait_cycles":101760,"child_traces":[["op",{"apply.queue_time_us":607,"apply.run_cpu_time_us":580,"apply.run_wall_time_us":580,"num_ops":2,"prepare.queue_time_us":5741,"prepare.run_cpu_time_us":756,"prepare.run_wall_time_us":1022,"raft.queue_time_us":777,"raft.run_cpu_time_us":3,"raft.run_wall_time_us":3,"replication_time_us":10949,"spinlock_wait_cycles":842496}]]}
I20240430 20:11:30.970440 18730 tablet_service.cc:1467] Processing CreateTablet for tablet 7683caba80594627900a0185f772c131 (DEFAULT_TABLE table=default.Table_2ec9639eca4a47439b7859929cd705bc [id=6ffe241384bd42b7a22d41154ac59e3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:30.970993 18730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7683caba80594627900a0185f772c131. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:30.972153 18735 tablet_service.cc:1467] Processing CreateTablet for tablet 2da900b6bbfa4fa99fe8494fd2e68da8 (DEFAULT_TABLE table=default.Table_19a91a7f17ff456ebb145e8dd89c0940 [id=67283dd3995a44c6a7a7bc8722e4f955]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:30.972558 18459 tablet_service.cc:1514] Processing DeleteTablet for tablet 47c37876694c4a4abb1091c631dab7a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:27 UTC) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:30.972836 19495 tablet_replica.cc:330] T 47c37876694c4a4abb1091c631dab7a5 P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:30.973060 19495 raft_consensus.cc:2229] T 47c37876694c4a4abb1091c631dab7a5 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:30.973238 19495 raft_consensus.cc:2258] T 47c37876694c4a4abb1091c631dab7a5 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:30.974232 19495 ts_tablet_manager.cc:1875] T 47c37876694c4a4abb1091c631dab7a5 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:30.976440 18983 tablet_bootstrap.cc:654] T 294644d7a16b4ca2bbb6d106bcb8c7ff P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:30.978837 18735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2da900b6bbfa4fa99fe8494fd2e68da8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:30.979821 18459 tablet_service.cc:1514] Processing DeleteTablet for tablet 68ca48351cd4412381f5195d81109016 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:26 UTC) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:30.980479 18459 tablet_service.cc:1514] Processing DeleteTablet for tablet d4508e4f6dbd42109d951c648b3dc7df with delete_type TABLET_DATA_DELETED (replaced by tablet 2e79df8744e74b5991a79ce978ff0202) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:30.980944 18459 tablet_service.cc:1514] Processing DeleteTablet for tablet 1398d4d2305b48e08fdb12432bd6c4b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:25 UTC) from {username='slave'} at 127.0.0.1:41680
W20240430 20:11:30.980916 19024 catalog_manager.cc:5676] Failed to send AlterTable RPC for tablet fcc9698abda4491b98e1e879589913a8 (table default.Table_e431f94220aa44ecba799a54dfb612ae [id=1e785b6c159b4312bd1b064271f55a49], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet fcc9698abda4491b98e1e879589913a8 consensus state has no leader
I20240430 20:11:30.981426 18459 tablet_service.cc:1514] Processing DeleteTablet for tablet aacabead3c874e77a47d2ecf19dbfa82 with delete_type TABLET_DATA_DELETED (replaced by tablet fd40967ab4a04001819e16c06a6c4300) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:30.981817 18459 tablet_service.cc:1514] Processing DeleteTablet for tablet a1aeee1b02214975a376cfd36957afe8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:26 UTC) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:30.984172 19185 raft_consensus.cc:482] T a05e32a33aa7488eb287cc6314c09c44 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:30.984489 19185 raft_consensus.cc:504] T a05e32a33aa7488eb287cc6314c09c44 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:30.985419 19185 leader_election.cc:288] T a05e32a33aa7488eb287cc6314c09c44 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:30.986253 18621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a05e32a33aa7488eb287cc6314c09c44" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:30.928606 19033 rpcz_store.cc:276] 0430 20:11:29.774681 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:30.824908 (+1050227us) service_pool.cc:225] Handling call
0430 20:11:30.825369 (+ 461us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0430 20:11:30.825500 (+ 131us) catalog_manager.cc:2749] Modifying in-memory table state
0430 20:11:30.825562 (+ 62us) catalog_manager.cc:2758] Locking tablets
0430 20:11:30.825699 (+ 137us) catalog_manager.cc:2772] Removing table and tablets from system table
0430 20:11:30.927376 (+101677us) catalog_manager.cc:2790] Removing table from by-name map
0430 20:11:30.927482 (+ 106us) catalog_manager.cc:2806] Committing in-memory state
0430 20:11:30.927529 (+ 47us) catalog_manager.cc:2810] Aborting table tasks
0430 20:11:30.928406 (+ 877us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:30.840967 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:30.841082 (+ 115us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:30.841088 (+ 6us) write_op.cc:427] Acquired schema lock
0430 20:11:30.841090 (+ 2us) tablet.cc:660] Decoding operations
0430 20:11:30.841195 (+ 105us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:30.841204 (+ 9us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:30.841208 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
0430 20:11:30.841317 (+ 109us) tablet.cc:699] Row locks acquired
0430 20:11:30.841320 (+ 3us) write_op.cc:260] PREPARE: finished
0430 20:11:30.841352 (+ 32us) write_op.cc:270] Start()
0430 20:11:30.841377 (+ 25us) write_op.cc:276] Timestamp: P: 1714507890841302 usec, L: 0
0430 20:11:30.841381 (+ 4us) op_driver.cc:331] REPLICATION: starting
0430 20:11:30.841525 (+ 144us) log.cc:842] Serialized 728 byte log entry
0430 20:11:30.851269 (+ 9744us) op_driver.cc:444] REPLICATION: finished
0430 20:11:30.852480 (+ 1211us) write_op.cc:301] APPLY: starting
0430 20:11:30.852490 (+ 10us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:30.852527 (+ 37us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:30.852557 (+ 30us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:30.852615 (+ 58us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:30.852644 (+ 29us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0430 20:11:30.852653 (+ 9us) write_op.cc:312] APPLY: finished
0430 20:11:30.852703 (+ 50us) log.cc:842] Serialized 29 byte log entry
0430 20:11:30.852739 (+ 36us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:30.852757 (+ 18us) write_op.cc:446] Released schema lock
0430 20:11:30.852762 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:30.852809 (+ 47us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:30.852814 (+ 5us) write_op.cc:446] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1182,"apply.run_cpu_time_us":373,"apply.run_wall_time_us":371,"num_ops":2,"prepare.queue_time_us":2411,"prepare.run_cpu_time_us":640,"prepare.run_wall_time_us":640,"raft.queue_time_us":91,"raft.run_cpu_time_us":3,"raft.run_wall_time_us":2,"replication_time_us":9874,"threads_started":1}]]}
I20240430 20:11:30.986637 19033 master_service.cc:946] ReplaceTablet: received request to replace tablet dd9b893d48684cfea7c242b93ecba01f from {username='slave'} at 127.0.0.1:51664
W20240430 20:11:30.986898 18415 leader_election.cc:341] T a05e32a33aa7488eb287cc6314c09c44 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:30.987417 18623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "724e51ff05ec4baebc2bb55e05c8eddc" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:30.989763 18622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd6083f12c7e4d1e8a2006b3ddf9aa3a" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:30.992409 18988 tablet_bootstrap.cc:492] T 12c6e131fa904d23865b8673b58a7e33 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:30.990659 18597 tablet_service.cc:1514] Processing DeleteTablet for tablet aacabead3c874e77a47d2ecf19dbfa82 with delete_type TABLET_DATA_DELETED (replaced by tablet fd40967ab4a04001819e16c06a6c4300) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:30.990908 18593 tablet_service.cc:1514] Processing DeleteTablet for tablet c5fd0730133c472299c93f4b0830e527 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:25 UTC) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:30.991080 18592 tablet_service.cc:1514] Processing DeleteTablet for tablet d9e578c99058424a9b86390641f9c34f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:26 UTC) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:30.991261 18589 tablet_service.cc:1514] Processing DeleteTablet for tablet 93d158de64994f3c9fa45a1250a4f533 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:25 UTC) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:30.991410 18599 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fe1247349344863a6273a769ce8ddf6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:24 UTC) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:30.991566 18602 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a9066d895744c5bb7d9532b2940603c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:29 UTC) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:30.991603 18587 tablet_service.cc:1514] Processing DeleteTablet for tablet 02176eee8e044a7db8cac48439e9f594 with delete_type TABLET_DATA_DELETED (replaced by tablet 09d92a4a6ebe44cbb3553ed660f143d0) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:30.991703 18603 tablet_service.cc:1514] Processing DeleteTablet for tablet 29974dc2c7dd45b99b6cc0712378fc30 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:26 UTC) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:30.991837 18590 tablet_service.cc:1514] Processing DeleteTablet for tablet bbe9a2429a434f3f95d6e7f314fcfe4c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:26 UTC) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:30.991974 18586 tablet_service.cc:1514] Processing DeleteTablet for tablet e6eac92dfa5f4456bc49571878f0f8dd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:29 UTC) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:30.992134 18583 tablet_service.cc:1514] Processing DeleteTablet for tablet 255c13ad8fae4ea4aa8e7b7aabb7602e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:27 UTC) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:30.994239 18415 leader_election.cc:302] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a; no voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710
I20240430 20:11:30.994817 19185 raft_consensus.cc:3037] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:30.995573 18988 ts_tablet_manager.cc:1367] T 12c6e131fa904d23865b8673b58a7e33 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.291s user 0.001s sys 0.004s
I20240430 20:11:30.996594 18988 raft_consensus.cc:348] T 12c6e131fa904d23865b8673b58a7e33 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:30.996846 18988 raft_consensus.cc:374] T 12c6e131fa904d23865b8673b58a7e33 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:30.996927 18988 raft_consensus.cc:729] T 12c6e131fa904d23865b8673b58a7e33 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:30.997221 18988 consensus_queue.cc:260] T 12c6e131fa904d23865b8673b58a7e33 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
W20240430 20:11:31.001298 19051 catalog_manager.cc:5676] Failed to send AlterTable RPC for tablet b72fe1f0713b4076a81a3fc1c74f7fa7 (table default.Table_16b082d140234e1c9a1e70654839b4cd [id=c0d8dc87d1f9426a85f6176f161595a1], current schema version=3): Not found: failed to reset TS proxy: No leader found: Tablet b72fe1f0713b4076a81a3fc1c74f7fa7 consensus state has no leader
I20240430 20:11:31.001616 18601 tablet_service.cc:1514] Processing DeleteTablet for tablet a9e0659564894b0e8cb7f600bb07a085 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:29 UTC) from {username='slave'} at 127.0.0.1:40218
W20240430 20:11:31.002931 19051 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.17.177.2:41749 (request call id 90) took 1477 ms (client timeout 1000 ms). Trace:
W20240430 20:11:31.003271 19051 rpcz_store.cc:269] 0430 20:11:29.525093 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:29.578290 (+ 53197us) service_pool.cc:225] Handling call
0430 20:11:29.596076 (+ 17786us) spinlock_profiling.cc:243] Waited 16 ms on lock 0x616000040654. stack: 00007feb16c74756 00007feb16c74510 00007feb15ca748d 0000000000955804 00000000009557cd 00007feb257b315d 00007feb257b3253 00007feb224e5d41 00007feb2a0d134d 00007feb2a258668 00007feb21043786 00007feb210434b5 00007feb1d97c04e 00007feb1d97af2e 00007feb1d981405 00007feb1d981d2f
0430 20:11:31.002856 (+1406780us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:30.410444 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:30.410584 (+ 140us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:30.410618 (+ 34us) write_op.cc:427] Acquired schema lock
0430 20:11:30.410620 (+ 2us) tablet.cc:660] Decoding operations
0430 20:11:30.410794 (+ 174us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:30.410805 (+ 11us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:30.410810 (+ 5us) tablet.cc:683] Acquiring locks for 3 operations
0430 20:11:30.410933 (+ 123us) tablet.cc:699] Row locks acquired
0430 20:11:30.410936 (+ 3us) write_op.cc:260] PREPARE: finished
0430 20:11:30.410972 (+ 36us) write_op.cc:270] Start()
0430 20:11:30.411003 (+ 31us) write_op.cc:276] Timestamp: P: 1714507890410920 usec, L: 0
0430 20:11:30.411006 (+ 3us) op_driver.cc:331] REPLICATION: starting
0430 20:11:30.411191 (+ 185us) log.cc:842] Serialized 1284 byte log entry
0430 20:11:30.420286 (+ 9095us) op_driver.cc:444] REPLICATION: finished
0430 20:11:30.420502 (+ 216us) write_op.cc:301] APPLY: starting
0430 20:11:30.420521 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:30.420585 (+ 64us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:30.420617 (+ 32us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:30.420731 (+ 114us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:30.420786 (+ 55us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
0430 20:11:30.420799 (+ 13us) write_op.cc:312] APPLY: finished
0430 20:11:30.420923 (+ 124us) log.cc:842] Serialized 35 byte log entry
0430 20:11:30.421003 (+ 80us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:30.421038 (+ 35us) write_op.cc:446] Released schema lock
0430 20:11:30.421050 (+ 12us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:30.421138 (+ 88us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:30.421144 (+ 6us) write_op.cc:446] Released schema lock
Metrics: {"mutex_wait_us":36010,"reported_tablets":76,"spinlock_wait_cycles":36686208,"child_traces":[["op",{"apply.queue_time_us":144,"apply.run_cpu_time_us":713,"apply.run_wall_time_us":712,"num_ops":3,"prepare.queue_time_us":156,"prepare.run_cpu_time_us":837,"prepare.run_wall_time_us":837,"raft.queue_time_us":95,"raft.run_cpu_time_us":4,"raft.run_wall_time_us":5,"replication_time_us":9249}]]}
I20240430 20:11:30.992223 18588 tablet_service.cc:1514] Processing DeleteTablet for tablet a1aeee1b02214975a376cfd36957afe8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:26 UTC) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:31.011529 18588 tablet_service.cc:1514] Processing DeleteTablet for tablet 34ab40d8d91e401db299ddee7175ba28 with delete_type TABLET_DATA_DELETED (replaced by tablet 5f02d8862e1f4ccfa9dbda73dbe18615) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:31.019183 18601 tablet_service.cc:1514] Processing DeleteTablet for tablet cace1398492f4d968dcb6899385d527c with delete_type TABLET_DATA_DELETED (replaced by tablet 01436e5b6232494babbb414342f03090) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:31.019562 18983 tablet_bootstrap.cc:492] T 294644d7a16b4ca2bbb6d106bcb8c7ff P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:31.019776 18983 ts_tablet_manager.cc:1367] T 294644d7a16b4ca2bbb6d106bcb8c7ff P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.142s user 0.002s sys 0.002s
W20240430 20:11:31.008723 19024 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.17.177.1:50353 (request call id 81) took 1643 ms (client timeout 1000 ms). Trace:
W20240430 20:11:31.020264 19024 rpcz_store.cc:269] 0430 20:11:29.365005 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:29.576405 (+211400us) service_pool.cc:225] Handling call
0430 20:11:29.598862 (+ 22457us) spinlock_profiling.cc:243] Waited 20.6 ms on lock 0x616000040654. stack: 00007feb16c74756 00007feb16c74510 00007feb15ca748d 0000000000955804 00000000009557cd 00007feb257b315d 00007feb257b3253 00007feb224e5d41 00007feb2a0d134d 00007feb2a258668 00007feb21043786 00007feb210434b5 00007feb1d97c04e 00007feb1d97af2e 00007feb1d981405 00007feb1d981d2f
0430 20:11:30.371814 (+772952us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:30.371824 (+ 10us) write_op.cc:446] Released schema lock
0430 20:11:31.008663 (+636839us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:30.362246 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:30.362409 (+ 163us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:30.362420 (+ 11us) write_op.cc:427] Acquired schema lock
0430 20:11:30.362436 (+ 16us) tablet.cc:660] Decoding operations
0430 20:11:30.362703 (+ 267us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:30.362718 (+ 15us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:30.362722 (+ 4us) tablet.cc:683] Acquiring locks for 4 operations
0430 20:11:30.362877 (+ 155us) tablet.cc:699] Row locks acquired
0430 20:11:30.362882 (+ 5us) write_op.cc:260] PREPARE: finished
0430 20:11:30.362936 (+ 54us) write_op.cc:270] Start()
0430 20:11:30.362987 (+ 51us) write_op.cc:276] Timestamp: P: 1714507890362883 usec, L: 0
0430 20:11:30.362994 (+ 7us) op_driver.cc:331] REPLICATION: starting
0430 20:11:30.363234 (+ 240us) log.cc:842] Serialized 1662 byte log entry
0430 20:11:30.368043 (+ 4809us) op_driver.cc:444] REPLICATION: finished
0430 20:11:30.368203 (+ 160us) write_op.cc:301] APPLY: starting
0430 20:11:30.368225 (+ 22us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:30.368337 (+ 112us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:30.368373 (+ 36us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:30.368519 (+ 146us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:30.368572 (+ 53us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=4
0430 20:11:30.368584 (+ 12us) write_op.cc:312] APPLY: finished
0430 20:11:30.368696 (+ 112us) log.cc:842] Serialized 41 byte log entry
0430 20:11:30.368779 (+ 83us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:30.368812 (+ 33us) write_op.cc:446] Released schema lock
0430 20:11:30.368822 (+ 10us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
0430 20:11:30.406962 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:30.407145 (+ 183us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:30.407155 (+ 10us) write_op.cc:427] Acquired schema lock
0430 20:11:30.407159 (+ 4us) tablet.cc:660] Decoding operations
0430 20:11:30.407289 (+ 130us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:30.407305 (+ 16us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:30.407311 (+ 6us) tablet.cc:683] Acquiring locks for 1 operations
0430 20:11:30.407387 (+ 76us) tablet.cc:699] Row locks acquired
0430 20:11:30.407389 (+ 2us) write_op.cc:260] PREPARE: finished
0430 20:11:30.407450 (+ 61us) write_op.cc:270] Start()
0430 20:11:30.407491 (+ 41us) write_op.cc:276] Timestamp: P: 1714507890407379 usec, L: 0
0430 20:11:30.407495 (+ 4us) op_driver.cc:331] REPLICATION: starting
0430 20:11:30.407867 (+ 372us) log.cc:842] Serialized 356 byte log entry
0430 20:11:30.411769 (+ 3902us) op_driver.cc:444] REPLICATION: finished
0430 20:11:30.411923 (+ 154us) write_op.cc:301] APPLY: starting
0430 20:11:30.411941 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:30.411984 (+ 43us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:30.412009 (+ 25us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:30.412065 (+ 56us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:30.412101 (+ 36us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0430 20:11:30.412108 (+ 7us) write_op.cc:312] APPLY: finished
0430 20:11:30.412178 (+ 70us) log.cc:842] Serialized 23 byte log entry
0430 20:11:30.412227 (+ 49us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:30.412245 (+ 18us) write_op.cc:446] Released schema lock
0430 20:11:30.412253 (+ 8us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:30.412312 (+ 59us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:30.412315 (+ 3us) write_op.cc:446] Released schema lock
Metrics: {"mutex_wait_us":6292,"reported_tablets":76,"spinlock_wait_cycles":47265024,"child_traces":[["op",{"apply.queue_time_us":102,"apply.run_cpu_time_us":688,"apply.run_wall_time_us":687,"num_ops":4,"prepare.queue_time_us":148,"prepare.run_cpu_time_us":1111,"prepare.run_wall_time_us":1124,"raft.queue_time_us":1384,"raft.run_cpu_time_us":298,"raft.run_wall_time_us":298,"replication_time_us":5016}],["op",{"apply.queue_time_us":108,"apply.run_cpu_time_us":440,"apply.run_wall_time_us":441,"num_ops":1,"prepare.queue_time_us":6368,"prepare.run_cpu_time_us":976,"prepare.run_wall_time_us":1725,"raft.queue_time_us":506,"raft.run_cpu_time_us":299,"raft.run_wall_time_us":299,"replication_time_us":4251}]]}
I20240430 20:11:31.022109 18983 raft_consensus.cc:348] T 294644d7a16b4ca2bbb6d106bcb8c7ff P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:31.022418 18983 raft_consensus.cc:374] T 294644d7a16b4ca2bbb6d106bcb8c7ff P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:31.022521 18983 raft_consensus.cc:729] T 294644d7a16b4ca2bbb6d106bcb8c7ff P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:31.022888 18983 consensus_queue.cc:260] T 294644d7a16b4ca2bbb6d106bcb8c7ff P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:31.022745 19051 master_service.cc:946] ReplaceTablet: received request to replace tablet 5bde72e2791a497db7dd08686ca36d80 from {username='slave'} at 127.0.0.1:51664
I20240430 20:11:31.023803 18983 ts_tablet_manager.cc:1398] T 294644d7a16b4ca2bbb6d106bcb8c7ff P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240430 20:11:31.024291 18983 tablet_bootstrap.cc:492] T 1555924b5f1444398f12ee82c3cb5b13 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:31.025429 18601 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e69ca12bd10494fb013614741bc166c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:27 UTC) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:31.026175 18983 tablet_bootstrap.cc:654] T 1555924b5f1444398f12ee82c3cb5b13 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:31.026383 18601 tablet_service.cc:1467] Processing CreateTablet for tablet 2da900b6bbfa4fa99fe8494fd2e68da8 (DEFAULT_TABLE table=default.Table_19a91a7f17ff456ebb145e8dd89c0940 [id=67283dd3995a44c6a7a7bc8722e4f955]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:31.026990 18601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2da900b6bbfa4fa99fe8494fd2e68da8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:31.037969 18588 tablet_service.cc:1514] Processing DeleteTablet for tablet f5236574672b477a9026e91adcaf70ec with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:26 UTC) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:31.038120 18583 tablet_service.cc:1514] Processing DeleteTablet for tablet eed6338ad3d64859af087ee15e1ec798 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:26 UTC) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:31.038720 18583 tablet_service.cc:1514] Processing DeleteTablet for tablet 13c1107ea4d04eb78a74d3f658c72fb3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:26 UTC) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:31.039842 18983 tablet_bootstrap.cc:492] T 1555924b5f1444398f12ee82c3cb5b13 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:31.040010 18983 ts_tablet_manager.cc:1367] T 1555924b5f1444398f12ee82c3cb5b13 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.004s
I20240430 20:11:30.962527 18620 raft_consensus.cc:2454] T 724e51ff05ec4baebc2bb55e05c8eddc P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 1.
I20240430 20:11:31.041611 18983 raft_consensus.cc:348] T 1555924b5f1444398f12ee82c3cb5b13 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:31.041884 18983 raft_consensus.cc:374] T 1555924b5f1444398f12ee82c3cb5b13 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:31.041980 18983 raft_consensus.cc:729] T 1555924b5f1444398f12ee82c3cb5b13 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:31.042263 18983 consensus_queue.cc:260] T 1555924b5f1444398f12ee82c3cb5b13 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:31.043433 18988 ts_tablet_manager.cc:1398] T 12c6e131fa904d23865b8673b58a7e33 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.048s user 0.001s sys 0.002s
I20240430 20:11:31.043990 18988 tablet_bootstrap.cc:492] T b76a4105a2974fe280e759349037d027 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:31.045290 18696 leader_election.cc:302] T 44fa6917a85c46f3aaa363df46ea586b P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
W20240430 20:11:31.045737 18696 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.177.2:46027, user_credentials={real_user=slave}} blocked reactor thread for 103620us
W20240430 20:11:31.046136 18699 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.177.1:41217, user_credentials={real_user=slave}} blocked reactor thread for 100352us
I20240430 20:11:31.039728 18591 tablet_service.cc:1467] Processing CreateTablet for tablet 7683caba80594627900a0185f772c131 (DEFAULT_TABLE table=default.Table_2ec9639eca4a47439b7859929cd705bc [id=6ffe241384bd42b7a22d41154ac59e3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:31.047108 19581 raft_consensus.cc:2781] T 44fa6917a85c46f3aaa363df46ea586b P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:31.047238 19581 raft_consensus.cc:482] T 44fa6917a85c46f3aaa363df46ea586b P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:31.047348 19581 raft_consensus.cc:3037] T 44fa6917a85c46f3aaa363df46ea586b P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:31.047685 18583 tablet_service.cc:1514] Processing DeleteTablet for tablet b688e03ea71940c69bda5520b62c1fc3 with delete_type TABLET_DATA_DELETED (replaced by tablet 7683caba80594627900a0185f772c131) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:31.048151 18583 tablet_service.cc:1514] Processing DeleteTablet for tablet e5111d7607dd42188a0176cf5be2ec05 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:29 UTC) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:31.048462 18583 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e503a898406412ab977d3e048ea4b5d with delete_type TABLET_DATA_DELETED (replaced by tablet 5bd0362ff41b4faca0205d136d596737) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:31.048702 18415 leader_election.cc:302] T 724e51ff05ec4baebc2bb55e05c8eddc P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:31.048799 18583 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f56ac2afd444d63a6bfd96988464bbb with delete_type TABLET_DATA_DELETED (replaced by tablet 2da900b6bbfa4fa99fe8494fd2e68da8) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:31.049664 18983 ts_tablet_manager.cc:1398] T 1555924b5f1444398f12ee82c3cb5b13 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.010s user 0.000s sys 0.003s
I20240430 20:11:31.050323 19581 raft_consensus.cc:504] T 44fa6917a85c46f3aaa363df46ea586b P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:31.051025 18447 tablet_service.cc:1514] Processing DeleteTablet for tablet 2cdccf4be7094ca5b863dd5c3432d55d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:30 UTC) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:31.056181 19581 leader_election.cc:288] T 44fa6917a85c46f3aaa363df46ea586b P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Requested vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:31.062686 19482 raft_consensus.cc:2781] T 724e51ff05ec4baebc2bb55e05c8eddc P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:31.064949 18591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7683caba80594627900a0185f772c131. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:31.053726 19495 ts_tablet_manager.cc:1888] T 47c37876694c4a4abb1091c631dab7a5 P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:31.072801 18622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44fa6917a85c46f3aaa363df46ea586b" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:31.073122 18622 raft_consensus.cc:3037] T 44fa6917a85c46f3aaa363df46ea586b P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:31.076409 18622 raft_consensus.cc:2454] T 44fa6917a85c46f3aaa363df46ea586b P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:31.077291 18983 tablet_bootstrap.cc:492] T dd9b893d48684cfea7c242b93ecba01f P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:31.079106 18983 tablet_bootstrap.cc:654] T dd9b893d48684cfea7c242b93ecba01f P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:31.095139 18454 tablet_service.cc:1514] Processing DeleteTablet for tablet 75ebf051383e42529dcb5f7dd0770a4b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:27 UTC) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:31.096535 19186 raft_consensus.cc:482] T b76a4105a2974fe280e759349037d027 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:31.096733 19186 raft_consensus.cc:504] T b76a4105a2974fe280e759349037d027 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:31.097604 19471 ts_tablet_manager.cc:1888] T d4508e4f6dbd42109d951c648b3dc7df P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:31.097790 19471 log.cc:1196] T d4508e4f6dbd42109d951c648b3dc7df P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/d4508e4f6dbd42109d951c648b3dc7df
I20240430 20:11:31.098259 19471 ts_tablet_manager.cc:1909] T d4508e4f6dbd42109d951c648b3dc7df P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:31.098862 19471 tablet_replica.cc:330] T 1398d4d2305b48e08fdb12432bd6c4b0 P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:31.101624 18999 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet d4508e4f6dbd42109d951c648b3dc7df (table default.Table_276bbdc7cfe542d1a2936485a2aad1ec [id=f146a15e6e8a4acbb4c734c1c863da57]) successfully deleted
I20240430 20:11:31.101640 18622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b76a4105a2974fe280e759349037d027" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:31.101845 18462 tablet_service.cc:1514] Processing DeleteTablet for tablet 02176eee8e044a7db8cac48439e9f594 with delete_type TABLET_DATA_DELETED (replaced by tablet 09d92a4a6ebe44cbb3553ed660f143d0) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:31.103195 18459 tablet_service.cc:1514] Processing DeleteTablet for tablet c9ff972ade694938888514979e1638aa with delete_type TABLET_DATA_DELETED (replaced by tablet a9e0659564894b0e8cb7f600bb07a085) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:31.104763 18696 leader_election.cc:302] T 44fa6917a85c46f3aaa363df46ea586b P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:31.105206 19581 raft_consensus.cc:2781] T 44fa6917a85c46f3aaa363df46ea586b P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:31.105554 19581 raft_consensus.cc:686] T 44fa6917a85c46f3aaa363df46ea586b P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Becoming Leader. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Running, Role: LEADER
I20240430 20:11:31.105933 19581 consensus_queue.cc:237] T 44fa6917a85c46f3aaa363df46ea586b P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:31.099025 18481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44fa6917a85c46f3aaa363df46ea586b" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:31.108760 18983 tablet_bootstrap.cc:492] T dd9b893d48684cfea7c242b93ecba01f P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:31.109829 18448 tablet_service.cc:1514] Processing DeleteTablet for tablet 15674aeb56b74210a4a2809608fd4457 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:25 UTC) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:31.110569 18983 ts_tablet_manager.cc:1367] T dd9b893d48684cfea7c242b93ecba01f P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.033s user 0.003s sys 0.008s
I20240430 20:11:31.110709 18458 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e69ca12bd10494fb013614741bc166c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:27 UTC) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:31.111227 18462 tablet_service.cc:1514] Processing DeleteTablet for tablet 34ab40d8d91e401db299ddee7175ba28 with delete_type TABLET_DATA_DELETED (replaced by tablet 5f02d8862e1f4ccfa9dbda73dbe18615) from {username='slave'} at 127.0.0.1:41680
W20240430 20:11:31.111868 18415 leader_election.cc:341] T b76a4105a2974fe280e759349037d027 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:31.111563 18454 tablet_service.cc:1467] Processing CreateTablet for tablet 2da900b6bbfa4fa99fe8494fd2e68da8 (DEFAULT_TABLE table=default.Table_19a91a7f17ff456ebb145e8dd89c0940 [id=67283dd3995a44c6a7a7bc8722e4f955]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:31.113157 19495 log.cc:1196] T 47c37876694c4a4abb1091c631dab7a5 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/47c37876694c4a4abb1091c631dab7a5
I20240430 20:11:31.113320 18445 tablet_service.cc:1514] Processing DeleteTablet for tablet c5fd0730133c472299c93f4b0830e527 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:25 UTC) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:31.113739 18447 tablet_service.cc:1514] Processing DeleteTablet for tablet d9e578c99058424a9b86390641f9c34f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:26 UTC) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:31.114349 18458 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a9066d895744c5bb7d9532b2940603c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:29 UTC) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:31.115027 19185 raft_consensus.cc:2726] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:31.115255 18481 raft_consensus.cc:3037] T 44fa6917a85c46f3aaa363df46ea586b P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:31.116294 18983 raft_consensus.cc:348] T dd9b893d48684cfea7c242b93ecba01f P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:31.116564 18983 raft_consensus.cc:374] T dd9b893d48684cfea7c242b93ecba01f P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:31.116657 18983 raft_consensus.cc:729] T dd9b893d48684cfea7c242b93ecba01f P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:31.118745 18481 raft_consensus.cc:2454] T 44fa6917a85c46f3aaa363df46ea586b P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:31.119974 18450 tablet_service.cc:1514] Processing DeleteTablet for tablet 255c13ad8fae4ea4aa8e7b7aabb7602e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:27 UTC) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:31.120471 18983 consensus_queue.cc:260] T dd9b893d48684cfea7c242b93ecba01f P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:31.121093 18452 tablet_service.cc:1514] Processing DeleteTablet for tablet cace1398492f4d968dcb6899385d527c with delete_type TABLET_DATA_DELETED (replaced by tablet 01436e5b6232494babbb414342f03090) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:31.121326 18448 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fe1247349344863a6273a769ce8ddf6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:24 UTC) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:31.121600 19186 leader_election.cc:288] T b76a4105a2974fe280e759349037d027 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:31.121821 18455 tablet_service.cc:1514] Processing DeleteTablet for tablet 93d158de64994f3c9fa45a1250a4f533 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:25 UTC) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:31.122049 18454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2da900b6bbfa4fa99fe8494fd2e68da8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:31.122581 18983 ts_tablet_manager.cc:1398] T dd9b893d48684cfea7c242b93ecba01f P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.011s user 0.000s sys 0.002s
I20240430 20:11:31.124459 18983 tablet_bootstrap.cc:492] T 20d166756819406ba85044f4fee9c4ce P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:31.124447 19186 raft_consensus.cc:482] T 20d166756819406ba85044f4fee9c4ce P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:31.124608 19186 raft_consensus.cc:504] T 20d166756819406ba85044f4fee9c4ce P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:31.125391 18459 tablet_service.cc:1467] Processing CreateTablet for tablet 7683caba80594627900a0185f772c131 (DEFAULT_TABLE table=default.Table_2ec9639eca4a47439b7859929cd705bc [id=6ffe241384bd42b7a22d41154ac59e3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:31.126194 18983 tablet_bootstrap.cc:654] T 20d166756819406ba85044f4fee9c4ce P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:31.126391 18459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7683caba80594627900a0185f772c131. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:31.127436 18622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20d166756819406ba85044f4fee9c4ce" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:31.135387 18450 tablet_service.cc:1514] Processing DeleteTablet for tablet a9e0659564894b0e8cb7f600bb07a085 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:29 UTC) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:31.140807 18988 tablet_bootstrap.cc:654] T b76a4105a2974fe280e759349037d027 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:31.142640 19029 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_4a09e48b08fa4c4387262a5b92b2a42d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:31.143352 19029 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4a09e48b08fa4c4387262a5b92b2a42d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:31.150174 19024 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_43c679fe95d64db79020bcef1cbe61cb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240430 20:11:31.099112 19471 raft_consensus.cc:2229] T 1398d4d2305b48e08fdb12432bd6c4b0 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:31.154935 19471 raft_consensus.cc:2258] T 1398d4d2305b48e08fdb12432bd6c4b0 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240430 20:11:31.155342 19024 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_43c679fe95d64db79020bcef1cbe61cb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:31.162063 19186 leader_election.cc:288] T 20d166756819406ba85044f4fee9c4ce P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:31.163623 19028 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_d3bf86d0283b4227a93a2a8aa241075b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240430 20:11:31.164489 18455 tablet_service.cc:1514] Processing DeleteTablet for tablet f77086a5675f44e99e8a867b694df4b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:29 UTC) from {username='slave'} at 127.0.0.1:41680
W20240430 20:11:31.165445 19028 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d3bf86d0283b4227a93a2a8aa241075b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:31.167531 19624 tablet.cc:1722] T 680553d4d0bc4fcb839235c1832757e1 P 5e1c6397f09747f091534faf93542710: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240430 20:11:31.173534 19471 ts_tablet_manager.cc:1875] T 1398d4d2305b48e08fdb12432bd6c4b0 P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240430 20:11:31.175146 18415 leader_election.cc:341] T 20d166756819406ba85044f4fee9c4ce P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Illegal state: must be running to vote when last-logged opid is not known
W20240430 20:11:31.175381 18415 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.177.2:46027, user_credentials={real_user=slave}} blocked reactor thread for 47328us
I20240430 20:11:31.177733 18985 tablet_bootstrap.cc:492] T 12c6e131fa904d23865b8673b58a7e33 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:31.185031 18461 tablet_service.cc:1514] Processing DeleteTablet for tablet e6eac92dfa5f4456bc49571878f0f8dd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:29 UTC) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:31.187202 18985 ts_tablet_manager.cc:1367] T 12c6e131fa904d23865b8673b58a7e33 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.338s user 0.020s sys 0.029s
I20240430 20:11:31.187469 19023 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: Sending DeleteTablet for 3 replicas of tablet 2a1e50f99c5d4b44bdbb51b58a92bc75
I20240430 20:11:31.188781 19023 catalog_manager.cc:6554] ReplaceTablet: tablet 2a1e50f99c5d4b44bdbb51b58a92bc75 deleted and replaced by tablet fbdeb4714b8c4b93ab72822c43329aae
I20240430 20:11:31.189080 19023 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:51664 (request call id 45) took 1377 ms. Trace:
I20240430 20:11:31.191843 19471 ts_tablet_manager.cc:1888] T 1398d4d2305b48e08fdb12432bd6c4b0 P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:31.189337 19023 rpcz_store.cc:276] 0430 20:11:29.811425 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:30.849768 (+1038343us) service_pool.cc:225] Handling call
0430 20:11:31.188993 (+339225us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:30.879595 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:30.879721 (+ 126us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:30.879728 (+ 7us) write_op.cc:427] Acquired schema lock
0430 20:11:30.879731 (+ 3us) tablet.cc:660] Decoding operations
0430 20:11:30.879849 (+ 118us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:30.879864 (+ 15us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:30.879871 (+ 7us) tablet.cc:683] Acquiring locks for 2 operations
0430 20:11:30.879974 (+ 103us) tablet.cc:699] Row locks acquired
0430 20:11:30.879976 (+ 2us) write_op.cc:260] PREPARE: finished
0430 20:11:30.880024 (+ 48us) write_op.cc:270] Start()
0430 20:11:30.880062 (+ 38us) write_op.cc:276] Timestamp: P: 1714507890879967 usec, L: 0
0430 20:11:30.880066 (+ 4us) op_driver.cc:331] REPLICATION: starting
0430 20:11:31.146004 (+265938us) log.cc:842] Serialized 625 byte log entry
0430 20:11:31.146202 (+ 198us) spinlock_profiling.cc:243] Waited 266 ms on lock 0x616000040654. stack: 00007feb16c74756 00007feb16c74510 00007feb15ca748d 0000000000955804 00000000009557cd 000000000095519a 00007feb224b9162 00007feb23821b02 00007feb23820629 00007feb2382853b 00007feb238283b8 0000000000a60bcc 00007feb16cbab74 00007feb16cbe19b 00007feb16cbe018 0000000000a60bcc
0430 20:11:31.184624 (+ 38422us) op_driver.cc:444] REPLICATION: finished
0430 20:11:31.186069 (+ 1445us) write_op.cc:301] APPLY: starting
0430 20:11:31.186094 (+ 25us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:31.186155 (+ 61us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:31.186159 (+ 4us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:31.186304 (+ 145us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:31.186378 (+ 74us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0430 20:11:31.186389 (+ 11us) write_op.cc:312] APPLY: finished
0430 20:11:31.186513 (+ 124us) log.cc:842] Serialized 29 byte log entry
0430 20:11:31.186622 (+ 109us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:31.186647 (+ 25us) write_op.cc:446] Released schema lock
0430 20:11:31.186656 (+ 9us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:31.186720 (+ 64us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:31.186723 (+ 3us) write_op.cc:446] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1352,"apply.run_cpu_time_us":726,"apply.run_wall_time_us":727,"num_ops":2,"prepare.queue_time_us":55,"prepare.run_cpu_time_us":3012,"prepare.run_wall_time_us":266643,"replication_time_us":304523,"spinlock_wait_cycles":611140480}]]}
I20240430 20:11:31.195679 19029 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:51664 (request call id 46) took 1318 ms. Trace:
I20240430 20:11:31.195822 19029 rpcz_store.cc:276] 0430 20:11:29.877448 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:30.882099 (+1004651us) service_pool.cc:225] Handling call
0430 20:11:31.142492 (+260393us) spinlock_profiling.cc:243] Waited 260 ms on lock 0x616000040654. stack: 00007feb16c74756 00007feb16c74510 00007feb15ca748d 0000000000955804 00000000009557cd 00007feb257b315d 00007feb257b3253 00007feb224e5d41 00007feb2a0d134d 00007feb2a25dcb3 00007feb210462a6 00007feb21045fd5 00007feb1d97c04e 00007feb1d97af2e 00007feb1d981405 00007feb1d981d2f
0430 20:11:31.143458 (+ 966us) catalog_manager.cc:2072] Acquired catalog manager lock
0430 20:11:31.144048 (+ 590us) catalog_manager.cc:2118] Created new table and tablet info
0430 20:11:31.195398 (+ 51350us) catalog_manager.cc:2178] Wrote table and tablets to system table
0430 20:11:31.195532 (+ 134us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0430 20:11:31.195641 (+ 109us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:31.146992 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:31.147075 (+ 83us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:31.147080 (+ 5us) write_op.cc:427] Acquired schema lock
0430 20:11:31.147083 (+ 3us) tablet.cc:660] Decoding operations
0430 20:11:31.147156 (+ 73us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:31.147165 (+ 9us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:31.147169 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
0430 20:11:31.147236 (+ 67us) tablet.cc:699] Row locks acquired
0430 20:11:31.147238 (+ 2us) write_op.cc:260] PREPARE: finished
0430 20:11:31.147263 (+ 25us) write_op.cc:270] Start()
0430 20:11:31.147288 (+ 25us) write_op.cc:276] Timestamp: P: 1714507891147239 usec, L: 0
0430 20:11:31.147291 (+ 3us) op_driver.cc:331] REPLICATION: starting
0430 20:11:31.147448 (+ 157us) log.cc:842] Serialized 433 byte log entry
0430 20:11:31.194298 (+ 46850us) op_driver.cc:444] REPLICATION: finished
0430 20:11:31.194680 (+ 382us) write_op.cc:301] APPLY: starting
0430 20:11:31.194699 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:31.194747 (+ 48us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:31.194811 (+ 64us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:31.194929 (+ 118us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:31.194964 (+ 35us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0430 20:11:31.194971 (+ 7us) write_op.cc:312] APPLY: finished
0430 20:11:31.195051 (+ 80us) log.cc:842] Serialized 29 byte log entry
0430 20:11:31.195289 (+ 238us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:31.195311 (+ 22us) write_op.cc:446] Released schema lock
0430 20:11:31.195318 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":598053248,"child_traces":[["op",{"apply.queue_time_us":305,"num_ops":2,"prepare.queue_time_us":2465,"prepare.run_cpu_time_us":511,"prepare.run_wall_time_us":510,"replication_time_us":46973}]]}
I20240430 20:11:31.198114 19023 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_c9603ec75ad0491592883980e5efc388"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:31.198979 19023 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c9603ec75ad0491592883980e5efc388 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:31.200619 19029 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_a2f72cd314144523b378f2c5b324eed9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240430 20:11:31.201220 19471 log.cc:1196] T 1398d4d2305b48e08fdb12432bd6c4b0 P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/1398d4d2305b48e08fdb12432bd6c4b0
W20240430 20:11:31.201381 19029 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a2f72cd314144523b378f2c5b324eed9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:31.201926 19471 ts_tablet_manager.cc:1909] T 1398d4d2305b48e08fdb12432bd6c4b0 P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:31.202710 18450 tablet_service.cc:1514] Processing DeleteTablet for tablet f5546f9edeea44d7a6c5cb954f5157ba with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:27 UTC) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:31.203114 19471 tablet_replica.cc:330] T 68ca48351cd4412381f5195d81109016 P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:31.203614 19471 raft_consensus.cc:2229] T 68ca48351cd4412381f5195d81109016 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:31.203696 19495 ts_tablet_manager.cc:1909] T 47c37876694c4a4abb1091c631dab7a5 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:31.203867 19471 raft_consensus.cc:2258] T 68ca48351cd4412381f5195d81109016 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:31.204528 19471 ts_tablet_manager.cc:1875] T 68ca48351cd4412381f5195d81109016 P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:31.206188 19051 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: Sending DeleteTablet for 3 replicas of tablet 5bde72e2791a497db7dd08686ca36d80
I20240430 20:11:31.206866 19051 catalog_manager.cc:6554] ReplaceTablet: tablet 5bde72e2791a497db7dd08686ca36d80 deleted and replaced by tablet 66dd767bd3f8494f94061fb89e5e5680
I20240430 20:11:31.207015 19051 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:51664 (request call id 49) took 1303 ms. Trace:
I20240430 20:11:31.207129 19051 rpcz_store.cc:276] 0430 20:11:29.903805 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:31.022552 (+1118747us) service_pool.cc:225] Handling call
0430 20:11:31.144832 (+122280us) spinlock_profiling.cc:243] Waited 120 ms on lock 0x616000040654. stack: 00007feb16c74756 00007feb16c74510 00007feb15ca748d 0000000000955804 00000000009557cd 00007feb257b315d 00007feb257b3253 00007feb224e5d41 00007feb2a0d134d 00007feb2a26a05b 00007feb21052be7 00007feb21052915 00007feb1d97c04e 00007feb1d97af2e 00007feb1d981405 00007feb1d981d2f
0430 20:11:31.206979 (+ 62147us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:31.147532 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:31.147617 (+ 85us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:31.147621 (+ 4us) write_op.cc:427] Acquired schema lock
0430 20:11:31.147623 (+ 2us) tablet.cc:660] Decoding operations
0430 20:11:31.147743 (+ 120us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:31.147751 (+ 8us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:31.147756 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0430 20:11:31.147836 (+ 80us) tablet.cc:699] Row locks acquired
0430 20:11:31.147839 (+ 3us) write_op.cc:260] PREPARE: finished
0430 20:11:31.147867 (+ 28us) write_op.cc:270] Start()
0430 20:11:31.147896 (+ 29us) write_op.cc:276] Timestamp: P: 1714507891147841 usec, L: 0
0430 20:11:31.147902 (+ 6us) op_driver.cc:331] REPLICATION: starting
0430 20:11:31.148090 (+ 188us) log.cc:842] Serialized 625 byte log entry
0430 20:11:31.197722 (+ 49632us) op_driver.cc:444] REPLICATION: finished
0430 20:11:31.204555 (+ 6833us) write_op.cc:301] APPLY: starting
0430 20:11:31.204573 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:31.204636 (+ 63us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:31.204665 (+ 29us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:31.205389 (+ 724us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:31.205484 (+ 95us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0430 20:11:31.205494 (+ 10us) write_op.cc:312] APPLY: finished
0430 20:11:31.205589 (+ 95us) log.cc:842] Serialized 29 byte log entry
0430 20:11:31.205850 (+ 261us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:31.205875 (+ 25us) write_op.cc:446] Released schema lock
0430 20:11:31.205882 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:31.205961 (+ 79us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:31.205964 (+ 3us) write_op.cc:446] Released schema lock
Metrics: {"spinlock_wait_cycles":277045504,"child_traces":[["op",{"apply.queue_time_us":6772,"apply.run_cpu_time_us":768,"apply.run_wall_time_us":1478,"num_ops":2,"prepare.queue_time_us":1807,"prepare.run_cpu_time_us":607,"prepare.run_wall_time_us":605,"replication_time_us":49795}]]}
I20240430 20:11:31.208180 19051 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_26cbad4b7efa400f946f15b31c9ff031"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240430 20:11:31.208539 18457 tablet_service.cc:1514] Processing DeleteTablet for tablet b688e03ea71940c69bda5520b62c1fc3 with delete_type TABLET_DATA_DELETED (replaced by tablet 7683caba80594627900a0185f772c131) from {username='slave'} at 127.0.0.1:41680
W20240430 20:11:31.209508 19051 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_26cbad4b7efa400f946f15b31c9ff031 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:31.214461 18450 tablet_service.cc:1514] Processing DeleteTablet for tablet 29974dc2c7dd45b99b6cc0712378fc30 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:26 UTC) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:31.217754 19024 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:51664 (request call id 50) took 1301 ms. Trace:
I20240430 20:11:31.217762 19180 consensus_queue.cc:1035] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:31.217931 19024 rpcz_store.cc:276] 0430 20:11:29.915808 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:31.028131 (+1112323us) service_pool.cc:225] Handling call
0430 20:11:31.150051 (+121920us) spinlock_profiling.cc:243] Waited 122 ms on lock 0x616000040654. stack: 00007feb16c74756 00007feb16c74510 00007feb15ca748d 0000000000955804 00000000009557cd 00007feb257b315d 00007feb257b3253 00007feb224e5d41 00007feb2a0d134d 00007feb2a25dcb3 00007feb210462a6 00007feb21045fd5 00007feb1d97c04e 00007feb1d97af2e 00007feb1d981405 00007feb1d981d2f
0430 20:11:31.155466 (+ 5415us) catalog_manager.cc:2072] Acquired catalog manager lock
0430 20:11:31.155984 (+ 518us) catalog_manager.cc:2118] Created new table and tablet info
0430 20:11:31.217449 (+ 61465us) catalog_manager.cc:2178] Wrote table and tablets to system table
0430 20:11:31.217577 (+ 128us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0430 20:11:31.217705 (+ 128us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:31.159391 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:31.159513 (+ 122us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:31.159520 (+ 7us) write_op.cc:427] Acquired schema lock
0430 20:11:31.159523 (+ 3us) tablet.cc:660] Decoding operations
0430 20:11:31.159626 (+ 103us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:31.159637 (+ 11us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:31.159642 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0430 20:11:31.159725 (+ 83us) tablet.cc:699] Row locks acquired
0430 20:11:31.159729 (+ 4us) write_op.cc:260] PREPARE: finished
0430 20:11:31.159771 (+ 42us) write_op.cc:270] Start()
0430 20:11:31.159809 (+ 38us) write_op.cc:276] Timestamp: P: 1714507891159741 usec, L: 0
0430 20:11:31.159814 (+ 5us) op_driver.cc:331] REPLICATION: starting
0430 20:11:31.160029 (+ 215us) log.cc:842] Serialized 433 byte log entry
0430 20:11:31.216513 (+ 56484us) op_driver.cc:444] REPLICATION: finished
0430 20:11:31.216645 (+ 132us) write_op.cc:301] APPLY: starting
0430 20:11:31.216666 (+ 21us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:31.216739 (+ 73us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:31.216775 (+ 36us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:31.216926 (+ 151us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:31.216962 (+ 36us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0430 20:11:31.216969 (+ 7us) write_op.cc:312] APPLY: finished
0430 20:11:31.217058 (+ 89us) log.cc:842] Serialized 29 byte log entry
0430 20:11:31.217325 (+ 267us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:31.217349 (+ 24us) write_op.cc:446] Released schema lock
0430 20:11:31.217356 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":279719168,"child_traces":[["op",{"apply.queue_time_us":71,"num_ops":2,"prepare.queue_time_us":2891,"prepare.run_cpu_time_us":726,"prepare.run_wall_time_us":726,"replication_time_us":56670}]]}
I20240430 20:11:31.218951 19024 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_0376ffa6507243ada0b3b67b10264ee0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:31.219564 19024 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0376ffa6507243ada0b3b67b10264ee0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:31.218199 18457 tablet_service.cc:1514] Processing DeleteTablet for tablet eed6338ad3d64859af087ee15e1ec798 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:26 UTC) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:31.231740 18601 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a1e50f99c5d4b44bdbb51b58a92bc75 with delete_type TABLET_DATA_DELETED (replaced by tablet fbdeb4714b8c4b93ab72822c43329aae) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:31.231822 18461 tablet_service.cc:1514] Processing DeleteTablet for tablet f5236574672b477a9026e91adcaf70ec with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:26 UTC) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:31.232074 19482 raft_consensus.cc:686] T 724e51ff05ec4baebc2bb55e05c8eddc P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Becoming Leader. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Running, Role: LEADER
I20240430 20:11:31.232558 18739 tablet_service.cc:1514] Processing DeleteTablet for tablet 68ca48351cd4412381f5195d81109016 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:26 UTC) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:31.232887 19489 tablet_replica.cc:330] T 68ca48351cd4412381f5195d81109016 P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:31.233126 19489 raft_consensus.cc:2229] T 68ca48351cd4412381f5195d81109016 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:31.233351 19489 raft_consensus.cc:2258] T 68ca48351cd4412381f5195d81109016 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:31.234511 18601 tablet_service.cc:1514] Processing DeleteTablet for tablet 5bde72e2791a497db7dd08686ca36d80 with delete_type TABLET_DATA_DELETED (replaced by tablet 66dd767bd3f8494f94061fb89e5e5680) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:31.234799 19230 raft_consensus.cc:482] T 1555924b5f1444398f12ee82c3cb5b13 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:31.235004 19230 raft_consensus.cc:504] T 1555924b5f1444398f12ee82c3cb5b13 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:31.235538 18735 tablet_service.cc:1514] Processing DeleteTablet for tablet 47c37876694c4a4abb1091c631dab7a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:27 UTC) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:31.235816 19230 leader_election.cc:288] T 1555924b5f1444398f12ee82c3cb5b13 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:31.237275 19033 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: Sending DeleteTablet for 3 replicas of tablet dd9b893d48684cfea7c242b93ecba01f
I20240430 20:11:31.237435 19629 tablet.cc:1722] T 77285afcafe04a829e485998bbbe00f0 P 5e1c6397f09747f091534faf93542710: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240430 20:11:31.237838 19033 catalog_manager.cc:6554] ReplaceTablet: tablet dd9b893d48684cfea7c242b93ecba01f deleted and replaced by tablet f6cfd089f08b47409fb2a6cf4f6b0385
I20240430 20:11:31.237974 19033 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:51664 (request call id 48) took 1352 ms. Trace:
I20240430 20:11:31.239022 19028 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:51664 (request call id 47) took 1358 ms. Trace:
I20240430 20:11:31.239079 19023 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:51664 (request call id 51) took 1315 ms. Trace:
I20240430 20:11:31.239178 19023 rpcz_store.cc:276] 0430 20:11:29.923790 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:31.192594 (+1268804us) service_pool.cc:225] Handling call
0430 20:11:31.197998 (+ 5404us) spinlock_profiling.cc:243] Waited 5.07 ms on lock 0x616000040654. stack: 00007feb16c74756 00007feb16c74510 00007feb15ca748d 0000000000955804 00000000009557cd 00007feb257b315d 00007feb257b3253 00007feb224e5d41 00007feb2a0d134d 00007feb2a25dcb3 00007feb210462a6 00007feb21045fd5 00007feb1d97c04e 00007feb1d97af2e 00007feb1d981405 00007feb1d981d2f
0430 20:11:31.199163 (+ 1165us) catalog_manager.cc:2072] Acquired catalog manager lock
0430 20:11:31.199727 (+ 564us) catalog_manager.cc:2118] Created new table and tablet info
0430 20:11:31.238827 (+ 39100us) catalog_manager.cc:2178] Wrote table and tablets to system table
0430 20:11:31.238968 (+ 141us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0430 20:11:31.239045 (+ 77us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:31.200338 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:31.200496 (+ 158us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:31.200503 (+ 7us) write_op.cc:427] Acquired schema lock
0430 20:11:31.200505 (+ 2us) tablet.cc:660] Decoding operations
0430 20:11:31.200609 (+ 104us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:31.200622 (+ 13us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:31.200626 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
0430 20:11:31.200711 (+ 85us) tablet.cc:699] Row locks acquired
0430 20:11:31.200713 (+ 2us) write_op.cc:260] PREPARE: finished
0430 20:11:31.200757 (+ 44us) write_op.cc:270] Start()
0430 20:11:31.200793 (+ 36us) write_op.cc:276] Timestamp: P: 1714507891200720 usec, L: 0
0430 20:11:31.200796 (+ 3us) op_driver.cc:331] REPLICATION: starting
0430 20:11:31.201013 (+ 217us) log.cc:842] Serialized 433 byte log entry
0430 20:11:31.236293 (+ 35280us) op_driver.cc:444] REPLICATION: finished
0430 20:11:31.237589 (+ 1296us) write_op.cc:301] APPLY: starting
0430 20:11:31.237597 (+ 8us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:31.237650 (+ 53us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:31.237678 (+ 28us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:31.237782 (+ 104us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:31.237816 (+ 34us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0430 20:11:31.237829 (+ 13us) write_op.cc:312] APPLY: finished
0430 20:11:31.237882 (+ 53us) log.cc:842] Serialized 29 byte log entry
0430 20:11:31.237920 (+ 38us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:31.237933 (+ 13us) write_op.cc:446] Released schema lock
0430 20:11:31.237937 (+ 4us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:31.237990 (+ 53us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:31.237995 (+ 5us) write_op.cc:446] Released schema lock
Metrics: {"spinlock_wait_cycles":11659008,"child_traces":[["op",{"apply.queue_time_us":1270,"apply.run_cpu_time_us":444,"apply.run_wall_time_us":442,"num_ops":2,"prepare.queue_time_us":66,"prepare.run_cpu_time_us":748,"prepare.run_wall_time_us":749,"replication_time_us":35486,"threads_started":1}]]}
I20240430 20:11:31.239131 19028 rpcz_store.cc:276] 0430 20:11:29.880305 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:30.963892 (+1083587us) service_pool.cc:225] Handling call
0430 20:11:31.163475 (+199583us) spinlock_profiling.cc:243] Waited 186 ms on lock 0x616000040654. stack: 00007feb16c74756 00007feb16c74510 00007feb15ca748d 0000000000955804 00000000009557cd 00007feb257b315d 00007feb257b3253 00007feb224e5d41 00007feb2a0d134d 00007feb2a25dcb3 00007feb210462a6 00007feb21045fd5 00007feb1d97c04e 00007feb1d97af2e 00007feb1d981405 00007feb1d981d2f
0430 20:11:31.165647 (+ 2172us) catalog_manager.cc:2072] Acquired catalog manager lock
0430 20:11:31.166129 (+ 482us) catalog_manager.cc:2118] Created new table and tablet info
0430 20:11:31.238694 (+ 72565us) catalog_manager.cc:2178] Wrote table and tablets to system table
0430 20:11:31.238854 (+ 160us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0430 20:11:31.238963 (+ 109us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:31.167378 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:31.167463 (+ 85us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:31.167468 (+ 5us) write_op.cc:427] Acquired schema lock
0430 20:11:31.167470 (+ 2us) tablet.cc:660] Decoding operations
0430 20:11:31.167555 (+ 85us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:31.167564 (+ 9us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:31.167568 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
0430 20:11:31.167630 (+ 62us) tablet.cc:699] Row locks acquired
0430 20:11:31.167632 (+ 2us) write_op.cc:260] PREPARE: finished
0430 20:11:31.167663 (+ 31us) write_op.cc:270] Start()
0430 20:11:31.167690 (+ 27us) write_op.cc:276] Timestamp: P: 1714507891167635 usec, L: 0
0430 20:11:31.167694 (+ 4us) op_driver.cc:331] REPLICATION: starting
0430 20:11:31.167839 (+ 145us) log.cc:842] Serialized 433 byte log entry
0430 20:11:31.236254 (+ 68415us) op_driver.cc:444] REPLICATION: finished
0430 20:11:31.237137 (+ 883us) write_op.cc:301] APPLY: starting
0430 20:11:31.237146 (+ 9us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:31.237178 (+ 32us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:31.237209 (+ 31us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:31.237311 (+ 102us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:31.237337 (+ 26us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0430 20:11:31.237345 (+ 8us) write_op.cc:312] APPLY: finished
0430 20:11:31.237402 (+ 57us) log.cc:842] Serialized 29 byte log entry
0430 20:11:31.237454 (+ 52us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:31.237468 (+ 14us) write_op.cc:446] Released schema lock
0430 20:11:31.237473 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:31.237518 (+ 45us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:31.237521 (+ 3us) write_op.cc:446] Released schema lock
Metrics: {"spinlock_wait_cycles":427182848,"child_traces":[["op",{"apply.queue_time_us":854,"apply.run_cpu_time_us":424,"apply.run_wall_time_us":422,"mutex_wait_us":778,"num_ops":2,"prepare.queue_time_us":816,"prepare.run_cpu_time_us":507,"prepare.run_wall_time_us":507,"replication_time_us":68547}]]}
I20240430 20:11:31.240437 19495 tablet_replica.cc:330] T 68ca48351cd4412381f5195d81109016 P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:31.238082 19033 rpcz_store.cc:276] 0430 20:11:29.885168 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:30.986539 (+1101371us) service_pool.cc:225] Handling call
0430 20:11:31.164124 (+177585us) spinlock_profiling.cc:243] Waited 159 ms on lock 0x616000040654. stack: 00007feb16c74756 00007feb16c74510 00007feb15ca748d 0000000000955804 00000000009557cd 00007feb257b315d 00007feb257b3253 00007feb224e5d41 00007feb2a0d134d 00007feb2a26a05b 00007feb21052be7 00007feb21052915 00007feb1d97c04e 00007feb1d97af2e 00007feb1d981405 00007feb1d981d2f
0430 20:11:31.237933 (+ 73809us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:31.166638 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:31.166737 (+ 99us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:31.166742 (+ 5us) write_op.cc:427] Acquired schema lock
0430 20:11:31.166745 (+ 3us) tablet.cc:660] Decoding operations
0430 20:11:31.166853 (+ 108us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:31.166863 (+ 10us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:31.166867 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
0430 20:11:31.166960 (+ 93us) tablet.cc:699] Row locks acquired
0430 20:11:31.166962 (+ 2us) write_op.cc:260] PREPARE: finished
0430 20:11:31.167001 (+ 39us) write_op.cc:270] Start()
0430 20:11:31.167034 (+ 33us) write_op.cc:276] Timestamp: P: 1714507891166971 usec, L: 0
0430 20:11:31.167037 (+ 3us) op_driver.cc:331] REPLICATION: starting
0430 20:11:31.167204 (+ 167us) log.cc:842] Serialized 625 byte log entry
0430 20:11:31.236197 (+ 68993us) op_driver.cc:444] REPLICATION: finished
0430 20:11:31.236478 (+ 281us) write_op.cc:301] APPLY: starting
0430 20:11:31.236497 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:31.236545 (+ 48us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:31.236586 (+ 41us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:31.236716 (+ 130us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:31.236779 (+ 63us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0430 20:11:31.236787 (+ 8us) write_op.cc:312] APPLY: finished
0430 20:11:31.236879 (+ 92us) log.cc:842] Serialized 29 byte log entry
0430 20:11:31.236955 (+ 76us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:31.236976 (+ 21us) write_op.cc:446] Released schema lock
0430 20:11:31.236983 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:31.237048 (+ 65us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:31.237051 (+ 3us) write_op.cc:446] Released schema lock
Metrics: {"spinlock_wait_cycles":365661952,"child_traces":[["op",{"apply.queue_time_us":229,"apply.run_cpu_time_us":624,"apply.run_wall_time_us":625,"num_ops":2,"prepare.queue_time_us":1554,"prepare.run_cpu_time_us":636,"prepare.run_wall_time_us":636,"replication_time_us":69133}]]}
I20240430 20:11:31.243438 19049 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet acd23f0a63d5448aaa7ed78422e67bad (table default.Table_972f13be4dca4cd19b9a04534dc59f51 [id=69f37db4de634a3185613e40013e0ed7]). Expected version 1 got 0
I20240430 20:11:31.247051 19049 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 24dcc20832804229a2664873bf6ddaa7 (table default.Table_be2ebe77f859489f9cd3e912cb1edc8b [id=2d44ec94131148048021b2ee6366ba30]). Expected version 2 got 0
I20240430 20:11:31.247491 19049 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet aec9c09cc0454acab6006bb3e10982b7 (table default.Table_e68dc52988f24f1788517de94d7d2a0e [id=26d4860fb33047b7b812ebd4309f2e28]). Expected version 1 got 0
I20240430 20:11:31.248906 18461 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e503a898406412ab977d3e048ea4b5d with delete_type TABLET_DATA_DELETED (replaced by tablet 5bd0362ff41b4faca0205d136d596737) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:31.250360 18741 tablet_service.cc:1514] Processing DeleteTablet for tablet 1398d4d2305b48e08fdb12432bd6c4b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:25 UTC) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:31.250504 18463 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f56ac2afd444d63a6bfd96988464bbb with delete_type TABLET_DATA_DELETED (replaced by tablet 2da900b6bbfa4fa99fe8494fd2e68da8) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:31.250618 18744 tablet_service.cc:1514] Processing DeleteTablet for tablet a1aeee1b02214975a376cfd36957afe8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:26 UTC) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:31.250730 19489 ts_tablet_manager.cc:1875] T 68ca48351cd4412381f5195d81109016 P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:31.251039 19618 raft_consensus.cc:482] T 68ca48351cd4412381f5195d81109016 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:31.251466 18983 tablet_bootstrap.cc:492] T 20d166756819406ba85044f4fee9c4ce P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:31.252216 18983 ts_tablet_manager.cc:1367] T 20d166756819406ba85044f4fee9c4ce P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.127s user 0.004s sys 0.002s
I20240430 20:11:31.253163 18983 raft_consensus.cc:348] T 20d166756819406ba85044f4fee9c4ce P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:31.253374 18983 raft_consensus.cc:374] T 20d166756819406ba85044f4fee9c4ce P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:31.253463 18983 raft_consensus.cc:729] T 20d166756819406ba85044f4fee9c4ce P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:31.253736 18983 consensus_queue.cc:260] T 20d166756819406ba85044f4fee9c4ce P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:31.259594 18734 tablet_service.cc:1514] Processing DeleteTablet for tablet 02176eee8e044a7db8cac48439e9f594 with delete_type TABLET_DATA_DELETED (replaced by tablet 09d92a4a6ebe44cbb3553ed660f143d0) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:31.260680 18738 tablet_service.cc:1514] Processing DeleteTablet for tablet 93d158de64994f3c9fa45a1250a4f533 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:25 UTC) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:31.261329 19049 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet a05e32a33aa7488eb287cc6314c09c44 (table default.Table_6551367971354d3195e4f7d47364ce7c [id=955d833ff8f14bf6957abadf9ef669e2]). Expected version 1 got 0
I20240430 20:11:31.261636 18730 tablet_service.cc:1514] Processing DeleteTablet for tablet 75ebf051383e42529dcb5f7dd0770a4b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:27 UTC) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:31.261648 19049 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 76d6e9d547ca4a848eae852944b1d117 (table default.Table_104ae015924a48fd8d27220792f07470 [id=ff86c0357c164dd0991098c416087773]). Expected version 2 got 0
I20240430 20:11:31.261890 18732 tablet_service.cc:1514] Processing DeleteTablet for tablet d4508e4f6dbd42109d951c648b3dc7df with delete_type TABLET_DATA_DELETED (replaced by tablet 2e79df8744e74b5991a79ce978ff0202) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:31.262061 18737 tablet_service.cc:1514] Processing DeleteTablet for tablet c5fd0730133c472299c93f4b0830e527 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:25 UTC) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:31.262214 18731 tablet_service.cc:1514] Processing DeleteTablet for tablet 255c13ad8fae4ea4aa8e7b7aabb7602e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:27 UTC) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:31.262266 18450 tablet_service.cc:1514] Processing DeleteTablet for tablet e5111d7607dd42188a0176cf5be2ec05 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:29 UTC) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:31.262662 18724 tablet_service.cc:1514] Processing DeleteTablet for tablet aacabead3c874e77a47d2ecf19dbfa82 with delete_type TABLET_DATA_DELETED (replaced by tablet fd40967ab4a04001819e16c06a6c4300) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:31.262673 18450 tablet_service.cc:1514] Processing DeleteTablet for tablet dd9b893d48684cfea7c242b93ecba01f with delete_type TABLET_DATA_DELETED (replaced by tablet f6cfd089f08b47409fb2a6cf4f6b0385) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:31.262909 18726 tablet_service.cc:1514] Processing DeleteTablet for tablet d9e578c99058424a9b86390641f9c34f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:26 UTC) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:31.263631 18738 tablet_service.cc:1514] Processing DeleteTablet for tablet 29974dc2c7dd45b99b6cc0712378fc30 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:26 UTC) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:31.264075 19004 catalog_manager.cc:4830] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): tablet 47c37876694c4a4abb1091c631dab7a5 (table default.Table_85072f20d90d4470bf3f65189b19fd51 [id=0fd2754282af4b2b9fcc7952be704cd3]) successfully deleted
I20240430 20:11:31.264467 18726 tablet_service.cc:1514] Processing DeleteTablet for tablet bbe9a2429a434f3f95d6e7f314fcfe4c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:26 UTC) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:31.264607 18457 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a1e50f99c5d4b44bdbb51b58a92bc75 with delete_type TABLET_DATA_DELETED (replaced by tablet fbdeb4714b8c4b93ab72822c43329aae) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:31.262483 19049 catalog_manager.cc:5484] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 5e1c6397f09747f091534faf93542710 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e1c6397f09747f091534faf93542710 (127.17.177.2). New cstate: current_term: 1 leader_uuid: "5e1c6397f09747f091534faf93542710" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } health_report { overall_health: UNKNOWN } } }
I20240430 20:11:31.264869 18738 tablet_service.cc:1514] Processing DeleteTablet for tablet 34ab40d8d91e401db299ddee7175ba28 with delete_type TABLET_DATA_DELETED (replaced by tablet 5f02d8862e1f4ccfa9dbda73dbe18615) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:31.265138 19049 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 6e826459c3f4437da81c9e6e09f0c22f (table default.Table_873c21f6dffd4d47a742b3699ab85268 [id=d7e2c4300ff94fa2acc65d6d6c932c65]). Expected version 1 got 0
I20240430 20:11:31.265395 19049 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 34130f2820064fa0898ef1f9d7de1c6d (table default.Table_4537f891b01048ceb51202abc3600911 [id=5a1c4a7be2fc465fac89a50854eb3322]). Expected version 1 got 0
I20240430 20:11:31.265866 19049 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 372b135dd01f44f98cd713ca8857ba61 (table default.Table_ee86b8fcf2064d8b8c3e327e357d3395 [id=7f3a15a442984827a7521d5d1c98dc3d]). Expected version 2 got 0
I20240430 20:11:31.265619 18759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1555924b5f1444398f12ee82c3cb5b13" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:31.266166 19049 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet fd40967ab4a04001819e16c06a6c4300 (table default.Table_fda07fd7279c484db2bee67e8c859855 [id=da9ad991953a40c4b77b31317597ad1d]). Expected version 3 got 0
I20240430 20:11:31.266302 18724 tablet_service.cc:1514] Processing DeleteTablet for tablet cace1398492f4d968dcb6899385d527c with delete_type TABLET_DATA_DELETED (replaced by tablet 01436e5b6232494babbb414342f03090) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:31.266439 19049 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 680553d4d0bc4fcb839235c1832757e1 (table default.Table_49a012a7c4c8457fadb0a8f0fc9654c6 [id=eecb2f8eeef547c5ab1abe6a2430e592]). Expected version 1 got 0
I20240430 20:11:31.266580 18999 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet 1398d4d2305b48e08fdb12432bd6c4b0 (table default.Table_eea9acffa45b4f3ab6d4524dc460853c [id=52cd3a55893547cabf418f4d01b11d08]) successfully deleted
I20240430 20:11:31.268303 18601 tablet_service.cc:1514] Processing DeleteTablet for tablet dd9b893d48684cfea7c242b93ecba01f with delete_type TABLET_DATA_DELETED (replaced by tablet f6cfd089f08b47409fb2a6cf4f6b0385) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:31.251209 19618 raft_consensus.cc:504] T 68ca48351cd4412381f5195d81109016 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:31.266870 19049 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 6e754af64cb64fd2a082adae7d5f24da (table default.Table_208b12b8e9bd4f3e8bc9c6d422ada649 [id=1df4e95702cd46f59ab21b101049a9ed]). Expected version 1 got 0
I20240430 20:11:31.268577 19471 ts_tablet_manager.cc:1888] T 68ca48351cd4412381f5195d81109016 P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:31.268802 19049 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet fcc9698abda4491b98e1e879589913a8 (table default.Table_e431f94220aa44ecba799a54dfb612ae [id=1e785b6c159b4312bd1b064271f55a49]). Expected version 1 got 0
I20240430 20:11:31.269021 19049 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet fae9a35a2ac844b4bc2ebdf214aa7171 (table default.Table_0ab5b2f09fa7468bb89ab703f6d3d25c [id=a5d580cffa2447d78fc9f2c8e31264f1]). Expected version 4 got 0
I20240430 20:11:31.270154 19618 leader_election.cc:288] T 68ca48351cd4412381f5195d81109016 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:31.271087 18759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b76a4105a2974fe280e759349037d027" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:31.270502 19029 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:51664 (request call id 52) took 1279 ms. Trace:
I20240430 20:11:31.272472 18724 tablet_service.cc:1514] Processing DeleteTablet for tablet f77086a5675f44e99e8a867b694df4b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:29 UTC) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:31.272504 19495 raft_consensus.cc:2229] T 68ca48351cd4412381f5195d81109016 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:31.272519 19028 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_d4469d8c485e427c8e245855b9d5328b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240430 20:11:31.272927 18726 tablet_service.cc:1514] Processing DeleteTablet for tablet 15674aeb56b74210a4a2809608fd4457 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:25 UTC) from {username='slave'} at 127.0.0.1:59830
W20240430 20:11:31.273138 19028 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d4469d8c485e427c8e245855b9d5328b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:31.273183 19049 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 53a899fa5a424a0d99f1b47925e8c7ad (table default.Table_deadf728d74a462fb79641891dfedd9c [id=608890834da843fa9af0cfc0d548029e]). Expected version 1 got 0
I20240430 20:11:31.273346 19049 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 32c823d1700949ab8b22adcd83b7bfd1 (table default.Table_e8c73c89623d4b49b8d7f0a01d729bdd [id=5cd10682ce52414cb5a103e4433a6a0d]). Expected version 1 got 0
I20240430 20:11:31.273594 19049 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 0b4d9e860d7c45fd91d7a566605c9283 (table default.Table_421069f84cb844368066c93102a8faf8 [id=d5cea90cbc2c48df9d8b80da1eccc949]). Expected version 2 got 0
I20240430 20:11:31.273305 18738 tablet_service.cc:1514] Processing DeleteTablet for tablet f5546f9edeea44d7a6c5cb954f5157ba with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:27 UTC) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:31.274166 18731 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fe1247349344863a6273a769ce8ddf6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:24 UTC) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:31.270921 19051 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:51664 (request call id 53) took 1278 ms. Trace:
I20240430 20:11:31.274329 19051 rpcz_store.cc:276] 0430 20:11:29.992107 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:31.207889 (+1215782us) service_pool.cc:225] Handling call
0430 20:11:31.209638 (+ 1749us) catalog_manager.cc:2072] Acquired catalog manager lock
0430 20:11:31.210207 (+ 569us) catalog_manager.cc:2118] Created new table and tablet info
0430 20:11:31.270664 (+ 60457us) catalog_manager.cc:2178] Wrote table and tablets to system table
0430 20:11:31.270783 (+ 119us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0430 20:11:31.270871 (+ 88us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:31.211748 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:31.211901 (+ 153us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:31.211920 (+ 19us) write_op.cc:427] Acquired schema lock
0430 20:11:31.211924 (+ 4us) tablet.cc:660] Decoding operations
0430 20:11:31.212049 (+ 125us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:31.212065 (+ 16us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:31.212072 (+ 7us) tablet.cc:683] Acquiring locks for 2 operations
0430 20:11:31.212179 (+ 107us) tablet.cc:699] Row locks acquired
0430 20:11:31.212183 (+ 4us) write_op.cc:260] PREPARE: finished
0430 20:11:31.212229 (+ 46us) write_op.cc:270] Start()
0430 20:11:31.212272 (+ 43us) write_op.cc:276] Timestamp: P: 1714507891212192 usec, L: 0
0430 20:11:31.212278 (+ 6us) op_driver.cc:331] REPLICATION: starting
0430 20:11:31.212514 (+ 236us) log.cc:842] Serialized 433 byte log entry
0430 20:11:31.269334 (+ 56820us) op_driver.cc:444] REPLICATION: finished
0430 20:11:31.269944 (+ 610us) write_op.cc:301] APPLY: starting
0430 20:11:31.269959 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:31.269995 (+ 36us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:31.270064 (+ 69us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:31.270196 (+ 132us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:31.270224 (+ 28us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0430 20:11:31.270232 (+ 8us) write_op.cc:312] APPLY: finished
0430 20:11:31.270316 (+ 84us) log.cc:842] Serialized 29 byte log entry
0430 20:11:31.270543 (+ 227us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:31.270562 (+ 19us) write_op.cc:446] Released schema lock
0430 20:11:31.270568 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:31.271035 (+ 467us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:31.271041 (+ 6us) write_op.cc:446] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":528,"apply.run_cpu_time_us":611,"apply.run_wall_time_us":1153,"num_ops":2,"prepare.queue_time_us":56,"prepare.run_cpu_time_us":866,"prepare.run_wall_time_us":867,"replication_time_us":57041}]]}
I20240430 20:11:31.275092 19051 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_e63f64a411094f8db2ef641f2257ca79" } modify_external_catalogs: true
I20240430 20:11:31.275297 19051 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_e63f64a411094f8db2ef641f2257ca79" } modify_external_catalogs: true
I20240430 20:11:31.275681 19495 raft_consensus.cc:2258] T 68ca48351cd4412381f5195d81109016 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:31.275918 19023 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_4bac430d51a7461e919b79d3a2f92a8e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240430 20:11:31.276434 19495 ts_tablet_manager.cc:1875] T 68ca48351cd4412381f5195d81109016 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240430 20:11:31.276455 19023 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4bac430d51a7461e919b79d3a2f92a8e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:31.276810 18726 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a1e50f99c5d4b44bdbb51b58a92bc75 with delete_type TABLET_DATA_DELETED (replaced by tablet fbdeb4714b8c4b93ab72822c43329aae) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:31.271757 19024 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:51664 (request call id 54) took 1270 ms. Trace:
I20240430 20:11:31.277652 19024 rpcz_store.cc:276] 0430 20:11:30.001737 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:31.218695 (+1216958us) service_pool.cc:225] Handling call
0430 20:11:31.219669 (+ 974us) catalog_manager.cc:2072] Acquired catalog manager lock
0430 20:11:31.220127 (+ 458us) catalog_manager.cc:2118] Created new table and tablet info
0430 20:11:31.271524 (+ 51397us) catalog_manager.cc:2178] Wrote table and tablets to system table
0430 20:11:31.271619 (+ 95us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0430 20:11:31.271716 (+ 97us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:31.221139 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:31.221240 (+ 101us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:31.221247 (+ 7us) write_op.cc:427] Acquired schema lock
0430 20:11:31.221250 (+ 3us) tablet.cc:660] Decoding operations
0430 20:11:31.221353 (+ 103us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:31.221365 (+ 12us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:31.221370 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0430 20:11:31.221452 (+ 82us) tablet.cc:699] Row locks acquired
0430 20:11:31.221456 (+ 4us) write_op.cc:260] PREPARE: finished
0430 20:11:31.221493 (+ 37us) write_op.cc:270] Start()
0430 20:11:31.221537 (+ 44us) write_op.cc:276] Timestamp: P: 1714507891221460 usec, L: 0
0430 20:11:31.221541 (+ 4us) op_driver.cc:331] REPLICATION: starting
0430 20:11:31.221714 (+ 173us) log.cc:842] Serialized 433 byte log entry
0430 20:11:31.270834 (+ 49120us) op_driver.cc:444] REPLICATION: finished
0430 20:11:31.270914 (+ 80us) write_op.cc:301] APPLY: starting
0430 20:11:31.271089 (+ 175us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:31.271127 (+ 38us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:31.271161 (+ 34us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:31.271252 (+ 91us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:31.271279 (+ 27us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0430 20:11:31.271286 (+ 7us) write_op.cc:312] APPLY: finished
0430 20:11:31.271343 (+ 57us) log.cc:842] Serialized 29 byte log entry
0430 20:11:31.271404 (+ 61us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:31.271444 (+ 40us) write_op.cc:446] Released schema lock
0430 20:11:31.271450 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:31.271893 (+ 443us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:31.271900 (+ 7us) write_op.cc:446] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":31,"apply.run_cpu_time_us":516,"apply.run_wall_time_us":1032,"num_ops":2,"prepare.queue_time_us":498,"prepare.run_cpu_time_us":646,"prepare.run_wall_time_us":647,"replication_time_us":49276}]]}
I20240430 20:11:31.278398 19024 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_327b9bc7f9c64a128209a7a24d2a410a" } new_table_name: "default.Table_453f21c4c2ad4ef7beca9aff68e47b35" modify_external_catalogs: true
I20240430 20:11:31.279753 19033 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_db56d63aef494592b69246a0c13006f2" } modify_external_catalogs: true
I20240430 20:11:31.279940 19033 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_db56d63aef494592b69246a0c13006f2" } modify_external_catalogs: true
I20240430 20:11:31.279839 19628 tablet.cc:1722] T 77285afcafe04a829e485998bbbe00f0 P 2efb542aa55e439b99ccb98d1c5c884a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240430 20:11:31.272329 19029 rpcz_store.cc:276] 0430 20:11:29.990957 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:31.196375 (+1205418us) service_pool.cc:225] Handling call
0430 20:11:31.200510 (+ 4135us) spinlock_profiling.cc:243] Waited 3.81 ms on lock 0x616000040654. stack: 00007feb16c74756 00007feb16c74510 00007feb15ca748d 0000000000955804 00000000009557cd 00007feb257b315d 00007feb257b3253 00007feb224e5d41 00007feb2a0d134d 00007feb2a25dcb3 00007feb210462a6 00007feb21045fd5 00007feb1d97c04e 00007feb1d97af2e 00007feb1d981405 00007feb1d981d2f
0430 20:11:31.201508 (+ 998us) catalog_manager.cc:2072] Acquired catalog manager lock
0430 20:11:31.202071 (+ 563us) catalog_manager.cc:2118] Created new table and tablet info
0430 20:11:31.270184 (+ 68113us) catalog_manager.cc:2178] Wrote table and tablets to system table
0430 20:11:31.270331 (+ 147us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0430 20:11:31.270449 (+ 118us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:31.203191 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:31.203290 (+ 99us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:31.203298 (+ 8us) write_op.cc:427] Acquired schema lock
0430 20:11:31.203302 (+ 4us) tablet.cc:660] Decoding operations
0430 20:11:31.203427 (+ 125us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:31.203440 (+ 13us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:31.203445 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0430 20:11:31.203524 (+ 79us) tablet.cc:699] Row locks acquired
0430 20:11:31.203527 (+ 3us) write_op.cc:260] PREPARE: finished
0430 20:11:31.203566 (+ 39us) write_op.cc:270] Start()
0430 20:11:31.203632 (+ 66us) write_op.cc:276] Timestamp: P: 1714507891203534 usec, L: 0
0430 20:11:31.203638 (+ 6us) op_driver.cc:331] REPLICATION: starting
0430 20:11:31.203878 (+ 240us) log.cc:842] Serialized 433 byte log entry
0430 20:11:31.269256 (+ 65378us) op_driver.cc:444] REPLICATION: finished
0430 20:11:31.269424 (+ 168us) write_op.cc:301] APPLY: starting
0430 20:11:31.269443 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:31.269498 (+ 55us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:31.269536 (+ 38us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:31.269658 (+ 122us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:31.269693 (+ 35us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0430 20:11:31.269701 (+ 8us) write_op.cc:312] APPLY: finished
0430 20:11:31.269794 (+ 93us) log.cc:842] Serialized 29 byte log entry
0430 20:11:31.270061 (+ 267us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:31.270085 (+ 24us) write_op.cc:446] Released schema lock
0430 20:11:31.270093 (+ 8us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:31.270694 (+ 601us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:31.270700 (+ 6us) write_op.cc:446] Released schema lock
Metrics: {"spinlock_wait_cycles":8757632,"child_traces":[["op",{"apply.queue_time_us":101,"apply.run_cpu_time_us":651,"apply.run_wall_time_us":1349,"num_ops":2,"prepare.queue_time_us":47,"prepare.run_cpu_time_us":770,"prepare.run_wall_time_us":769,"replication_time_us":65590}]]}
I20240430 20:11:31.283622 19028 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:51664 (request call id 56) took 1268 ms. Trace:
I20240430 20:11:31.283716 19028 rpcz_store.cc:276] 0430 20:11:30.015240 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:31.240033 (+1224793us) service_pool.cc:225] Handling call
0430 20:11:31.272399 (+ 32366us) spinlock_profiling.cc:243] Waited 32.1 ms on lock 0x616000040654. stack: 00007feb16c74756 00007feb16c74510 00007feb15ca748d 0000000000955804 00000000009557cd 00007feb257b315d 00007feb257b3253 00007feb224e5d41 00007feb2a0d134d 00007feb2a25dcb3 00007feb210462a6 00007feb21045fd5 00007feb1d97c04e 00007feb1d97af2e 00007feb1d981405 00007feb1d981d2f
0430 20:11:31.273256 (+ 857us) catalog_manager.cc:2072] Acquired catalog manager lock
0430 20:11:31.273752 (+ 496us) catalog_manager.cc:2118] Created new table and tablet info
0430 20:11:31.283321 (+ 9569us) catalog_manager.cc:2178] Wrote table and tablets to system table
0430 20:11:31.283462 (+ 141us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0430 20:11:31.283562 (+ 100us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:31.274331 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:31.274447 (+ 116us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:31.274452 (+ 5us) write_op.cc:427] Acquired schema lock
0430 20:11:31.274455 (+ 3us) tablet.cc:660] Decoding operations
0430 20:11:31.274546 (+ 91us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:31.274559 (+ 13us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:31.274564 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0430 20:11:31.274679 (+ 115us) tablet.cc:699] Row locks acquired
0430 20:11:31.274681 (+ 2us) write_op.cc:260] PREPARE: finished
0430 20:11:31.274718 (+ 37us) write_op.cc:270] Start()
0430 20:11:31.274748 (+ 30us) write_op.cc:276] Timestamp: P: 1714507891274679 usec, L: 0
0430 20:11:31.274751 (+ 3us) op_driver.cc:331] REPLICATION: starting
0430 20:11:31.274946 (+ 195us) log.cc:842] Serialized 433 byte log entry
0430 20:11:31.282398 (+ 7452us) op_driver.cc:444] REPLICATION: finished
0430 20:11:31.282505 (+ 107us) write_op.cc:301] APPLY: starting
0430 20:11:31.282520 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:31.282605 (+ 85us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:31.282632 (+ 27us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:31.282762 (+ 130us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:31.282806 (+ 44us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0430 20:11:31.282817 (+ 11us) write_op.cc:312] APPLY: finished
0430 20:11:31.282909 (+ 92us) log.cc:842] Serialized 29 byte log entry
0430 20:11:31.282980 (+ 71us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:31.283003 (+ 23us) write_op.cc:446] Released schema lock
0430 20:11:31.283009 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:31.283059 (+ 50us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:31.283063 (+ 4us) write_op.cc:446] Released schema lock
Metrics: {"spinlock_wait_cycles":73799424,"child_traces":[["op",{"apply.queue_time_us":30,"apply.run_cpu_time_us":603,"apply.run_wall_time_us":603,"num_ops":2,"prepare.queue_time_us":49,"prepare.run_cpu_time_us":737,"prepare.run_wall_time_us":738,"raft.queue_time_us":68,"raft.run_cpu_time_us":287,"raft.run_wall_time_us":6801,"replication_time_us":7626}]]}
I20240430 20:11:31.285012 18764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20d166756819406ba85044f4fee9c4ce" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:31.285909 18738 tablet_service.cc:1514] Processing DeleteTablet for tablet e6eac92dfa5f4456bc49571878f0f8dd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:29 UTC) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:31.286139 18731 tablet_service.cc:1514] Processing DeleteTablet for tablet 5bde72e2791a497db7dd08686ca36d80 with delete_type TABLET_DATA_DELETED (replaced by tablet 66dd767bd3f8494f94061fb89e5e5680) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:31.288527 18731 tablet_service.cc:1514] Processing DeleteTablet for tablet a9e0659564894b0e8cb7f600bb07a085 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:29 UTC) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:31.288808 18724 tablet_service.cc:1514] Processing DeleteTablet for tablet dd9b893d48684cfea7c242b93ecba01f with delete_type TABLET_DATA_DELETED (replaced by tablet f6cfd089f08b47409fb2a6cf4f6b0385) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:31.289113 18764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68ca48351cd4412381f5195d81109016" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:31.289535 18726 tablet_service.cc:1514] Processing DeleteTablet for tablet eed6338ad3d64859af087ee15e1ec798 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:26 UTC) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:31.289788 18738 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e69ca12bd10494fb013614741bc166c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:27 UTC) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:31.290721 19029 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_6551367971354d3195e4f7d47364ce7c" } new_table_name: "default.Table_2d1cfc78363a47aa8dba7ef7d80f8c9a" modify_external_catalogs: true
I20240430 20:11:31.291296 19049 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet b72fe1f0713b4076a81a3fc1c74f7fa7 (table default.Table_16b082d140234e1c9a1e70654839b4cd [id=c0d8dc87d1f9426a85f6176f161595a1]). Expected version 3 got 0
I20240430 20:11:31.292052 19049 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 77285afcafe04a829e485998bbbe00f0 (table default.Table_e63f64a411094f8db2ef641f2257ca79 [id=040dec3a3d804cdfbdada359bbe5123b]). Expected version 1 got 0
I20240430 20:11:31.292471 19049 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet c66c142419ca4e1399ecd1c8d1e62a59 (table default.Table_0443359dbeff437fb6cffb0db33cbb30 [id=93c44abdf6b147228224388398cd7aeb]). Expected version 2 got 0
I20240430 20:11:31.292667 19049 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 724e51ff05ec4baebc2bb55e05c8eddc (table default.Table_0c5d4686f2ed40fb8d62c31b0166dc05 [id=4dce8a767f0541e1beb5464b2ed95d06]). Expected version 1 got 0
I20240430 20:11:31.292938 19049 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 294644d7a16b4ca2bbb6d106bcb8c7ff (table default.Table_190b2d86515a44898fa38a49517b9df1 [id=97d8a36ddfae4bb5b99c04a7a08544af]). Expected version 2 got 0
W20240430 20:11:31.294255 19026 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.17.177.3:42157 (request call id 78) took 1949 ms (client timeout 1000 ms). Trace:
W20240430 20:11:31.294365 19026 rpcz_store.cc:269] 0430 20:11:29.345058 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:29.561793 (+216735us) service_pool.cc:225] Handling call
0430 20:11:29.606472 (+ 44679us) spinlock_profiling.cc:243] Waited 42.6 ms on lock 0x616000040654. stack: 00007feb16c74756 00007feb16c74510 00007feb15ca748d 0000000000955804 00000000009557cd 00007feb257b315d 00007feb257b3253 00007feb224e5d41 00007feb2a0d134d 00007feb2a258668 00007feb21043786 00007feb210434b5 00007feb1d97c04e 00007feb1d97af2e 00007feb1d981405 00007feb1d981d2f
0430 20:11:31.294186 (+1687714us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:30.551448 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:30.551580 (+ 132us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:30.551587 (+ 7us) write_op.cc:427] Acquired schema lock
0430 20:11:30.551590 (+ 3us) tablet.cc:660] Decoding operations
0430 20:11:30.551704 (+ 114us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:30.551720 (+ 16us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:30.551726 (+ 6us) tablet.cc:683] Acquiring locks for 1 operations
0430 20:11:30.551803 (+ 77us) tablet.cc:699] Row locks acquired
0430 20:11:30.551806 (+ 3us) write_op.cc:260] PREPARE: finished
0430 20:11:30.551845 (+ 39us) write_op.cc:270] Start()
0430 20:11:30.551886 (+ 41us) write_op.cc:276] Timestamp: P: 1714507890551822 usec, L: 0
0430 20:11:30.551890 (+ 4us) op_driver.cc:331] REPLICATION: starting
0430 20:11:30.552096 (+ 206us) log.cc:842] Serialized 530 byte log entry
0430 20:11:30.565178 (+ 13082us) op_driver.cc:444] REPLICATION: finished
0430 20:11:30.565345 (+ 167us) write_op.cc:301] APPLY: starting
0430 20:11:30.565366 (+ 21us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:30.565418 (+ 52us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:30.565476 (+ 58us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:30.565556 (+ 80us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:30.565609 (+ 53us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0430 20:11:30.565621 (+ 12us) write_op.cc:312] APPLY: finished
0430 20:11:30.565740 (+ 119us) log.cc:842] Serialized 23 byte log entry
0430 20:11:30.565813 (+ 73us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:30.565836 (+ 23us) write_op.cc:446] Released schema lock
0430 20:11:30.565845 (+ 9us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:30.565905 (+ 60us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:30.565909 (+ 4us) write_op.cc:446] Released schema lock
Metrics: {"reported_tablets":75,"spinlock_wait_cycles":97890560,"child_traces":[["op",{"apply.queue_time_us":111,"apply.run_cpu_time_us":632,"apply.run_wall_time_us":631,"num_ops":1,"prepare.queue_time_us":56,"prepare.run_cpu_time_us":776,"prepare.run_wall_time_us":777,"raft.queue_time_us":433,"raft.run_cpu_time_us":261,"raft.run_wall_time_us":260,"replication_time_us":13256}]]}
I20240430 20:11:31.295010 19023 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:51664 (request call id 55) took 1279 ms. Trace:
I20240430 20:11:31.295219 19028 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_d997b723a0dd48f6bc2d42c9bfd6bc14" } modify_external_catalogs: true
I20240430 20:11:31.295269 19026 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_fe1773cf77654e2a86e95444437b7201" } modify_external_catalogs: true
I20240430 20:11:31.295468 19026 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_fe1773cf77654e2a86e95444437b7201" } modify_external_catalogs: true
I20240430 20:11:31.295413 19028 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_d997b723a0dd48f6bc2d42c9bfd6bc14" } modify_external_catalogs: true
I20240430 20:11:31.283334 18620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68ca48351cd4412381f5195d81109016" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
W20240430 20:11:31.287854 18557 leader_election.cc:341] T 1555924b5f1444398f12ee82c3cb5b13 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:31.288800 19474 raft_consensus.cc:482] T b688e03ea71940c69bda5520b62c1fc3 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:31.296870 19471 log.cc:1196] T 68ca48351cd4412381f5195d81109016 P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/68ca48351cd4412381f5195d81109016
I20240430 20:11:31.295151 19023 rpcz_store.cc:276] 0430 20:11:30.015100 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:31.239833 (+1224733us) service_pool.cc:225] Handling call
0430 20:11:31.275820 (+ 35987us) spinlock_profiling.cc:243] Waited 35.7 ms on lock 0x616000040654. stack: 00007feb16c74756 00007feb16c74510 00007feb15ca748d 0000000000955804 00000000009557cd 00007feb257b315d 00007feb257b3253 00007feb224e5d41 00007feb2a0d134d 00007feb2a25dcb3 00007feb210462a6 00007feb21045fd5 00007feb1d97c04e 00007feb1d97af2e 00007feb1d981405 00007feb1d981d2f
0430 20:11:31.276546 (+ 726us) catalog_manager.cc:2072] Acquired catalog manager lock
0430 20:11:31.276980 (+ 434us) catalog_manager.cc:2118] Created new table and tablet info
0430 20:11:31.294685 (+ 17705us) catalog_manager.cc:2178] Wrote table and tablets to system table
0430 20:11:31.294856 (+ 171us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0430 20:11:31.294960 (+ 104us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:31.277585 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:31.277709 (+ 124us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:31.277720 (+ 11us) write_op.cc:427] Acquired schema lock
0430 20:11:31.277723 (+ 3us) tablet.cc:660] Decoding operations
0430 20:11:31.277802 (+ 79us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:31.277813 (+ 11us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:31.277819 (+ 6us) tablet.cc:683] Acquiring locks for 2 operations
0430 20:11:31.277912 (+ 93us) tablet.cc:699] Row locks acquired
0430 20:11:31.277915 (+ 3us) write_op.cc:260] PREPARE: finished
0430 20:11:31.277948 (+ 33us) write_op.cc:270] Start()
0430 20:11:31.277981 (+ 33us) write_op.cc:276] Timestamp: P: 1714507891277910 usec, L: 0
0430 20:11:31.277985 (+ 4us) op_driver.cc:331] REPLICATION: starting
0430 20:11:31.278155 (+ 170us) log.cc:842] Serialized 433 byte log entry
0430 20:11:31.288139 (+ 9984us) op_driver.cc:444] REPLICATION: finished
0430 20:11:31.288222 (+ 83us) write_op.cc:301] APPLY: starting
0430 20:11:31.288237 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:31.288278 (+ 41us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:31.288310 (+ 32us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:31.288591 (+ 281us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:31.288627 (+ 36us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0430 20:11:31.288635 (+ 8us) write_op.cc:312] APPLY: finished
0430 20:11:31.288708 (+ 73us) log.cc:842] Serialized 29 byte log entry
0430 20:11:31.288779 (+ 71us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:31.288799 (+ 20us) write_op.cc:446] Released schema lock
0430 20:11:31.288804 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:31.288873 (+ 69us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:31.288877 (+ 4us) write_op.cc:446] Released schema lock
Metrics: {"spinlock_wait_cycles":82079488,"child_traces":[["op",{"apply.queue_time_us":34,"apply.run_cpu_time_us":702,"apply.run_wall_time_us":702,"num_ops":2,"prepare.queue_time_us":166,"prepare.run_cpu_time_us":633,"prepare.run_wall_time_us":633,"replication_time_us":10129}]]}
I20240430 20:11:31.296989 19474 raft_consensus.cc:504] T b688e03ea71940c69bda5520b62c1fc3 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:31.297318 19471 ts_tablet_manager.cc:1909] T 68ca48351cd4412381f5195d81109016 P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:31.297377 19023 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_0ab5b2f09fa7468bb89ab703f6d3d25c" } new_table_name: "default.Table_730a215d7af74a5bb753adca39dea676" modify_external_catalogs: true
I20240430 20:11:31.297752 19474 leader_election.cc:288] T b688e03ea71940c69bda5520b62c1fc3 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:31.297909 19474 raft_consensus.cc:482] T 294644d7a16b4ca2bbb6d106bcb8c7ff P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:31.298075 19474 raft_consensus.cc:504] T 294644d7a16b4ca2bbb6d106bcb8c7ff P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:31.298660 19474 leader_election.cc:288] T 294644d7a16b4ca2bbb6d106bcb8c7ff P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:31.301167 18983 ts_tablet_manager.cc:1398] T 20d166756819406ba85044f4fee9c4ce P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.049s user 0.002s sys 0.002s
I20240430 20:11:31.301499 18983 tablet_bootstrap.cc:492] T c9e7660eccc448dbb99d135f82bfd261 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:31.303592 19471 tablet_replica.cc:330] T aacabead3c874e77a47d2ecf19dbfa82 P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:31.303819 19471 raft_consensus.cc:2229] T aacabead3c874e77a47d2ecf19dbfa82 P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Raft consensus shutting down.
I20240430 20:11:31.304194 19471 raft_consensus.cc:2258] T aacabead3c874e77a47d2ecf19dbfa82 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:31.304852 19471 ts_tablet_manager.cc:1875] T aacabead3c874e77a47d2ecf19dbfa82 P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:31.305667 18738 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a9066d895744c5bb7d9532b2940603c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:29 UTC) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:31.248742 18985 raft_consensus.cc:348] T 12c6e131fa904d23865b8673b58a7e33 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:31.306845 18762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b688e03ea71940c69bda5520b62c1fc3" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:31.307207 18738 tablet_service.cc:1514] Processing DeleteTablet for tablet b688e03ea71940c69bda5520b62c1fc3 with delete_type TABLET_DATA_DELETED (replaced by tablet 7683caba80594627900a0185f772c131) from {username='slave'} at 127.0.0.1:59830
W20240430 20:11:31.307752 18557 leader_election.cc:341] T b688e03ea71940c69bda5520b62c1fc3 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:31.309128 18762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "294644d7a16b4ca2bbb6d106bcb8c7ff" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
W20240430 20:11:31.309974 18557 leader_election.cc:341] T 294644d7a16b4ca2bbb6d106bcb8c7ff P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:31.310557 18738 tablet_service.cc:1514] Processing DeleteTablet for tablet e5111d7607dd42188a0176cf5be2ec05 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:29 UTC) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:31.312252 18738 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e503a898406412ab977d3e048ea4b5d with delete_type TABLET_DATA_DELETED (replaced by tablet 5bd0362ff41b4faca0205d136d596737) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:31.312708 18738 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f56ac2afd444d63a6bfd96988464bbb with delete_type TABLET_DATA_DELETED (replaced by tablet 2da900b6bbfa4fa99fe8494fd2e68da8) from {username='slave'} at 127.0.0.1:59830
W20240430 20:11:31.314481 18417 leader_election.cc:341] T b76a4105a2974fe280e759349037d027 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:31.314777 18985 raft_consensus.cc:374] T 12c6e131fa904d23865b8673b58a7e33 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20240430 20:11:31.324631 18415 leader_election.cc:341] T 68ca48351cd4412381f5195d81109016 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Illegal state: Tablet not RUNNING: STOPPED
I20240430 20:11:31.325641 19489 ts_tablet_manager.cc:1888] T 68ca48351cd4412381f5195d81109016 P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:31.325888 19489 log.cc:1196] T 68ca48351cd4412381f5195d81109016 P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/68ca48351cd4412381f5195d81109016
I20240430 20:11:31.326457 19489 ts_tablet_manager.cc:1909] T 68ca48351cd4412381f5195d81109016 P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:31.327256 19001 catalog_manager.cc:4830] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): tablet 68ca48351cd4412381f5195d81109016 (table default.Table_172b295132814722b733e25f446730a7 [id=7168f3d7e00348ce8fa33100c98652f4]) successfully deleted
W20240430 20:11:31.328476 19001 catalog_manager.cc:4810] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): delete failed for tablet 47c37876694c4a4abb1091c631dab7a5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 47c37876694c4a4abb1091c631dab7a5 already in progress: creating tablet
I20240430 20:11:31.328986 19489 tablet_replica.cc:330] T 1398d4d2305b48e08fdb12432bd6c4b0 P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:31.329411 19489 raft_consensus.cc:2229] T 1398d4d2305b48e08fdb12432bd6c4b0 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:31.329728 19489 raft_consensus.cc:2258] T 1398d4d2305b48e08fdb12432bd6c4b0 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:31.332031 19489 ts_tablet_manager.cc:1875] T 1398d4d2305b48e08fdb12432bd6c4b0 P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:31.318042 19482 consensus_queue.cc:237] T 724e51ff05ec4baebc2bb55e05c8eddc P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:31.336498 18417 leader_election.cc:302] T b76a4105a2974fe280e759349037d027 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a; no voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710
I20240430 20:11:31.336966 19618 raft_consensus.cc:2726] T b76a4105a2974fe280e759349037d027 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240430 20:11:31.337496 18417 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.177.3:44867, user_credentials={real_user=slave}} blocked reactor thread for 46885us
W20240430 20:11:31.337906 18417 leader_election.cc:341] T 20d166756819406ba85044f4fee9c4ce P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:31.339069 18417 leader_election.cc:302] T 20d166756819406ba85044f4fee9c4ce P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a; no voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710
I20240430 20:11:31.339412 19618 raft_consensus.cc:2726] T 20d166756819406ba85044f4fee9c4ce P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240430 20:11:31.340651 18417 leader_election.cc:341] T 68ca48351cd4412381f5195d81109016 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: Tablet not RUNNING: STOPPED
I20240430 20:11:31.342197 18758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a05e32a33aa7488eb287cc6314c09c44" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:31.344115 19625 tablet.cc:1722] T 680553d4d0bc4fcb839235c1832757e1 P 3c51c33d5616482cafb68e26688820f4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240430 20:11:31.344426 18983 tablet_bootstrap.cc:654] T c9e7660eccc448dbb99d135f82bfd261 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:31.341009 18985 raft_consensus.cc:729] T 12c6e131fa904d23865b8673b58a7e33 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:31.343358 18463 tablet_service.cc:1514] Processing DeleteTablet for tablet 5bde72e2791a497db7dd08686ca36d80 with delete_type TABLET_DATA_DELETED (replaced by tablet 66dd767bd3f8494f94061fb89e5e5680) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:31.346406 19631 tablet.cc:1722] T 372b135dd01f44f98cd713ca8857ba61 P 3c51c33d5616482cafb68e26688820f4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240430 20:11:31.353331 19471 ts_tablet_manager.cc:1888] T aacabead3c874e77a47d2ecf19dbfa82 P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:31.353495 19471 log.cc:1196] T aacabead3c874e77a47d2ecf19dbfa82 P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/aacabead3c874e77a47d2ecf19dbfa82
I20240430 20:11:31.353981 19471 ts_tablet_manager.cc:1909] T aacabead3c874e77a47d2ecf19dbfa82 P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:31.355299 19624 tablet.cc:1722] T 372b135dd01f44f98cd713ca8857ba61 P 5e1c6397f09747f091534faf93542710: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240430 20:11:31.355996 18999 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet 68ca48351cd4412381f5195d81109016 (table default.Table_172b295132814722b733e25f446730a7 [id=7168f3d7e00348ce8fa33100c98652f4]) successfully deleted
W20240430 20:11:31.356297 18999 catalog_manager.cc:4810] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): delete failed for tablet f5546f9edeea44d7a6c5cb954f5157ba because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f5546f9edeea44d7a6c5cb954f5157ba already in progress: creating tablet
W20240430 20:11:31.356565 18999 catalog_manager.cc:4797] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): delete failed for tablet f77086a5675f44e99e8a867b694df4b8 because the tablet was not found. No further retry: Not found: Tablet not found: f77086a5675f44e99e8a867b694df4b8
W20240430 20:11:31.356837 18999 catalog_manager.cc:4810] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): delete failed for tablet 2cdccf4be7094ca5b863dd5c3432d55d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2cdccf4be7094ca5b863dd5c3432d55d already in progress: creating tablet
I20240430 20:11:31.357290 18985 consensus_queue.cc:260] T 12c6e131fa904d23865b8673b58a7e33 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:31.352510 18479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1555924b5f1444398f12ee82c3cb5b13" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
W20240430 20:11:31.361810 18558 leader_election.cc:341] T 1555924b5f1444398f12ee82c3cb5b13 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:31.362046 18558 leader_election.cc:302] T 1555924b5f1444398f12ee82c3cb5b13 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:31.362757 19471 tablet_replica.cc:330] T c5fd0730133c472299c93f4b0830e527 P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:31.363025 19180 raft_consensus.cc:2726] T 1555924b5f1444398f12ee82c3cb5b13 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:31.363071 19471 raft_consensus.cc:2229] T c5fd0730133c472299c93f4b0830e527 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:31.363497 19471 raft_consensus.cc:2258] T c5fd0730133c472299c93f4b0830e527 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:31.364323 19471 ts_tablet_manager.cc:1875] T c5fd0730133c472299c93f4b0830e527 P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:31.370736 18999 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet aacabead3c874e77a47d2ecf19dbfa82 (table default.Table_fda07fd7279c484db2bee67e8c859855 [id=da9ad991953a40c4b77b31317597ad1d]) successfully deleted
I20240430 20:11:31.373324 18417 leader_election.cc:302] T 68ca48351cd4412381f5195d81109016 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a; no voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710
I20240430 20:11:31.356843 18478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b688e03ea71940c69bda5520b62c1fc3" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:31.374919 18983 tablet_bootstrap.cc:492] T c9e7660eccc448dbb99d135f82bfd261 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:31.358290 18477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "294644d7a16b4ca2bbb6d106bcb8c7ff" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:31.375690 18983 ts_tablet_manager.cc:1367] T c9e7660eccc448dbb99d135f82bfd261 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.074s user 0.002s sys 0.010s
I20240430 20:11:31.376561 18983 raft_consensus.cc:348] T c9e7660eccc448dbb99d135f82bfd261 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:31.376706 18985 ts_tablet_manager.cc:1398] T 12c6e131fa904d23865b8673b58a7e33 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.187s user 0.003s sys 0.000s
I20240430 20:11:31.376804 18983 raft_consensus.cc:374] T c9e7660eccc448dbb99d135f82bfd261 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:31.376888 18983 raft_consensus.cc:729] T c9e7660eccc448dbb99d135f82bfd261 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:31.377190 18983 consensus_queue.cc:260] T c9e7660eccc448dbb99d135f82bfd261 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:31.378630 18529 heartbeater.cc:503] Master 127.17.177.60:37225 was elected leader, sending a full tablet report...
W20240430 20:11:31.379274 18558 leader_election.cc:341] T b688e03ea71940c69bda5520b62c1fc3 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:31.379433 18558 leader_election.cc:302] T b688e03ea71940c69bda5520b62c1fc3 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4
W20240430 20:11:31.379848 18558 leader_election.cc:341] T 294644d7a16b4ca2bbb6d106bcb8c7ff P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:31.379987 18558 leader_election.cc:302] T 294644d7a16b4ca2bbb6d106bcb8c7ff P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:31.380721 19386 raft_consensus.cc:2726] T b688e03ea71940c69bda5520b62c1fc3 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:31.380987 19386 raft_consensus.cc:2726] T 294644d7a16b4ca2bbb6d106bcb8c7ff P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:31.382170 19629 tablet.cc:1722] T fae9a35a2ac844b4bc2ebdf214aa7171 P 5e1c6397f09747f091534faf93542710: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20240430 20:11:31.398954 18988 tablet_bootstrap.cc:492] T b76a4105a2974fe280e759349037d027 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:31.399153 18988 ts_tablet_manager.cc:1367] T b76a4105a2974fe280e759349037d027 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.355s user 0.008s sys 0.008s
I20240430 20:11:31.399926 18988 raft_consensus.cc:348] T b76a4105a2974fe280e759349037d027 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:31.400156 18988 raft_consensus.cc:374] T b76a4105a2974fe280e759349037d027 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:31.400259 18988 raft_consensus.cc:729] T b76a4105a2974fe280e759349037d027 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:31.400489 18988 consensus_queue.cc:260] T b76a4105a2974fe280e759349037d027 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:31.401063 18988 ts_tablet_manager.cc:1398] T b76a4105a2974fe280e759349037d027 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240430 20:11:31.401357 18988 tablet_bootstrap.cc:492] T 294644d7a16b4ca2bbb6d106bcb8c7ff P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:31.402276 19180 raft_consensus.cc:482] T 01436e5b6232494babbb414342f03090 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:31.402451 19180 raft_consensus.cc:504] T 01436e5b6232494babbb414342f03090 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:31.403278 19180 leader_election.cc:288] T 01436e5b6232494babbb414342f03090 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:31.403355 18738 tablet_service.cc:1467] Processing CreateTablet for tablet 99203cc522de424184c6f97453f166fd (DEFAULT_TABLE table=default.Table_9f4b05e2470349c7bbc8c4ae15e778b2 [id=3f8690a437ae49a3beb009fa7899d5d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:31.403932 18738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99203cc522de424184c6f97453f166fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:31.411206 19629 tablet.cc:1722] T aec9c09cc0454acab6006bb3e10982b7 P 5e1c6397f09747f091534faf93542710: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240430 20:11:31.416214 18988 tablet_bootstrap.cc:654] T 294644d7a16b4ca2bbb6d106bcb8c7ff P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:31.421299 19489 ts_tablet_manager.cc:1888] T 1398d4d2305b48e08fdb12432bd6c4b0 P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:31.421490 19489 log.cc:1196] T 1398d4d2305b48e08fdb12432bd6c4b0 P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/1398d4d2305b48e08fdb12432bd6c4b0
I20240430 20:11:31.421969 19489 ts_tablet_manager.cc:1909] T 1398d4d2305b48e08fdb12432bd6c4b0 P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:31.423197 19001 catalog_manager.cc:4830] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): tablet 1398d4d2305b48e08fdb12432bd6c4b0 (table default.Table_eea9acffa45b4f3ab6d4524dc460853c [id=52cd3a55893547cabf418f4d01b11d08]) successfully deleted
W20240430 20:11:31.423219 19000 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet a05e32a33aa7488eb287cc6314c09c44 (table default.Table_6551367971354d3195e4f7d47364ce7c [id=955d833ff8f14bf6957abadf9ef669e2], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet a05e32a33aa7488eb287cc6314c09c44 consensus state has no leader
I20240430 20:11:31.428293 18591 tablet_service.cc:1467] Processing CreateTablet for tablet 99203cc522de424184c6f97453f166fd (DEFAULT_TABLE table=default.Table_9f4b05e2470349c7bbc8c4ae15e778b2 [id=3f8690a437ae49a3beb009fa7899d5d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:31.428854 18591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99203cc522de424184c6f97453f166fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:31.428921 18759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01436e5b6232494babbb414342f03090" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
W20240430 20:11:31.429572 18557 leader_election.cc:341] T 01436e5b6232494babbb414342f03090 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:31.429903 19489 tablet_replica.cc:330] T a1aeee1b02214975a376cfd36957afe8 P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:31.430135 19489 raft_consensus.cc:2229] T a1aeee1b02214975a376cfd36957afe8 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:31.430349 19489 raft_consensus.cc:2258] T a1aeee1b02214975a376cfd36957afe8 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:31.431052 19489 ts_tablet_manager.cc:1875] T a1aeee1b02214975a376cfd36957afe8 P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:31.431672 19027 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet acd23f0a63d5448aaa7ed78422e67bad (table default.Table_972f13be4dca4cd19b9a04534dc59f51 [id=69f37db4de634a3185613e40013e0ed7]). Expected version 1 got 0
I20240430 20:11:31.434549 19027 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 24dcc20832804229a2664873bf6ddaa7 (table default.Table_be2ebe77f859489f9cd3e912cb1edc8b [id=2d44ec94131148048021b2ee6366ba30]). Expected version 2 got 0
I20240430 20:11:31.435099 19625 tablet.cc:1722] T fae9a35a2ac844b4bc2ebdf214aa7171 P 3c51c33d5616482cafb68e26688820f4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20240430 20:11:31.437047 18983 ts_tablet_manager.cc:1398] T c9e7660eccc448dbb99d135f82bfd261 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.056s user 0.003s sys 0.000s
I20240430 20:11:31.439795 19027 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet a05e32a33aa7488eb287cc6314c09c44 (table default.Table_6551367971354d3195e4f7d47364ce7c [id=955d833ff8f14bf6957abadf9ef669e2]). Expected version 1 got 0
I20240430 20:11:31.442478 18985 tablet_bootstrap.cc:492] T 1555924b5f1444398f12ee82c3cb5b13 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:31.445751 18983 tablet_bootstrap.cc:492] T 802336602d8f4ac4bd7e0872d6f3dae9 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:31.447959 18983 tablet_bootstrap.cc:654] T 802336602d8f4ac4bd7e0872d6f3dae9 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:31.451056 19628 tablet.cc:1722] T 6e826459c3f4437da81c9e6e09f0c22f P 2efb542aa55e439b99ccb98d1c5c884a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20240430 20:11:31.451463 18415 consensus_peers.cc:485] T 1398d4d2305b48e08fdb12432bd6c4b0 P 2efb542aa55e439b99ccb98d1c5c884a -> Peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Couldn't send request to peer 5e1c6397f09747f091534faf93542710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1398d4d2305b48e08fdb12432bd6c4b0. This is attempt 1: this message will repeat every 5th retry.
I20240430 20:11:31.453891 19027 catalog_manager.cc:5484] T 0b4d9e860d7c45fd91d7a566605c9283 P 3c51c33d5616482cafb68e26688820f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c51c33d5616482cafb68e26688820f4 (127.17.177.3). New cstate: current_term: 1 leader_uuid: "3c51c33d5616482cafb68e26688820f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } health_report { overall_health: HEALTHY } } }
I20240430 20:11:31.454465 19027 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 0b4d9e860d7c45fd91d7a566605c9283 (table default.Table_421069f84cb844368066c93102a8faf8 [id=d5cea90cbc2c48df9d8b80da1eccc949]). Expected version 2 got 0
I20240430 20:11:31.436158 19026 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: Sending DeleteTablet for 3 replicas of tablet 5bd0362ff41b4faca0205d136d596737
I20240430 20:11:31.456081 19026 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_6f2fd99337344da2ac5a0c83d9b56bb8" } new_table_name: "default.Table_e34613952dcb46618c3bc7cecb31fdaf" modify_external_catalogs: true
I20240430 20:11:31.458417 18738 tablet_service.cc:1514] Processing DeleteTablet for tablet 5bd0362ff41b4faca0205d136d596737 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:31 UTC) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:31.459663 18481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01436e5b6232494babbb414342f03090" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
W20240430 20:11:31.460031 18417 raft_consensus.cc:2670] T 68ca48351cd4412381f5195d81109016 P 2efb542aa55e439b99ccb98d1c5c884a: unable to run election callback: Service unavailable: Thread pool token was shut down
I20240430 20:11:31.459926 19636 tablet.cc:1722] T aec9c09cc0454acab6006bb3e10982b7 P 2efb542aa55e439b99ccb98d1c5c884a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20240430 20:11:31.460255 18417 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.177.3:44867, user_credentials={real_user=slave}} blocked reactor thread for 119639us
W20240430 20:11:31.460603 18417 leader_election.cc:341] T a05e32a33aa7488eb287cc6314c09c44 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:31.460738 18417 leader_election.cc:302] T a05e32a33aa7488eb287cc6314c09c44 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a; no voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710
I20240430 20:11:31.462332 18601 tablet_service.cc:1514] Processing DeleteTablet for tablet 5bd0362ff41b4faca0205d136d596737 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:31 UTC) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:31.463918 19615 raft_consensus.cc:2726] T a05e32a33aa7488eb287cc6314c09c44 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:31.465456 19386 raft_consensus.cc:482] T 20d166756819406ba85044f4fee9c4ce P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:31.466213 19386 raft_consensus.cc:504] T 20d166756819406ba85044f4fee9c4ce P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:31.467020 19386 leader_election.cc:288] T 20d166756819406ba85044f4fee9c4ce P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:31.467906 18759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20d166756819406ba85044f4fee9c4ce" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
W20240430 20:11:31.468534 18557 leader_election.cc:341] T 20d166756819406ba85044f4fee9c4ce P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
W20240430 20:11:31.471388 18696 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.177.2:46027, user_credentials={real_user=slave}} blocked reactor thread for 48782.8us
I20240430 20:11:31.473822 19615 raft_consensus.cc:482] T eed6338ad3d64859af087ee15e1ec798 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:31.474006 19615 raft_consensus.cc:504] T eed6338ad3d64859af087ee15e1ec798 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:31.474781 19625 tablet.cc:1722] T 77285afcafe04a829e485998bbbe00f0 P 3c51c33d5616482cafb68e26688820f4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240430 20:11:31.474812 19615 leader_election.cc:288] T eed6338ad3d64859af087ee15e1ec798 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:31.475821 18623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eed6338ad3d64859af087ee15e1ec798" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:31.476097 18463 tablet_service.cc:1467] Processing CreateTablet for tablet 99203cc522de424184c6f97453f166fd (DEFAULT_TABLE table=default.Table_9f4b05e2470349c7bbc8c4ae15e778b2 [id=3f8690a437ae49a3beb009fa7899d5d0]), partition=RANGE (key) PARTITION UNBOUNDED
W20240430 20:11:31.476434 18415 leader_election.cc:341] T eed6338ad3d64859af087ee15e1ec798 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:31.476586 18463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99203cc522de424184c6f97453f166fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:31.476691 19631 tablet.cc:1722] T aec9c09cc0454acab6006bb3e10982b7 P 3c51c33d5616482cafb68e26688820f4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240430 20:11:31.479302 18759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eed6338ad3d64859af087ee15e1ec798" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:31.480067 18810 heartbeater.cc:503] Master 127.17.177.60:37225 was elected leader, sending a full tablet report...
W20240430 20:11:31.490397 18558 leader_election.cc:341] T 01436e5b6232494babbb414342f03090 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:31.490617 18558 leader_election.cc:302] T 01436e5b6232494babbb414342f03090 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:31.490849 19629 tablet.cc:1722] T 6e826459c3f4437da81c9e6e09f0c22f P 5e1c6397f09747f091534faf93542710: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240430 20:11:31.491572 19230 raft_consensus.cc:2726] T 01436e5b6232494babbb414342f03090 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:31.489897 19471 ts_tablet_manager.cc:1888] T c5fd0730133c472299c93f4b0830e527 P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:31.493136 19471 log.cc:1196] T c5fd0730133c472299c93f4b0830e527 P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/c5fd0730133c472299c93f4b0830e527
I20240430 20:11:31.493280 19027 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 372b135dd01f44f98cd713ca8857ba61 (table default.Table_ee86b8fcf2064d8b8c3e327e357d3395 [id=7f3a15a442984827a7521d5d1c98dc3d]). Expected version 2 got 0
I20240430 20:11:31.493634 19027 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet fd40967ab4a04001819e16c06a6c4300 (table default.Table_fda07fd7279c484db2bee67e8c859855 [id=da9ad991953a40c4b77b31317597ad1d]). Expected version 3 got 0
I20240430 20:11:31.493968 19027 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 680553d4d0bc4fcb839235c1832757e1 (table default.Table_49a012a7c4c8457fadb0a8f0fc9654c6 [id=eecb2f8eeef547c5ab1abe6a2430e592]). Expected version 1 got 0
I20240430 20:11:31.494001 18481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20d166756819406ba85044f4fee9c4ce" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:31.494377 19027 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 76d6e9d547ca4a848eae852944b1d117 (table default.Table_104ae015924a48fd8d27220792f07470 [id=ff86c0357c164dd0991098c416087773]). Expected version 2 got 0
I20240430 20:11:31.494912 19027 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 6e754af64cb64fd2a082adae7d5f24da (table default.Table_208b12b8e9bd4f3e8bc9c6d422ada649 [id=1df4e95702cd46f59ab21b101049a9ed]). Expected version 1 got 0
I20240430 20:11:31.495622 19027 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 34130f2820064fa0898ef1f9d7de1c6d (table default.Table_4537f891b01048ceb51202abc3600911 [id=5a1c4a7be2fc465fac89a50854eb3322]). Expected version 1 got 0
I20240430 20:11:31.496385 19027 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 32c823d1700949ab8b22adcd83b7bfd1 (table default.Table_e8c73c89623d4b49b8d7f0a01d729bdd [id=5cd10682ce52414cb5a103e4433a6a0d]). Expected version 1 got 0
I20240430 20:11:31.496762 19027 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 6e826459c3f4437da81c9e6e09f0c22f (table default.Table_873c21f6dffd4d47a742b3699ab85268 [id=d7e2c4300ff94fa2acc65d6d6c932c65]). Expected version 1 got 0
I20240430 20:11:31.531287 18454 tablet_service.cc:1514] Processing DeleteTablet for tablet 5bd0362ff41b4faca0205d136d596737 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:31 UTC) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:31.535842 19631 tablet.cc:1722] T 6e826459c3f4437da81c9e6e09f0c22f P 3c51c33d5616482cafb68e26688820f4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240430 20:11:31.537195 18988 tablet_bootstrap.cc:492] T 294644d7a16b4ca2bbb6d106bcb8c7ff P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:31.538416 19489 ts_tablet_manager.cc:1888] T a1aeee1b02214975a376cfd36957afe8 P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:31.540848 18988 ts_tablet_manager.cc:1367] T 294644d7a16b4ca2bbb6d106bcb8c7ff P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.139s user 0.003s sys 0.002s
I20240430 20:11:31.541776 19489 log.cc:1196] T a1aeee1b02214975a376cfd36957afe8 P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/a1aeee1b02214975a376cfd36957afe8
I20240430 20:11:31.544008 19489 ts_tablet_manager.cc:1909] T a1aeee1b02214975a376cfd36957afe8 P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:31.548805 19489 tablet_replica.cc:330] T 02176eee8e044a7db8cac48439e9f594 P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:31.549154 19489 raft_consensus.cc:2229] T 02176eee8e044a7db8cac48439e9f594 P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Raft consensus shutting down.
I20240430 20:11:31.535915 19636 tablet.cc:1722] T 680553d4d0bc4fcb839235c1832757e1 P 2efb542aa55e439b99ccb98d1c5c884a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240430 20:11:31.550380 19489 raft_consensus.cc:2258] T 02176eee8e044a7db8cac48439e9f594 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:31.552091 19489 ts_tablet_manager.cc:1875] T 02176eee8e044a7db8cac48439e9f594 P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240430 20:11:31.555238 18417 consensus_peers.cc:485] T 1398d4d2305b48e08fdb12432bd6c4b0 P 2efb542aa55e439b99ccb98d1c5c884a -> Peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Couldn't send request to peer 3c51c33d5616482cafb68e26688820f4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1398d4d2305b48e08fdb12432bd6c4b0. This is attempt 1: this message will repeat every 5th retry.
I20240430 20:11:31.555313 18481 raft_consensus.cc:2454] T 20d166756819406ba85044f4fee9c4ce P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 0.
W20240430 20:11:31.555423 18417 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.177.3:44867, user_credentials={real_user=slave}} blocked reactor thread for 75148.8us
I20240430 20:11:31.550873 18988 raft_consensus.cc:348] T 294644d7a16b4ca2bbb6d106bcb8c7ff P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:31.555562 19646 tablet.cc:1722] T fae9a35a2ac844b4bc2ebdf214aa7171 P 2efb542aa55e439b99ccb98d1c5c884a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
W20240430 20:11:31.556006 18417 leader_election.cc:341] T eed6338ad3d64859af087ee15e1ec798 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:31.556154 18417 leader_election.cc:302] T eed6338ad3d64859af087ee15e1ec798 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a; no voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710
I20240430 20:11:31.555260 18985 tablet_bootstrap.cc:654] T 1555924b5f1444398f12ee82c3cb5b13 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:31.555740 18988 raft_consensus.cc:374] T 294644d7a16b4ca2bbb6d106bcb8c7ff P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:31.557143 18988 raft_consensus.cc:729] T 294644d7a16b4ca2bbb6d106bcb8c7ff P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:31.557467 19617 raft_consensus.cc:2726] T eed6338ad3d64859af087ee15e1ec798 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:31.548523 19001 catalog_manager.cc:4830] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): tablet a1aeee1b02214975a376cfd36957afe8 (table default.Table_31d8d978e96c4e12a0fbd5500c7b93cd [id=8b480956f4d146e08dcfed0d0a0e4e0d]) successfully deleted
I20240430 20:11:31.561926 19027 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet b72fe1f0713b4076a81a3fc1c74f7fa7 (table default.Table_16b082d140234e1c9a1e70654839b4cd [id=c0d8dc87d1f9426a85f6176f161595a1]). Expected version 3 got 0
I20240430 20:11:31.562187 19027 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 53a899fa5a424a0d99f1b47925e8c7ad (table default.Table_deadf728d74a462fb79641891dfedd9c [id=608890834da843fa9af0cfc0d548029e]). Expected version 1 got 0
I20240430 20:11:31.562673 19027 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet fcc9698abda4491b98e1e879589913a8 (table default.Table_e431f94220aa44ecba799a54dfb612ae [id=1e785b6c159b4312bd1b064271f55a49]). Expected version 1 got 0
I20240430 20:11:31.562880 19027 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet fae9a35a2ac844b4bc2ebdf214aa7171 (table default.Table_0ab5b2f09fa7468bb89ab703f6d3d25c [id=a5d580cffa2447d78fc9f2c8e31264f1]). Expected version 4 got 0
I20240430 20:11:31.563745 19027 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 724e51ff05ec4baebc2bb55e05c8eddc (table default.Table_0c5d4686f2ed40fb8d62c31b0166dc05 [id=4dce8a767f0541e1beb5464b2ed95d06]). Expected version 1 got 0
I20240430 20:11:31.563943 19027 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet c66c142419ca4e1399ecd1c8d1e62a59 (table default.Table_0443359dbeff437fb6cffb0db33cbb30 [id=93c44abdf6b147228224388398cd7aeb]). Expected version 2 got 0
I20240430 20:11:31.564234 19027 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 294644d7a16b4ca2bbb6d106bcb8c7ff (table default.Table_190b2d86515a44898fa38a49517b9df1 [id=97d8a36ddfae4bb5b99c04a7a08544af]). Expected version 2 got 0
I20240430 20:11:31.564455 19027 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet aec9c09cc0454acab6006bb3e10982b7 (table default.Table_e68dc52988f24f1788517de94d7d2a0e [id=26d4860fb33047b7b812ebd4309f2e28]). Expected version 1 got 0
I20240430 20:11:31.564862 19051 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: Sending DeleteTablet for 3 replicas of tablet 77285afcafe04a829e485998bbbe00f0
I20240430 20:11:31.557482 18988 consensus_queue.cc:260] T 294644d7a16b4ca2bbb6d106bcb8c7ff P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:31.567121 18988 ts_tablet_manager.cc:1398] T 294644d7a16b4ca2bbb6d106bcb8c7ff P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.026s user 0.002s sys 0.000s
I20240430 20:11:31.567341 19026 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_e233554a5e1d41389948fecaf498a856" } new_table_name: "default.Table_685a26737bb046fb85ced15df7ae3423" modify_external_catalogs: true
I20240430 20:11:31.567466 18988 tablet_bootstrap.cc:492] T 53a899fa5a424a0d99f1b47925e8c7ad P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:31.504348 18983 tablet_bootstrap.cc:492] T 802336602d8f4ac4bd7e0872d6f3dae9 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:31.569944 18983 ts_tablet_manager.cc:1367] T 802336602d8f4ac4bd7e0872d6f3dae9 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.126s user 0.004s sys 0.004s
I20240430 20:11:31.570391 19471 ts_tablet_manager.cc:1909] T c5fd0730133c472299c93f4b0830e527 P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:31.570780 18983 raft_consensus.cc:348] T 802336602d8f4ac4bd7e0872d6f3dae9 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:31.571002 18983 raft_consensus.cc:374] T 802336602d8f4ac4bd7e0872d6f3dae9 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:31.571076 18983 raft_consensus.cc:729] T 802336602d8f4ac4bd7e0872d6f3dae9 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:31.571292 18983 consensus_queue.cc:260] T 802336602d8f4ac4bd7e0872d6f3dae9 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:31.571749 18983 ts_tablet_manager.cc:1398] T 802336602d8f4ac4bd7e0872d6f3dae9 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240430 20:11:31.572058 18983 tablet_bootstrap.cc:492] T 12c6e131fa904d23865b8673b58a7e33 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:31.573609 19048 catalog_manager.cc:6029] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: default.Table_88c44b20f2ce4eeca7ed9583a70c7c65 [id=73caec5fb65e4cb090e7ae8c391501fd] alter complete (version 1)
I20240430 20:11:31.574044 19471 tablet_replica.cc:330] T d9e578c99058424a9b86390641f9c34f P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:31.574106 19028 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: Sending DeleteTablet for 3 replicas of tablet 802336602d8f4ac4bd7e0872d6f3dae9
I20240430 20:11:31.574898 19028 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_0604805f009c4d6486965426c82dff10"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:31.575520 19028 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0604805f009c4d6486965426c82dff10 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240430 20:11:31.575832 19024 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a59ef6c545dd41a28265a4c1b8ce10e0 consensus state has no leader
I20240430 20:11:31.515200 19645 consensus_queue.cc:1035] T c66c142419ca4e1399ecd1c8d1e62a59 P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:31.576594 18463 tablet_service.cc:1514] Processing DeleteTablet for tablet 77285afcafe04a829e485998bbbe00f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:31 UTC) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:31.560319 18669 heartbeater.cc:503] Master 127.17.177.60:37225 was elected leader, sending a full tablet report...
I20240430 20:11:31.578516 18988 tablet_bootstrap.cc:654] T 53a899fa5a424a0d99f1b47925e8c7ad P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:31.563104 18558 leader_election.cc:302] T 20d166756819406ba85044f4fee9c4ce P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters: 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:31.580456 19386 raft_consensus.cc:2781] T 20d166756819406ba85044f4fee9c4ce P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:31.580579 19386 raft_consensus.cc:482] T 20d166756819406ba85044f4fee9c4ce P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:31.580722 19386 raft_consensus.cc:3037] T 20d166756819406ba85044f4fee9c4ce P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:31.582090 18454 tablet_service.cc:1514] Processing DeleteTablet for tablet 802336602d8f4ac4bd7e0872d6f3dae9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:31 UTC) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:31.582542 19180 consensus_queue.cc:1035] T c66c142419ca4e1399ecd1c8d1e62a59 P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:31.584436 19471 raft_consensus.cc:2229] T d9e578c99058424a9b86390641f9c34f P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Raft consensus shutting down.
I20240430 20:11:31.584936 19386 raft_consensus.cc:504] T 20d166756819406ba85044f4fee9c4ce P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:31.585186 19471 raft_consensus.cc:2258] T d9e578c99058424a9b86390641f9c34f P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:31.585549 18999 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet c5fd0730133c472299c93f4b0830e527 (table default.Table_ea83912975774b6089fd85c252f45ea5 [id=99cbf7b92c7943838fce9f3ba42babde]) successfully deleted
I20240430 20:11:31.585803 19471 ts_tablet_manager.cc:1875] T d9e578c99058424a9b86390641f9c34f P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:31.586074 19386 leader_election.cc:288] T 20d166756819406ba85044f4fee9c4ce P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:31.586057 18759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20d166756819406ba85044f4fee9c4ce" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:31.586800 18738 tablet_service.cc:1514] Processing DeleteTablet for tablet 77285afcafe04a829e485998bbbe00f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:31 UTC) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:31.587024 18726 tablet_service.cc:1514] Processing DeleteTablet for tablet 802336602d8f4ac4bd7e0872d6f3dae9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:31 UTC) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:31.587430 18983 tablet_bootstrap.cc:654] T 12c6e131fa904d23865b8673b58a7e33 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
W20240430 20:11:31.587548 18557 leader_election.cc:341] T 20d166756819406ba85044f4fee9c4ce P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
W20240430 20:11:31.591590 19029 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a05e32a33aa7488eb287cc6314c09c44 consensus state has no leader
I20240430 20:11:31.594316 19489 ts_tablet_manager.cc:1888] T 02176eee8e044a7db8cac48439e9f594 P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:31.594483 19489 log.cc:1196] T 02176eee8e044a7db8cac48439e9f594 P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/02176eee8e044a7db8cac48439e9f594
I20240430 20:11:31.597404 19024 master_service.cc:946] ReplaceTablet: received request to replace tablet 7dfdecba959543a795dfcd3eeda29e58 from {username='slave'} at 127.0.0.1:51664
I20240430 20:11:31.608754 18480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20d166756819406ba85044f4fee9c4ce" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:31.609117 18480 raft_consensus.cc:3037] T 20d166756819406ba85044f4fee9c4ce P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:31.611152 19028 master_service.cc:946] ReplaceTablet: received request to replace tablet 26ed12f498644d15bf45e884dcad172e from {username='slave'} at 127.0.0.1:51664
I20240430 20:11:31.612419 18480 raft_consensus.cc:2454] T 20d166756819406ba85044f4fee9c4ce P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
I20240430 20:11:31.612725 19628 tablet.cc:1722] T 372b135dd01f44f98cd713ca8857ba61 P 2efb542aa55e439b99ccb98d1c5c884a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240430 20:11:31.613178 19495 ts_tablet_manager.cc:1888] T 68ca48351cd4412381f5195d81109016 P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:31.613313 19495 log.cc:1196] T 68ca48351cd4412381f5195d81109016 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/68ca48351cd4412381f5195d81109016
I20240430 20:11:31.613793 19495 ts_tablet_manager.cc:1909] T 68ca48351cd4412381f5195d81109016 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:31.615353 19004 catalog_manager.cc:4830] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): tablet 68ca48351cd4412381f5195d81109016 (table default.Table_172b295132814722b733e25f446730a7 [id=7168f3d7e00348ce8fa33100c98652f4]) successfully deleted
W20240430 20:11:31.615726 19004 catalog_manager.cc:4797] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): delete failed for tablet d4508e4f6dbd42109d951c648b3dc7df because the tablet was not found. No further retry: Not found: Tablet not found: d4508e4f6dbd42109d951c648b3dc7df
I20240430 20:11:31.615960 18988 tablet_bootstrap.cc:492] T 53a899fa5a424a0d99f1b47925e8c7ad P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:31.616138 18988 ts_tablet_manager.cc:1367] T 53a899fa5a424a0d99f1b47925e8c7ad P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.049s user 0.002s sys 0.005s
I20240430 20:11:31.616518 19495 tablet_replica.cc:330] T 1398d4d2305b48e08fdb12432bd6c4b0 P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:31.617048 18988 raft_consensus.cc:348] T 53a899fa5a424a0d99f1b47925e8c7ad P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:31.617267 18988 raft_consensus.cc:374] T 53a899fa5a424a0d99f1b47925e8c7ad P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:31.617357 18988 raft_consensus.cc:729] T 53a899fa5a424a0d99f1b47925e8c7ad P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:31.617410 18985 tablet_bootstrap.cc:492] T 1555924b5f1444398f12ee82c3cb5b13 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:31.617795 18985 ts_tablet_manager.cc:1367] T 1555924b5f1444398f12ee82c3cb5b13 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.241s user 0.014s sys 0.000s
I20240430 20:11:31.617594 19495 raft_consensus.cc:2229] T 1398d4d2305b48e08fdb12432bd6c4b0 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Raft consensus shutting down.
I20240430 20:11:31.619910 19489 ts_tablet_manager.cc:1909] T 02176eee8e044a7db8cac48439e9f594 P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:31.620471 19495 raft_consensus.cc:2258] T 1398d4d2305b48e08fdb12432bd6c4b0 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:31.624886 19489 tablet_replica.cc:330] T 93d158de64994f3c9fa45a1250a4f533 P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:31.624975 19001 catalog_manager.cc:4830] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): tablet 02176eee8e044a7db8cac48439e9f594 (table default.Table_3b327aece0ea4ec4b882f2264e3f5f0c [id=1fb10bddcfe94ec488ecc71e43ca339d]) successfully deleted
I20240430 20:11:31.625133 19489 raft_consensus.cc:2229] T 93d158de64994f3c9fa45a1250a4f533 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:31.625341 19489 raft_consensus.cc:2258] T 93d158de64994f3c9fa45a1250a4f533 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:31.625931 19489 ts_tablet_manager.cc:1875] T 93d158de64994f3c9fa45a1250a4f533 P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:31.626622 18985 raft_consensus.cc:348] T 1555924b5f1444398f12ee82c3cb5b13 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:31.627816 18985 raft_consensus.cc:374] T 1555924b5f1444398f12ee82c3cb5b13 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:31.627970 18985 raft_consensus.cc:729] T 1555924b5f1444398f12ee82c3cb5b13 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:31.627452 19495 ts_tablet_manager.cc:1875] T 1398d4d2305b48e08fdb12432bd6c4b0 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:31.632359 18988 consensus_queue.cc:260] T 53a899fa5a424a0d99f1b47925e8c7ad P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:31.636694 18988 ts_tablet_manager.cc:1398] T 53a899fa5a424a0d99f1b47925e8c7ad P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.020s user 0.001s sys 0.001s
I20240430 20:11:31.639981 18988 tablet_bootstrap.cc:492] T a05e32a33aa7488eb287cc6314c09c44 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:31.641819 19028 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: Sending DeleteTablet for 3 replicas of tablet 26ed12f498644d15bf45e884dcad172e
I20240430 20:11:31.642305 19028 catalog_manager.cc:6554] ReplaceTablet: tablet 26ed12f498644d15bf45e884dcad172e deleted and replaced by tablet c6378f1495144c458af49b3c19a2b67c
I20240430 20:11:31.642974 18454 tablet_service.cc:1514] Processing DeleteTablet for tablet 26ed12f498644d15bf45e884dcad172e with delete_type TABLET_DATA_DELETED (replaced by tablet c6378f1495144c458af49b3c19a2b67c) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:31.643177 18726 tablet_service.cc:1514] Processing DeleteTablet for tablet 26ed12f498644d15bf45e884dcad172e with delete_type TABLET_DATA_DELETED (replaced by tablet c6378f1495144c458af49b3c19a2b67c) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:31.644061 18985 consensus_queue.cc:260] T 1555924b5f1444398f12ee82c3cb5b13 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:31.644434 18988 tablet_bootstrap.cc:654] T a05e32a33aa7488eb287cc6314c09c44 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:31.645036 19028 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_772ce91c27ce4119b6d1b1993e417f21" } modify_external_catalogs: true
I20240430 20:11:31.645226 19028 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_772ce91c27ce4119b6d1b1993e417f21" } modify_external_catalogs: true
I20240430 20:11:31.645426 19631 tablet.cc:1722] T e2d90ff17e574f88b2061b1c117d4b7d P 3c51c33d5616482cafb68e26688820f4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240430 20:11:31.646239 19628 tablet.cc:1722] T e2d90ff17e574f88b2061b1c117d4b7d P 2efb542aa55e439b99ccb98d1c5c884a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240430 20:11:31.647609 18985 ts_tablet_manager.cc:1398] T 1555924b5f1444398f12ee82c3cb5b13 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.030s user 0.000s sys 0.002s
I20240430 20:11:31.648195 18985 tablet_bootstrap.cc:492] T e3e74e76ab2b4fe492ffcc1838656fd1 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:31.649955 18985 tablet_bootstrap.cc:654] T e3e74e76ab2b4fe492ffcc1838656fd1 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:31.650185 19024 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: Sending DeleteTablet for 3 replicas of tablet 7dfdecba959543a795dfcd3eeda29e58
I20240430 20:11:31.650832 19024 catalog_manager.cc:6554] ReplaceTablet: tablet 7dfdecba959543a795dfcd3eeda29e58 deleted and replaced by tablet 95383d9d5a234a4d9d841a8b114428b0
I20240430 20:11:31.651286 19024 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_f323a3bead0249b5a4a3cbb07a5363b8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:31.651885 19024 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f323a3bead0249b5a4a3cbb07a5363b8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:31.654284 18726 tablet_service.cc:1514] Processing DeleteTablet for tablet 7dfdecba959543a795dfcd3eeda29e58 with delete_type TABLET_DATA_DELETED (replaced by tablet 95383d9d5a234a4d9d841a8b114428b0) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:31.658999 18454 tablet_service.cc:1514] Processing DeleteTablet for tablet 7dfdecba959543a795dfcd3eeda29e58 with delete_type TABLET_DATA_DELETED (replaced by tablet 95383d9d5a234a4d9d841a8b114428b0) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:31.672600 19489 ts_tablet_manager.cc:1888] T 93d158de64994f3c9fa45a1250a4f533 P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:31.675915 19489 log.cc:1196] T 93d158de64994f3c9fa45a1250a4f533 P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/93d158de64994f3c9fa45a1250a4f533
I20240430 20:11:31.676688 19489 ts_tablet_manager.cc:1909] T 93d158de64994f3c9fa45a1250a4f533 P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:31.677593 18601 tablet_service.cc:1514] Processing DeleteTablet for tablet 802336602d8f4ac4bd7e0872d6f3dae9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:31 UTC) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:31.677561 18558 leader_election.cc:302] T 20d166756819406ba85044f4fee9c4ce P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters: 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:31.680857 19180 raft_consensus.cc:482] T dd9b893d48684cfea7c242b93ecba01f P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:31.681524 19180 raft_consensus.cc:504] T dd9b893d48684cfea7c242b93ecba01f P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:31.682284 19180 leader_election.cc:288] T dd9b893d48684cfea7c242b93ecba01f P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:31.682996 19645 raft_consensus.cc:2781] T 20d166756819406ba85044f4fee9c4ce P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:31.682994 18761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd9b893d48684cfea7c242b93ecba01f" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:31.678141 18988 tablet_bootstrap.cc:492] T a05e32a33aa7488eb287cc6314c09c44 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:31.683990 18988 ts_tablet_manager.cc:1367] T a05e32a33aa7488eb287cc6314c09c44 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.047s user 0.002s sys 0.002s
I20240430 20:11:31.684383 19645 raft_consensus.cc:686] T 20d166756819406ba85044f4fee9c4ce P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Becoming Leader. State: Replica: 5e1c6397f09747f091534faf93542710, State: Running, Role: LEADER
I20240430 20:11:31.684727 19645 consensus_queue.cc:237] T 20d166756819406ba85044f4fee9c4ce P 5e1c6397f09747f091534faf93542710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:31.685118 18988 raft_consensus.cc:348] T a05e32a33aa7488eb287cc6314c09c44 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:31.685418 18988 raft_consensus.cc:374] T a05e32a33aa7488eb287cc6314c09c44 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:31.685503 18988 raft_consensus.cc:729] T a05e32a33aa7488eb287cc6314c09c44 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:31.685874 18988 consensus_queue.cc:260] T a05e32a33aa7488eb287cc6314c09c44 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
W20240430 20:11:31.686204 18557 leader_election.cc:341] T dd9b893d48684cfea7c242b93ecba01f P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
W20240430 20:11:31.686460 18558 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.177.1:41217, user_credentials={real_user=slave}} blocked reactor thread for 68443.8us
I20240430 20:11:31.687197 18601 tablet_service.cc:1514] Processing DeleteTablet for tablet 26ed12f498644d15bf45e884dcad172e with delete_type TABLET_DATA_DELETED (replaced by tablet c6378f1495144c458af49b3c19a2b67c) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:31.688200 18480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd9b893d48684cfea7c242b93ecba01f" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:31.688760 18583 tablet_service.cc:1514] Processing DeleteTablet for tablet 7dfdecba959543a795dfcd3eeda29e58 with delete_type TABLET_DATA_DELETED (replaced by tablet 95383d9d5a234a4d9d841a8b114428b0) from {username='slave'} at 127.0.0.1:40218
W20240430 20:11:31.690984 18558 leader_election.cc:341] T dd9b893d48684cfea7c242b93ecba01f P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:31.691146 18558 leader_election.cc:302] T dd9b893d48684cfea7c242b93ecba01f P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:31.691430 19474 raft_consensus.cc:2726] T dd9b893d48684cfea7c242b93ecba01f P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:31.696321 18591 tablet_service.cc:1514] Processing DeleteTablet for tablet 77285afcafe04a829e485998bbbe00f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:31 UTC) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:31.696709 19624 tablet.cc:1722] T e2d90ff17e574f88b2061b1c117d4b7d P 5e1c6397f09747f091534faf93542710: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240430 20:11:31.698199 18988 ts_tablet_manager.cc:1398] T a05e32a33aa7488eb287cc6314c09c44 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.014s user 0.001s sys 0.001s
I20240430 20:11:31.703449 19648 consensus_queue.cc:1035] T 7dfdecba959543a795dfcd3eeda29e58 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:31.703742 18988 tablet_bootstrap.cc:492] T eed6338ad3d64859af087ee15e1ec798 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:31.705533 18988 tablet_bootstrap.cc:654] T eed6338ad3d64859af087ee15e1ec798 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:31.707016 19648 raft_consensus.cc:482] T 12c6e131fa904d23865b8673b58a7e33 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:31.707219 19648 raft_consensus.cc:504] T 12c6e131fa904d23865b8673b58a7e33 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:31.707974 19489 tablet_replica.cc:330] T c5fd0730133c472299c93f4b0830e527 P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:31.708035 19648 leader_election.cc:288] T 12c6e131fa904d23865b8673b58a7e33 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:31.708063 19024 master_service.cc:946] ReplaceTablet: received request to replace tablet fcc9698abda4491b98e1e879589913a8 from {username='slave'} at 127.0.0.1:51664
I20240430 20:11:31.708314 19489 raft_consensus.cc:2229] T c5fd0730133c472299c93f4b0830e527 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:31.708331 19001 catalog_manager.cc:4830] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): tablet 93d158de64994f3c9fa45a1250a4f533 (table default.Table_9d39d94262a7481aad7df7868a2aff26 [id=06b16d46a7484d0dbc7971f070d0f37e]) successfully deleted
I20240430 20:11:31.708572 19489 raft_consensus.cc:2258] T c5fd0730133c472299c93f4b0830e527 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240430 20:11:31.708846 19001 catalog_manager.cc:4810] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): delete failed for tablet 75ebf051383e42529dcb5f7dd0770a4b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 75ebf051383e42529dcb5f7dd0770a4b already in progress: creating tablet
W20240430 20:11:31.709158 19001 catalog_manager.cc:4797] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): delete failed for tablet d4508e4f6dbd42109d951c648b3dc7df because the tablet was not found. No further retry: Not found: Tablet not found: d4508e4f6dbd42109d951c648b3dc7df
I20240430 20:11:31.709239 19489 ts_tablet_manager.cc:1875] T c5fd0730133c472299c93f4b0830e527 P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:31.709175 18622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12c6e131fa904d23865b8673b58a7e33" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
W20240430 20:11:31.709730 18696 leader_election.cc:341] T 12c6e131fa904d23865b8673b58a7e33 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:31.710258 18477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12c6e131fa904d23865b8673b58a7e33" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:31.710518 18477 raft_consensus.cc:2454] T 12c6e131fa904d23865b8673b58a7e33 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:31.711128 18699 leader_election.cc:302] T 12c6e131fa904d23865b8673b58a7e33 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters: 5e1c6397f09747f091534faf93542710
I20240430 20:11:31.711663 19648 raft_consensus.cc:2781] T 12c6e131fa904d23865b8673b58a7e33 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:31.711788 19648 raft_consensus.cc:482] T 12c6e131fa904d23865b8673b58a7e33 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:31.711917 19648 raft_consensus.cc:3037] T 12c6e131fa904d23865b8673b58a7e33 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:31.715770 19648 raft_consensus.cc:504] T 12c6e131fa904d23865b8673b58a7e33 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:31.716816 19648 leader_election.cc:288] T 12c6e131fa904d23865b8673b58a7e33 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Requested vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:31.717363 18477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12c6e131fa904d23865b8673b58a7e33" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:31.717615 18477 raft_consensus.cc:3037] T 12c6e131fa904d23865b8673b58a7e33 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:31.717684 18622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12c6e131fa904d23865b8673b58a7e33" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:31.718889 18983 tablet_bootstrap.cc:492] T 12c6e131fa904d23865b8673b58a7e33 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:31.719060 18983 ts_tablet_manager.cc:1367] T 12c6e131fa904d23865b8673b58a7e33 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.147s user 0.003s sys 0.009s
I20240430 20:11:31.719957 18983 raft_consensus.cc:348] T 12c6e131fa904d23865b8673b58a7e33 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:31.720185 18983 raft_consensus.cc:374] T 12c6e131fa904d23865b8673b58a7e33 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:31.720242 18983 raft_consensus.cc:729] T 12c6e131fa904d23865b8673b58a7e33 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:31.720515 18983 consensus_queue.cc:260] T 12c6e131fa904d23865b8673b58a7e33 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:31.721174 18477 raft_consensus.cc:2454] T 12c6e131fa904d23865b8673b58a7e33 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:31.722332 18983 ts_tablet_manager.cc:1398] T 12c6e131fa904d23865b8673b58a7e33 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240430 20:11:31.722330 18699 leader_election.cc:302] T 12c6e131fa904d23865b8673b58a7e33 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters:
I20240430 20:11:31.723637 18983 tablet_bootstrap.cc:492] T b72fe1f0713b4076a81a3fc1c74f7fa7 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:31.723704 18583 tablet_service.cc:1467] Processing CreateTablet for tablet a9f37f0f7ce8425f9b24699513d41d61 (DEFAULT_TABLE table=default.Table_43c679fe95d64db79020bcef1cbe61cb [id=bec003e8d2234bf6b02a2d026788a22d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:31.724159 18583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9f37f0f7ce8425f9b24699513d41d61. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:31.724262 18454 tablet_service.cc:1467] Processing CreateTablet for tablet 06e82c706b9c4f35bf53faddf57b3fd7 (DEFAULT_TABLE table=default.Table_4bac430d51a7461e919b79d3a2f92a8e [id=afaa8b1a82404015ba960f018dd7550b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:31.724678 18454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06e82c706b9c4f35bf53faddf57b3fd7. 1 dirs total, 0 dirs full, 0 dirs failed
W20240430 20:11:31.725003 18696 leader_election.cc:341] T 12c6e131fa904d23865b8673b58a7e33 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:31.725437 18983 tablet_bootstrap.cc:654] T b72fe1f0713b4076a81a3fc1c74f7fa7 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:31.726918 18463 tablet_service.cc:1467] Processing CreateTablet for tablet a9f37f0f7ce8425f9b24699513d41d61 (DEFAULT_TABLE table=default.Table_43c679fe95d64db79020bcef1cbe61cb [id=bec003e8d2234bf6b02a2d026788a22d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:31.727313 18463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9f37f0f7ce8425f9b24699513d41d61. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:31.729662 19474 raft_consensus.cc:482] T 802336602d8f4ac4bd7e0872d6f3dae9 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:31.729928 19648 raft_consensus.cc:2781] T 12c6e131fa904d23865b8673b58a7e33 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:31.730976 18459 tablet_service.cc:1467] Processing CreateTablet for tablet 06d28b6f48a34b17b0d7fb3102df2f4c (DEFAULT_TABLE table=default.Table_4a09e48b08fa4c4387262a5b92b2a42d [id=565591f1f0e84bc89c8d366ef53ae969]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:31.731418 18459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06d28b6f48a34b17b0d7fb3102df2f4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:31.731447 19648 raft_consensus.cc:686] T 12c6e131fa904d23865b8673b58a7e33 P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Becoming Leader. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Running, Role: LEADER
I20240430 20:11:31.731752 19648 consensus_queue.cc:237] T 12c6e131fa904d23865b8673b58a7e33 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:31.733299 19474 raft_consensus.cc:504] T 802336602d8f4ac4bd7e0872d6f3dae9 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:31.734054 19474 leader_election.cc:288] T 802336602d8f4ac4bd7e0872d6f3dae9 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:31.734529 18601 tablet_service.cc:1467] Processing CreateTablet for tablet 06d28b6f48a34b17b0d7fb3102df2f4c (DEFAULT_TABLE table=default.Table_4a09e48b08fa4c4387262a5b92b2a42d [id=565591f1f0e84bc89c8d366ef53ae969]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:31.734733 18761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "802336602d8f4ac4bd7e0872d6f3dae9" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:31.734905 18457 tablet_service.cc:1467] Processing CreateTablet for tablet ae20f44df50343289b0a512a4ea53692 (DEFAULT_TABLE table=default.Table_d4469d8c485e427c8e245855b9d5328b [id=71da9acc3cff4d95bbac4a686b708fd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:31.734992 18601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06d28b6f48a34b17b0d7fb3102df2f4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:31.735332 18457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae20f44df50343289b0a512a4ea53692. 1 dirs total, 0 dirs full, 0 dirs failed
W20240430 20:11:31.735435 18557 leader_election.cc:341] T 802336602d8f4ac4bd7e0872d6f3dae9 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:31.736583 19489 ts_tablet_manager.cc:1888] T c5fd0730133c472299c93f4b0830e527 P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:31.736733 19489 log.cc:1196] T c5fd0730133c472299c93f4b0830e527 P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/c5fd0730133c472299c93f4b0830e527
I20240430 20:11:31.738883 19489 ts_tablet_manager.cc:1909] T c5fd0730133c472299c93f4b0830e527 P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:31.739787 18988 tablet_bootstrap.cc:492] T eed6338ad3d64859af087ee15e1ec798 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:31.739924 18988 ts_tablet_manager.cc:1367] T eed6338ad3d64859af087ee15e1ec798 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.036s user 0.002s sys 0.006s
I20240430 20:11:31.740480 18477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "802336602d8f4ac4bd7e0872d6f3dae9" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:31.740650 18988 raft_consensus.cc:348] T eed6338ad3d64859af087ee15e1ec798 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:31.740847 18988 raft_consensus.cc:374] T eed6338ad3d64859af087ee15e1ec798 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:31.740918 18988 raft_consensus.cc:729] T eed6338ad3d64859af087ee15e1ec798 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:31.741168 18988 consensus_queue.cc:260] T eed6338ad3d64859af087ee15e1ec798 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:31.743366 18738 tablet_service.cc:1467] Processing CreateTablet for tablet a9f37f0f7ce8425f9b24699513d41d61 (DEFAULT_TABLE table=default.Table_43c679fe95d64db79020bcef1cbe61cb [id=bec003e8d2234bf6b02a2d026788a22d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:31.743817 18724 tablet_service.cc:1467] Processing CreateTablet for tablet 06d28b6f48a34b17b0d7fb3102df2f4c (DEFAULT_TABLE table=default.Table_4a09e48b08fa4c4387262a5b92b2a42d [id=565591f1f0e84bc89c8d366ef53ae969]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:31.744189 18731 tablet_service.cc:1467] Processing CreateTablet for tablet ae20f44df50343289b0a512a4ea53692 (DEFAULT_TABLE table=default.Table_d4469d8c485e427c8e245855b9d5328b [id=71da9acc3cff4d95bbac4a686b708fd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:31.744551 18737 tablet_service.cc:1467] Processing CreateTablet for tablet ae24e75a0d42441ebc97400ffa94145e (DEFAULT_TABLE table=default.Table_c9603ec75ad0491592883980e5efc388 [id=ce2d781a514e4c10a2fde5b9d4551ff5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:31.745019 18726 tablet_service.cc:1467] Processing CreateTablet for tablet 06e82c706b9c4f35bf53faddf57b3fd7 (DEFAULT_TABLE table=default.Table_4bac430d51a7461e919b79d3a2f92a8e [id=afaa8b1a82404015ba960f018dd7550b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:31.745865 18988 ts_tablet_manager.cc:1398] T eed6338ad3d64859af087ee15e1ec798 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240430 20:11:31.746438 18738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9f37f0f7ce8425f9b24699513d41d61. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:31.746721 18588 tablet_service.cc:1467] Processing CreateTablet for tablet ae20f44df50343289b0a512a4ea53692 (DEFAULT_TABLE table=default.Table_d4469d8c485e427c8e245855b9d5328b [id=71da9acc3cff4d95bbac4a686b708fd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:31.747181 18588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae20f44df50343289b0a512a4ea53692. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:31.748157 18591 tablet_service.cc:1467] Processing CreateTablet for tablet 06e82c706b9c4f35bf53faddf57b3fd7 (DEFAULT_TABLE table=default.Table_4bac430d51a7461e919b79d3a2f92a8e [id=afaa8b1a82404015ba960f018dd7550b]), partition=RANGE (key) PARTITION UNBOUNDED
W20240430 20:11:31.753401 18558 leader_election.cc:341] T 802336602d8f4ac4bd7e0872d6f3dae9 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:31.753682 18558 leader_election.cc:302] T 802336602d8f4ac4bd7e0872d6f3dae9 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:31.753850 19024 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: Sending DeleteTablet for 3 replicas of tablet fcc9698abda4491b98e1e879589913a8
I20240430 20:11:31.754328 19024 catalog_manager.cc:6554] ReplaceTablet: tablet fcc9698abda4491b98e1e879589913a8 deleted and replaced by tablet de1dfe6c82d34a4d8d02e466c55327ff
I20240430 20:11:31.754997 18731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae20f44df50343289b0a512a4ea53692. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:31.755304 19492 consensus_queue.cc:1035] T 7dfdecba959543a795dfcd3eeda29e58 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:31.758862 18737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae24e75a0d42441ebc97400ffa94145e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:31.759110 18450 tablet_service.cc:1514] Processing DeleteTablet for tablet fcc9698abda4491b98e1e879589913a8 with delete_type TABLET_DATA_DELETED (replaced by tablet de1dfe6c82d34a4d8d02e466c55327ff) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:31.759327 19474 raft_consensus.cc:2726] T 802336602d8f4ac4bd7e0872d6f3dae9 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240430 20:11:31.759434 18999 consensus_peers.cc:485] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33 -> Peer 88b3f98793354c7aaa5c106e8ec31ed8 (127.17.177.62:36441): Couldn't send request to peer 88b3f98793354c7aaa5c106e8ec31ed8. Status: Network error: Client connection negotiation failed: client connection to 127.17.177.62:36441: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240430 20:11:31.761812 18450 tablet_service.cc:1467] Processing CreateTablet for tablet ae24e75a0d42441ebc97400ffa94145e (DEFAULT_TABLE table=default.Table_c9603ec75ad0491592883980e5efc388 [id=ce2d781a514e4c10a2fde5b9d4551ff5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:31.762148 19495 ts_tablet_manager.cc:1888] T 1398d4d2305b48e08fdb12432bd6c4b0 P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:31.762307 19495 log.cc:1196] T 1398d4d2305b48e08fdb12432bd6c4b0 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/1398d4d2305b48e08fdb12432bd6c4b0
I20240430 20:11:31.762312 18450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae24e75a0d42441ebc97400ffa94145e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:31.762815 19495 ts_tablet_manager.cc:1909] T 1398d4d2305b48e08fdb12432bd6c4b0 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:31.763592 18724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06d28b6f48a34b17b0d7fb3102df2f4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:31.765756 18591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06e82c706b9c4f35bf53faddf57b3fd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:31.766844 18726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06e82c706b9c4f35bf53faddf57b3fd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:31.768615 18461 tablet_service.cc:1467] Processing CreateTablet for tablet 677481aee5fa4eb4b189322d7a3f8f2e (DEFAULT_TABLE table=default.Table_0376ffa6507243ada0b3b67b10264ee0 [id=d268c31d82df4871bc1a78aea5871b33]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:31.769091 18461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 677481aee5fa4eb4b189322d7a3f8f2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:31.771896 18732 tablet_service.cc:1467] Processing CreateTablet for tablet 677481aee5fa4eb4b189322d7a3f8f2e (DEFAULT_TABLE table=default.Table_0376ffa6507243ada0b3b67b10264ee0 [id=d268c31d82df4871bc1a78aea5871b33]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:31.772307 18732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 677481aee5fa4eb4b189322d7a3f8f2e. 1 dirs total, 0 dirs full, 0 dirs failed
W20240430 20:11:31.778782 18417 consensus_peers.cc:485] T c5fd0730133c472299c93f4b0830e527 P 2efb542aa55e439b99ccb98d1c5c884a -> Peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Couldn't send request to peer 3c51c33d5616482cafb68e26688820f4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c5fd0730133c472299c93f4b0830e527. This is attempt 1: this message will repeat every 5th retry.
I20240430 20:11:31.781968 19471 ts_tablet_manager.cc:1888] T d9e578c99058424a9b86390641f9c34f P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:31.776858 18985 tablet_bootstrap.cc:492] T e3e74e76ab2b4fe492ffcc1838656fd1 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:31.782673 18985 ts_tablet_manager.cc:1367] T e3e74e76ab2b4fe492ffcc1838656fd1 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.135s user 0.007s sys 0.002s
I20240430 20:11:31.784600 19471 log.cc:1196] T d9e578c99058424a9b86390641f9c34f P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/d9e578c99058424a9b86390641f9c34f
I20240430 20:11:31.785528 18985 raft_consensus.cc:348] T e3e74e76ab2b4fe492ffcc1838656fd1 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:31.785849 18985 raft_consensus.cc:374] T e3e74e76ab2b4fe492ffcc1838656fd1 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:31.786166 18985 raft_consensus.cc:729] T e3e74e76ab2b4fe492ffcc1838656fd1 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:31.786800 19471 ts_tablet_manager.cc:1909] T d9e578c99058424a9b86390641f9c34f P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:31.789024 19471 tablet_replica.cc:330] T 93d158de64994f3c9fa45a1250a4f533 P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:31.789592 19471 raft_consensus.cc:2229] T 93d158de64994f3c9fa45a1250a4f533 P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Raft consensus shutting down.
I20240430 20:11:31.789989 18999 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet d9e578c99058424a9b86390641f9c34f (table default.Table_40857db02a904104a4850e51d5996da2 [id=64a1f78e3f3f425d9c350ced9e169ac5]) successfully deleted
I20240430 20:11:31.790527 18601 tablet_service.cc:1467] Processing CreateTablet for tablet ae24e75a0d42441ebc97400ffa94145e (DEFAULT_TABLE table=default.Table_c9603ec75ad0491592883980e5efc388 [id=ce2d781a514e4c10a2fde5b9d4551ff5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:31.791009 18601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae24e75a0d42441ebc97400ffa94145e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:31.794458 18983 tablet_bootstrap.cc:492] T b72fe1f0713b4076a81a3fc1c74f7fa7 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:31.794677 18983 ts_tablet_manager.cc:1367] T b72fe1f0713b4076a81a3fc1c74f7fa7 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.071s user 0.002s sys 0.004s
I20240430 20:11:31.795290 18583 tablet_service.cc:1467] Processing CreateTablet for tablet 677481aee5fa4eb4b189322d7a3f8f2e (DEFAULT_TABLE table=default.Table_0376ffa6507243ada0b3b67b10264ee0 [id=d268c31d82df4871bc1a78aea5871b33]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:31.795863 18985 consensus_queue.cc:260] T e3e74e76ab2b4fe492ffcc1838656fd1 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:31.795493 19471 raft_consensus.cc:2258] T 93d158de64994f3c9fa45a1250a4f533 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:31.795694 18583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 677481aee5fa4eb4b189322d7a3f8f2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:31.801810 18731 tablet_service.cc:1467] Processing CreateTablet for tablet eb76f60448774fe1967fffe7c6873cfc (DEFAULT_TABLE table=default.Table_d3bf86d0283b4227a93a2a8aa241075b [id=4cd17de5c1504aab878e51dea02917e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:31.802522 18449 tablet_service.cc:1467] Processing CreateTablet for tablet eb76f60448774fe1967fffe7c6873cfc (DEFAULT_TABLE table=default.Table_d3bf86d0283b4227a93a2a8aa241075b [id=4cd17de5c1504aab878e51dea02917e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:31.803038 18449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb76f60448774fe1967fffe7c6873cfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:31.805403 18985 ts_tablet_manager.cc:1398] T e3e74e76ab2b4fe492ffcc1838656fd1 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.023s user 0.002s sys 0.000s
I20240430 20:11:31.805735 18985 tablet_bootstrap.cc:492] T dd9b893d48684cfea7c242b93ecba01f P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:31.806212 18731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb76f60448774fe1967fffe7c6873cfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:31.808812 19471 ts_tablet_manager.cc:1875] T 93d158de64994f3c9fa45a1250a4f533 P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:31.810190 18446 tablet_service.cc:1467] Processing CreateTablet for tablet 4f6f9ae722da49e58c9b72107f6da2fa (DEFAULT_TABLE table=default.Table_a2f72cd314144523b378f2c5b324eed9 [id=42130cfc762b4a6293494164a770f18f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:31.806582 18738 tablet_service.cc:1514] Processing DeleteTablet for tablet fcc9698abda4491b98e1e879589913a8 with delete_type TABLET_DATA_DELETED (replaced by tablet de1dfe6c82d34a4d8d02e466c55327ff) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:31.810649 18446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f6f9ae722da49e58c9b72107f6da2fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:31.805717 18726 tablet_service.cc:1467] Processing CreateTablet for tablet f6cfd089f08b47409fb2a6cf4f6b0385 (DEFAULT_TABLE table=default.Table_4bce1dae3391497ebea5f1b536700d62 [id=8f0cb0ef65d84fb8964209dd501c0b5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:31.809116 18730 tablet_service.cc:1467] Processing CreateTablet for tablet 4f6f9ae722da49e58c9b72107f6da2fa (DEFAULT_TABLE table=default.Table_a2f72cd314144523b378f2c5b324eed9 [id=42130cfc762b4a6293494164a770f18f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:31.811427 18730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f6f9ae722da49e58c9b72107f6da2fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:31.815248 18983 raft_consensus.cc:348] T b72fe1f0713b4076a81a3fc1c74f7fa7 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:31.815485 18983 raft_consensus.cc:374] T b72fe1f0713b4076a81a3fc1c74f7fa7 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:31.815572 18983 raft_consensus.cc:729] T b72fe1f0713b4076a81a3fc1c74f7fa7 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:31.818791 18983 consensus_queue.cc:260] T b72fe1f0713b4076a81a3fc1c74f7fa7 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:31.819368 19581 raft_consensus.cc:482] T b76a4105a2974fe280e759349037d027 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:31.819540 19581 raft_consensus.cc:504] T b76a4105a2974fe280e759349037d027 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:31.820348 19581 leader_election.cc:288] T b76a4105a2974fe280e759349037d027 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:31.821072 18477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b76a4105a2974fe280e759349037d027" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:31.821352 18477 raft_consensus.cc:2454] T b76a4105a2974fe280e759349037d027 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:31.821875 18699 leader_election.cc:302] T b76a4105a2974fe280e759349037d027 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters:
I20240430 20:11:31.822300 19581 raft_consensus.cc:2781] T b76a4105a2974fe280e759349037d027 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:31.822403 19581 raft_consensus.cc:482] T b76a4105a2974fe280e759349037d027 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:31.822472 19581 raft_consensus.cc:3037] T b76a4105a2974fe280e759349037d027 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:31.809587 18988 tablet_bootstrap.cc:492] T 1555924b5f1444398f12ee82c3cb5b13 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:31.825349 18985 tablet_bootstrap.cc:654] T dd9b893d48684cfea7c242b93ecba01f P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:31.825582 18588 tablet_service.cc:1467] Processing CreateTablet for tablet eb76f60448774fe1967fffe7c6873cfc (DEFAULT_TABLE table=default.Table_d3bf86d0283b4227a93a2a8aa241075b [id=4cd17de5c1504aab878e51dea02917e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:31.825868 19581 raft_consensus.cc:504] T b76a4105a2974fe280e759349037d027 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:31.826062 18588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb76f60448774fe1967fffe7c6873cfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:31.826659 18463 tablet_service.cc:1467] Processing CreateTablet for tablet f6cfd089f08b47409fb2a6cf4f6b0385 (DEFAULT_TABLE table=default.Table_4bce1dae3391497ebea5f1b536700d62 [id=8f0cb0ef65d84fb8964209dd501c0b5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:31.827045 19581 leader_election.cc:288] T b76a4105a2974fe280e759349037d027 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Requested vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:31.827162 18463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6cfd089f08b47409fb2a6cf4f6b0385. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:31.827304 18481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b76a4105a2974fe280e759349037d027" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:31.827425 18726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6cfd089f08b47409fb2a6cf4f6b0385. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:31.827535 18481 raft_consensus.cc:3037] T b76a4105a2974fe280e759349037d027 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:31.831688 18481 raft_consensus.cc:2454] T b76a4105a2974fe280e759349037d027 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:31.835148 18449 tablet_service.cc:1467] Processing CreateTablet for tablet fbdeb4714b8c4b93ab72822c43329aae (DEFAULT_TABLE table=default.Table_c12ca6938d424fbfa2e1df5cdb00fdbc [id=0fb93f36bc17408ea56bc4cafcfd8580]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:31.835464 18586 tablet_service.cc:1467] Processing CreateTablet for tablet 4f6f9ae722da49e58c9b72107f6da2fa (DEFAULT_TABLE table=default.Table_a2f72cd314144523b378f2c5b324eed9 [id=42130cfc762b4a6293494164a770f18f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:31.835736 18449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbdeb4714b8c4b93ab72822c43329aae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:31.835916 18586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f6f9ae722da49e58c9b72107f6da2fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:31.840343 19001 catalog_manager.cc:4830] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): tablet c5fd0730133c472299c93f4b0830e527 (table default.Table_ea83912975774b6089fd85c252f45ea5 [id=99cbf7b92c7943838fce9f3ba42babde]) successfully deleted
I20240430 20:11:31.841785 19051 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet acd23f0a63d5448aaa7ed78422e67bad (table default.Table_972f13be4dca4cd19b9a04534dc59f51 [id=69f37db4de634a3185613e40013e0ed7]). Expected version 1 got 0
W20240430 20:11:31.842978 19001 catalog_manager.cc:4810] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): delete failed for tablet 255c13ad8fae4ea4aa8e7b7aabb7602e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 255c13ad8fae4ea4aa8e7b7aabb7602e already in progress: creating tablet
I20240430 20:11:31.833879 18988 tablet_bootstrap.cc:654] T 1555924b5f1444398f12ee82c3cb5b13 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:31.843226 18622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b76a4105a2974fe280e759349037d027" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:31.843358 18591 tablet_service.cc:1514] Processing DeleteTablet for tablet fcc9698abda4491b98e1e879589913a8 with delete_type TABLET_DATA_DELETED (replaced by tablet de1dfe6c82d34a4d8d02e466c55327ff) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:31.843650 19051 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 2e79df8744e74b5991a79ce978ff0202 (table default.Table_276bbdc7cfe542d1a2936485a2aad1ec [id=f146a15e6e8a4acbb4c734c1c863da57]). Expected version 1 got 0
I20240430 20:11:31.843850 19051 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 24dcc20832804229a2664873bf6ddaa7 (table default.Table_be2ebe77f859489f9cd3e912cb1edc8b [id=2d44ec94131148048021b2ee6366ba30]). Expected version 2 got 0
I20240430 20:11:31.844327 19051 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 294644d7a16b4ca2bbb6d106bcb8c7ff (table default.Table_190b2d86515a44898fa38a49517b9df1 [id=97d8a36ddfae4bb5b99c04a7a08544af]). Expected version 2 got 0
I20240430 20:11:31.844274 18620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b76a4105a2974fe280e759349037d027" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:31.844591 19051 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet c66c142419ca4e1399ecd1c8d1e62a59 (table default.Table_0443359dbeff437fb6cffb0db33cbb30 [id=93c44abdf6b147228224388398cd7aeb]). Expected version 2 got 0
I20240430 20:11:31.844841 19051 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 724e51ff05ec4baebc2bb55e05c8eddc (table default.Table_0c5d4686f2ed40fb8d62c31b0166dc05 [id=4dce8a767f0541e1beb5464b2ed95d06]). Expected version 1 got 0
I20240430 20:11:31.845301 19051 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 76d6e9d547ca4a848eae852944b1d117 (table default.Table_104ae015924a48fd8d27220792f07470 [id=ff86c0357c164dd0991098c416087773]). Expected version 2 got 0
I20240430 20:11:31.845494 19051 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet a59ef6c545dd41a28265a4c1b8ce10e0 (table default.Table_453f21c4c2ad4ef7beca9aff68e47b35 [id=86df5d161d8641c89070e7eed9a39360]). Expected version 1 got 0
I20240430 20:11:31.834729 18699 leader_election.cc:302] T b76a4105a2974fe280e759349037d027 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters:
I20240430 20:11:31.845691 19051 catalog_manager.cc:5484] T 44fa6917a85c46f3aaa363df46ea586b P 3c51c33d5616482cafb68e26688820f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c51c33d5616482cafb68e26688820f4 (127.17.177.3). New cstate: current_term: 1 leader_uuid: "3c51c33d5616482cafb68e26688820f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } health_report { overall_health: UNKNOWN } } }
I20240430 20:11:31.846647 19581 raft_consensus.cc:2781] T b76a4105a2974fe280e759349037d027 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:31.846899 18983 ts_tablet_manager.cc:1398] T b72fe1f0713b4076a81a3fc1c74f7fa7 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.042s user 0.003s sys 0.000s
I20240430 20:11:31.848217 18983 tablet_bootstrap.cc:492] T eed6338ad3d64859af087ee15e1ec798 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:31.848140 18583 tablet_service.cc:1467] Processing CreateTablet for tablet f6cfd089f08b47409fb2a6cf4f6b0385 (DEFAULT_TABLE table=default.Table_4bce1dae3391497ebea5f1b536700d62 [id=8f0cb0ef65d84fb8964209dd501c0b5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:31.850625 18583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6cfd089f08b47409fb2a6cf4f6b0385. 1 dirs total, 0 dirs full, 0 dirs failed
W20240430 20:11:31.858629 18696 leader_election.cc:341] T b76a4105a2974fe280e759349037d027 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Illegal state: must be running to vote when last-logged opid is not known
W20240430 20:11:31.859464 18696 leader_election.cc:341] T b76a4105a2974fe280e759349037d027 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:31.860121 18454 tablet_service.cc:1467] Processing CreateTablet for tablet 2873af6da928463f9809864b8eaf0c4a (DEFAULT_TABLE table=default.Table_26cbad4b7efa400f946f15b31c9ff031 [id=b7706c6195364115b5ba240651c22940]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:31.861369 18454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2873af6da928463f9809864b8eaf0c4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:31.863454 18590 tablet_service.cc:1467] Processing CreateTablet for tablet 66dd767bd3f8494f94061fb89e5e5680 (DEFAULT_TABLE table=default.Table_9c1e6182953140bb9cd594f08458c7d9 [id=5227206f44d04fa89b4842b899f33afb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:31.834496 18724 tablet_service.cc:1467] Processing CreateTablet for tablet fbdeb4714b8c4b93ab72822c43329aae (DEFAULT_TABLE table=default.Table_c12ca6938d424fbfa2e1df5cdb00fdbc [id=0fb93f36bc17408ea56bc4cafcfd8580]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:31.864398 18724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbdeb4714b8c4b93ab72822c43329aae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:31.865280 18450 tablet_service.cc:1467] Processing CreateTablet for tablet 66dd767bd3f8494f94061fb89e5e5680 (DEFAULT_TABLE table=default.Table_9c1e6182953140bb9cd594f08458c7d9 [id=5227206f44d04fa89b4842b899f33afb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:31.866185 18450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66dd767bd3f8494f94061fb89e5e5680. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:31.864187 18603 tablet_service.cc:1467] Processing CreateTablet for tablet 2873af6da928463f9809864b8eaf0c4a (DEFAULT_TABLE table=default.Table_26cbad4b7efa400f946f15b31c9ff031 [id=b7706c6195364115b5ba240651c22940]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:31.873236 18590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66dd767bd3f8494f94061fb89e5e5680. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:31.871191 18591 tablet_service.cc:1467] Processing CreateTablet for tablet fbdeb4714b8c4b93ab72822c43329aae (DEFAULT_TABLE table=default.Table_c12ca6938d424fbfa2e1df5cdb00fdbc [id=0fb93f36bc17408ea56bc4cafcfd8580]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:31.841980 18730 tablet_service.cc:1467] Processing CreateTablet for tablet 2873af6da928463f9809864b8eaf0c4a (DEFAULT_TABLE table=default.Table_26cbad4b7efa400f946f15b31c9ff031 [id=b7706c6195364115b5ba240651c22940]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:31.874168 18730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2873af6da928463f9809864b8eaf0c4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:31.839145 18737 tablet_service.cc:1467] Processing CreateTablet for tablet 66dd767bd3f8494f94061fb89e5e5680 (DEFAULT_TABLE table=default.Table_9c1e6182953140bb9cd594f08458c7d9 [id=5227206f44d04fa89b4842b899f33afb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:31.880539 18737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66dd767bd3f8494f94061fb89e5e5680. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:31.882234 18603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2873af6da928463f9809864b8eaf0c4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:31.883138 18983 tablet_bootstrap.cc:654] T eed6338ad3d64859af087ee15e1ec798 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:31.883560 18591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbdeb4714b8c4b93ab72822c43329aae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:31.887740 19051 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet fd40967ab4a04001819e16c06a6c4300 (table default.Table_fda07fd7279c484db2bee67e8c859855 [id=da9ad991953a40c4b77b31317597ad1d]). Expected version 3 got 0
I20240430 20:11:31.888922 19051 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 6e754af64cb64fd2a082adae7d5f24da (table default.Table_208b12b8e9bd4f3e8bc9c6d422ada649 [id=1df4e95702cd46f59ab21b101049a9ed]). Expected version 1 got 0
I20240430 20:11:31.890622 19051 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 2da900b6bbfa4fa99fe8494fd2e68da8 (table default.Table_19a91a7f17ff456ebb145e8dd89c0940 [id=67283dd3995a44c6a7a7bc8722e4f955]). Expected version 1 got 0
I20240430 20:11:31.896026 19489 tablet_replica.cc:330] T aacabead3c874e77a47d2ecf19dbfa82 P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:31.896417 19489 raft_consensus.cc:2229] T aacabead3c874e77a47d2ecf19dbfa82 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:31.896744 19489 raft_consensus.cc:2258] T aacabead3c874e77a47d2ecf19dbfa82 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:31.896832 19051 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 34130f2820064fa0898ef1f9d7de1c6d (table default.Table_4537f891b01048ceb51202abc3600911 [id=5a1c4a7be2fc465fac89a50854eb3322]). Expected version 1 got 0
I20240430 20:11:31.897548 19489 ts_tablet_manager.cc:1875] T aacabead3c874e77a47d2ecf19dbfa82 P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:31.899097 19051 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 6e826459c3f4437da81c9e6e09f0c22f (table default.Table_873c21f6dffd4d47a742b3699ab85268 [id=d7e2c4300ff94fa2acc65d6d6c932c65]). Expected version 1 got 0
I20240430 20:11:31.901975 19051 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet b72fe1f0713b4076a81a3fc1c74f7fa7 (table default.Table_16b082d140234e1c9a1e70654839b4cd [id=c0d8dc87d1f9426a85f6176f161595a1]). Expected version 3 got 0
I20240430 20:11:31.902379 19051 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 32c823d1700949ab8b22adcd83b7bfd1 (table default.Table_e8c73c89623d4b49b8d7f0a01d729bdd [id=5cd10682ce52414cb5a103e4433a6a0d]). Expected version 1 got 0
I20240430 20:11:31.903110 19051 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 53a899fa5a424a0d99f1b47925e8c7ad (table default.Table_deadf728d74a462fb79641891dfedd9c [id=608890834da843fa9af0cfc0d548029e]). Expected version 1 got 0
I20240430 20:11:31.908432 19495 tablet_replica.cc:330] T aacabead3c874e77a47d2ecf19dbfa82 P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:31.908305 19004 catalog_manager.cc:4830] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): tablet 1398d4d2305b48e08fdb12432bd6c4b0 (table default.Table_eea9acffa45b4f3ab6d4524dc460853c [id=52cd3a55893547cabf418f4d01b11d08]) successfully deleted
I20240430 20:11:31.908674 19495 raft_consensus.cc:2229] T aacabead3c874e77a47d2ecf19dbfa82 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:31.908908 19495 raft_consensus.cc:2258] T aacabead3c874e77a47d2ecf19dbfa82 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:31.909245 19051 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 0b4d9e860d7c45fd91d7a566605c9283 (table default.Table_421069f84cb844368066c93102a8faf8 [id=d5cea90cbc2c48df9d8b80da1eccc949]). Expected version 2 got 0
I20240430 20:11:31.909605 19495 ts_tablet_manager.cc:1875] T aacabead3c874e77a47d2ecf19dbfa82 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:31.909750 19051 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet a05e32a33aa7488eb287cc6314c09c44 (table default.Table_2d1cfc78363a47aa8dba7ef7d80f8c9a [id=955d833ff8f14bf6957abadf9ef669e2]). Expected version 2 got 0
I20240430 20:11:31.910511 19051 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet e2d90ff17e574f88b2061b1c117d4b7d (table default.Table_e34613952dcb46618c3bc7cecb31fdaf [id=8f1ad1f8d0224b3daa5d84dc6916227b]). Expected version 1 got 0
W20240430 20:11:31.922708 18699 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.177.1:41217, user_credentials={real_user=slave}} blocked reactor thread for 76423.8us
I20240430 20:11:31.927992 19581 raft_consensus.cc:686] T b76a4105a2974fe280e759349037d027 P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Becoming Leader. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Running, Role: LEADER
I20240430 20:11:31.928370 19581 consensus_queue.cc:237] T b76a4105a2974fe280e759349037d027 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:31.943864 19674 consensus_queue.cc:1035] T 44fa6917a85c46f3aaa363df46ea586b P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240430 20:11:31.949350 19029 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 24dcc20832804229a2664873bf6ddaa7 (table default.Table_be2ebe77f859489f9cd3e912cb1edc8b [id=2d44ec94131148048021b2ee6366ba30]). Expected version 2 got 0
I20240430 20:11:31.949623 18988 tablet_bootstrap.cc:492] T 1555924b5f1444398f12ee82c3cb5b13 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:31.949858 18988 ts_tablet_manager.cc:1367] T 1555924b5f1444398f12ee82c3cb5b13 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.140s user 0.000s sys 0.003s
I20240430 20:11:31.950929 18988 raft_consensus.cc:348] T 1555924b5f1444398f12ee82c3cb5b13 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:31.951293 18988 raft_consensus.cc:374] T 1555924b5f1444398f12ee82c3cb5b13 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:31.951395 18988 raft_consensus.cc:729] T 1555924b5f1444398f12ee82c3cb5b13 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:31.951747 18988 consensus_queue.cc:260] T 1555924b5f1444398f12ee82c3cb5b13 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:31.952025 19471 ts_tablet_manager.cc:1888] T 93d158de64994f3c9fa45a1250a4f533 P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:31.952180 19029 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 2e79df8744e74b5991a79ce978ff0202 (table default.Table_276bbdc7cfe542d1a2936485a2aad1ec [id=f146a15e6e8a4acbb4c734c1c863da57]). Expected version 1 got 0
I20240430 20:11:31.952409 18988 ts_tablet_manager.cc:1398] T 1555924b5f1444398f12ee82c3cb5b13 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240430 20:11:31.952440 19029 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet acd23f0a63d5448aaa7ed78422e67bad (table default.Table_972f13be4dca4cd19b9a04534dc59f51 [id=69f37db4de634a3185613e40013e0ed7]). Expected version 1 got 0
I20240430 20:11:31.953275 18985 tablet_bootstrap.cc:492] T dd9b893d48684cfea7c242b93ecba01f P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:31.953685 18988 tablet_bootstrap.cc:492] T e3e74e76ab2b4fe492ffcc1838656fd1 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:31.953732 19029 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 294644d7a16b4ca2bbb6d106bcb8c7ff (table default.Table_190b2d86515a44898fa38a49517b9df1 [id=97d8a36ddfae4bb5b99c04a7a08544af]). Expected version 2 got 0
I20240430 20:11:31.954282 19029 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet c66c142419ca4e1399ecd1c8d1e62a59 (table default.Table_0443359dbeff437fb6cffb0db33cbb30 [id=93c44abdf6b147228224388398cd7aeb]). Expected version 2 got 0
I20240430 20:11:31.955281 18985 ts_tablet_manager.cc:1367] T dd9b893d48684cfea7c242b93ecba01f P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.150s user 0.011s sys 0.000s
I20240430 20:11:31.955459 18988 tablet_bootstrap.cc:654] T e3e74e76ab2b4fe492ffcc1838656fd1 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:31.955860 19029 catalog_manager.cc:5484] T 724e51ff05ec4baebc2bb55e05c8eddc P 2efb542aa55e439b99ccb98d1c5c884a reported cstate change: term changed from 0 to 1, leader changed from <none> to 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1). New cstate: current_term: 1 leader_uuid: "2efb542aa55e439b99ccb98d1c5c884a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } health_report { overall_health: HEALTHY } } }
I20240430 20:11:31.956156 18985 raft_consensus.cc:348] T dd9b893d48684cfea7c242b93ecba01f P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:31.956375 18985 raft_consensus.cc:374] T dd9b893d48684cfea7c242b93ecba01f P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:31.956447 18985 raft_consensus.cc:729] T dd9b893d48684cfea7c242b93ecba01f P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:31.956776 18985 consensus_queue.cc:260] T dd9b893d48684cfea7c242b93ecba01f P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:31.959925 18985 ts_tablet_manager.cc:1398] T dd9b893d48684cfea7c242b93ecba01f P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240430 20:11:31.961934 19471 log.cc:1196] T 93d158de64994f3c9fa45a1250a4f533 P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/93d158de64994f3c9fa45a1250a4f533
I20240430 20:11:31.962152 18985 tablet_bootstrap.cc:492] T 2a1e50f99c5d4b44bdbb51b58a92bc75 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:31.962738 19471 ts_tablet_manager.cc:1909] T 93d158de64994f3c9fa45a1250a4f533 P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:31.966809 19029 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 724e51ff05ec4baebc2bb55e05c8eddc (table default.Table_0c5d4686f2ed40fb8d62c31b0166dc05 [id=4dce8a767f0541e1beb5464b2ed95d06]). Expected version 1 got 0
I20240430 20:11:31.967464 19029 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet a05e32a33aa7488eb287cc6314c09c44 (table default.Table_2d1cfc78363a47aa8dba7ef7d80f8c9a [id=955d833ff8f14bf6957abadf9ef669e2]). Expected version 2 got 0
I20240430 20:11:31.967847 19029 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 76d6e9d547ca4a848eae852944b1d117 (table default.Table_104ae015924a48fd8d27220792f07470 [id=ff86c0357c164dd0991098c416087773]). Expected version 2 got 0
I20240430 20:11:31.968212 19029 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 0b4d9e860d7c45fd91d7a566605c9283 (table default.Table_421069f84cb844368066c93102a8faf8 [id=d5cea90cbc2c48df9d8b80da1eccc949]). Expected version 2 got 0
I20240430 20:11:31.970475 18985 tablet_bootstrap.cc:654] T 2a1e50f99c5d4b44bdbb51b58a92bc75 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:31.970678 19029 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 6e754af64cb64fd2a082adae7d5f24da (table default.Table_208b12b8e9bd4f3e8bc9c6d422ada649 [id=1df4e95702cd46f59ab21b101049a9ed]). Expected version 1 got 0
I20240430 20:11:31.971091 19029 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 32c823d1700949ab8b22adcd83b7bfd1 (table default.Table_e8c73c89623d4b49b8d7f0a01d729bdd [id=5cd10682ce52414cb5a103e4433a6a0d]). Expected version 1 got 0
I20240430 20:11:31.953503 18983 tablet_bootstrap.cc:492] T eed6338ad3d64859af087ee15e1ec798 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:31.971697 18983 ts_tablet_manager.cc:1367] T eed6338ad3d64859af087ee15e1ec798 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.124s user 0.006s sys 0.000s
I20240430 20:11:31.972179 18999 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet 93d158de64994f3c9fa45a1250a4f533 (table default.Table_9d39d94262a7481aad7df7868a2aff26 [id=06b16d46a7484d0dbc7971f070d0f37e]) successfully deleted
I20240430 20:11:31.972671 18983 raft_consensus.cc:348] T eed6338ad3d64859af087ee15e1ec798 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:31.973109 18983 raft_consensus.cc:374] T eed6338ad3d64859af087ee15e1ec798 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:31.973265 18983 raft_consensus.cc:729] T eed6338ad3d64859af087ee15e1ec798 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:31.973667 18983 consensus_queue.cc:260] T eed6338ad3d64859af087ee15e1ec798 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:31.974264 18983 ts_tablet_manager.cc:1398] T eed6338ad3d64859af087ee15e1ec798 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20240430 20:11:31.974725 18999 catalog_manager.cc:4797] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): delete failed for tablet 0fe1247349344863a6273a769ce8ddf6 because the tablet was not found. No further retry: Not found: Tablet not found: 0fe1247349344863a6273a769ce8ddf6
W20240430 20:11:31.975033 18999 catalog_manager.cc:4810] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): delete failed for tablet 0a9066d895744c5bb7d9532b2940603c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0a9066d895744c5bb7d9532b2940603c already in progress: creating tablet
I20240430 20:11:31.975461 19471 tablet_replica.cc:330] T 02176eee8e044a7db8cac48439e9f594 P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:31.975525 18983 tablet_bootstrap.cc:492] T fcc9698abda4491b98e1e879589913a8 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:31.975971 19029 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 2da900b6bbfa4fa99fe8494fd2e68da8 (table default.Table_19a91a7f17ff456ebb145e8dd89c0940 [id=67283dd3995a44c6a7a7bc8722e4f955]). Expected version 1 got 0
I20240430 20:11:31.977715 19029 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet a59ef6c545dd41a28265a4c1b8ce10e0 (table default.Table_453f21c4c2ad4ef7beca9aff68e47b35 [id=86df5d161d8641c89070e7eed9a39360]). Expected version 1 got 0
I20240430 20:11:31.977921 18983 tablet_bootstrap.cc:654] T fcc9698abda4491b98e1e879589913a8 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:31.979137 19029 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 34130f2820064fa0898ef1f9d7de1c6d (table default.Table_4537f891b01048ceb51202abc3600911 [id=5a1c4a7be2fc465fac89a50854eb3322]). Expected version 1 got 0
I20240430 20:11:31.981046 19029 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 680553d4d0bc4fcb839235c1832757e1 (table default.Table_49a012a7c4c8457fadb0a8f0fc9654c6 [id=eecb2f8eeef547c5ab1abe6a2430e592]). Expected version 1 got 0
I20240430 20:11:31.981567 19029 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet b72fe1f0713b4076a81a3fc1c74f7fa7 (table default.Table_16b082d140234e1c9a1e70654839b4cd [id=c0d8dc87d1f9426a85f6176f161595a1]). Expected version 3 got 0
I20240430 20:11:31.982611 19029 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 53a899fa5a424a0d99f1b47925e8c7ad (table default.Table_deadf728d74a462fb79641891dfedd9c [id=608890834da843fa9af0cfc0d548029e]). Expected version 1 got 0
I20240430 20:11:31.984078 19029 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet fd40967ab4a04001819e16c06a6c4300 (table default.Table_fda07fd7279c484db2bee67e8c859855 [id=da9ad991953a40c4b77b31317597ad1d]). Expected version 3 got 0
I20240430 20:11:31.984380 19029 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 372b135dd01f44f98cd713ca8857ba61 (table default.Table_ee86b8fcf2064d8b8c3e327e357d3395 [id=7f3a15a442984827a7521d5d1c98dc3d]). Expected version 2 got 0
I20240430 20:11:31.986131 19029 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet e2d90ff17e574f88b2061b1c117d4b7d (table default.Table_e34613952dcb46618c3bc7cecb31fdaf [id=8f1ad1f8d0224b3daa5d84dc6916227b]). Expected version 1 got 0
I20240430 20:11:31.990561 19471 raft_consensus.cc:2229] T 02176eee8e044a7db8cac48439e9f594 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:31.990932 19471 raft_consensus.cc:2258] T 02176eee8e044a7db8cac48439e9f594 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:31.991639 19471 ts_tablet_manager.cc:1875] T 02176eee8e044a7db8cac48439e9f594 P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:31.996615 19492 consensus_queue.cc:1035] T 44fa6917a85c46f3aaa363df46ea586b P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:32.013126 19024 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet acd23f0a63d5448aaa7ed78422e67bad (table default.Table_972f13be4dca4cd19b9a04534dc59f51 [id=69f37db4de634a3185613e40013e0ed7]). Expected version 1 got 0
I20240430 20:11:32.014081 18985 tablet_bootstrap.cc:492] T 2a1e50f99c5d4b44bdbb51b58a92bc75 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:32.014415 18983 tablet_bootstrap.cc:492] T fcc9698abda4491b98e1e879589913a8 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:32.014616 18983 ts_tablet_manager.cc:1367] T fcc9698abda4491b98e1e879589913a8 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.039s user 0.001s sys 0.003s
I20240430 20:11:32.014624 19024 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 2e79df8744e74b5991a79ce978ff0202 (table default.Table_276bbdc7cfe542d1a2936485a2aad1ec [id=f146a15e6e8a4acbb4c734c1c863da57]). Expected version 1 got 0
I20240430 20:11:32.014968 19024 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 24dcc20832804229a2664873bf6ddaa7 (table default.Table_be2ebe77f859489f9cd3e912cb1edc8b [id=2d44ec94131148048021b2ee6366ba30]). Expected version 2 got 0
I20240430 20:11:32.015209 18985 ts_tablet_manager.cc:1367] T 2a1e50f99c5d4b44bdbb51b58a92bc75 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.055s user 0.004s sys 0.000s
I20240430 20:11:32.015599 18983 raft_consensus.cc:348] T fcc9698abda4491b98e1e879589913a8 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:32.015818 18983 raft_consensus.cc:374] T fcc9698abda4491b98e1e879589913a8 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:32.015939 18983 raft_consensus.cc:729] T fcc9698abda4491b98e1e879589913a8 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:32.016192 19024 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 294644d7a16b4ca2bbb6d106bcb8c7ff (table default.Table_190b2d86515a44898fa38a49517b9df1 [id=97d8a36ddfae4bb5b99c04a7a08544af]). Expected version 2 got 0
I20240430 20:11:32.016181 18983 consensus_queue.cc:260] T fcc9698abda4491b98e1e879589913a8 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:32.016724 18983 ts_tablet_manager.cc:1398] T fcc9698abda4491b98e1e879589913a8 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240430 20:11:32.017047 18983 tablet_bootstrap.cc:492] T 4e69ca12bd10494fb013614741bc166c P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:32.018222 18988 tablet_bootstrap.cc:492] T e3e74e76ab2b4fe492ffcc1838656fd1 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:32.018440 18985 raft_consensus.cc:348] T 2a1e50f99c5d4b44bdbb51b58a92bc75 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:32.016709 19024 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 724e51ff05ec4baebc2bb55e05c8eddc (table default.Table_0c5d4686f2ed40fb8d62c31b0166dc05 [id=4dce8a767f0541e1beb5464b2ed95d06]). Expected version 1 got 0
I20240430 20:11:32.018999 18983 tablet_bootstrap.cc:654] T 4e69ca12bd10494fb013614741bc166c P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:32.018385 18988 ts_tablet_manager.cc:1367] T e3e74e76ab2b4fe492ffcc1838656fd1 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.065s user 0.000s sys 0.003s
I20240430 20:11:32.019138 18985 raft_consensus.cc:374] T 2a1e50f99c5d4b44bdbb51b58a92bc75 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:32.019488 19024 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 76d6e9d547ca4a848eae852944b1d117 (table default.Table_104ae015924a48fd8d27220792f07470 [id=ff86c0357c164dd0991098c416087773]). Expected version 2 got 0
I20240430 20:11:32.019693 18985 raft_consensus.cc:729] T 2a1e50f99c5d4b44bdbb51b58a92bc75 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:32.020375 19024 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet a59ef6c545dd41a28265a4c1b8ce10e0 (table default.Table_453f21c4c2ad4ef7beca9aff68e47b35 [id=86df5d161d8641c89070e7eed9a39360]). Expected version 1 got 0
I20240430 20:11:32.020782 19024 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 0b4d9e860d7c45fd91d7a566605c9283 (table default.Table_421069f84cb844368066c93102a8faf8 [id=d5cea90cbc2c48df9d8b80da1eccc949]). Expected version 2 got 0
I20240430 20:11:32.020030 18985 consensus_queue.cc:260] T 2a1e50f99c5d4b44bdbb51b58a92bc75 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:32.021401 18988 raft_consensus.cc:348] T e3e74e76ab2b4fe492ffcc1838656fd1 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:32.021653 18988 raft_consensus.cc:374] T e3e74e76ab2b4fe492ffcc1838656fd1 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:32.021754 18988 raft_consensus.cc:729] T e3e74e76ab2b4fe492ffcc1838656fd1 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:32.022699 19495 ts_tablet_manager.cc:1888] T aacabead3c874e77a47d2ecf19dbfa82 P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:32.025177 19024 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 6e754af64cb64fd2a082adae7d5f24da (table default.Table_208b12b8e9bd4f3e8bc9c6d422ada649 [id=1df4e95702cd46f59ab21b101049a9ed]). Expected version 1 got 0
I20240430 20:11:32.027513 19024 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 2da900b6bbfa4fa99fe8494fd2e68da8 (table default.Table_19a91a7f17ff456ebb145e8dd89c0940 [id=67283dd3995a44c6a7a7bc8722e4f955]). Expected version 1 got 0
I20240430 20:11:32.028019 19024 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 34130f2820064fa0898ef1f9d7de1c6d (table default.Table_4537f891b01048ceb51202abc3600911 [id=5a1c4a7be2fc465fac89a50854eb3322]). Expected version 1 got 0
I20240430 20:11:32.028541 19024 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet b72fe1f0713b4076a81a3fc1c74f7fa7 (table default.Table_16b082d140234e1c9a1e70654839b4cd [id=c0d8dc87d1f9426a85f6176f161595a1]). Expected version 3 got 0
I20240430 20:11:32.029063 19471 ts_tablet_manager.cc:1888] T 02176eee8e044a7db8cac48439e9f594 P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:32.030771 19024 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 32c823d1700949ab8b22adcd83b7bfd1 (table default.Table_e8c73c89623d4b49b8d7f0a01d729bdd [id=5cd10682ce52414cb5a103e4433a6a0d]). Expected version 1 got 0
I20240430 20:11:32.031318 19471 log.cc:1196] T 02176eee8e044a7db8cac48439e9f594 P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/02176eee8e044a7db8cac48439e9f594
I20240430 20:11:32.031838 19471 ts_tablet_manager.cc:1909] T 02176eee8e044a7db8cac48439e9f594 P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:32.031749 19024 catalog_manager.cc:5484] T c66c142419ca4e1399ecd1c8d1e62a59 P 5e1c6397f09747f091534faf93542710 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e1c6397f09747f091534faf93542710 (127.17.177.2). New cstate: current_term: 1 leader_uuid: "5e1c6397f09747f091534faf93542710" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } health_report { overall_health: UNKNOWN } } }
I20240430 20:11:32.032385 19024 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet c66c142419ca4e1399ecd1c8d1e62a59 (table default.Table_0443359dbeff437fb6cffb0db33cbb30 [id=93c44abdf6b147228224388398cd7aeb]). Expected version 2 got 0
I20240430 20:11:32.032691 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 52) took 1042 ms. Trace:
I20240430 20:11:32.032912 19024 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 53a899fa5a424a0d99f1b47925e8c7ad (table default.Table_deadf728d74a462fb79641891dfedd9c [id=608890834da843fa9af0cfc0d548029e]). Expected version 1 got 0
I20240430 20:11:32.032840 19471 rpcz_store.cc:276] 0430 20:11:30.990635 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:30.991519 (+ 884us) service_pool.cc:225] Handling call
0430 20:11:31.975214 (+983695us) ts_tablet_manager.cc:1120] Deleting tablet 02176eee8e044a7db8cac48439e9f594
0430 20:11:31.975223 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:32.008723 (+ 33500us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:32.028908 (+ 20185us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:32.032649 (+ 3741us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":981664}
I20240430 20:11:32.033332 19471 tablet_replica.cc:330] T 29974dc2c7dd45b99b6cc0712378fc30 P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:32.033561 19471 raft_consensus.cc:2229] T 29974dc2c7dd45b99b6cc0712378fc30 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:32.033738 19471 raft_consensus.cc:2258] T 29974dc2c7dd45b99b6cc0712378fc30 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:32.033996 18999 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet 02176eee8e044a7db8cac48439e9f594 (table default.Table_3b327aece0ea4ec4b882f2264e3f5f0c [id=1fb10bddcfe94ec488ecc71e43ca339d]) successfully deleted
I20240430 20:11:32.034303 19471 ts_tablet_manager.cc:1875] T 29974dc2c7dd45b99b6cc0712378fc30 P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:32.033746 19024 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet fd40967ab4a04001819e16c06a6c4300 (table default.Table_fda07fd7279c484db2bee67e8c859855 [id=da9ad991953a40c4b77b31317597ad1d]). Expected version 3 got 0
I20240430 20:11:32.035403 19024 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet a05e32a33aa7488eb287cc6314c09c44 (table default.Table_2d1cfc78363a47aa8dba7ef7d80f8c9a [id=955d833ff8f14bf6957abadf9ef669e2]). Expected version 2 got 0
I20240430 20:11:32.039156 19024 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet e2d90ff17e574f88b2061b1c117d4b7d (table default.Table_e34613952dcb46618c3bc7cecb31fdaf [id=8f1ad1f8d0224b3daa5d84dc6916227b]). Expected version 1 got 0
I20240430 20:11:32.039783 19674 raft_consensus.cc:482] T 0fe1247349344863a6273a769ce8ddf6 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5e1c6397f09747f091534faf93542710)
I20240430 20:11:32.039897 19051 catalog_manager.cc:6029] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: default.Table_e68dc52988f24f1788517de94d7d2a0e [id=26d4860fb33047b7b812ebd4309f2e28] alter complete (version 1)
I20240430 20:11:32.040068 19674 raft_consensus.cc:504] T 0fe1247349344863a6273a769ce8ddf6 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:32.041013 19674 leader_election.cc:288] T 0fe1247349344863a6273a769ce8ddf6 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:32.041088 18983 tablet_bootstrap.cc:492] T 4e69ca12bd10494fb013614741bc166c P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:32.041306 18983 ts_tablet_manager.cc:1367] T 4e69ca12bd10494fb013614741bc166c P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.024s user 0.004s sys 0.000s
W20240430 20:11:32.042029 18699 leader_election.cc:341] T 0fe1247349344863a6273a769ce8ddf6 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Not found: Tablet not found: 0fe1247349344863a6273a769ce8ddf6
W20240430 20:11:32.042196 18696 leader_election.cc:341] T 0fe1247349344863a6273a769ce8ddf6 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Not found: Tablet not found: 0fe1247349344863a6273a769ce8ddf6
I20240430 20:11:32.042340 18696 leader_election.cc:302] T 0fe1247349344863a6273a769ce8ddf6 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710
I20240430 20:11:32.042822 19674 raft_consensus.cc:2726] T 0fe1247349344863a6273a769ce8ddf6 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240430 20:11:32.043017 18983 raft_consensus.cc:348] T 4e69ca12bd10494fb013614741bc166c P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:32.043329 18983 raft_consensus.cc:374] T 4e69ca12bd10494fb013614741bc166c P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:32.043426 18983 raft_consensus.cc:729] T 4e69ca12bd10494fb013614741bc166c P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:32.043749 19489 ts_tablet_manager.cc:1888] T aacabead3c874e77a47d2ecf19dbfa82 P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:32.043880 19489 log.cc:1196] T aacabead3c874e77a47d2ecf19dbfa82 P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/aacabead3c874e77a47d2ecf19dbfa82
I20240430 20:11:32.043824 18983 consensus_queue.cc:260] T 4e69ca12bd10494fb013614741bc166c P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:32.044935 18983 ts_tablet_manager.cc:1398] T 4e69ca12bd10494fb013614741bc166c P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240430 20:11:32.044878 19631 tablet.cc:1722] T 0b4d9e860d7c45fd91d7a566605c9283 P 3c51c33d5616482cafb68e26688820f4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240430 20:11:32.047055 18983 tablet_bootstrap.cc:492] T 47c37876694c4a4abb1091c631dab7a5 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:32.050030 19624 tablet.cc:1722] T 0b4d9e860d7c45fd91d7a566605c9283 P 5e1c6397f09747f091534faf93542710: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240430 20:11:32.054059 19028 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: Sending DeleteTablet for 3 replicas of tablet 5d09baed576e4136bf6d756ca39d08b2
I20240430 20:11:32.054539 18983 tablet_bootstrap.cc:654] T 47c37876694c4a4abb1091c631dab7a5 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:32.055297 18737 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d09baed576e4136bf6d756ca39d08b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:31 UTC) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:32.055709 19028 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_067c3b36dd2e4cfd828ab8ff6bbcbd2e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:32.056324 19028 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_067c3b36dd2e4cfd828ab8ff6bbcbd2e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:32.062120 18591 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d09baed576e4136bf6d756ca39d08b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:31 UTC) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:32.043217 18988 consensus_queue.cc:260] T e3e74e76ab2b4fe492ffcc1838656fd1 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:32.063800 18988 ts_tablet_manager.cc:1398] T e3e74e76ab2b4fe492ffcc1838656fd1 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.045s user 0.001s sys 0.001s
I20240430 20:11:32.065587 19489 ts_tablet_manager.cc:1909] T aacabead3c874e77a47d2ecf19dbfa82 P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:32.065791 19495 log.cc:1196] T aacabead3c874e77a47d2ecf19dbfa82 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/aacabead3c874e77a47d2ecf19dbfa82
I20240430 20:11:32.066291 19495 ts_tablet_manager.cc:1909] T aacabead3c874e77a47d2ecf19dbfa82 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:32.066649 19645 raft_consensus.cc:482] T c9e7660eccc448dbb99d135f82bfd261 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:32.066798 19645 raft_consensus.cc:504] T c9e7660eccc448dbb99d135f82bfd261 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:32.067492 19645 leader_election.cc:288] T c9e7660eccc448dbb99d135f82bfd261 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:32.068118 18761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9e7660eccc448dbb99d135f82bfd261" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:32.068598 19648 raft_consensus.cc:482] T 1555924b5f1444398f12ee82c3cb5b13 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240430 20:11:32.068696 18557 leader_election.cc:341] T c9e7660eccc448dbb99d135f82bfd261 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:32.068744 19648 raft_consensus.cc:504] T 1555924b5f1444398f12ee82c3cb5b13 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:32.069109 19001 catalog_manager.cc:4830] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): tablet aacabead3c874e77a47d2ecf19dbfa82 (table default.Table_fda07fd7279c484db2bee67e8c859855 [id=da9ad991953a40c4b77b31317597ad1d]) successfully deleted
I20240430 20:11:32.069408 19648 leader_election.cc:288] T 1555924b5f1444398f12ee82c3cb5b13 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:32.070514 18620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1555924b5f1444398f12ee82c3cb5b13" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:32.070900 18620 raft_consensus.cc:2454] T 1555924b5f1444398f12ee82c3cb5b13 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:32.071130 19028 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_640bb0fa95ca44d49921276d665b9ca3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240430 20:11:32.071444 18696 leader_election.cc:302] T 1555924b5f1444398f12ee82c3cb5b13 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:32.071799 19648 raft_consensus.cc:2781] T 1555924b5f1444398f12ee82c3cb5b13 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:32.071925 19648 raft_consensus.cc:482] T 1555924b5f1444398f12ee82c3cb5b13 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:32.072027 19648 raft_consensus.cc:3037] T 1555924b5f1444398f12ee82c3cb5b13 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
W20240430 20:11:32.072463 19028 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_640bb0fa95ca44d49921276d665b9ca3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:32.075942 19648 raft_consensus.cc:504] T 1555924b5f1444398f12ee82c3cb5b13 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:32.077369 18622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1555924b5f1444398f12ee82c3cb5b13" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:32.078564 19648 leader_election.cc:288] T 1555924b5f1444398f12ee82c3cb5b13 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Requested vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:32.079099 19028 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_b3741e0be6594cfab0de42dec8e1f2fe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:32.080133 19028 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b3741e0be6594cfab0de42dec8e1f2fe in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240430 20:11:32.081562 19026 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c9e7660eccc448dbb99d135f82bfd261 consensus state has no leader
I20240430 20:11:32.081743 19026 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:51664 (request call id 65) took 1012 ms. Trace:
I20240430 20:11:32.081816 19023 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:51664 (request call id 63) took 1045 ms. Trace:
I20240430 20:11:32.081907 19023 rpcz_store.cc:276] 0430 20:11:31.036199 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:31.297200 (+261001us) service_pool.cc:225] Handling call
0430 20:11:31.297549 (+ 349us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0430 20:11:31.297763 (+ 214us) catalog_manager.cc:3613] Apply alter schema
0430 20:11:31.297962 (+ 199us) catalog_manager.cc:3636] Acquired catalog manager lock
0430 20:11:31.298022 (+ 60us) catalog_manager.cc:3783] Updating metadata on disk
0430 20:11:31.338037 (+ 40015us) catalog_manager.cc:3814] Committing alterations to in-memory state
0430 20:11:32.081766 (+743729us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:31.316984 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:31.317072 (+ 88us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:31.317077 (+ 5us) write_op.cc:427] Acquired schema lock
0430 20:11:31.317079 (+ 2us) tablet.cc:660] Decoding operations
0430 20:11:31.317165 (+ 86us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:31.317172 (+ 7us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:31.317177 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0430 20:11:31.317220 (+ 43us) tablet.cc:699] Row locks acquired
0430 20:11:31.317222 (+ 2us) write_op.cc:260] PREPARE: finished
0430 20:11:31.317248 (+ 26us) write_op.cc:270] Start()
0430 20:11:31.317272 (+ 24us) write_op.cc:276] Timestamp: P: 1714507891317209 usec, L: 0
0430 20:11:31.317276 (+ 4us) op_driver.cc:331] REPLICATION: starting
0430 20:11:31.317406 (+ 130us) log.cc:842] Serialized 405 byte log entry
0430 20:11:31.327315 (+ 9909us) op_driver.cc:444] REPLICATION: finished
0430 20:11:31.328405 (+ 1090us) write_op.cc:301] APPLY: starting
0430 20:11:31.328416 (+ 11us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:31.328442 (+ 26us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:31.328473 (+ 31us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:31.328509 (+ 36us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:31.328534 (+ 25us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0430 20:11:31.328541 (+ 7us) write_op.cc:312] APPLY: finished
0430 20:11:31.328588 (+ 47us) log.cc:842] Serialized 23 byte log entry
0430 20:11:31.328622 (+ 34us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:31.328639 (+ 17us) write_op.cc:446] Released schema lock
0430 20:11:31.328645 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:31.328692 (+ 47us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:31.328695 (+ 3us) write_op.cc:446] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1056,"apply.run_cpu_time_us":331,"apply.run_wall_time_us":330,"num_ops":1,"prepare.queue_time_us":18554,"prepare.run_cpu_time_us":512,"prepare.run_wall_time_us":855,"raft.queue_time_us":34,"raft.run_cpu_time_us":3,"raft.run_wall_time_us":4,"replication_time_us":10027}]]}
I20240430 20:11:32.082652 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 48) took 1101 ms. Trace:
I20240430 20:11:32.082813 19495 rpcz_store.cc:276] 0430 20:11:30.981323 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:30.981362 (+ 39us) service_pool.cc:225] Handling call
0430 20:11:31.908328 (+926966us) ts_tablet_manager.cc:1120] Deleting tablet aacabead3c874e77a47d2ecf19dbfa82
0430 20:11:31.908340 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:31.961584 (+ 53244us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:32.022531 (+ 60947us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:32.082550 (+ 60019us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":926793}
I20240430 20:11:32.083029 19023 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_21208fd7cdca4c73a4857bc4692187f6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240430 20:11:32.083462 18988 tablet_bootstrap.cc:492] T 01436e5b6232494babbb414342f03090 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
W20240430 20:11:32.083729 19023 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_21208fd7cdca4c73a4857bc4692187f6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:32.055138 18450 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d09baed576e4136bf6d756ca39d08b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:31 UTC) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:32.062419 19628 tablet.cc:1722] T 0b4d9e860d7c45fd91d7a566605c9283 P 2efb542aa55e439b99ccb98d1c5c884a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240430 20:11:32.070221 18477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1555924b5f1444398f12ee82c3cb5b13" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:32.085379 18477 raft_consensus.cc:2454] T 1555924b5f1444398f12ee82c3cb5b13 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:32.085490 18988 tablet_bootstrap.cc:654] T 01436e5b6232494babbb414342f03090 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:32.085848 19004 catalog_manager.cc:4830] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): tablet aacabead3c874e77a47d2ecf19dbfa82 (table default.Table_fda07fd7279c484db2bee67e8c859855 [id=da9ad991953a40c4b77b31317597ad1d]) successfully deleted
I20240430 20:11:32.081842 19026 rpcz_store.cc:276] 0430 20:11:31.069462 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:31.567044 (+497582us) service_pool.cc:225] Handling call
0430 20:11:31.567695 (+ 651us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0430 20:11:31.568342 (+ 647us) catalog_manager.cc:3613] Apply alter schema
0430 20:11:31.596994 (+ 28652us) catalog_manager.cc:3636] Acquired catalog manager lock
0430 20:11:31.604709 (+ 7715us) catalog_manager.cc:3783] Updating metadata on disk
0430 20:11:31.625923 (+ 21214us) catalog_manager.cc:3814] Committing alterations to in-memory state
0430 20:11:32.081683 (+455760us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:31.605430 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:31.605559 (+ 129us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:31.605565 (+ 6us) write_op.cc:427] Acquired schema lock
0430 20:11:31.605567 (+ 2us) tablet.cc:660] Decoding operations
0430 20:11:31.605679 (+ 112us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:31.605690 (+ 11us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:31.605694 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0430 20:11:31.605766 (+ 72us) tablet.cc:699] Row locks acquired
0430 20:11:31.605771 (+ 5us) write_op.cc:260] PREPARE: finished
0430 20:11:31.605814 (+ 43us) write_op.cc:270] Start()
0430 20:11:31.605857 (+ 43us) write_op.cc:276] Timestamp: P: 1714507891605794 usec, L: 0
0430 20:11:31.605863 (+ 6us) op_driver.cc:331] REPLICATION: starting
0430 20:11:31.606090 (+ 227us) log.cc:842] Serialized 405 byte log entry
0430 20:11:31.624735 (+ 18645us) op_driver.cc:444] REPLICATION: finished
0430 20:11:31.624894 (+ 159us) write_op.cc:301] APPLY: starting
0430 20:11:31.624914 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:31.624968 (+ 54us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:31.625006 (+ 38us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:31.625059 (+ 53us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:31.625103 (+ 44us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0430 20:11:31.625111 (+ 8us) write_op.cc:312] APPLY: finished
0430 20:11:31.625192 (+ 81us) log.cc:842] Serialized 23 byte log entry
0430 20:11:31.625248 (+ 56us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:31.625264 (+ 16us) write_op.cc:446] Released schema lock
0430 20:11:31.625270 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:31.625329 (+ 59us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:31.625333 (+ 4us) write_op.cc:446] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":110,"apply.run_cpu_time_us":500,"apply.run_wall_time_us":489,"num_ops":1,"prepare.queue_time_us":67,"prepare.run_cpu_time_us":756,"prepare.run_wall_time_us":756,"raft.queue_time_us":88,"raft.run_cpu_time_us":7,"raft.run_wall_time_us":8,"replication_time_us":18851}]]}
I20240430 20:11:32.090911 19489 tablet_replica.cc:330] T d9e578c99058424a9b86390641f9c34f P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:32.091229 19489 raft_consensus.cc:2229] T d9e578c99058424a9b86390641f9c34f P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:32.091454 19489 raft_consensus.cc:2258] T d9e578c99058424a9b86390641f9c34f P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:32.092046 19489 ts_tablet_manager.cc:1875] T d9e578c99058424a9b86390641f9c34f P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:32.092996 18454 tablet_service.cc:1514] Processing DeleteTablet for tablet 7542c7fe6cad452ab718f9ef54dbbbbe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:29 UTC) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:32.093807 18457 tablet_service.cc:1514] Processing DeleteTablet for tablet aacabead3c874e77a47d2ecf19dbfa82 with delete_type TABLET_DATA_DELETED (replaced by tablet fd40967ab4a04001819e16c06a6c4300) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:32.094292 18450 tablet_service.cc:1514] Processing DeleteTablet for tablet 47c37876694c4a4abb1091c631dab7a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:27 UTC) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:32.095106 19495 tablet_replica.cc:330] T a1aeee1b02214975a376cfd36957afe8 P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:32.095319 19495 raft_consensus.cc:2229] T a1aeee1b02214975a376cfd36957afe8 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:32.095490 19495 raft_consensus.cc:2258] T a1aeee1b02214975a376cfd36957afe8 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:32.096063 18450 tablet_service.cc:1514] Processing DeleteTablet for tablet 1398d4d2305b48e08fdb12432bd6c4b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:25 UTC) from {username='slave'} at 127.0.0.1:41680
W20240430 20:11:32.096951 19048 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.17.177.1:50353 (request call id 82) took 1723 ms (client timeout 1000 ms). Trace:
I20240430 20:11:32.097414 19495 ts_tablet_manager.cc:1875] T a1aeee1b02214975a376cfd36957afe8 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240430 20:11:32.097064 19048 rpcz_store.cc:269] 0430 20:11:30.373783 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:30.453846 (+ 80063us) service_pool.cc:225] Handling call
0430 20:11:32.096873 (+1643027us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:30.831514 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:30.831612 (+ 98us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:30.831618 (+ 6us) write_op.cc:427] Acquired schema lock
0430 20:11:30.831620 (+ 2us) tablet.cc:660] Decoding operations
0430 20:11:30.831709 (+ 89us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:30.831719 (+ 10us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:30.831723 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0430 20:11:30.831781 (+ 58us) tablet.cc:699] Row locks acquired
0430 20:11:30.831784 (+ 3us) write_op.cc:260] PREPARE: finished
0430 20:11:30.831816 (+ 32us) write_op.cc:270] Start()
0430 20:11:30.831844 (+ 28us) write_op.cc:276] Timestamp: P: 1714507890831770 usec, L: 0
0430 20:11:30.831847 (+ 3us) op_driver.cc:331] REPLICATION: starting
0430 20:11:30.831992 (+ 145us) log.cc:842] Serialized 530 byte log entry
0430 20:11:30.843260 (+ 11268us) op_driver.cc:444] REPLICATION: finished
0430 20:11:30.843416 (+ 156us) write_op.cc:301] APPLY: starting
0430 20:11:30.843434 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:30.843477 (+ 43us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:30.843514 (+ 37us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:30.843578 (+ 64us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:30.843614 (+ 36us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0430 20:11:30.843623 (+ 9us) write_op.cc:312] APPLY: finished
0430 20:11:30.844328 (+ 705us) log.cc:842] Serialized 23 byte log entry
0430 20:11:30.844402 (+ 74us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:30.844421 (+ 19us) write_op.cc:446] Released schema lock
0430 20:11:30.844428 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:30.844496 (+ 68us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:30.844500 (+ 4us) write_op.cc:446] Released schema lock
Related trace 'op':
0430 20:11:31.146276 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:31.146397 (+ 121us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:31.146403 (+ 6us) write_op.cc:427] Acquired schema lock
0430 20:11:31.146406 (+ 3us) tablet.cc:660] Decoding operations
0430 20:11:31.146534 (+ 128us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:31.146548 (+ 14us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:31.146553 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0430 20:11:31.146656 (+ 103us) tablet.cc:699] Row locks acquired
0430 20:11:31.146658 (+ 2us) write_op.cc:260] PREPARE: finished
0430 20:11:31.146695 (+ 37us) write_op.cc:270] Start()
0430 20:11:31.146731 (+ 36us) write_op.cc:276] Timestamp: P: 1714507891146668 usec, L: 0
0430 20:11:31.146736 (+ 5us) op_driver.cc:331] REPLICATION: starting
0430 20:11:31.146889 (+ 153us) log.cc:842] Serialized 356 byte log entry
0430 20:11:31.184727 (+ 37838us) op_driver.cc:444] REPLICATION: finished
0430 20:11:31.186816 (+ 2089us) write_op.cc:301] APPLY: starting
0430 20:11:31.186826 (+ 10us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:31.186857 (+ 31us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:31.186897 (+ 40us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:31.186944 (+ 47us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:31.186969 (+ 25us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0430 20:11:31.186975 (+ 6us) write_op.cc:312] APPLY: finished
0430 20:11:31.187030 (+ 55us) log.cc:842] Serialized 23 byte log entry
0430 20:11:31.187066 (+ 36us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:31.187078 (+ 12us) write_op.cc:446] Released schema lock
0430 20:11:31.187082 (+ 4us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:31.187148 (+ 66us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:31.187151 (+ 3us) write_op.cc:446] Released schema lock
Metrics: {"mutex_wait_us":1375,"reported_tablets":88,"child_traces":[["op",{"apply.queue_time_us":109,"apply.run_cpu_time_us":1138,"apply.run_wall_time_us":1137,"mutex_wait_us":7725,"num_ops":1,"prepare.queue_time_us":448,"prepare.run_cpu_time_us":602,"prepare.run_wall_time_us":8315,"raft.queue_time_us":8304,"raft.run_cpu_time_us":7,"raft.run_wall_time_us":8,"replication_time_us":11388}],["op",{"apply.queue_time_us":2061,"apply.run_cpu_time_us":371,"apply.run_wall_time_us":370,"num_ops":1,"prepare.queue_time_us":95441,"prepare.run_cpu_time_us":675,"prepare.run_wall_time_us":674,"replication_time_us":37972}]]}
I20240430 20:11:32.099632 19033 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: Sending DeleteTablet for 3 replicas of tablet 2f61e3f88a6448dfa323204bd8d14cc5
I20240430 20:11:32.100190 19033 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:51664 (request call id 57) took 1597 ms. Trace:
I20240430 20:11:32.100577 18737 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f61e3f88a6448dfa323204bd8d14cc5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:31 UTC) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:32.100308 19033 rpcz_store.cc:276] 0430 20:11:30.502363 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:31.241045 (+738682us) service_pool.cc:225] Handling call
0430 20:11:31.279704 (+ 38659us) spinlock_profiling.cc:243] Waited 38.4 ms on lock 0x616000040654. stack: 00007feb16c74756 00007feb16c74510 00007feb15ca748d 0000000000955804 00000000009557cd 00007feb257b315d 00007feb257b3253 00007feb224e5d41 00007feb2a0d134d 00007feb2a25eca6 00007feb21047f66 00007feb21047c95 00007feb1d97c04e 00007feb1d97af2e 00007feb1d981405 00007feb1d981d2f
0430 20:11:31.280042 (+ 338us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0430 20:11:31.280168 (+ 126us) catalog_manager.cc:2749] Modifying in-memory table state
0430 20:11:31.280221 (+ 53us) catalog_manager.cc:2758] Locking tablets
0430 20:11:31.397266 (+117045us) catalog_manager.cc:2772] Removing table and tablets from system table
0430 20:11:31.408479 (+ 11213us) catalog_manager.cc:2790] Removing table from by-name map
0430 20:11:31.410933 (+ 2454us) catalog_manager.cc:2806] Committing in-memory state
0430 20:11:32.099581 (+688648us) catalog_manager.cc:2810] Aborting table tasks
0430 20:11:32.100130 (+ 549us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:31.399147 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:31.399270 (+ 123us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:31.399277 (+ 7us) write_op.cc:427] Acquired schema lock
0430 20:11:31.399281 (+ 4us) tablet.cc:660] Decoding operations
0430 20:11:31.399423 (+ 142us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:31.399437 (+ 14us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:31.399441 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
0430 20:11:31.399610 (+ 169us) tablet.cc:699] Row locks acquired
0430 20:11:31.399613 (+ 3us) write_op.cc:260] PREPARE: finished
0430 20:11:31.399654 (+ 41us) write_op.cc:270] Start()
0430 20:11:31.399690 (+ 36us) write_op.cc:276] Timestamp: P: 1714507891399603 usec, L: 0
0430 20:11:31.399695 (+ 5us) op_driver.cc:331] REPLICATION: starting
0430 20:11:31.399895 (+ 200us) log.cc:842] Serialized 728 byte log entry
0430 20:11:31.407451 (+ 7556us) op_driver.cc:444] REPLICATION: finished
0430 20:11:31.407647 (+ 196us) write_op.cc:301] APPLY: starting
0430 20:11:31.407667 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:31.407717 (+ 50us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:31.407760 (+ 43us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:31.407833 (+ 73us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:31.407881 (+ 48us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0430 20:11:31.407893 (+ 12us) write_op.cc:312] APPLY: finished
0430 20:11:31.407988 (+ 95us) log.cc:842] Serialized 29 byte log entry
0430 20:11:31.408061 (+ 73us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:31.408089 (+ 28us) write_op.cc:446] Released schema lock
0430 20:11:31.408098 (+ 9us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:31.408156 (+ 58us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:31.408160 (+ 4us) write_op.cc:446] Released schema lock
Metrics: {"spinlock_wait_cycles":88374016,"child_traces":[["op",{"apply.queue_time_us":136,"apply.run_cpu_time_us":573,"apply.run_wall_time_us":573,"num_ops":2,"prepare.queue_time_us":1033,"prepare.run_cpu_time_us":877,"prepare.run_wall_time_us":877,"raft.queue_time_us":460,"raft.run_cpu_time_us":292,"raft.run_wall_time_us":292,"replication_time_us":7725}]]}
I20240430 20:11:32.100865 18450 tablet_service.cc:1514] Processing DeleteTablet for tablet 68ca48351cd4412381f5195d81109016 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:26 UTC) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:32.102689 19026 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_891f568fffb349c097f751981343f2b1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:32.103389 19026 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_891f568fffb349c097f751981343f2b1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:32.105026 18450 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f61e3f88a6448dfa323204bd8d14cc5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:31 UTC) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:32.108721 19033 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_2a72e434cca94803ae3e8dfcd2e47235"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240430 20:11:32.024394 18985 ts_tablet_manager.cc:1398] T 2a1e50f99c5d4b44bdbb51b58a92bc75 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240430 20:11:32.079036 18481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1555924b5f1444398f12ee82c3cb5b13" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
W20240430 20:11:32.109275 19033 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2a72e434cca94803ae3e8dfcd2e47235 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:32.109352 18481 raft_consensus.cc:3037] T 1555924b5f1444398f12ee82c3cb5b13 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:32.079571 18480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9e7660eccc448dbb99d135f82bfd261" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:32.111275 19048 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_925990f1c73245818b39aafeb0de55e3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240430 20:11:32.112076 18481 raft_consensus.cc:2454] T 1555924b5f1444398f12ee82c3cb5b13 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:32.113209 18985 tablet_bootstrap.cc:492] T 294644d7a16b4ca2bbb6d106bcb8c7ff P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
W20240430 20:11:32.113253 19048 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_925990f1c73245818b39aafeb0de55e3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:32.114826 18985 tablet_bootstrap.cc:654] T 294644d7a16b4ca2bbb6d106bcb8c7ff P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:32.116147 18699 leader_election.cc:302] T 1555924b5f1444398f12ee82c3cb5b13 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters:
I20240430 20:11:32.116261 19051 catalog_manager.cc:6029] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: default.Table_49a012a7c4c8457fadb0a8f0fc9654c6 [id=eecb2f8eeef547c5ab1abe6a2430e592] alter complete (version 1)
I20240430 20:11:32.120378 19489 ts_tablet_manager.cc:1888] T d9e578c99058424a9b86390641f9c34f P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:32.128039 19023 master_service.cc:946] ReplaceTablet: received request to replace tablet d9e578c99058424a9b86390641f9c34f from {username='slave'} at 127.0.0.1:51664
I20240430 20:11:32.130396 19495 ts_tablet_manager.cc:1888] T a1aeee1b02214975a376cfd36957afe8 P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:32.130622 19495 log.cc:1196] T a1aeee1b02214975a376cfd36957afe8 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/a1aeee1b02214975a376cfd36957afe8
I20240430 20:11:32.120517 19674 raft_consensus.cc:2781] T 1555924b5f1444398f12ee82c3cb5b13 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:32.131839 19674 raft_consensus.cc:686] T 1555924b5f1444398f12ee82c3cb5b13 P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Becoming Leader. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Running, Role: LEADER
I20240430 20:11:32.132187 19489 log.cc:1196] T d9e578c99058424a9b86390641f9c34f P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/d9e578c99058424a9b86390641f9c34f
W20240430 20:11:32.132230 19049 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.17.177.2:41749 (request call id 91) took 1581 ms (client timeout 1000 ms). Trace:
I20240430 20:11:32.126571 18737 tablet_service.cc:1514] Processing DeleteTablet for tablet 47c37876694c4a4abb1091c631dab7a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:27 UTC) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:32.127043 18731 tablet_service.cc:1514] Processing DeleteTablet for tablet 75ebf051383e42529dcb5f7dd0770a4b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:27 UTC) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:32.122014 18761 raft_consensus.cc:1261] T 724e51ff05ec4baebc2bb55e05c8eddc P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Refusing update from remote peer 2efb542aa55e439b99ccb98d1c5c884a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240430 20:11:32.134375 18726 tablet_service.cc:1514] Processing DeleteTablet for tablet 255c13ad8fae4ea4aa8e7b7aabb7602e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:27 UTC) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:32.134698 18731 tablet_service.cc:1514] Processing DeleteTablet for tablet 93d158de64994f3c9fa45a1250a4f533 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:25 UTC) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:32.135542 18726 tablet_service.cc:1514] Processing DeleteTablet for tablet 02176eee8e044a7db8cac48439e9f594 with delete_type TABLET_DATA_DELETED (replaced by tablet 09d92a4a6ebe44cbb3553ed660f143d0) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:32.135779 18731 tablet_service.cc:1514] Processing DeleteTablet for tablet c5fd0730133c472299c93f4b0830e527 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:25 UTC) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:32.136337 18737 tablet_service.cc:1514] Processing DeleteTablet for tablet 7542c7fe6cad452ab718f9ef54dbbbbe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:29 UTC) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:32.132247 19674 consensus_queue.cc:237] T 1555924b5f1444398f12ee82c3cb5b13 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
W20240430 20:11:32.133957 19049 rpcz_store.cc:269] 0430 20:11:30.550951 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:30.820974 (+270023us) service_pool.cc:225] Handling call
0430 20:11:31.144778 (+323804us) spinlock_profiling.cc:243] Waited 263 ms on lock 0x616000040654. stack: 00007feb16c74756 00007feb16c74510 00007feb15ca748d 0000000000955804 00000000009557cd 00007feb257b315d 00007feb257b3253 00007feb224e5d41 00007feb2a0d134d 00007feb2a258668 00007feb21043786 00007feb210434b5 00007feb1d97c04e 00007feb1d97af2e 00007feb1d981405 00007feb1d981d2f
0430 20:11:32.132166 (+987388us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:31.311825 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:31.312026 (+ 201us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:31.312035 (+ 9us) write_op.cc:427] Acquired schema lock
0430 20:11:31.312039 (+ 4us) tablet.cc:660] Decoding operations
0430 20:11:31.312189 (+ 150us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:31.312208 (+ 19us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:31.312216 (+ 8us) tablet.cc:683] Acquiring locks for 1 operations
0430 20:11:31.312300 (+ 84us) tablet.cc:699] Row locks acquired
0430 20:11:31.312302 (+ 2us) write_op.cc:260] PREPARE: finished
0430 20:11:31.312353 (+ 51us) write_op.cc:270] Start()
0430 20:11:31.312394 (+ 41us) write_op.cc:276] Timestamp: P: 1714507891312308 usec, L: 0
0430 20:11:31.312398 (+ 4us) op_driver.cc:331] REPLICATION: starting
0430 20:11:31.315089 (+ 2691us) log.cc:842] Serialized 530 byte log entry
0430 20:11:31.315677 (+ 588us) spinlock_profiling.cc:243] Waited 2.44 ms on lock 0x616000040654. stack: 00007feb16c74756 00007feb16c74510 00007feb15ca748d 0000000000955804 00000000009557cd 000000000095519a 00007feb224b9162 00007feb23821b02 00007feb23820629 00007feb2382853b 00007feb238283b8 0000000000a60bcc 00007feb16cbab74 00007feb16cbe19b 00007feb16cbe018 0000000000a60bcc
0430 20:11:31.327195 (+ 11518us) op_driver.cc:444] REPLICATION: finished
0430 20:11:31.327424 (+ 229us) write_op.cc:301] APPLY: starting
0430 20:11:31.327442 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:31.327492 (+ 50us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:31.327494 (+ 2us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:31.327552 (+ 58us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:31.327586 (+ 34us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0430 20:11:31.327593 (+ 7us) write_op.cc:312] APPLY: finished
0430 20:11:31.327683 (+ 90us) log.cc:842] Serialized 23 byte log entry
0430 20:11:31.327753 (+ 70us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:31.327773 (+ 20us) write_op.cc:446] Released schema lock
0430 20:11:31.327781 (+ 8us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:31.327841 (+ 60us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:31.327844 (+ 3us) write_op.cc:446] Released schema lock
Metrics: {"reported_tablets":87,"spinlock_wait_cycles":604407168,"child_traces":[["op",{"apply.queue_time_us":175,"apply.run_cpu_time_us":485,"apply.run_wall_time_us":484,"num_ops":1,"prepare.queue_time_us":18132,"prepare.run_cpu_time_us":1217,"prepare.run_wall_time_us":3988,"raft.queue_time_us":39,"raft.run_cpu_time_us":5,"raft.run_wall_time_us":6,"replication_time_us":14766,"spinlock_wait_cycles":5613952}]]}
I20240430 20:11:32.138324 18737 tablet_service.cc:1514] Processing DeleteTablet for tablet a1aeee1b02214975a376cfd36957afe8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:26 UTC) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:32.139334 18737 tablet_service.cc:1514] Processing DeleteTablet for tablet aacabead3c874e77a47d2ecf19dbfa82 with delete_type TABLET_DATA_DELETED (replaced by tablet fd40967ab4a04001819e16c06a6c4300) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:32.139796 18737 tablet_service.cc:1514] Processing DeleteTablet for tablet 1398d4d2305b48e08fdb12432bd6c4b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:25 UTC) from {username='slave'} at 127.0.0.1:59830
W20240430 20:11:32.142235 19027 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.17.177.3:42157 (request call id 79) took 1665 ms (client timeout 1000 ms). Trace:
I20240430 20:11:32.143541 18737 tablet_service.cc:1514] Processing DeleteTablet for tablet 68ca48351cd4412381f5195d81109016 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:26 UTC) from {username='slave'} at 127.0.0.1:59830
W20240430 20:11:32.142422 19027 rpcz_store.cc:269] 0430 20:11:30.476340 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:30.820961 (+344621us) service_pool.cc:225] Handling call
0430 20:11:31.160393 (+339432us) spinlock_profiling.cc:243] Waited 229 ms on lock 0x616000040654. stack: 00007feb16c74756 00007feb16c74510 00007feb15ca748d 0000000000955804 00000000009557cd 00007feb257b315d 00007feb257b3253 00007feb224e5d41 00007feb2a0d134d 00007feb2a258668 00007feb21043786 00007feb210434b5 00007feb1d97c04e 00007feb1d97af2e 00007feb1d981405 00007feb1d981d2f
0430 20:11:32.142187 (+981794us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:31.623858 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:31.624004 (+ 146us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:31.624011 (+ 7us) write_op.cc:427] Acquired schema lock
0430 20:11:31.624013 (+ 2us) tablet.cc:660] Decoding operations
0430 20:11:31.624132 (+ 119us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:31.624145 (+ 13us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:31.624150 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0430 20:11:31.624230 (+ 80us) tablet.cc:699] Row locks acquired
0430 20:11:31.624233 (+ 3us) write_op.cc:260] PREPARE: finished
0430 20:11:31.624276 (+ 43us) write_op.cc:270] Start()
0430 20:11:31.624316 (+ 40us) write_op.cc:276] Timestamp: P: 1714507891624255 usec, L: 0
0430 20:11:31.624320 (+ 4us) op_driver.cc:331] REPLICATION: starting
0430 20:11:31.624535 (+ 215us) log.cc:842] Serialized 530 byte log entry
0430 20:11:31.633581 (+ 9046us) op_driver.cc:444] REPLICATION: finished
0430 20:11:31.633743 (+ 162us) write_op.cc:301] APPLY: starting
0430 20:11:31.633760 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:31.633804 (+ 44us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:31.633837 (+ 33us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:31.633892 (+ 55us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:31.633927 (+ 35us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0430 20:11:31.633935 (+ 8us) write_op.cc:312] APPLY: finished
0430 20:11:31.634018 (+ 83us) log.cc:842] Serialized 23 byte log entry
0430 20:11:31.634077 (+ 59us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:31.634099 (+ 22us) write_op.cc:446] Released schema lock
0430 20:11:31.634106 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:31.634163 (+ 57us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:31.634169 (+ 6us) write_op.cc:446] Released schema lock
Metrics: {"mutex_wait_us":5479,"reported_tablets":86,"spinlock_wait_cycles":527578880,"child_traces":[["op",{"apply.queue_time_us":115,"apply.run_cpu_time_us":482,"apply.run_wall_time_us":482,"num_ops":1,"prepare.queue_time_us":15855,"prepare.run_cpu_time_us":799,"prepare.run_wall_time_us":799,"raft.queue_time_us":62,"raft.run_cpu_time_us":3,"raft.run_wall_time_us":4,"replication_time_us":9234}]]}
I20240430 20:11:32.144661 19489 ts_tablet_manager.cc:1909] T d9e578c99058424a9b86390641f9c34f P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:32.145042 18622 raft_consensus.cc:3037] T 1555924b5f1444398f12ee82c3cb5b13 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:32.145448 19027 master_service.cc:946] ReplaceTablet: received request to replace tablet fae9a35a2ac844b4bc2ebdf214aa7171 from {username='slave'} at 127.0.0.1:51664
I20240430 20:11:32.148185 18587 tablet_service.cc:1514] Processing DeleteTablet for tablet 93d158de64994f3c9fa45a1250a4f533 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:25 UTC) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:32.148725 18586 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f61e3f88a6448dfa323204bd8d14cc5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:31 UTC) from {username='slave'} at 127.0.0.1:40218
W20240430 20:11:32.149128 18557 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.177.3:44867, user_credentials={real_user=slave}} blocked reactor thread for 50157.1us
I20240430 20:11:32.133699 19617 consensus_queue.cc:1035] T 724e51ff05ec4baebc2bb55e05c8eddc P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:32.150846 19001 catalog_manager.cc:4830] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): tablet d9e578c99058424a9b86390641f9c34f (table default.Table_40857db02a904104a4850e51d5996da2 [id=64a1f78e3f3f425d9c350ced9e169ac5]) successfully deleted
I20240430 20:11:32.152482 19495 ts_tablet_manager.cc:1909] T a1aeee1b02214975a376cfd36957afe8 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:32.153993 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 49) took 1172 ms. Trace:
I20240430 20:11:32.154639 19489 tablet_replica.cc:330] T 29974dc2c7dd45b99b6cc0712378fc30 P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:32.155187 19489 raft_consensus.cc:2229] T 29974dc2c7dd45b99b6cc0712378fc30 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:32.156760 19489 raft_consensus.cc:2258] T 29974dc2c7dd45b99b6cc0712378fc30 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:32.157074 19495 rpcz_store.cc:276] 0430 20:11:30.981704 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:30.981743 (+ 39us) service_pool.cc:225] Handling call
0430 20:11:32.095038 (+1113295us) ts_tablet_manager.cc:1120] Deleting tablet a1aeee1b02214975a376cfd36957afe8
0430 20:11:32.095048 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:32.104376 (+ 9328us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:32.130204 (+ 25828us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:32.153928 (+ 23724us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1113123}
I20240430 20:11:32.165601 18588 tablet_service.cc:1514] Processing DeleteTablet for tablet 7542c7fe6cad452ab718f9ef54dbbbbe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:29 UTC) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:32.165719 18591 tablet_service.cc:1514] Processing DeleteTablet for tablet 02176eee8e044a7db8cac48439e9f594 with delete_type TABLET_DATA_DELETED (replaced by tablet 09d92a4a6ebe44cbb3553ed660f143d0) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:32.167834 18583 tablet_service.cc:1514] Processing DeleteTablet for tablet 47c37876694c4a4abb1091c631dab7a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:27 UTC) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:32.169035 18586 tablet_service.cc:1514] Processing DeleteTablet for tablet d4508e4f6dbd42109d951c648b3dc7df with delete_type TABLET_DATA_DELETED (replaced by tablet 2e79df8744e74b5991a79ce978ff0202) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:32.169490 18587 tablet_service.cc:1514] Processing DeleteTablet for tablet 68ca48351cd4412381f5195d81109016 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:26 UTC) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:32.169659 18601 tablet_service.cc:1514] Processing DeleteTablet for tablet d9e578c99058424a9b86390641f9c34f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:26 UTC) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:32.170230 18602 tablet_service.cc:1514] Processing DeleteTablet for tablet c5fd0730133c472299c93f4b0830e527 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:25 UTC) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:32.170483 18603 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a9066d895744c5bb7d9532b2940603c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:29 UTC) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:32.170830 18590 tablet_service.cc:1514] Processing DeleteTablet for tablet 75ebf051383e42529dcb5f7dd0770a4b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:27 UTC) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:32.172464 18620 raft_consensus.cc:1261] T 724e51ff05ec4baebc2bb55e05c8eddc P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Refusing update from remote peer 2efb542aa55e439b99ccb98d1c5c884a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240430 20:11:32.173619 18588 tablet_service.cc:1514] Processing DeleteTablet for tablet f5546f9edeea44d7a6c5cb954f5157ba with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:27 UTC) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:32.174064 19624 tablet.cc:1722] T fae9a35a2ac844b4bc2ebdf214aa7171 P 5e1c6397f09747f091534faf93542710: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20240430 20:11:32.174369 18583 tablet_service.cc:1514] Processing DeleteTablet for tablet 1398d4d2305b48e08fdb12432bd6c4b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:25 UTC) from {username='slave'} at 127.0.0.1:40218
W20240430 20:11:32.174787 18557 consensus_peers.cc:485] T a1aeee1b02214975a376cfd36957afe8 P 5e1c6397f09747f091534faf93542710 -> Peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Couldn't send request to peer 3c51c33d5616482cafb68e26688820f4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a1aeee1b02214975a376cfd36957afe8. This is attempt 1: this message will repeat every 5th retry.
I20240430 20:11:32.175149 18591 tablet_service.cc:1514] Processing DeleteTablet for tablet aacabead3c874e77a47d2ecf19dbfa82 with delete_type TABLET_DATA_DELETED (replaced by tablet fd40967ab4a04001819e16c06a6c4300) from {username='slave'} at 127.0.0.1:40218
W20240430 20:11:32.178283 18558 leader_election.cc:341] T c9e7660eccc448dbb99d135f82bfd261 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:32.178490 18558 leader_election.cc:302] T c9e7660eccc448dbb99d135f82bfd261 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:32.180371 19474 raft_consensus.cc:2726] T c9e7660eccc448dbb99d135f82bfd261 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:32.182790 19049 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_849e6e38123a4d8cae466d841850d6f9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:32.183458 19049 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_849e6e38123a4d8cae466d841850d6f9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:32.182790 19028 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_5924e4520c6c49988ccef21ca04d25d4" } modify_external_catalogs: true
I20240430 20:11:32.184914 19028 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_5924e4520c6c49988ccef21ca04d25d4" } modify_external_catalogs: true
I20240430 20:11:32.196162 19471 ts_tablet_manager.cc:1888] T 29974dc2c7dd45b99b6cc0712378fc30 P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:32.196933 19471 log.cc:1196] T 29974dc2c7dd45b99b6cc0712378fc30 P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/29974dc2c7dd45b99b6cc0712378fc30
I20240430 20:11:32.197446 19471 ts_tablet_manager.cc:1909] T 29974dc2c7dd45b99b6cc0712378fc30 P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:32.197827 19026 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_16b082d140234e1c9a1e70654839b4cd" } modify_external_catalogs: true
I20240430 20:11:32.198030 19026 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_16b082d140234e1c9a1e70654839b4cd" } modify_external_catalogs: true
I20240430 20:11:32.198344 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 57) took 1207 ms. Trace:
I20240430 20:11:32.198612 18622 raft_consensus.cc:2454] T 1555924b5f1444398f12ee82c3cb5b13 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:32.160698 19631 tablet.cc:1722] T fae9a35a2ac844b4bc2ebdf214aa7171 P 3c51c33d5616482cafb68e26688820f4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20240430 20:11:32.180006 19674 raft_consensus.cc:482] T a05e32a33aa7488eb287cc6314c09c44 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:32.199229 19674 raft_consensus.cc:504] T a05e32a33aa7488eb287cc6314c09c44 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:32.200042 19674 leader_election.cc:288] T a05e32a33aa7488eb287cc6314c09c44 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:32.204414 19051 catalog_manager.cc:6029] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: default.Table_ee86b8fcf2064d8b8c3e327e357d3395 [id=7f3a15a442984827a7521d5d1c98dc3d] alter complete (version 2)
I20240430 20:11:32.207336 19489 ts_tablet_manager.cc:1875] T 29974dc2c7dd45b99b6cc0712378fc30 P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:32.165103 19617 raft_consensus.cc:482] T dd9b893d48684cfea7c242b93ecba01f P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:32.208021 19617 raft_consensus.cc:504] T dd9b893d48684cfea7c242b93ecba01f P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:32.208760 19617 leader_election.cc:288] T dd9b893d48684cfea7c242b93ecba01f P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:32.166287 19628 tablet.cc:1722] T 724e51ff05ec4baebc2bb55e05c8eddc P 2efb542aa55e439b99ccb98d1c5c884a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240430 20:11:32.173710 19186 consensus_queue.cc:1035] T 724e51ff05ec4baebc2bb55e05c8eddc P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:32.209903 18622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a05e32a33aa7488eb287cc6314c09c44" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
W20240430 20:11:32.211195 18696 leader_election.cc:341] T a05e32a33aa7488eb287cc6314c09c44 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:32.180344 19695 tablet.cc:1722] T 724e51ff05ec4baebc2bb55e05c8eddc P 3c51c33d5616482cafb68e26688820f4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240430 20:11:32.184881 19648 raft_consensus.cc:482] T 294644d7a16b4ca2bbb6d106bcb8c7ff P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:32.213541 19648 raft_consensus.cc:504] T 294644d7a16b4ca2bbb6d106bcb8c7ff P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:32.213739 19645 raft_consensus.cc:482] T fcc9698abda4491b98e1e879589913a8 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:32.213923 19645 raft_consensus.cc:504] T fcc9698abda4491b98e1e879589913a8 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:32.214402 19648 leader_election.cc:288] T 294644d7a16b4ca2bbb6d106bcb8c7ff P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:32.214701 19645 leader_election.cc:288] T fcc9698abda4491b98e1e879589913a8 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:32.215178 18477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "294644d7a16b4ca2bbb6d106bcb8c7ff" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
W20240430 20:11:32.216280 18699 leader_election.cc:341] T 294644d7a16b4ca2bbb6d106bcb8c7ff P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:32.218801 18761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcc9698abda4491b98e1e879589913a8" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
W20240430 20:11:32.219511 18557 leader_election.cc:341] T fcc9698abda4491b98e1e879589913a8 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:32.220093 18622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd9b893d48684cfea7c242b93ecba01f" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:32.220445 18622 raft_consensus.cc:2454] T dd9b893d48684cfea7c242b93ecba01f P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 0.
I20240430 20:11:32.223628 18983 tablet_bootstrap.cc:492] T 47c37876694c4a4abb1091c631dab7a5 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:32.223793 18983 ts_tablet_manager.cc:1367] T 47c37876694c4a4abb1091c631dab7a5 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.177s user 0.013s sys 0.052s
I20240430 20:11:32.198452 19471 rpcz_store.cc:276] 0430 20:11:30.991073 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:30.991652 (+ 579us) service_pool.cc:225] Handling call
0430 20:11:32.033273 (+1041621us) ts_tablet_manager.cc:1120] Deleting tablet 29974dc2c7dd45b99b6cc0712378fc30
0430 20:11:32.033282 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:32.058654 (+ 25372us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:32.195680 (+137026us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:32.198277 (+ 2597us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1039644}
I20240430 20:11:32.224525 18983 raft_consensus.cc:348] T 47c37876694c4a4abb1091c631dab7a5 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:32.224756 18983 raft_consensus.cc:374] T 47c37876694c4a4abb1091c631dab7a5 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:32.224869 18983 raft_consensus.cc:729] T 47c37876694c4a4abb1091c631dab7a5 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:32.224938 19471 tablet_replica.cc:330] T bbe9a2429a434f3f95d6e7f314fcfe4c P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:32.225092 18983 consensus_queue.cc:260] T 47c37876694c4a4abb1091c631dab7a5 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:32.225236 19471 raft_consensus.cc:2229] T bbe9a2429a434f3f95d6e7f314fcfe4c P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:32.225468 19471 raft_consensus.cc:2258] T bbe9a2429a434f3f95d6e7f314fcfe4c P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:32.225656 18983 ts_tablet_manager.cc:1398] T 47c37876694c4a4abb1091c631dab7a5 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240430 20:11:32.227447 18415 leader_election.cc:302] T dd9b893d48684cfea7c242b93ecba01f P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:32.227571 19471 ts_tablet_manager.cc:1875] T bbe9a2429a434f3f95d6e7f314fcfe4c P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:32.228224 18983 tablet_bootstrap.cc:492] T a05e32a33aa7488eb287cc6314c09c44 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:32.228264 19186 raft_consensus.cc:2781] T dd9b893d48684cfea7c242b93ecba01f P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:32.228389 19186 raft_consensus.cc:482] T dd9b893d48684cfea7c242b93ecba01f P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:32.228477 19186 raft_consensus.cc:3037] T dd9b893d48684cfea7c242b93ecba01f P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:32.229965 18983 tablet_bootstrap.cc:654] T a05e32a33aa7488eb287cc6314c09c44 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:32.233592 18477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcc9698abda4491b98e1e879589913a8" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:32.234206 19628 tablet.cc:1722] T fae9a35a2ac844b4bc2ebdf214aa7171 P 2efb542aa55e439b99ccb98d1c5c884a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
W20240430 20:11:32.234694 19519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240430 20:11:32.235131 19519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240430 20:11:32.235292 19519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240430 20:11:32.236344 18999 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet 29974dc2c7dd45b99b6cc0712378fc30 (table default.Table_28b63364f56b484eb48763e662dda489 [id=cc553a8f7052423b9724a3b28c63592d]) successfully deleted
I20240430 20:11:32.242543 18761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd9b893d48684cfea7c242b93ecba01f" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
W20240430 20:11:32.243132 18417 leader_election.cc:341] T dd9b893d48684cfea7c242b93ecba01f P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:32.184151 18985 tablet_bootstrap.cc:492] T 294644d7a16b4ca2bbb6d106bcb8c7ff P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:32.201929 18481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a05e32a33aa7488eb287cc6314c09c44" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:32.244534 18481 raft_consensus.cc:2454] T a05e32a33aa7488eb287cc6314c09c44 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:32.245064 18699 leader_election.cc:302] T a05e32a33aa7488eb287cc6314c09c44 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters: 5e1c6397f09747f091534faf93542710
I20240430 20:11:32.245087 19023 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_d5b955117a0a4c1eb8ed8151e155b32e" } new_table_name: "default.Table_4e56923d440944978f0aacc140a0dd8e" modify_external_catalogs: true
I20240430 20:11:32.245512 19648 raft_consensus.cc:2781] T a05e32a33aa7488eb287cc6314c09c44 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:32.245642 19648 raft_consensus.cc:482] T a05e32a33aa7488eb287cc6314c09c44 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:32.246309 19648 raft_consensus.cc:3037] T a05e32a33aa7488eb287cc6314c09c44 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:32.247290 19048 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_7a9f9a8c27bb4392acddd2a7ea3cc3b2" } modify_external_catalogs: true
I20240430 20:11:32.247538 19048 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_7a9f9a8c27bb4392acddd2a7ea3cc3b2" } modify_external_catalogs: true
I20240430 20:11:32.248301 19033 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_a327a8eced314cdf95731f1181f52fb1" } new_table_name: "default.Table_1e2196b2ecb846caa42407f194800886" modify_external_catalogs: true
W20240430 20:11:32.248759 19519 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240430 20:11:32.248865 19519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240430 20:11:32.248937 19519 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240430 20:11:32.249023 19519 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240430 20:11:32.249552 18985 ts_tablet_manager.cc:1367] T 294644d7a16b4ca2bbb6d106bcb8c7ff P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.137s user 0.001s sys 0.007s
I20240430 20:11:32.253772 19674 raft_consensus.cc:482] T 53a899fa5a424a0d99f1b47925e8c7ad P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:32.253995 19674 raft_consensus.cc:504] T 53a899fa5a424a0d99f1b47925e8c7ad P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:32.254705 19674 leader_election.cc:288] T 53a899fa5a424a0d99f1b47925e8c7ad P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:32.255221 18985 raft_consensus.cc:348] T 294644d7a16b4ca2bbb6d106bcb8c7ff P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:32.256145 18988 tablet_bootstrap.cc:492] T 01436e5b6232494babbb414342f03090 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:32.256285 18481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53a899fa5a424a0d99f1b47925e8c7ad" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:32.257339 18985 raft_consensus.cc:374] T 294644d7a16b4ca2bbb6d106bcb8c7ff P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:32.257577 18985 raft_consensus.cc:729] T 294644d7a16b4ca2bbb6d106bcb8c7ff P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:32.258124 18985 consensus_queue.cc:260] T 294644d7a16b4ca2bbb6d106bcb8c7ff P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
W20240430 20:11:32.257105 18699 leader_election.cc:341] T 53a899fa5a424a0d99f1b47925e8c7ad P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:32.258751 18988 ts_tablet_manager.cc:1367] T 01436e5b6232494babbb414342f03090 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.175s user 0.003s sys 0.008s
I20240430 20:11:32.258775 19029 catalog_manager.cc:6029] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: default.Table_873c21f6dffd4d47a742b3699ab85268 [id=d7e2c4300ff94fa2acc65d6d6c932c65] alter complete (version 1)
I20240430 20:11:32.260227 18988 raft_consensus.cc:348] T 01436e5b6232494babbb414342f03090 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:32.260464 18988 raft_consensus.cc:374] T 01436e5b6232494babbb414342f03090 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:32.260540 18988 raft_consensus.cc:729] T 01436e5b6232494babbb414342f03090 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:32.264565 18985 ts_tablet_manager.cc:1398] T 294644d7a16b4ca2bbb6d106bcb8c7ff P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.015s user 0.001s sys 0.001s
I20240430 20:11:32.264958 18985 tablet_bootstrap.cc:492] T acd23f0a63d5448aaa7ed78422e67bad P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:32.265372 19519 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.177.20:42079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/master-0/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.17.177.62:36441,127.17.177.61:34777,127.17.177.60:37225
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.177.62:36441
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.177.62
--webserver_port=33023
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type FASTDEBUG
built by None at 30 Apr 2024 19:43:22 UTC on efd1337a2aa3
build id 63
ASAN enabled
I20240430 20:11:32.267122 19519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240430 20:11:32.267581 19027 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: Sending DeleteTablet for 3 replicas of tablet fae9a35a2ac844b4bc2ebdf214aa7171
I20240430 20:11:32.268105 19027 catalog_manager.cc:6554] ReplaceTablet: tablet fae9a35a2ac844b4bc2ebdf214aa7171 deleted and replaced by tablet b22aaedb0a6c4fd9ad0ea8fa019001e1
I20240430 20:11:32.271065 19519 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240430 20:11:32.272619 19648 raft_consensus.cc:504] T a05e32a33aa7488eb287cc6314c09c44 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
W20240430 20:11:32.272948 19023 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 46251c08277e498da04a41d009310c62 consensus state has no leader
I20240430 20:11:32.273547 18450 tablet_service.cc:1514] Processing DeleteTablet for tablet fae9a35a2ac844b4bc2ebdf214aa7171 with delete_type TABLET_DATA_DELETED (replaced by tablet b22aaedb0a6c4fd9ad0ea8fa019001e1) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:32.273828 18481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a05e32a33aa7488eb287cc6314c09c44" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:32.274070 18481 raft_consensus.cc:3037] T a05e32a33aa7488eb287cc6314c09c44 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:32.274070 18459 tablet_service.cc:1467] Processing CreateTablet for tablet 3ec3993042f5498b818bdf32d5087928 (DEFAULT_TABLE table=default.Table_f323a3bead0249b5a4a3cbb07a5363b8 [id=27217d463b9c40ec926a1862a31f1f24]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:32.274580 18449 tablet_service.cc:1467] Processing CreateTablet for tablet 95383d9d5a234a4d9d841a8b114428b0 (DEFAULT_TABLE table=default.Table_da5216d025844afda5a83ada555bc120 [id=dc55b25c3ca14863a21042e805e3bf9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:32.274334 18457 tablet_service.cc:1467] Processing CreateTablet for tablet a0baee91f5b44945b595d0e234af018e (DEFAULT_TABLE table=default.Table_0604805f009c4d6486965426c82dff10 [id=b3d1c2f97c4648518dbea83a46cadd85]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:32.275000 19648 leader_election.cc:288] T a05e32a33aa7488eb287cc6314c09c44 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Requested vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:32.275081 18463 tablet_service.cc:1467] Processing CreateTablet for tablet de1dfe6c82d34a4d8d02e466c55327ff (DEFAULT_TABLE table=default.Table_e431f94220aa44ecba799a54dfb612ae [id=1e785b6c159b4312bd1b064271f55a49]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:32.275413 19186 raft_consensus.cc:504] T dd9b893d48684cfea7c242b93ecba01f P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:32.276863 18761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd9b893d48684cfea7c242b93ecba01f" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
W20240430 20:11:32.277456 18417 leader_election.cc:341] T dd9b893d48684cfea7c242b93ecba01f P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:32.277612 19186 leader_election.cc:288] T dd9b893d48684cfea7c242b93ecba01f P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Requested vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:32.278928 18737 tablet_service.cc:1514] Processing DeleteTablet for tablet fae9a35a2ac844b4bc2ebdf214aa7171 with delete_type TABLET_DATA_DELETED (replaced by tablet b22aaedb0a6c4fd9ad0ea8fa019001e1) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:32.279399 18731 tablet_service.cc:1467] Processing CreateTablet for tablet 3ec3993042f5498b818bdf32d5087928 (DEFAULT_TABLE table=default.Table_f323a3bead0249b5a4a3cbb07a5363b8 [id=27217d463b9c40ec926a1862a31f1f24]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:32.279906 18731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ec3993042f5498b818bdf32d5087928. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:32.280772 18724 tablet_service.cc:1467] Processing CreateTablet for tablet 95383d9d5a234a4d9d841a8b114428b0 (DEFAULT_TABLE table=default.Table_da5216d025844afda5a83ada555bc120 [id=dc55b25c3ca14863a21042e805e3bf9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:32.281284 18724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95383d9d5a234a4d9d841a8b114428b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:32.282963 18726 tablet_service.cc:1467] Processing CreateTablet for tablet a0baee91f5b44945b595d0e234af018e (DEFAULT_TABLE table=default.Table_0604805f009c4d6486965426c82dff10 [id=b3d1c2f97c4648518dbea83a46cadd85]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:32.283464 18726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0baee91f5b44945b595d0e234af018e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:32.286981 18730 tablet_service.cc:1467] Processing CreateTablet for tablet de1dfe6c82d34a4d8d02e466c55327ff (DEFAULT_TABLE table=default.Table_e431f94220aa44ecba799a54dfb612ae [id=1e785b6c159b4312bd1b064271f55a49]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:32.287549 18730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de1dfe6c82d34a4d8d02e466c55327ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:32.290683 18737 tablet_service.cc:1467] Processing CreateTablet for tablet c6378f1495144c458af49b3c19a2b67c (DEFAULT_TABLE table=default.Table_3f42877693b04aaa837c89b8983d5a72 [id=6d499a625cee4df2a4f40ee189cc6220]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:32.291170 18737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6378f1495144c458af49b3c19a2b67c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:32.291481 18450 tablet_service.cc:1467] Processing CreateTablet for tablet c6378f1495144c458af49b3c19a2b67c (DEFAULT_TABLE table=default.Table_3f42877693b04aaa837c89b8983d5a72 [id=6d499a625cee4df2a4f40ee189cc6220]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:32.306114 18459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ec3993042f5498b818bdf32d5087928. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:32.306481 18449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95383d9d5a234a4d9d841a8b114428b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:32.307070 18463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de1dfe6c82d34a4d8d02e466c55327ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:32.308096 18457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0baee91f5b44945b595d0e234af018e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:32.308676 18450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6378f1495144c458af49b3c19a2b67c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:32.318295 19004 catalog_manager.cc:4830] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): tablet a1aeee1b02214975a376cfd36957afe8 (table default.Table_31d8d978e96c4e12a0fbd5500c7b93cd [id=8b480956f4d146e08dcfed0d0a0e4e0d]) successfully deleted
W20240430 20:11:32.318454 18558 leader_election.cc:341] T fcc9698abda4491b98e1e879589913a8 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:32.318526 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 41) took 1360 ms. Trace:
I20240430 20:11:32.318647 18558 leader_election.cc:302] T fcc9698abda4491b98e1e879589913a8 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4
W20240430 20:11:32.319041 18558 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.177.1:41217, user_credentials={real_user=slave}} blocked reactor thread for 79989.8us
I20240430 20:11:32.319608 18988 consensus_queue.cc:260] T 01436e5b6232494babbb414342f03090 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:32.320137 19674 raft_consensus.cc:482] T 01436e5b6232494babbb414342f03090 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:32.320328 19674 raft_consensus.cc:504] T 01436e5b6232494babbb414342f03090 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:32.321231 19674 leader_election.cc:288] T 01436e5b6232494babbb414342f03090 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:32.323238 19180 raft_consensus.cc:2726] T fcc9698abda4491b98e1e879589913a8 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:32.323024 18622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "294644d7a16b4ca2bbb6d106bcb8c7ff" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:32.323933 18591 tablet_service.cc:1514] Processing DeleteTablet for tablet fae9a35a2ac844b4bc2ebdf214aa7171 with delete_type TABLET_DATA_DELETED (replaced by tablet b22aaedb0a6c4fd9ad0ea8fa019001e1) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:32.324661 19023 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_78678179d193432ebe9e2484951c2102"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:32.325333 19023 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_78678179d193432ebe9e2484951c2102 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:32.322582 18477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01436e5b6232494babbb414342f03090" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
W20240430 20:11:32.326695 18699 leader_election.cc:341] T 01436e5b6232494babbb414342f03090 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:32.318647 19495 rpcz_store.cc:276] 0430 20:11:30.958451 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:31.012728 (+ 54277us) service_pool.cc:225] Handling call
0430 20:11:32.318325 (+1305597us) ts_tablet_manager.cc:1120] Deleting tablet 2cdccf4be7094ca5b863dd5c3432d55d
0430 20:11:32.318337 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:32.318454 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1267154}
I20240430 20:11:32.327999 19049 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_42c1cf2f2e9542feaaa80106ced02347" } new_table_name: "default.Table_2f122c3c1c5949108231e55f6326e934" modify_external_catalogs: true
I20240430 20:11:32.328295 18988 ts_tablet_manager.cc:1398] T 01436e5b6232494babbb414342f03090 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.069s user 0.000s sys 0.002s
I20240430 20:11:32.328713 18988 tablet_bootstrap.cc:492] T dd9b893d48684cfea7c242b93ecba01f P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
W20240430 20:11:32.329412 19004 catalog_manager.cc:4810] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): delete failed for tablet 2cdccf4be7094ca5b863dd5c3432d55d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2cdccf4be7094ca5b863dd5c3432d55d already in progress: creating tablet
I20240430 20:11:32.330103 19027 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_c6d5c8ecce28428790b07d228879a0e4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:32.330637 19027 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c6d5c8ecce28428790b07d228879a0e4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:32.334481 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 43) took 1362 ms. Trace:
I20240430 20:11:32.334638 19495 rpcz_store.cc:276] 0430 20:11:30.972318 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:31.002552 (+ 30234us) service_pool.cc:225] Handling call
0430 20:11:32.334251 (+1331699us) ts_tablet_manager.cc:1120] Deleting tablet 75ebf051383e42529dcb5f7dd0770a4b
0430 20:11:32.334265 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:32.334412 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1237973}
W20240430 20:11:32.335700 19004 catalog_manager.cc:4810] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): delete failed for tablet 75ebf051383e42529dcb5f7dd0770a4b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 75ebf051383e42529dcb5f7dd0770a4b already in progress: creating tablet
I20240430 20:11:32.336704 18622 raft_consensus.cc:2454] T 294644d7a16b4ca2bbb6d106bcb8c7ff P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
W20240430 20:11:32.342253 19033 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d61c279d20b0471a967942525ceb4fcb consensus state has no leader
I20240430 20:11:32.342696 19033 master_service.cc:946] ReplaceTablet: received request to replace tablet 87ac32a0ddb84676adc83038ef664591 from {username='slave'} at 127.0.0.1:51664
I20240430 20:11:32.342767 18590 tablet_service.cc:1467] Processing CreateTablet for tablet 95383d9d5a234a4d9d841a8b114428b0 (DEFAULT_TABLE table=default.Table_da5216d025844afda5a83ada555bc120 [id=dc55b25c3ca14863a21042e805e3bf9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:32.345754 18621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a05e32a33aa7488eb287cc6314c09c44" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:32.346148 18588 tablet_service.cc:1467] Processing CreateTablet for tablet 3ec3993042f5498b818bdf32d5087928 (DEFAULT_TABLE table=default.Table_f323a3bead0249b5a4a3cbb07a5363b8 [id=27217d463b9c40ec926a1862a31f1f24]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:32.346370 19028 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: Sending DeleteTablet for 3 replicas of tablet 44fa6917a85c46f3aaa363df46ea586b
I20240430 20:11:32.346922 18696 leader_election.cc:302] T 294644d7a16b4ca2bbb6d106bcb8c7ff P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a
I20240430 20:11:32.347069 19028 master_service.cc:946] ReplaceTablet: received request to replace tablet 26ed12f498644d15bf45e884dcad172e from {username='slave'} at 127.0.0.1:51664
W20240430 20:11:32.347462 18696 leader_election.cc:341] T a05e32a33aa7488eb287cc6314c09c44 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:32.347548 19028 master_service.cc:946] ReplaceTablet: received request to replace tablet f940d17b02b541289121463e05b0bb5f from {username='slave'} at 127.0.0.1:51664
I20240430 20:11:32.347591 18481 raft_consensus.cc:2454] T a05e32a33aa7488eb287cc6314c09c44 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:32.348278 18699 leader_election.cc:302] T a05e32a33aa7488eb287cc6314c09c44 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters: 5e1c6397f09747f091534faf93542710
I20240430 20:11:32.348790 19674 raft_consensus.cc:2781] T 294644d7a16b4ca2bbb6d106bcb8c7ff P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:32.348883 19048 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: Sending DeleteTablet for 3 replicas of tablet 73453a7fa30342e29db552b17b62f0c1
I20240430 20:11:32.348898 19674 raft_consensus.cc:482] T 294644d7a16b4ca2bbb6d106bcb8c7ff P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:32.349000 19674 raft_consensus.cc:3037] T 294644d7a16b4ca2bbb6d106bcb8c7ff P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:32.349114 19648 raft_consensus.cc:2781] T a05e32a33aa7488eb287cc6314c09c44 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:32.349311 19648 raft_consensus.cc:686] T a05e32a33aa7488eb287cc6314c09c44 P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Becoming Leader. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Running, Role: LEADER
I20240430 20:11:32.349568 19048 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_b331bf8aeaae4911ac01ed3fbd2627bd" } modify_external_catalogs: true
I20240430 20:11:32.349776 19048 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_b331bf8aeaae4911ac01ed3fbd2627bd" } modify_external_catalogs: true
I20240430 20:11:32.349665 19648 consensus_queue.cc:237] T a05e32a33aa7488eb287cc6314c09c44 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:32.350317 19026 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: Sending DeleteTablet for 3 replicas of tablet b72fe1f0713b4076a81a3fc1c74f7fa7
I20240430 20:11:32.351596 19026 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_3a35534b6ef2416b968740d28b156dd0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:32.352198 19026 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3a35534b6ef2416b968740d28b156dd0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:32.354032 18454 tablet_service.cc:1514] Processing DeleteTablet for tablet 44fa6917a85c46f3aaa363df46ea586b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:32 UTC) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:32.356515 18730 tablet_service.cc:1514] Processing DeleteTablet for tablet 44fa6917a85c46f3aaa363df46ea586b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:32 UTC) from {username='slave'} at 127.0.0.1:59830
W20240430 20:11:32.363080 19029 catalog_manager.cc:5676] Failed to send AlterTable RPC for tablet 2e79df8744e74b5991a79ce978ff0202 (table default.Table_276bbdc7cfe542d1a2936485a2aad1ec [id=f146a15e6e8a4acbb4c734c1c863da57], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 2e79df8744e74b5991a79ce978ff0202 consensus state has no leader
I20240430 20:11:32.363112 18454 tablet_service.cc:1514] Processing DeleteTablet for tablet b72fe1f0713b4076a81a3fc1c74f7fa7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:32 UTC) from {username='slave'} at 127.0.0.1:41680
W20240430 20:11:32.364940 19029 catalog_manager.cc:5676] Failed to send AlterTable RPC for tablet 2da900b6bbfa4fa99fe8494fd2e68da8 (table default.Table_19a91a7f17ff456ebb145e8dd89c0940 [id=67283dd3995a44c6a7a7bc8722e4f955], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 2da900b6bbfa4fa99fe8494fd2e68da8 consensus state has no leader
I20240430 20:11:32.370914 18454 tablet_service.cc:1514] Processing DeleteTablet for tablet 75ebf051383e42529dcb5f7dd0770a4b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:27 UTC) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:32.371708 18461 tablet_service.cc:1514] Processing DeleteTablet for tablet 115c4a12440647a4bb46eb6824f7ea98 with delete_type TABLET_DATA_DELETED (Sending initial creation of tablet) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:32.372819 18454 tablet_service.cc:1514] Processing DeleteTablet for tablet 2cdccf4be7094ca5b863dd5c3432d55d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:30 UTC) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:32.373214 18461 tablet_service.cc:1514] Processing DeleteTablet for tablet a1aeee1b02214975a376cfd36957afe8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:26 UTC) from {username='slave'} at 127.0.0.1:41680
W20240430 20:11:32.374542 19029 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.17.177.1:50353 (request call id 83) took 780 ms (client timeout 1000 ms). Trace:
W20240430 20:11:32.374933 19029 rpcz_store.cc:269] 0430 20:11:31.594485 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:31.597268 (+ 2783us) service_pool.cc:225] Handling call
0430 20:11:31.998528 (+401260us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:31.998535 (+ 7us) write_op.cc:446] Released schema lock
0430 20:11:32.374495 (+375960us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:31.990991 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:31.991145 (+ 154us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:31.991156 (+ 11us) write_op.cc:427] Acquired schema lock
0430 20:11:31.991159 (+ 3us) tablet.cc:660] Decoding operations
0430 20:11:31.991285 (+ 126us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:31.991301 (+ 16us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:31.991305 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0430 20:11:31.991390 (+ 85us) tablet.cc:699] Row locks acquired
0430 20:11:31.991392 (+ 2us) write_op.cc:260] PREPARE: finished
0430 20:11:31.991435 (+ 43us) write_op.cc:270] Start()
0430 20:11:31.991475 (+ 40us) write_op.cc:276] Timestamp: P: 1714507891991382 usec, L: 0
0430 20:11:31.991479 (+ 4us) op_driver.cc:331] REPLICATION: starting
0430 20:11:31.991709 (+ 230us) log.cc:842] Serialized 530 byte log entry
0430 20:11:31.995499 (+ 3790us) op_driver.cc:444] REPLICATION: finished
0430 20:11:31.995646 (+ 147us) write_op.cc:301] APPLY: starting
0430 20:11:31.995662 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:31.995709 (+ 47us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:31.995741 (+ 32us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:31.995830 (+ 89us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:31.995865 (+ 35us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0430 20:11:31.995873 (+ 8us) write_op.cc:312] APPLY: finished
0430 20:11:31.995952 (+ 79us) log.cc:842] Serialized 23 byte log entry
0430 20:11:31.996012 (+ 60us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:31.996034 (+ 22us) write_op.cc:446] Released schema lock
0430 20:11:31.996041 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
0430 20:11:32.232832 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:32.232968 (+ 136us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:32.232975 (+ 7us) write_op.cc:427] Acquired schema lock
0430 20:11:32.232978 (+ 3us) tablet.cc:660] Decoding operations
0430 20:11:32.233099 (+ 121us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:32.233114 (+ 15us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:32.233119 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0430 20:11:32.233191 (+ 72us) tablet.cc:699] Row locks acquired
0430 20:11:32.233193 (+ 2us) write_op.cc:260] PREPARE: finished
0430 20:11:32.233240 (+ 47us) write_op.cc:270] Start()
0430 20:11:32.233280 (+ 40us) write_op.cc:276] Timestamp: P: 1714507892233202 usec, L: 0
0430 20:11:32.233285 (+ 5us) op_driver.cc:331] REPLICATION: starting
0430 20:11:32.246966 (+ 13681us) log.cc:842] Serialized 356 byte log entry
0430 20:11:32.249958 (+ 2992us) spinlock_profiling.cc:243] Waited 13.5 ms on lock 0x616000040654. stack: 00007feb16c74756 00007feb16c74510 00007feb15ca748d 0000000000955804 00000000009557cd 000000000095519a 00007feb224b9162 00007feb23821b02 00007feb23820629 00007feb2382853b 00007feb238283b8 0000000000a60bcc 00007feb16cbab74 00007feb16cbe19b 00007feb16cbe018 0000000000a60bcc
0430 20:11:32.257481 (+ 7523us) op_driver.cc:444] REPLICATION: finished
0430 20:11:32.257678 (+ 197us) write_op.cc:301] APPLY: starting
0430 20:11:32.257696 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:32.257750 (+ 54us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:32.257753 (+ 3us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:32.257811 (+ 58us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:32.257844 (+ 33us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0430 20:11:32.257850 (+ 6us) write_op.cc:312] APPLY: finished
0430 20:11:32.257984 (+ 134us) log.cc:842] Serialized 23 byte log entry
0430 20:11:32.258044 (+ 60us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:32.258060 (+ 16us) write_op.cc:446] Released schema lock
0430 20:11:32.258065 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:32.258122 (+ 57us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:32.258126 (+ 4us) write_op.cc:446] Released schema lock
Metrics: {"reported_tablets":102,"child_traces":[["op",{"apply.queue_time_us":94,"apply.run_cpu_time_us":453,"apply.run_wall_time_us":452,"num_ops":1,"prepare.queue_time_us":3343,"prepare.run_cpu_time_us":860,"prepare.run_wall_time_us":861,"raft.queue_time_us":1109,"raft.run_cpu_time_us":259,"raft.run_wall_time_us":258,"replication_time_us":3992}],["op",{"apply.queue_time_us":138,"apply.run_cpu_time_us":504,"apply.run_wall_time_us":504,"num_ops":1,"prepare.queue_time_us":58248,"prepare.run_cpu_time_us":1127,"prepare.run_wall_time_us":18405,"raft.queue_time_us":88,"raft.run_cpu_time_us":374,"raft.run_wall_time_us":377,"replication_time_us":24173,"spinlock_wait_cycles":31030144}]]}
I20240430 20:11:32.378293 19029 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_f4b025d97b144efa9449e830a6f44126"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:32.379765 19029 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f4b025d97b144efa9449e830a6f44126 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:32.380450 18583 tablet_service.cc:1467] Processing CreateTablet for tablet a0baee91f5b44945b595d0e234af018e (DEFAULT_TABLE table=default.Table_0604805f009c4d6486965426c82dff10 [id=b3d1c2f97c4648518dbea83a46cadd85]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:32.380988 19645 raft_consensus.cc:482] T 47c37876694c4a4abb1091c631dab7a5 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:32.380959 18623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd9b893d48684cfea7c242b93ecba01f" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:32.381204 18623 raft_consensus.cc:3037] T dd9b893d48684cfea7c242b93ecba01f P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:32.381182 19645 raft_consensus.cc:504] T 47c37876694c4a4abb1091c631dab7a5 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:32.381994 19645 leader_election.cc:288] T 47c37876694c4a4abb1091c631dab7a5 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:32.383468 18591 tablet_service.cc:1467] Processing CreateTablet for tablet c6378f1495144c458af49b3c19a2b67c (DEFAULT_TABLE table=default.Table_3f42877693b04aaa837c89b8983d5a72 [id=6d499a625cee4df2a4f40ee189cc6220]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:32.383877 18619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01436e5b6232494babbb414342f03090" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:32.384227 18619 raft_consensus.cc:2454] T 01436e5b6232494babbb414342f03090 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:32.384899 18603 tablet_service.cc:1467] Processing CreateTablet for tablet de1dfe6c82d34a4d8d02e466c55327ff (DEFAULT_TABLE table=default.Table_e431f94220aa44ecba799a54dfb612ae [id=1e785b6c159b4312bd1b064271f55a49]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:32.385412 18622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53a899fa5a424a0d99f1b47925e8c7ad" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:32.387245 18463 tablet_service.cc:1514] Processing DeleteTablet for tablet 73453a7fa30342e29db552b17b62f0c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:32 UTC) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:32.389189 18590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95383d9d5a234a4d9d841a8b114428b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:32.392565 18602 tablet_service.cc:1514] Processing DeleteTablet for tablet 44fa6917a85c46f3aaa363df46ea586b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:32 UTC) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:32.392802 18601 tablet_service.cc:1514] Processing DeleteTablet for tablet 73453a7fa30342e29db552b17b62f0c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:32 UTC) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:32.393304 18587 tablet_service.cc:1514] Processing DeleteTablet for tablet b72fe1f0713b4076a81a3fc1c74f7fa7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:32 UTC) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:32.393505 18583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0baee91f5b44945b595d0e234af018e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:32.394304 18591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6378f1495144c458af49b3c19a2b67c. 1 dirs total, 0 dirs full, 0 dirs failed
W20240430 20:11:32.394941 18415 consensus_peers.cc:485] T c5fd0730133c472299c93f4b0830e527 P 2efb542aa55e439b99ccb98d1c5c884a -> Peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Couldn't send request to peer 5e1c6397f09747f091534faf93542710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c5fd0730133c472299c93f4b0830e527. This is attempt 1: this message will repeat every 5th retry.
I20240430 20:11:32.395159 19624 tablet.cc:1722] T 724e51ff05ec4baebc2bb55e05c8eddc P 5e1c6397f09747f091534faf93542710: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240430 20:11:32.409065 18983 tablet_bootstrap.cc:492] T a05e32a33aa7488eb287cc6314c09c44 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:32.410090 18730 tablet_service.cc:1514] Processing DeleteTablet for tablet 73453a7fa30342e29db552b17b62f0c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:32 UTC) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:32.410429 19581 raft_consensus.cc:482] T eed6338ad3d64859af087ee15e1ec798 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:32.411515 18761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c37876694c4a4abb1091c631dab7a5" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
W20240430 20:11:32.411918 18558 leader_election.cc:341] T 47c37876694c4a4abb1091c631dab7a5 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Not found: Tablet not found: 47c37876694c4a4abb1091c631dab7a5
I20240430 20:11:32.412487 18696 leader_election.cc:302] T 01436e5b6232494babbb414342f03090 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a
W20240430 20:11:32.412575 18557 leader_election.cc:341] T 47c37876694c4a4abb1091c631dab7a5 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:32.412698 18557 leader_election.cc:302] T 47c37876694c4a4abb1091c631dab7a5 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:32.413772 18603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de1dfe6c82d34a4d8d02e466c55327ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:32.413952 19474 raft_consensus.cc:2726] T 47c37876694c4a4abb1091c631dab7a5 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:32.413914 19581 raft_consensus.cc:504] T eed6338ad3d64859af087ee15e1ec798 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
W20240430 20:11:32.414121 18696 leader_election.cc:341] T 53a899fa5a424a0d99f1b47925e8c7ad P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:32.414240 18696 leader_election.cc:302] T 53a899fa5a424a0d99f1b47925e8c7ad P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710
I20240430 20:11:32.414633 19581 leader_election.cc:288] T eed6338ad3d64859af087ee15e1ec798 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:32.414820 19581 raft_consensus.cc:2726] T 53a899fa5a424a0d99f1b47925e8c7ad P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:32.414846 19492 raft_consensus.cc:2781] T 01436e5b6232494babbb414342f03090 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:32.414938 19492 raft_consensus.cc:482] T 01436e5b6232494babbb414342f03090 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:32.415041 19492 raft_consensus.cc:3037] T 01436e5b6232494babbb414342f03090 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:32.415138 18621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eed6338ad3d64859af087ee15e1ec798" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:32.415401 18621 raft_consensus.cc:2454] T eed6338ad3d64859af087ee15e1ec798 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:32.415779 18724 tablet_service.cc:1514] Processing DeleteTablet for tablet b72fe1f0713b4076a81a3fc1c74f7fa7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:32 UTC) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:32.416368 18477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eed6338ad3d64859af087ee15e1ec798" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:32.416697 18477 raft_consensus.cc:2454] T eed6338ad3d64859af087ee15e1ec798 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:32.418988 18588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ec3993042f5498b818bdf32d5087928. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:32.420050 18699 leader_election.cc:302] T eed6338ad3d64859af087ee15e1ec798 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters:
W20240430 20:11:32.420145 18558 consensus_peers.cc:485] T a1aeee1b02214975a376cfd36957afe8 P 5e1c6397f09747f091534faf93542710 -> Peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Couldn't send request to peer 2efb542aa55e439b99ccb98d1c5c884a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a1aeee1b02214975a376cfd36957afe8. This is attempt 1: this message will repeat every 5th retry.
I20240430 20:11:32.420830 19581 raft_consensus.cc:2781] T eed6338ad3d64859af087ee15e1ec798 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:32.420962 19581 raft_consensus.cc:482] T eed6338ad3d64859af087ee15e1ec798 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:32.421070 19581 raft_consensus.cc:3037] T eed6338ad3d64859af087ee15e1ec798 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:32.423648 19648 raft_consensus.cc:482] T e3e74e76ab2b4fe492ffcc1838656fd1 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:32.423821 19648 raft_consensus.cc:504] T e3e74e76ab2b4fe492ffcc1838656fd1 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:32.424649 18983 ts_tablet_manager.cc:1367] T a05e32a33aa7488eb287cc6314c09c44 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.181s user 0.002s sys 0.026s
I20240430 20:11:32.424764 19648 leader_election.cc:288] T e3e74e76ab2b4fe492ffcc1838656fd1 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:32.425133 18620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3e74e76ab2b4fe492ffcc1838656fd1" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
W20240430 20:11:32.425477 19004 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 46251c08277e498da04a41d009310c62 (table default.Table_4e56923d440944978f0aacc140a0dd8e [id=b1312a6cd1a54003b1db727a2a9d125d], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 46251c08277e498da04a41d009310c62 consensus state has no leader
W20240430 20:11:32.425734 18696 leader_election.cc:341] T e3e74e76ab2b4fe492ffcc1838656fd1 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:32.426342 18481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3e74e76ab2b4fe492ffcc1838656fd1" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:32.426630 18481 raft_consensus.cc:2454] T e3e74e76ab2b4fe492ffcc1838656fd1 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:32.427076 18699 leader_election.cc:302] T e3e74e76ab2b4fe492ffcc1838656fd1 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters: 5e1c6397f09747f091534faf93542710
I20240430 20:11:32.427443 19648 raft_consensus.cc:2781] T e3e74e76ab2b4fe492ffcc1838656fd1 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:32.427548 19648 raft_consensus.cc:482] T e3e74e76ab2b4fe492ffcc1838656fd1 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:32.427651 19648 raft_consensus.cc:3037] T e3e74e76ab2b4fe492ffcc1838656fd1 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:32.427743 19495 tablet_replica.cc:330] T 02176eee8e044a7db8cac48439e9f594 P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:32.427623 18983 raft_consensus.cc:348] T a05e32a33aa7488eb287cc6314c09c44 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:32.428020 19495 raft_consensus.cc:2229] T 02176eee8e044a7db8cac48439e9f594 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:32.428606 19495 raft_consensus.cc:2258] T 02176eee8e044a7db8cac48439e9f594 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:32.429507 19495 ts_tablet_manager.cc:1875] T 02176eee8e044a7db8cac48439e9f594 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240430 20:11:32.429976 19049 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e87fb2f0361d4aa3ada27bbdfa88e94c consensus state has no leader
I20240430 20:11:32.430756 18983 raft_consensus.cc:374] T a05e32a33aa7488eb287cc6314c09c44 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:32.430894 18983 raft_consensus.cc:729] T a05e32a33aa7488eb287cc6314c09c44 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:32.431159 18983 consensus_queue.cc:260] T a05e32a33aa7488eb287cc6314c09c44 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:32.431737 18983 ts_tablet_manager.cc:1398] T a05e32a33aa7488eb287cc6314c09c44 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20240430 20:11:32.432068 18983 tablet_bootstrap.cc:492] T 75ebf051383e42529dcb5f7dd0770a4b P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:32.432893 19023 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_26cbad4b7efa400f946f15b31c9ff031" } new_table_name: "default.Table_9dc9f3b0b0c8452a9da7c85e62d87bdc" modify_external_catalogs: true
I20240430 20:11:32.433683 18983 tablet_bootstrap.cc:654] T 75ebf051383e42529dcb5f7dd0770a4b P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:32.434610 19027 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_fda07fd7279c484db2bee67e8c859855" } new_table_name: "default.Table_4130caf534004edfb413056d7f6cfd52" modify_external_catalogs: true
I20240430 20:11:32.435638 19033 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: Sending DeleteTablet for 3 replicas of tablet 87ac32a0ddb84676adc83038ef664591
I20240430 20:11:32.436281 19033 catalog_manager.cc:6554] ReplaceTablet: tablet 87ac32a0ddb84676adc83038ef664591 deleted and replaced by tablet 331c031d44854d679a27d10d9151d0ea
I20240430 20:11:32.436622 19033 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_388d774462294cab8342c6de331f9712" } modify_external_catalogs: true
I20240430 20:11:32.436833 19033 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_388d774462294cab8342c6de331f9712" } modify_external_catalogs: true
I20240430 20:11:32.436923 18983 tablet_bootstrap.cc:492] T 75ebf051383e42529dcb5f7dd0770a4b P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:32.437067 18983 ts_tablet_manager.cc:1367] T 75ebf051383e42529dcb5f7dd0770a4b P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20240430 20:11:32.437867 18983 raft_consensus.cc:348] T 75ebf051383e42529dcb5f7dd0770a4b P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:32.438058 18983 raft_consensus.cc:374] T 75ebf051383e42529dcb5f7dd0770a4b P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:32.438151 18983 raft_consensus.cc:729] T 75ebf051383e42529dcb5f7dd0770a4b P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:32.438200 18459 tablet_service.cc:1514] Processing DeleteTablet for tablet 87ac32a0ddb84676adc83038ef664591 with delete_type TABLET_DATA_DELETED (replaced by tablet 331c031d44854d679a27d10d9151d0ea) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:32.439253 18724 tablet_service.cc:1514] Processing DeleteTablet for tablet 87ac32a0ddb84676adc83038ef664591 with delete_type TABLET_DATA_DELETED (replaced by tablet 331c031d44854d679a27d10d9151d0ea) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:32.440367 18983 consensus_queue.cc:260] T 75ebf051383e42529dcb5f7dd0770a4b P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:32.441329 18623 raft_consensus.cc:2454] T dd9b893d48684cfea7c242b93ecba01f P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 1.
I20240430 20:11:32.445583 19674 raft_consensus.cc:504] T 294644d7a16b4ca2bbb6d106bcb8c7ff P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:32.445936 18983 ts_tablet_manager.cc:1398] T 75ebf051383e42529dcb5f7dd0770a4b P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240430 20:11:32.446292 18983 tablet_bootstrap.cc:492] T 255c13ad8fae4ea4aa8e7b7aabb7602e P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:32.446467 19674 leader_election.cc:288] T 294644d7a16b4ca2bbb6d106bcb8c7ff P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Requested vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:32.447032 18415 leader_election.cc:302] T dd9b893d48684cfea7c242b93ecba01f P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters: 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:32.447417 19700 raft_consensus.cc:2781] T dd9b893d48684cfea7c242b93ecba01f P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:32.447559 19700 raft_consensus.cc:686] T dd9b893d48684cfea7c242b93ecba01f P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Becoming Leader. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Running, Role: LEADER
I20240430 20:11:32.448004 18983 tablet_bootstrap.cc:654] T 255c13ad8fae4ea4aa8e7b7aabb7602e P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:32.447839 19700 consensus_queue.cc:237] T dd9b893d48684cfea7c242b93ecba01f P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:32.451319 18623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "294644d7a16b4ca2bbb6d106bcb8c7ff" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:32.451584 18623 raft_consensus.cc:3037] T 294644d7a16b4ca2bbb6d106bcb8c7ff P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:32.452170 18481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "294644d7a16b4ca2bbb6d106bcb8c7ff" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:32.452436 18481 raft_consensus.cc:3037] T 294644d7a16b4ca2bbb6d106bcb8c7ff P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:32.453071 19028 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: Sending DeleteTablet for 3 replicas of tablet f940d17b02b541289121463e05b0bb5f
I20240430 20:11:32.453526 18983 tablet_bootstrap.cc:492] T 255c13ad8fae4ea4aa8e7b7aabb7602e P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:32.453564 19028 catalog_manager.cc:6554] ReplaceTablet: tablet f940d17b02b541289121463e05b0bb5f deleted and replaced by tablet 53da4b3855344b6d92211af767195dc8
I20240430 20:11:32.453696 18983 ts_tablet_manager.cc:1367] T 255c13ad8fae4ea4aa8e7b7aabb7602e P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20240430 20:11:32.453995 19028 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_47779624a66e46468c8e9e35a8e9487f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240430 20:11:32.454452 18983 raft_consensus.cc:348] T 255c13ad8fae4ea4aa8e7b7aabb7602e P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
W20240430 20:11:32.454581 19028 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_47779624a66e46468c8e9e35a8e9487f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:32.454696 18983 raft_consensus.cc:374] T 255c13ad8fae4ea4aa8e7b7aabb7602e P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:32.454774 18983 raft_consensus.cc:729] T 255c13ad8fae4ea4aa8e7b7aabb7602e P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:32.456260 19026 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_c264c250888a459694c33184912ab76d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240430 20:11:32.456832 18988 tablet_bootstrap.cc:654] T dd9b893d48684cfea7c242b93ecba01f P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
W20240430 20:11:32.456831 19026 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c264c250888a459694c33184912ab76d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:32.456861 18983 consensus_queue.cc:260] T 255c13ad8fae4ea4aa8e7b7aabb7602e P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:32.459383 18983 ts_tablet_manager.cc:1398] T 255c13ad8fae4ea4aa8e7b7aabb7602e P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240430 20:11:32.459757 18983 tablet_bootstrap.cc:492] T 53a899fa5a424a0d99f1b47925e8c7ad P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:32.460119 19029 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_0858797306c54d81aab9d77e8591e502"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:32.460718 19029 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0858797306c54d81aab9d77e8591e502 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:32.461326 18983 tablet_bootstrap.cc:654] T 53a899fa5a424a0d99f1b47925e8c7ad P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:32.466799 18983 tablet_bootstrap.cc:492] T 53a899fa5a424a0d99f1b47925e8c7ad P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:32.466907 18587 tablet_service.cc:1514] Processing DeleteTablet for tablet 87ac32a0ddb84676adc83038ef664591 with delete_type TABLET_DATA_DELETED (replaced by tablet 331c031d44854d679a27d10d9151d0ea) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:32.467005 18983 ts_tablet_manager.cc:1367] T 53a899fa5a424a0d99f1b47925e8c7ad P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.001s
I20240430 20:11:32.467221 18586 tablet_service.cc:1514] Processing DeleteTablet for tablet f940d17b02b541289121463e05b0bb5f with delete_type TABLET_DATA_DELETED (replaced by tablet 53da4b3855344b6d92211af767195dc8) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:32.467861 18983 raft_consensus.cc:348] T 53a899fa5a424a0d99f1b47925e8c7ad P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:32.468111 18983 raft_consensus.cc:374] T 53a899fa5a424a0d99f1b47925e8c7ad P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:32.468199 18983 raft_consensus.cc:729] T 53a899fa5a424a0d99f1b47925e8c7ad P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:32.468446 18983 consensus_queue.cc:260] T 53a899fa5a424a0d99f1b47925e8c7ad P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:32.469040 18983 ts_tablet_manager.cc:1398] T 53a899fa5a424a0d99f1b47925e8c7ad P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240430 20:11:32.469331 18983 tablet_bootstrap.cc:492] T 26ed12f498644d15bf45e884dcad172e P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:32.471594 18983 tablet_bootstrap.cc:654] T 26ed12f498644d15bf45e884dcad172e P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
W20240430 20:11:32.477006 19023 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2873af6da928463f9809864b8eaf0c4a consensus state has no leader
I20240430 20:11:32.477504 19023 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_dd819f96762949c786839dda00487cc2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:32.478132 19023 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dd819f96762949c786839dda00487cc2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240430 20:11:32.479545 19027 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fd40967ab4a04001819e16c06a6c4300 consensus state has no leader
I20240430 20:11:32.480010 18983 tablet_bootstrap.cc:492] T 26ed12f498644d15bf45e884dcad172e P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:32.480176 18983 ts_tablet_manager.cc:1367] T 26ed12f498644d15bf45e884dcad172e P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.003s
I20240430 20:11:32.480086 19027 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_01e4bb745b8749a28f6182742fd38168"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:32.480703 19027 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_01e4bb745b8749a28f6182742fd38168 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:32.480890 19474 raft_consensus.cc:482] T 255c13ad8fae4ea4aa8e7b7aabb7602e P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:32.480952 18983 raft_consensus.cc:348] T 26ed12f498644d15bf45e884dcad172e P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:32.481065 19474 raft_consensus.cc:504] T 255c13ad8fae4ea4aa8e7b7aabb7602e P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:32.481166 18983 raft_consensus.cc:374] T 26ed12f498644d15bf45e884dcad172e P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:32.481240 18983 raft_consensus.cc:729] T 26ed12f498644d15bf45e884dcad172e P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:32.481876 19474 leader_election.cc:288] T 255c13ad8fae4ea4aa8e7b7aabb7602e P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:32.483400 18477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "255c13ad8fae4ea4aa8e7b7aabb7602e" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:32.483428 18983 consensus_queue.cc:260] T 26ed12f498644d15bf45e884dcad172e P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:32.484092 19617 raft_consensus.cc:482] T c9ff972ade694938888514979e1638aa P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240430 20:11:32.484217 18558 leader_election.cc:341] T 255c13ad8fae4ea4aa8e7b7aabb7602e P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:32.484311 19617 raft_consensus.cc:504] T c9ff972ade694938888514979e1638aa P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:32.484030 18983 ts_tablet_manager.cc:1398] T 26ed12f498644d15bf45e884dcad172e P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240430 20:11:32.485102 19617 leader_election.cc:288] T c9ff972ade694938888514979e1638aa P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:32.485419 18983 tablet_bootstrap.cc:492] T e3e74e76ab2b4fe492ffcc1838656fd1 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:32.487253 18983 tablet_bootstrap.cc:654] T e3e74e76ab2b4fe492ffcc1838656fd1 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
W20240430 20:11:32.487696 18415 leader_election.cc:341] T c9ff972ade694938888514979e1638aa P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Not found: Tablet not found: c9ff972ade694938888514979e1638aa
I20240430 20:11:32.491148 18586 tablet_service.cc:1514] Processing DeleteTablet for tablet 115c4a12440647a4bb46eb6824f7ea98 with delete_type TABLET_DATA_DELETED (Sending initial creation of tablet) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:32.492728 19026 master_service.cc:946] ReplaceTablet: received request to replace tablet b688e03ea71940c69bda5520b62c1fc3 from {username='slave'} at 127.0.0.1:51664
I20240430 20:11:32.323032 18985 tablet_bootstrap.cc:654] T acd23f0a63d5448aaa7ed78422e67bad P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:32.499437 19028 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_3106434588bd4241a314064d6a8749c3" } new_table_name: "default.Table_c671e4288ad041dda677eb1024151188" modify_external_catalogs: true
I20240430 20:11:32.500816 19029 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_8a6b5152d20b4890918aedb454764ee9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:32.501366 19029 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8a6b5152d20b4890918aedb454764ee9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240430 20:11:32.504240 19024 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.17.177.2:41749 (request call id 92) took 796 ms (client timeout 1000 ms). Trace:
I20240430 20:11:32.506700 18724 tablet_service.cc:1514] Processing DeleteTablet for tablet f940d17b02b541289121463e05b0bb5f with delete_type TABLET_DATA_DELETED (replaced by tablet 53da4b3855344b6d92211af767195dc8) from {username='slave'} at 127.0.0.1:59830
W20240430 20:11:32.511350 18417 leader_election.cc:341] T c9ff972ade694938888514979e1638aa P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Not found: Tablet not found: c9ff972ade694938888514979e1638aa
I20240430 20:11:32.511575 18417 leader_election.cc:302] T c9ff972ade694938888514979e1638aa P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a; no voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710
I20240430 20:11:32.512291 19617 raft_consensus.cc:2726] T c9ff972ade694938888514979e1638aa P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:32.514139 18461 tablet_service.cc:1467] Processing CreateTablet for tablet b22aaedb0a6c4fd9ad0ea8fa019001e1 (DEFAULT_TABLE table=default.Table_730a215d7af74a5bb753adca39dea676 [id=a5d580cffa2447d78fc9f2c8e31264f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:32.514689 18461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b22aaedb0a6c4fd9ad0ea8fa019001e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:32.512748 19049 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet c66c142419ca4e1399ecd1c8d1e62a59 (table default.Table_0443359dbeff437fb6cffb0db33cbb30 [id=93c44abdf6b147228224388398cd7aeb]). Expected version 2 got 0
I20240430 20:11:32.516534 19049 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 294644d7a16b4ca2bbb6d106bcb8c7ff (table default.Table_190b2d86515a44898fa38a49517b9df1 [id=97d8a36ddfae4bb5b99c04a7a08544af]). Expected version 2 got 0
W20240430 20:11:32.516106 19024 rpcz_store.cc:269] 0430 20:11:31.707692 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:31.754509 (+ 46817us) service_pool.cc:225] Handling call
0430 20:11:32.504178 (+749669us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:32.191335 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:32.191529 (+ 194us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:32.191543 (+ 14us) write_op.cc:427] Acquired schema lock
0430 20:11:32.191550 (+ 7us) tablet.cc:660] Decoding operations
0430 20:11:32.191803 (+ 253us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:32.191824 (+ 21us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:32.191833 (+ 9us) tablet.cc:683] Acquiring locks for 1 operations
0430 20:11:32.191955 (+ 122us) tablet.cc:699] Row locks acquired
0430 20:11:32.191962 (+ 7us) write_op.cc:260] PREPARE: finished
0430 20:11:32.192029 (+ 67us) write_op.cc:270] Start()
0430 20:11:32.192094 (+ 65us) write_op.cc:276] Timestamp: P: 1714507892191986 usec, L: 0
0430 20:11:32.192102 (+ 8us) op_driver.cc:331] REPLICATION: starting
0430 20:11:32.192360 (+ 258us) log.cc:842] Serialized 530 byte log entry
0430 20:11:32.198897 (+ 6537us) op_driver.cc:444] REPLICATION: finished
0430 20:11:32.202679 (+ 3782us) write_op.cc:301] APPLY: starting
0430 20:11:32.202689 (+ 10us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:32.202718 (+ 29us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:32.202749 (+ 31us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:32.202785 (+ 36us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:32.202811 (+ 26us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0430 20:11:32.202818 (+ 7us) write_op.cc:312] APPLY: finished
0430 20:11:32.202887 (+ 69us) log.cc:842] Serialized 23 byte log entry
0430 20:11:32.202927 (+ 40us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:32.202939 (+ 12us) write_op.cc:446] Released schema lock
0430 20:11:32.202945 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:32.202990 (+ 45us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:32.202994 (+ 4us) write_op.cc:446] Released schema lock
Metrics: {"reported_tablets":98,"child_traces":[["op",{"apply.queue_time_us":3748,"apply.run_cpu_time_us":355,"apply.run_wall_time_us":354,"num_ops":1,"prepare.queue_time_us":61476,"prepare.run_cpu_time_us":1114,"prepare.run_wall_time_us":1486,"raft.queue_time_us":29,"raft.run_cpu_time_us":3,"raft.run_wall_time_us":3,"replication_time_us":6776}]]}
I20240430 20:11:32.519879 18459 tablet_service.cc:1514] Processing DeleteTablet for tablet f940d17b02b541289121463e05b0bb5f with delete_type TABLET_DATA_DELETED (replaced by tablet 53da4b3855344b6d92211af767195dc8) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:32.523519 18587 tablet_service.cc:1514] Processing DeleteTablet for tablet 2cdccf4be7094ca5b863dd5c3432d55d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:30 UTC) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:32.525550 18463 tablet_service.cc:1467] Processing CreateTablet for tablet 04b3dff86d5c48a09d03da58bd48f360 (DEFAULT_TABLE table=default.Table_640bb0fa95ca44d49921276d665b9ca3 [id=9fa22833979f4c2f978de651145acef9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:32.527485 18454 tablet_service.cc:1467] Processing CreateTablet for tablet 0a16d98f700c46e29f6dd8568752ef29 (DEFAULT_TABLE table=default.Table_925990f1c73245818b39aafeb0de55e3 [id=5979225bccda4e35bea4bec2c67b22df]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:32.528065 18446 tablet_service.cc:1467] Processing CreateTablet for tablet 22fef8bb6b034db098c3372ae992f7a7 (DEFAULT_TABLE table=default.Table_891f568fffb349c097f751981343f2b1 [id=7d186ca20c094cc19aef792fe1c094e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:32.529310 18457 tablet_service.cc:1467] Processing CreateTablet for tablet c6e3d09402574f1e8fe9e4160c70a3ee (DEFAULT_TABLE table=default.Table_2a72e434cca94803ae3e8dfcd2e47235 [id=450f5d1c7e084a1bb88d035c0ccfcf82]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:32.529784 18463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04b3dff86d5c48a09d03da58bd48f360. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:32.530089 18446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22fef8bb6b034db098c3372ae992f7a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:32.533989 18454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a16d98f700c46e29f6dd8568752ef29. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:32.538664 18455 tablet_service.cc:1467] Processing CreateTablet for tablet 8e33df70a1504dbd852fe0b3c3f61622 (DEFAULT_TABLE table=default.Table_21208fd7cdca4c73a4857bc4692187f6 [id=da22cb339ab647e8afda7aa92f3548a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:32.539119 18455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e33df70a1504dbd852fe0b3c3f61622. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:32.538656 18449 tablet_service.cc:1467] Processing CreateTablet for tablet a482f4622f774d60b4efd6827b8be7d7 (DEFAULT_TABLE table=default.Table_b3741e0be6594cfab0de42dec8e1f2fe [id=c3dc9f05316d46d6a3ef4d03f40877ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:32.546047 18449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a482f4622f774d60b4efd6827b8be7d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:32.548929 18457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6e3d09402574f1e8fe9e4160c70a3ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:32.549226 19023 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_453f21c4c2ad4ef7beca9aff68e47b35" } new_table_name: "default.Table_07e40af8f1c546048975c8edf88170d7" modify_external_catalogs: true
I20240430 20:11:32.550665 19027 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_d3bf86d0283b4227a93a2a8aa241075b" } modify_external_catalogs: true
I20240430 20:11:32.550861 19027 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_d3bf86d0283b4227a93a2a8aa241075b" } modify_external_catalogs: true
W20240430 20:11:32.515543 19051 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.17.177.3:42157 (request call id 80) took 953 ms (client timeout 1000 ms). Trace:
W20240430 20:11:32.553210 19051 rpcz_store.cc:269] 0430 20:11:31.561609 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:31.565492 (+ 3883us) service_pool.cc:225] Handling call
0430 20:11:31.918553 (+353061us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:31.918560 (+ 7us) write_op.cc:446] Released schema lock
0430 20:11:31.964977 (+ 46417us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:31.964983 (+ 6us) write_op.cc:446] Released schema lock
0430 20:11:32.053809 (+ 88826us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:32.053815 (+ 6us) write_op.cc:446] Released schema lock
0430 20:11:32.515504 (+461689us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:31.911800 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:31.911929 (+ 129us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:31.911939 (+ 10us) write_op.cc:427] Acquired schema lock
0430 20:11:31.911943 (+ 4us) tablet.cc:660] Decoding operations
0430 20:11:31.912057 (+ 114us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:31.912072 (+ 15us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:31.912077 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0430 20:11:31.912146 (+ 69us) tablet.cc:699] Row locks acquired
0430 20:11:31.912149 (+ 3us) write_op.cc:260] PREPARE: finished
0430 20:11:31.912187 (+ 38us) write_op.cc:270] Start()
0430 20:11:31.912222 (+ 35us) write_op.cc:276] Timestamp: P: 1714507891912142 usec, L: 0
0430 20:11:31.912228 (+ 6us) op_driver.cc:331] REPLICATION: starting
0430 20:11:31.912436 (+ 208us) log.cc:842] Serialized 530 byte log entry
0430 20:11:31.916488 (+ 4052us) op_driver.cc:444] REPLICATION: finished
0430 20:11:31.916640 (+ 152us) write_op.cc:301] APPLY: starting
0430 20:11:31.916660 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:31.916706 (+ 46us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:31.916742 (+ 36us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:31.916801 (+ 59us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:31.916838 (+ 37us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0430 20:11:31.916845 (+ 7us) write_op.cc:312] APPLY: finished
0430 20:11:31.916930 (+ 85us) log.cc:842] Serialized 23 byte log entry
0430 20:11:31.916997 (+ 67us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:31.917021 (+ 24us) write_op.cc:446] Released schema lock
0430 20:11:31.917027 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
0430 20:11:31.956412 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:31.956543 (+ 131us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:31.956553 (+ 10us) write_op.cc:427] Acquired schema lock
0430 20:11:31.956556 (+ 3us) tablet.cc:660] Decoding operations
0430 20:11:31.956681 (+ 125us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:31.956696 (+ 15us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:31.956700 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0430 20:11:31.956776 (+ 76us) tablet.cc:699] Row locks acquired
0430 20:11:31.956778 (+ 2us) write_op.cc:260] PREPARE: finished
0430 20:11:31.956821 (+ 43us) write_op.cc:270] Start()
0430 20:11:31.956858 (+ 37us) write_op.cc:276] Timestamp: P: 1714507891956771 usec, L: 0
0430 20:11:31.956862 (+ 4us) op_driver.cc:331] REPLICATION: starting
0430 20:11:31.957092 (+ 230us) log.cc:842] Serialized 356 byte log entry
0430 20:11:31.961395 (+ 4303us) op_driver.cc:444] REPLICATION: finished
0430 20:11:31.961841 (+ 446us) write_op.cc:301] APPLY: starting
0430 20:11:31.961859 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:31.961909 (+ 50us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:31.961950 (+ 41us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:31.962020 (+ 70us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:31.962060 (+ 40us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0430 20:11:31.962066 (+ 6us) write_op.cc:312] APPLY: finished
0430 20:11:31.962157 (+ 91us) log.cc:842] Serialized 23 byte log entry
0430 20:11:31.962226 (+ 69us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:31.962248 (+ 22us) write_op.cc:446] Released schema lock
0430 20:11:31.962254 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
0430 20:11:32.046306 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:32.046457 (+ 151us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:32.046465 (+ 8us) write_op.cc:427] Acquired schema lock
0430 20:11:32.046467 (+ 2us) tablet.cc:660] Decoding operations
0430 20:11:32.046631 (+ 164us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:32.046646 (+ 15us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:32.046651 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0430 20:11:32.046725 (+ 74us) tablet.cc:699] Row locks acquired
0430 20:11:32.046727 (+ 2us) write_op.cc:260] PREPARE: finished
0430 20:11:32.046767 (+ 40us) write_op.cc:270] Start()
0430 20:11:32.046803 (+ 36us) write_op.cc:276] Timestamp: P: 1714507892046709 usec, L: 0
0430 20:11:32.046807 (+ 4us) op_driver.cc:331] REPLICATION: starting
0430 20:11:32.047019 (+ 212us) log.cc:842] Serialized 356 byte log entry
0430 20:11:32.050881 (+ 3862us) op_driver.cc:444] REPLICATION: finished
0430 20:11:32.051021 (+ 140us) write_op.cc:301] APPLY: starting
0430 20:11:32.051038 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:32.051083 (+ 45us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:32.051111 (+ 28us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:32.051165 (+ 54us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:32.051200 (+ 35us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0430 20:11:32.051207 (+ 7us) write_op.cc:312] APPLY: finished
0430 20:11:32.051282 (+ 75us) log.cc:842] Serialized 23 byte log entry
0430 20:11:32.051353 (+ 71us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:32.051373 (+ 20us) write_op.cc:446] Released schema lock
0430 20:11:32.051380 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
0430 20:11:32.190152 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:32.190310 (+ 158us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:32.190319 (+ 9us) write_op.cc:427] Acquired schema lock
0430 20:11:32.190324 (+ 5us) tablet.cc:660] Decoding operations
0430 20:11:32.190454 (+ 130us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:32.190469 (+ 15us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:32.190477 (+ 8us) tablet.cc:683] Acquiring locks for 1 operations
0430 20:11:32.190557 (+ 80us) tablet.cc:699] Row locks acquired
0430 20:11:32.190560 (+ 3us) write_op.cc:260] PREPARE: finished
0430 20:11:32.190621 (+ 61us) write_op.cc:270] Start()
0430 20:11:32.190659 (+ 38us) write_op.cc:276] Timestamp: P: 1714507892190588 usec, L: 0
0430 20:11:32.190665 (+ 6us) op_driver.cc:331] REPLICATION: starting
0430 20:11:32.190869 (+ 204us) log.cc:842] Serialized 356 byte log entry
0430 20:11:32.198815 (+ 7946us) op_driver.cc:444] REPLICATION: finished
0430 20:11:32.202130 (+ 3315us) write_op.cc:301] APPLY: starting
0430 20:11:32.202152 (+ 22us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:32.202198 (+ 46us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:32.202223 (+ 25us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:32.202280 (+ 57us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:32.202319 (+ 39us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0430 20:11:32.202326 (+ 7us) write_op.cc:312] APPLY: finished
0430 20:11:32.202418 (+ 92us) log.cc:842] Serialized 23 byte log entry
0430 20:11:32.202483 (+ 65us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:32.202504 (+ 21us) write_op.cc:446] Released schema lock
0430 20:11:32.202511 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:32.202572 (+ 61us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:32.202575 (+ 3us) write_op.cc:446] Released schema lock
Metrics: {"mutex_wait_us":2687,"reported_tablets":100,"child_traces":[["op",{"apply.queue_time_us":98,"apply.run_cpu_time_us":454,"apply.run_wall_time_us":454,"num_ops":1,"prepare.queue_time_us":53,"prepare.run_cpu_time_us":750,"prepare.run_wall_time_us":752,"raft.queue_time_us":1205,"raft.run_cpu_time_us":255,"raft.run_wall_time_us":256,"replication_time_us":4235}],["op",{"apply.queue_time_us":384,"apply.run_cpu_time_us":481,"apply.run_wall_time_us":480,"num_ops":1,"prepare.queue_time_us":10684,"prepare.run_cpu_time_us":805,"prepare.run_wall_time_us":806,"raft.queue_time_us":1139,"raft.run_cpu_time_us":230,"raft.run_wall_time_us":231,"replication_time_us":4507}],["op",{"apply.queue_time_us":91,"apply.run_cpu_time_us":414,"apply.run_wall_time_us":414,"num_ops":1,"prepare.queue_time_us":52,"prepare.run_cpu_time_us":819,"prepare.run_wall_time_us":822,"raft.queue_time_us":1174,"raft.run_cpu_time_us":253,"raft.run_wall_time_us":254,"replication_time_us":4052}],["op",{"apply.queue_time_us":3250,"apply.run_cpu_time_us":524,"apply.run_wall_time_us":524,"num_ops":1,"prepare.queue_time_us":71765,"prepare.run_cpu_time_us":804,"prepare.run_wall_time_us":1154,"raft.queue_time_us":1402,"raft.run_cpu_time_us":6,"raft.run_wall_time_us":7,"replication_time_us":8128}]]}
I20240430 20:11:32.556368 18586 tablet_service.cc:1514] Processing DeleteTablet for tablet 29974dc2c7dd45b99b6cc0712378fc30 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:26 UTC) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:32.564921 18623 raft_consensus.cc:2454] T 294644d7a16b4ca2bbb6d106bcb8c7ff P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
W20240430 20:11:32.566804 18415 consensus_peers.cc:485] T 29974dc2c7dd45b99b6cc0712378fc30 P 2efb542aa55e439b99ccb98d1c5c884a -> Peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Couldn't send request to peer 5e1c6397f09747f091534faf93542710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 29974dc2c7dd45b99b6cc0712378fc30. This is attempt 1: this message will repeat every 5th retry.
I20240430 20:11:32.572366 19471 ts_tablet_manager.cc:1888] T bbe9a2429a434f3f95d6e7f314fcfe4c P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20240430 20:11:32.562135 19028 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 12c6e131fa904d23865b8673b58a7e33 consensus state has no leader
I20240430 20:11:32.571842 19029 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_873c21f6dffd4d47a742b3699ab85268" } modify_external_catalogs: true
I20240430 20:11:32.572788 19029 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_873c21f6dffd4d47a742b3699ab85268" } modify_external_catalogs: true
I20240430 20:11:32.573307 19474 raft_consensus.cc:482] T 13c1107ea4d04eb78a74d3f658c72fb3 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:32.573468 19700 raft_consensus.cc:482] T 2a1e50f99c5d4b44bdbb51b58a92bc75 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:32.573446 19474 raft_consensus.cc:504] T 13c1107ea4d04eb78a74d3f658c72fb3 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:32.573633 19700 raft_consensus.cc:504] T 2a1e50f99c5d4b44bdbb51b58a92bc75 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:32.574019 19474 leader_election.cc:288] T 13c1107ea4d04eb78a74d3f658c72fb3 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:32.574167 18601 tablet_service.cc:1467] Processing CreateTablet for tablet 04b3dff86d5c48a09d03da58bd48f360 (DEFAULT_TABLE table=default.Table_640bb0fa95ca44d49921276d665b9ca3 [id=9fa22833979f4c2f978de651145acef9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:32.574294 19700 leader_election.cc:288] T 2a1e50f99c5d4b44bdbb51b58a92bc75 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:32.574554 18601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04b3dff86d5c48a09d03da58bd48f360. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:32.575229 18623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a1e50f99c5d4b44bdbb51b58a92bc75" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:32.575834 19024 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_9dc9f3b0b0c8452a9da7c85e62d87bdc" } modify_external_catalogs: true
I20240430 20:11:32.576006 19024 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_9dc9f3b0b0c8452a9da7c85e62d87bdc" } modify_external_catalogs: true
I20240430 20:11:32.573120 18587 tablet_service.cc:1467] Processing CreateTablet for tablet 283e5e73cbc04daba0de19f4541f9fd0 (DEFAULT_TABLE table=default.Table_849e6e38123a4d8cae466d841850d6f9 [id=c1c646a7c27e4d77af05d1ac92aa6ddc]), partition=RANGE (key) PARTITION UNBOUNDED
W20240430 20:11:32.577432 18415 leader_election.cc:341] T 2a1e50f99c5d4b44bdbb51b58a92bc75 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:32.579429 19048 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: Sending DeleteTablet for 3 replicas of tablet 20d166756819406ba85044f4fee9c4ce
I20240430 20:11:32.581180 18456 tablet_service.cc:1514] Processing DeleteTablet for tablet 20d166756819406ba85044f4fee9c4ce with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:32 UTC) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:32.583725 18586 tablet_service.cc:1467] Processing CreateTablet for tablet a482f4622f774d60b4efd6827b8be7d7 (DEFAULT_TABLE table=default.Table_b3741e0be6594cfab0de42dec8e1f2fe [id=c3dc9f05316d46d6a3ef4d03f40877ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:32.584959 19471 log.cc:1196] T bbe9a2429a434f3f95d6e7f314fcfe4c P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/bbe9a2429a434f3f95d6e7f314fcfe4c
I20240430 20:11:32.585157 18481 raft_consensus.cc:2454] T 294644d7a16b4ca2bbb6d106bcb8c7ff P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:32.587872 18450 tablet_service.cc:1467] Processing CreateTablet for tablet 283e5e73cbc04daba0de19f4541f9fd0 (DEFAULT_TABLE table=default.Table_849e6e38123a4d8cae466d841850d6f9 [id=c1c646a7c27e4d77af05d1ac92aa6ddc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:32.588374 18450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 283e5e73cbc04daba0de19f4541f9fd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:32.588418 19033 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: Sending DeleteTablet for 3 replicas of tablet 1555924b5f1444398f12ee82c3cb5b13
I20240430 20:11:32.587967 18599 tablet_service.cc:1467] Processing CreateTablet for tablet b22aaedb0a6c4fd9ad0ea8fa019001e1 (DEFAULT_TABLE table=default.Table_730a215d7af74a5bb753adca39dea676 [id=a5d580cffa2447d78fc9f2c8e31264f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:32.589365 18456 tablet_service.cc:1514] Processing DeleteTablet for tablet 1555924b5f1444398f12ee82c3cb5b13 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:32 UTC) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:32.586529 19474 raft_consensus.cc:482] T b72fe1f0713b4076a81a3fc1c74f7fa7 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:32.590344 18602 tablet_service.cc:1467] Processing CreateTablet for tablet c6e3d09402574f1e8fe9e4160c70a3ee (DEFAULT_TABLE table=default.Table_2a72e434cca94803ae3e8dfcd2e47235 [id=450f5d1c7e084a1bb88d035c0ccfcf82]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:32.591466 18587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 283e5e73cbc04daba0de19f4541f9fd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:32.591837 18985 tablet_bootstrap.cc:492] T acd23f0a63d5448aaa7ed78422e67bad P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:32.592028 18985 ts_tablet_manager.cc:1367] T acd23f0a63d5448aaa7ed78422e67bad P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.327s user 0.004s sys 0.000s
I20240430 20:11:32.592895 18985 raft_consensus.cc:348] T acd23f0a63d5448aaa7ed78422e67bad P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:32.593145 18985 raft_consensus.cc:374] T acd23f0a63d5448aaa7ed78422e67bad P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:32.593247 18985 raft_consensus.cc:729] T acd23f0a63d5448aaa7ed78422e67bad P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:32.593514 18985 consensus_queue.cc:260] T acd23f0a63d5448aaa7ed78422e67bad P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:32.594100 18985 ts_tablet_manager.cc:1398] T acd23f0a63d5448aaa7ed78422e67bad P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240430 20:11:32.594393 18985 tablet_bootstrap.cc:492] T b688e03ea71940c69bda5520b62c1fc3 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:32.594894 18459 tablet_service.cc:1467] Processing CreateTablet for tablet 246fb63a5e9842df8263800e28a5935c (DEFAULT_TABLE table=default.Table_067c3b36dd2e4cfd828ab8ff6bbcbd2e [id=3a030fd1a2134783a02c9a01911e64e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:32.595336 18459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 246fb63a5e9842df8263800e28a5935c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:32.595986 18593 tablet_service.cc:1467] Processing CreateTablet for tablet 22fef8bb6b034db098c3372ae992f7a7 (DEFAULT_TABLE table=default.Table_891f568fffb349c097f751981343f2b1 [id=7d186ca20c094cc19aef792fe1c094e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:32.596491 18597 tablet_service.cc:1467] Processing CreateTablet for tablet 8e33df70a1504dbd852fe0b3c3f61622 (DEFAULT_TABLE table=default.Table_21208fd7cdca4c73a4857bc4692187f6 [id=da22cb339ab647e8afda7aa92f3548a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:32.596781 18592 tablet_service.cc:1467] Processing CreateTablet for tablet 0a16d98f700c46e29f6dd8568752ef29 (DEFAULT_TABLE table=default.Table_925990f1c73245818b39aafeb0de55e3 [id=5979225bccda4e35bea4bec2c67b22df]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:32.597479 18602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6e3d09402574f1e8fe9e4160c70a3ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:32.598016 19474 raft_consensus.cc:504] T b72fe1f0713b4076a81a3fc1c74f7fa7 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:32.599499 19049 catalog_manager.cc:6029] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: default.Table_421069f84cb844368066c93102a8faf8 [id=d5cea90cbc2c48df9d8b80da1eccc949] alter complete (version 2)
I20240430 20:11:32.600601 19474 leader_election.cc:288] T b72fe1f0713b4076a81a3fc1c74f7fa7 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:32.600783 18589 tablet_service.cc:1467] Processing CreateTablet for tablet 246fb63a5e9842df8263800e28a5935c (DEFAULT_TABLE table=default.Table_067c3b36dd2e4cfd828ab8ff6bbcbd2e [id=3a030fd1a2134783a02c9a01911e64e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:32.601951 19471 ts_tablet_manager.cc:1909] T bbe9a2429a434f3f95d6e7f314fcfe4c P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:32.605129 18599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b22aaedb0a6c4fd9ad0ea8fa019001e1. 1 dirs total, 0 dirs full, 0 dirs failed
W20240430 20:11:32.606988 19023 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a59ef6c545dd41a28265a4c1b8ce10e0 consensus state has no leader
I20240430 20:11:32.608191 18480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b72fe1f0713b4076a81a3fc1c74f7fa7" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:32.608510 18985 tablet_bootstrap.cc:654] T b688e03ea71940c69bda5520b62c1fc3 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:32.610359 18586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a482f4622f774d60b4efd6827b8be7d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:32.611413 18595 tablet_service.cc:1514] Processing DeleteTablet for tablet 1555924b5f1444398f12ee82c3cb5b13 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:32 UTC) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:32.611568 18600 tablet_service.cc:1514] Processing DeleteTablet for tablet 20d166756819406ba85044f4fee9c4ce with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:32 UTC) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:32.613168 18622 raft_consensus.cc:3037] T 12c6e131fa904d23865b8673b58a7e33 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:32.614524 19027 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: Sending DeleteTablet for 3 replicas of tablet eb76f60448774fe1967fffe7c6873cfc
W20240430 20:11:32.614753 18558 leader_election.cc:341] T b72fe1f0713b4076a81a3fc1c74f7fa7 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
W20240430 20:11:32.615061 18558 leader_election.cc:341] T 13c1107ea4d04eb78a74d3f658c72fb3 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Not found: Tablet not found: 13c1107ea4d04eb78a74d3f658c72fb3
I20240430 20:11:32.615823 18456 tablet_service.cc:1514] Processing DeleteTablet for tablet eb76f60448774fe1967fffe7c6873cfc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:32 UTC) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:32.617440 18600 tablet_service.cc:1514] Processing DeleteTablet for tablet eb76f60448774fe1967fffe7c6873cfc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:32 UTC) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:32.620519 19474 consensus_queue.cc:1035] T 20d166756819406ba85044f4fee9c4ce P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240430 20:11:32.622555 19029 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: Sending DeleteTablet for 3 replicas of tablet 6e826459c3f4437da81c9e6e09f0c22f
I20240430 20:11:32.623344 18456 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e826459c3f4437da81c9e6e09f0c22f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:32 UTC) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:32.632234 19029 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_deadf728d74a462fb79641891dfedd9c" } new_table_name: "default.Table_ffb3192f443d437ab63f56f628e596ad" modify_external_catalogs: true
I20240430 20:11:32.634034 19024 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: Sending DeleteTablet for 3 replicas of tablet 2873af6da928463f9809864b8eaf0c4a
I20240430 20:11:32.635385 18456 tablet_service.cc:1514] Processing DeleteTablet for tablet 2873af6da928463f9809864b8eaf0c4a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:32 UTC) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:32.635514 18600 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e826459c3f4437da81c9e6e09f0c22f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:32 UTC) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:32.638760 19645 raft_consensus.cc:482] T 4e69ca12bd10494fb013614741bc166c P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240430 20:11:32.642632 19722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240430 20:11:32.643186 19495 ts_tablet_manager.cc:1888] T 02176eee8e044a7db8cac48439e9f594 P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:32.643406 19495 log.cc:1196] T 02176eee8e044a7db8cac48439e9f594 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/02176eee8e044a7db8cac48439e9f594
I20240430 20:11:32.643944 19495 ts_tablet_manager.cc:1909] T 02176eee8e044a7db8cac48439e9f594 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:32.653199 19474 raft_consensus.cc:482] T 12c6e131fa904d23865b8673b58a7e33 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:32.653458 19645 raft_consensus.cc:504] T 4e69ca12bd10494fb013614741bc166c P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:32.656592 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 58) took 1654 ms. Trace:
I20240430 20:11:32.658277 18600 tablet_service.cc:1514] Processing DeleteTablet for tablet 2873af6da928463f9809864b8eaf0c4a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:32 UTC) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:32.659387 19474 raft_consensus.cc:504] T 12c6e131fa904d23865b8673b58a7e33 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:32.660032 18985 tablet_bootstrap.cc:492] T b688e03ea71940c69bda5520b62c1fc3 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:32.660326 19474 leader_election.cc:288] T 12c6e131fa904d23865b8673b58a7e33 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:32.660460 18985 ts_tablet_manager.cc:1367] T b688e03ea71940c69bda5520b62c1fc3 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.066s user 0.009s sys 0.005s
I20240430 20:11:32.662845 18597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e33df70a1504dbd852fe0b3c3f61622. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:32.662555 18985 raft_consensus.cc:348] T b688e03ea71940c69bda5520b62c1fc3 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:32.663224 18985 raft_consensus.cc:374] T b688e03ea71940c69bda5520b62c1fc3 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:32.664024 18477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e69ca12bd10494fb013614741bc166c" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:32.663643 18985 raft_consensus.cc:729] T b688e03ea71940c69bda5520b62c1fc3 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:32.666916 18456 tablet_service.cc:1467] Processing CreateTablet for tablet 47bf64ecc41b445da4ff697e141243dd (DEFAULT_TABLE table=default.Table_0858797306c54d81aab9d77e8591e502 [id=838fc8b61a864dea8877a6f144e4cb4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:32.667357 18456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47bf64ecc41b445da4ff697e141243dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:32.668200 19027 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_85bed92bc6b94df3bd1f89734050f578"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:32.668784 19027 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_85bed92bc6b94df3bd1f89734050f578 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:32.670823 19645 leader_election.cc:288] T 4e69ca12bd10494fb013614741bc166c P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:32.670873 18593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22fef8bb6b034db098c3372ae992f7a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:32.671967 18481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12c6e131fa904d23865b8673b58a7e33" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:32.672286 18481 raft_consensus.cc:2454] T 12c6e131fa904d23865b8673b58a7e33 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
I20240430 20:11:32.672952 18448 tablet_service.cc:1467] Processing CreateTablet for tablet f5eaf2413e1d4e69b56fae61f768f0ce (DEFAULT_TABLE table=default.Table_c264c250888a459694c33184912ab76d [id=8489b872074a42b6abb713e420968141]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:32.673349 18448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5eaf2413e1d4e69b56fae61f768f0ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:32.666395 18985 consensus_queue.cc:260] T b688e03ea71940c69bda5520b62c1fc3 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:32.676712 18985 ts_tablet_manager.cc:1398] T b688e03ea71940c69bda5520b62c1fc3 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.016s user 0.003s sys 0.000s
I20240430 20:11:32.678002 19023 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_68528229d5a94981a7fc0677d48003d1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:32.678617 19023 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_68528229d5a94981a7fc0677d48003d1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:32.678925 19048 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_20f3b06af23d476f8ebc0471d08f8659"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240430 20:11:32.679085 18985 tablet_bootstrap.cc:492] T 26ed12f498644d15bf45e884dcad172e P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
W20240430 20:11:32.679487 19048 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_20f3b06af23d476f8ebc0471d08f8659 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:32.680801 19033 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_45d50fc81325443097375ae6e65a1633"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:32.681324 19033 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_45d50fc81325443097375ae6e65a1633 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:32.684403 19051 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_9f4b05e2470349c7bbc8c4ae15e778b2" } new_table_name: "default.Table_cb2a4b9ceaf4431facfff2c9bcf94c5d" modify_external_catalogs: true
I20240430 20:11:32.666232 19024 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 12c6e131fa904d23865b8673b58a7e33 (table default.Table_c671e4288ad041dda677eb1024151188 [id=19f4e6decedf4ee59fccc0a24a05cedd]). Expected version 1 got 0
I20240430 20:11:32.688277 19028 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_c671e4288ad041dda677eb1024151188" } new_table_name: "default.Table_a9dc8fc6a4aa46dc914d0e327a4de554" modify_external_catalogs: true
I20240430 20:11:32.688699 19471 rpcz_store.cc:276] 0430 20:11:30.991145 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:30.991783 (+ 638us) service_pool.cc:225] Handling call
0430 20:11:32.224830 (+1233047us) ts_tablet_manager.cc:1120] Deleting tablet bbe9a2429a434f3f95d6e7f314fcfe4c
0430 20:11:32.224841 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:32.441185 (+216344us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:32.540089 (+ 98904us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:32.645491 (+105402us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1231072}
I20240430 20:11:32.689283 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 60) took 1697 ms. Trace:
I20240430 20:11:32.689407 19471 rpcz_store.cc:276] 0430 20:11:30.991315 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:30.991915 (+ 600us) service_pool.cc:225] Handling call
0430 20:11:32.689103 (+1697188us) ts_tablet_manager.cc:1120] Deleting tablet e6eac92dfa5f4456bc49571878f0f8dd
0430 20:11:32.689115 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:32.689243 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1695257}
I20240430 20:11:32.689711 19471 tablet_replica.cc:330] T 255c13ad8fae4ea4aa8e7b7aabb7602e P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:32.689987 19471 raft_consensus.cc:2229] T 255c13ad8fae4ea4aa8e7b7aabb7602e P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:32.690188 19471 raft_consensus.cc:2258] T 255c13ad8fae4ea4aa8e7b7aabb7602e P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:32.690755 18699 leader_election.cc:302] T 294644d7a16b4ca2bbb6d106bcb8c7ff P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters:
I20240430 20:11:32.691073 19674 consensus_queue.cc:1035] T 12c6e131fa904d23865b8673b58a7e33 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.091s
I20240430 20:11:32.691569 19674 raft_consensus.cc:2781] T 294644d7a16b4ca2bbb6d106bcb8c7ff P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:32.691707 19674 raft_consensus.cc:686] T 294644d7a16b4ca2bbb6d106bcb8c7ff P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Becoming Leader. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Running, Role: LEADER
I20240430 20:11:32.691954 19674 consensus_queue.cc:237] T 294644d7a16b4ca2bbb6d106bcb8c7ff P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:32.692771 18726 tablet_service.cc:1467] Processing CreateTablet for tablet 283e5e73cbc04daba0de19f4541f9fd0 (DEFAULT_TABLE table=default.Table_849e6e38123a4d8cae466d841850d6f9 [id=c1c646a7c27e4d77af05d1ac92aa6ddc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:32.693197 18726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 283e5e73cbc04daba0de19f4541f9fd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:32.697750 19024 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet a05e32a33aa7488eb287cc6314c09c44 (table default.Table_2d1cfc78363a47aa8dba7ef7d80f8c9a [id=955d833ff8f14bf6957abadf9ef669e2]). Expected version 2 got 0
I20240430 20:11:32.698526 19024 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet c66c142419ca4e1399ecd1c8d1e62a59 (table default.Table_0443359dbeff437fb6cffb0db33cbb30 [id=93c44abdf6b147228224388398cd7aeb]). Expected version 2 got 0
I20240430 20:11:32.698904 19024 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 53a899fa5a424a0d99f1b47925e8c7ad (table default.Table_deadf728d74a462fb79641891dfedd9c [id=608890834da843fa9af0cfc0d548029e]). Expected version 1 got 0
I20240430 20:11:32.700728 18728 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e826459c3f4437da81c9e6e09f0c22f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:32 UTC) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:32.701161 19049 catalog_manager.cc:6029] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: default.Table_0c5d4686f2ed40fb8d62c31b0166dc05 [id=4dce8a767f0541e1beb5464b2ed95d06] alter complete (version 1)
W20240430 20:11:32.702461 19029 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 53a899fa5a424a0d99f1b47925e8c7ad consensus state has no leader
I20240430 20:11:32.702754 18737 tablet_service.cc:1514] Processing DeleteTablet for tablet 2cdccf4be7094ca5b863dd5c3432d55d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:30 UTC) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:32.702962 18736 tablet_service.cc:1467] Processing CreateTablet for tablet 22fef8bb6b034db098c3372ae992f7a7 (DEFAULT_TABLE table=default.Table_891f568fffb349c097f751981343f2b1 [id=7d186ca20c094cc19aef792fe1c094e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:32.703420 18736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22fef8bb6b034db098c3372ae992f7a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:32.705389 19700 raft_consensus.cc:482] T acd23f0a63d5448aaa7ed78422e67bad P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:32.705559 19700 raft_consensus.cc:504] T acd23f0a63d5448aaa7ed78422e67bad P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:32.706257 19700 leader_election.cc:288] T acd23f0a63d5448aaa7ed78422e67bad P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:32.706669 18725 tablet_service.cc:1514] Processing DeleteTablet for tablet 20d166756819406ba85044f4fee9c4ce with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:32 UTC) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:32.707000 18739 tablet_service.cc:1467] Processing CreateTablet for tablet 8e33df70a1504dbd852fe0b3c3f61622 (DEFAULT_TABLE table=default.Table_21208fd7cdca4c73a4857bc4692187f6 [id=da22cb339ab647e8afda7aa92f3548a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:32.689968 19492 raft_consensus.cc:504] T 01436e5b6232494babbb414342f03090 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:32.709712 18735 tablet_service.cc:1467] Processing CreateTablet for tablet 0a16d98f700c46e29f6dd8568752ef29 (DEFAULT_TABLE table=default.Table_925990f1c73245818b39aafeb0de55e3 [id=5979225bccda4e35bea4bec2c67b22df]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:32.709897 18733 tablet_service.cc:1514] Processing DeleteTablet for tablet eb76f60448774fe1967fffe7c6873cfc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:32 UTC) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:32.710706 18985 tablet_bootstrap.cc:654] T 26ed12f498644d15bf45e884dcad172e P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:32.711237 18734 tablet_service.cc:1467] Processing CreateTablet for tablet c6e3d09402574f1e8fe9e4160c70a3ee (DEFAULT_TABLE table=default.Table_2a72e434cca94803ae3e8dfcd2e47235 [id=450f5d1c7e084a1bb88d035c0ccfcf82]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:32.711643 18730 tablet_service.cc:1514] Processing DeleteTablet for tablet 115c4a12440647a4bb46eb6824f7ea98 with delete_type TABLET_DATA_DELETED (Sending initial creation of tablet) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:32.710346 18731 tablet_service.cc:1514] Processing DeleteTablet for tablet d9e578c99058424a9b86390641f9c34f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:26 UTC) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:32.711998 18724 tablet_service.cc:1467] Processing CreateTablet for tablet 331c031d44854d679a27d10d9151d0ea (DEFAULT_TABLE table=default.Table_999c55e7bef74339b7b3ab1142599447 [id=61c8a5b99ef34a0baeeb343fef7cfd05]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:32.714336 18738 tablet_service.cc:1467] Processing CreateTablet for tablet a482f4622f774d60b4efd6827b8be7d7 (DEFAULT_TABLE table=default.Table_b3741e0be6594cfab0de42dec8e1f2fe [id=c3dc9f05316d46d6a3ef4d03f40877ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:32.714639 18740 tablet_service.cc:1514] Processing DeleteTablet for tablet 2873af6da928463f9809864b8eaf0c4a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:32 UTC) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:32.714833 18732 tablet_service.cc:1467] Processing CreateTablet for tablet 04b3dff86d5c48a09d03da58bd48f360 (DEFAULT_TABLE table=default.Table_640bb0fa95ca44d49921276d665b9ca3 [id=9fa22833979f4c2f978de651145acef9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:32.715046 18744 tablet_service.cc:1467] Processing CreateTablet for tablet b22aaedb0a6c4fd9ad0ea8fa019001e1 (DEFAULT_TABLE table=default.Table_730a215d7af74a5bb753adca39dea676 [id=a5d580cffa2447d78fc9f2c8e31264f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:32.715260 18742 tablet_service.cc:1514] Processing DeleteTablet for tablet 1555924b5f1444398f12ee82c3cb5b13 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:32 UTC) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:32.715444 18729 tablet_service.cc:1467] Processing CreateTablet for tablet 47bf64ecc41b445da4ff697e141243dd (DEFAULT_TABLE table=default.Table_0858797306c54d81aab9d77e8591e502 [id=838fc8b61a864dea8877a6f144e4cb4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:32.716389 19672 consensus_queue.cc:1035] T 12c6e131fa904d23865b8673b58a7e33 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.007s
I20240430 20:11:32.717211 18810 heartbeater.cc:503] Master 127.17.177.60:37225 was elected leader, sending a full tablet report...
I20240430 20:11:32.711549 18759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "255c13ad8fae4ea4aa8e7b7aabb7602e" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:32.713407 18743 tablet_service.cc:1467] Processing CreateTablet for tablet f5eaf2413e1d4e69b56fae61f768f0ce (DEFAULT_TABLE table=default.Table_c264c250888a459694c33184912ab76d [id=8489b872074a42b6abb713e420968141]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:32.719506 18741 tablet_service.cc:1467] Processing CreateTablet for tablet 246fb63a5e9842df8263800e28a5935c (DEFAULT_TABLE table=default.Table_067c3b36dd2e4cfd828ab8ff6bbcbd2e [id=3a030fd1a2134783a02c9a01911e64e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:32.720268 19492 leader_election.cc:288] T 01436e5b6232494babbb414342f03090 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Requested vote from peers 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:32.721534 18739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e33df70a1504dbd852fe0b3c3f61622. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:32.725570 19026 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_e20b13b4103849b9bc5e5f96f9b911e4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:32.726256 19026 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e20b13b4103849b9bc5e5f96f9b911e4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:32.725603 19027 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_5b6a4bb3832e497888af8e8db7b96c12"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:32.728276 19027 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5b6a4bb3832e497888af8e8db7b96c12 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:32.729094 19029 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_3127eaa465c84dcab3eb9935d15667ce"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:32.729629 19029 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3127eaa465c84dcab3eb9935d15667ce in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:32.738340 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 51) took 1689 ms. Trace:
I20240430 20:11:32.739204 19495 rpcz_store.cc:276] 0430 20:11:31.047473 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:31.048145 (+ 672us) service_pool.cc:225] Handling call
0430 20:11:32.427656 (+1379511us) ts_tablet_manager.cc:1120] Deleting tablet 02176eee8e044a7db8cac48439e9f594
0430 20:11:32.427667 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:32.638048 (+210381us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:32.642390 (+ 4342us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:32.736431 (+ 94041us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1325506}
I20240430 20:11:32.740298 19495 tablet_replica.cc:330] T c9ff972ade694938888514979e1638aa P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:32.740983 19004 catalog_manager.cc:4830] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): tablet 02176eee8e044a7db8cac48439e9f594 (table default.Table_3b327aece0ea4ec4b882f2264e3f5f0c [id=1fb10bddcfe94ec488ecc71e43ca339d]) successfully deleted
I20240430 20:11:32.741415 19495 raft_consensus.cc:2229] T c9ff972ade694938888514979e1638aa P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:32.741608 19495 raft_consensus.cc:2258] T c9ff972ade694938888514979e1638aa P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:32.745028 18592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a16d98f700c46e29f6dd8568752ef29. 1 dirs total, 0 dirs full, 0 dirs failed
W20240430 20:11:32.745491 18558 leader_election.cc:341] T 4e69ca12bd10494fb013614741bc166c P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:32.745831 19495 ts_tablet_manager.cc:1875] T c9ff972ade694938888514979e1638aa P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:32.748518 18743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5eaf2413e1d4e69b56fae61f768f0ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:32.749519 19645 raft_consensus.cc:482] T a05e32a33aa7488eb287cc6314c09c44 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:32.749722 19645 raft_consensus.cc:504] T a05e32a33aa7488eb287cc6314c09c44 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:32.751188 19645 leader_election.cc:288] T a05e32a33aa7488eb287cc6314c09c44 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
W20240430 20:11:32.753294 18558 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.177.1:41217, user_credentials={real_user=slave}} blocked reactor thread for 73559.9us
I20240430 20:11:32.757063 18999 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet bbe9a2429a434f3f95d6e7f314fcfe4c (table default.Table_3acf40c6d58741d4a8e94ca96d642249 [id=71226c1fc71242a6aad13625c9ca3c23]) successfully deleted
W20240430 20:11:32.757381 18999 catalog_manager.cc:4810] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): delete failed for tablet e6eac92dfa5f4456bc49571878f0f8dd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e6eac92dfa5f4456bc49571878f0f8dd already in progress: creating tablet
I20240430 20:11:32.758108 18481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a05e32a33aa7488eb287cc6314c09c44" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:32.758419 18985 tablet_bootstrap.cc:492] T 26ed12f498644d15bf45e884dcad172e P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:32.760406 18481 raft_consensus.cc:2379] T a05e32a33aa7488eb287cc6314c09c44 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e1c6397f09747f091534faf93542710 in current term 1: Already voted for candidate 3c51c33d5616482cafb68e26688820f4 in this term.
I20240430 20:11:32.760414 18985 ts_tablet_manager.cc:1367] T 26ed12f498644d15bf45e884dcad172e P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.082s user 0.009s sys 0.008s
I20240430 20:11:32.762109 18734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6e3d09402574f1e8fe9e4160c70a3ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:32.762923 18985 raft_consensus.cc:348] T 26ed12f498644d15bf45e884dcad172e P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:32.763274 18985 raft_consensus.cc:374] T 26ed12f498644d15bf45e884dcad172e P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:32.763459 18985 raft_consensus.cc:729] T 26ed12f498644d15bf45e884dcad172e P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:32.763792 18985 consensus_queue.cc:260] T 26ed12f498644d15bf45e884dcad172e P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:32.763998 18600 tablet_service.cc:1467] Processing CreateTablet for tablet 47bf64ecc41b445da4ff697e141243dd (DEFAULT_TABLE table=default.Table_0858797306c54d81aab9d77e8591e502 [id=838fc8b61a864dea8877a6f144e4cb4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:32.764657 19471 ts_tablet_manager.cc:1875] T 255c13ad8fae4ea4aa8e7b7aabb7602e P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:32.764796 18985 ts_tablet_manager.cc:1398] T 26ed12f498644d15bf45e884dcad172e P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240430 20:11:32.765367 18985 tablet_bootstrap.cc:492] T 53a899fa5a424a0d99f1b47925e8c7ad P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:32.765779 18589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 246fb63a5e9842df8263800e28a5935c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:32.766716 18595 tablet_service.cc:1467] Processing CreateTablet for tablet f5eaf2413e1d4e69b56fae61f768f0ce (DEFAULT_TABLE table=default.Table_c264c250888a459694c33184912ab76d [id=8489b872074a42b6abb713e420968141]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:32.766806 18558 leader_election.cc:302] T 12c6e131fa904d23865b8673b58a7e33 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:32.757001 19648 raft_consensus.cc:504] T e3e74e76ab2b4fe492ffcc1838656fd1 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:32.769340 18985 tablet_bootstrap.cc:654] T 53a899fa5a424a0d99f1b47925e8c7ad P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:32.767444 18622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acd23f0a63d5448aaa7ed78422e67bad" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
W20240430 20:11:32.771457 18415 leader_election.cc:341] T acd23f0a63d5448aaa7ed78422e67bad P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:32.772444 18732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04b3dff86d5c48a09d03da58bd48f360. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:32.773881 18735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a16d98f700c46e29f6dd8568752ef29. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:32.775970 19645 raft_consensus.cc:2781] T 12c6e131fa904d23865b8673b58a7e33 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240430 20:11:32.776947 18585 tablet_service.cc:1467] Processing CreateTablet for tablet 331c031d44854d679a27d10d9151d0ea (DEFAULT_TABLE table=default.Table_999c55e7bef74339b7b3ab1142599447 [id=61c8a5b99ef34a0baeeb343fef7cfd05]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:32.777621 19645 raft_consensus.cc:482] T 12c6e131fa904d23865b8673b58a7e33 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:32.777786 19645 raft_consensus.cc:3037] T 12c6e131fa904d23865b8673b58a7e33 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Advancing to term 2
I20240430 20:11:32.784400 18595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5eaf2413e1d4e69b56fae61f768f0ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:32.792235 18741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 246fb63a5e9842df8263800e28a5935c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:32.794674 18585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 331c031d44854d679a27d10d9151d0ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:32.812445 18600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47bf64ecc41b445da4ff697e141243dd. 1 dirs total, 0 dirs full, 0 dirs failed
W20240430 20:11:32.825769 18696 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.177.2:46027, user_credentials={real_user=slave}} blocked reactor thread for 134505us
I20240430 20:11:32.828269 19648 leader_election.cc:288] T e3e74e76ab2b4fe492ffcc1838656fd1 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Requested vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:32.828554 18744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b22aaedb0a6c4fd9ad0ea8fa019001e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:32.828833 19474 raft_consensus.cc:482] T 53a899fa5a424a0d99f1b47925e8c7ad P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:32.829038 19474 raft_consensus.cc:504] T 53a899fa5a424a0d99f1b47925e8c7ad P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:32.832927 18481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53a899fa5a424a0d99f1b47925e8c7ad" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:32.834141 19474 leader_election.cc:288] T 53a899fa5a424a0d99f1b47925e8c7ad P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
W20240430 20:11:32.840327 18558 leader_election.cc:341] T 53a899fa5a424a0d99f1b47925e8c7ad P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:32.842048 18729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47bf64ecc41b445da4ff697e141243dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:32.842672 18738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a482f4622f774d60b4efd6827b8be7d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:32.845155 19495 ts_tablet_manager.cc:1888] T c9ff972ade694938888514979e1638aa P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:32.845322 19495 log.cc:1196] T c9ff972ade694938888514979e1638aa P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/c9ff972ade694938888514979e1638aa
I20240430 20:11:32.845700 19495 ts_tablet_manager.cc:1909] T c9ff972ade694938888514979e1638aa P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:32.846109 18724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 331c031d44854d679a27d10d9151d0ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:32.848744 19581 raft_consensus.cc:504] T eed6338ad3d64859af087ee15e1ec798 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:32.849522 19581 leader_election.cc:288] T eed6338ad3d64859af087ee15e1ec798 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Requested vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:32.850219 19489 ts_tablet_manager.cc:1888] T 29974dc2c7dd45b99b6cc0712378fc30 P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:32.850366 19489 log.cc:1196] T 29974dc2c7dd45b99b6cc0712378fc30 P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/29974dc2c7dd45b99b6cc0712378fc30
I20240430 20:11:32.850963 19489 ts_tablet_manager.cc:1909] T 29974dc2c7dd45b99b6cc0712378fc30 P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:32.857440 18988 tablet_bootstrap.cc:492] T dd9b893d48684cfea7c242b93ecba01f P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:32.857733 18988 ts_tablet_manager.cc:1367] T dd9b893d48684cfea7c242b93ecba01f P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.529s user 0.013s sys 0.040s
I20240430 20:11:32.861593 18988 raft_consensus.cc:348] T dd9b893d48684cfea7c242b93ecba01f P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:32.862795 18988 raft_consensus.cc:374] T dd9b893d48684cfea7c242b93ecba01f P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:32.870815 18621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eed6338ad3d64859af087ee15e1ec798" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:32.871206 18985 tablet_bootstrap.cc:492] T 53a899fa5a424a0d99f1b47925e8c7ad P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:32.871384 18985 ts_tablet_manager.cc:1367] T 53a899fa5a424a0d99f1b47925e8c7ad P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.106s user 0.012s sys 0.036s
I20240430 20:11:32.872233 18985 raft_consensus.cc:348] T 53a899fa5a424a0d99f1b47925e8c7ad P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:32.872481 18985 raft_consensus.cc:374] T 53a899fa5a424a0d99f1b47925e8c7ad P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:32.872573 18985 raft_consensus.cc:729] T 53a899fa5a424a0d99f1b47925e8c7ad P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:32.872833 18985 consensus_queue.cc:260] T 53a899fa5a424a0d99f1b47925e8c7ad P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:32.873395 18985 ts_tablet_manager.cc:1398] T 53a899fa5a424a0d99f1b47925e8c7ad P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240430 20:11:32.873744 18985 tablet_bootstrap.cc:492] T 75ebf051383e42529dcb5f7dd0770a4b P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:32.875388 18985 tablet_bootstrap.cc:654] T 75ebf051383e42529dcb5f7dd0770a4b P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:32.882202 18622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01436e5b6232494babbb414342f03090" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:32.882539 18622 raft_consensus.cc:3037] T 01436e5b6232494babbb414342f03090 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:32.883813 18620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3e74e76ab2b4fe492ffcc1838656fd1" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:32.884274 18621 raft_consensus.cc:3037] T eed6338ad3d64859af087ee15e1ec798 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:32.884629 18985 tablet_bootstrap.cc:492] T 75ebf051383e42529dcb5f7dd0770a4b P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:32.884792 18985 ts_tablet_manager.cc:1367] T 75ebf051383e42529dcb5f7dd0770a4b P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20240430 20:11:32.886142 18985 raft_consensus.cc:348] T 75ebf051383e42529dcb5f7dd0770a4b P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:32.886363 18985 raft_consensus.cc:374] T 75ebf051383e42529dcb5f7dd0770a4b P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:32.886471 18985 raft_consensus.cc:729] T 75ebf051383e42529dcb5f7dd0770a4b P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:32.886716 18985 consensus_queue.cc:260] T 75ebf051383e42529dcb5f7dd0770a4b P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:32.887301 18985 ts_tablet_manager.cc:1398] T 75ebf051383e42529dcb5f7dd0770a4b P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240430 20:11:32.887642 18985 tablet_bootstrap.cc:492] T b72fe1f0713b4076a81a3fc1c74f7fa7 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:32.889370 18985 tablet_bootstrap.cc:654] T b72fe1f0713b4076a81a3fc1c74f7fa7 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:32.894647 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 56) took 1799 ms. Trace:
I20240430 20:11:32.894793 19495 rpcz_store.cc:276] 0430 20:11:31.095472 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:31.095526 (+ 54us) service_pool.cc:225] Handling call
0430 20:11:32.740218 (+1644692us) ts_tablet_manager.cc:1120] Deleting tablet c9ff972ade694938888514979e1638aa
0430 20:11:32.740228 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:32.841676 (+101448us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:32.845026 (+ 3350us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:32.894548 (+ 49522us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1636864}
I20240430 20:11:32.896075 19495 tablet_replica.cc:330] T 15674aeb56b74210a4a2809608fd4457 P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:32.896301 19495 raft_consensus.cc:2229] T 15674aeb56b74210a4a2809608fd4457 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Raft consensus shutting down.
I20240430 20:11:32.896435 18985 tablet_bootstrap.cc:492] T b72fe1f0713b4076a81a3fc1c74f7fa7 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:32.896580 18985 ts_tablet_manager.cc:1367] T b72fe1f0713b4076a81a3fc1c74f7fa7 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.001s
I20240430 20:11:32.896683 19495 raft_consensus.cc:2258] T 15674aeb56b74210a4a2809608fd4457 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:32.897445 19495 ts_tablet_manager.cc:1875] T 15674aeb56b74210a4a2809608fd4457 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:32.898181 18985 raft_consensus.cc:348] T b72fe1f0713b4076a81a3fc1c74f7fa7 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:32.898444 18985 raft_consensus.cc:374] T b72fe1f0713b4076a81a3fc1c74f7fa7 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:32.899461 18985 raft_consensus.cc:729] T b72fe1f0713b4076a81a3fc1c74f7fa7 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:32.899713 18985 consensus_queue.cc:260] T b72fe1f0713b4076a81a3fc1c74f7fa7 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:32.900768 18985 ts_tablet_manager.cc:1398] T b72fe1f0713b4076a81a3fc1c74f7fa7 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240430 20:11:32.901067 18985 tablet_bootstrap.cc:492] T fcc9698abda4491b98e1e879589913a8 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:32.903029 19004 catalog_manager.cc:4830] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): tablet c9ff972ade694938888514979e1638aa (table default.Table_3c5df662747b48bf8beb6ebe4d2ea618 [id=663c8efbd7ac4a148d5f7626f1522033]) successfully deleted
I20240430 20:11:32.904486 18985 tablet_bootstrap.cc:654] T fcc9698abda4491b98e1e879589913a8 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:32.910480 18985 tablet_bootstrap.cc:492] T fcc9698abda4491b98e1e879589913a8 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:32.910671 18985 ts_tablet_manager.cc:1367] T fcc9698abda4491b98e1e879589913a8 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.002s
I20240430 20:11:32.911486 18985 raft_consensus.cc:348] T fcc9698abda4491b98e1e879589913a8 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:32.911700 18985 raft_consensus.cc:374] T fcc9698abda4491b98e1e879589913a8 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:32.911788 18985 raft_consensus.cc:729] T fcc9698abda4491b98e1e879589913a8 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:32.912037 18985 consensus_queue.cc:260] T fcc9698abda4491b98e1e879589913a8 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:32.912582 18985 ts_tablet_manager.cc:1398] T fcc9698abda4491b98e1e879589913a8 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240430 20:11:32.915627 18985 tablet_bootstrap.cc:492] T 802336602d8f4ac4bd7e0872d6f3dae9 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:32.917310 18985 tablet_bootstrap.cc:654] T 802336602d8f4ac4bd7e0872d6f3dae9 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:32.920178 18988 raft_consensus.cc:729] T dd9b893d48684cfea7c242b93ecba01f P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
W20240430 20:11:32.920248 18696 consensus_peers.cc:485] T b76a4105a2974fe280e759349037d027 P 3c51c33d5616482cafb68e26688820f4 -> Peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Couldn't send request to peer 5e1c6397f09747f091534faf93542710. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20240430 20:11:32.924839 18696 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.177.2:46027, user_credentials={real_user=slave}} blocked reactor thread for 55577.8us
W20240430 20:11:32.925222 18696 leader_election.cc:341] T e3e74e76ab2b4fe492ffcc1838656fd1 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:32.928359 18985 tablet_bootstrap.cc:492] T 802336602d8f4ac4bd7e0872d6f3dae9 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:32.928572 18985 ts_tablet_manager.cc:1367] T 802336602d8f4ac4bd7e0872d6f3dae9 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.002s
I20240430 20:11:32.929370 18985 raft_consensus.cc:348] T 802336602d8f4ac4bd7e0872d6f3dae9 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:32.929591 18985 raft_consensus.cc:374] T 802336602d8f4ac4bd7e0872d6f3dae9 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:32.929649 18985 raft_consensus.cc:729] T 802336602d8f4ac4bd7e0872d6f3dae9 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:32.929865 18985 consensus_queue.cc:260] T 802336602d8f4ac4bd7e0872d6f3dae9 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:32.929874 18988 consensus_queue.cc:260] T dd9b893d48684cfea7c242b93ecba01f P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:32.930382 18985 ts_tablet_manager.cc:1398] T 802336602d8f4ac4bd7e0872d6f3dae9 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240430 20:11:32.933657 18985 tablet_bootstrap.cc:492] T 24dcc20832804229a2664873bf6ddaa7 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:32.937520 18985 tablet_bootstrap.cc:654] T 24dcc20832804229a2664873bf6ddaa7 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:32.938117 18988 ts_tablet_manager.cc:1398] T dd9b893d48684cfea7c242b93ecba01f P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.080s user 0.001s sys 0.001s
I20240430 20:11:32.945091 18985 tablet_bootstrap.cc:492] T 24dcc20832804229a2664873bf6ddaa7 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:32.945276 18985 ts_tablet_manager.cc:1367] T 24dcc20832804229a2664873bf6ddaa7 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.004s
I20240430 20:11:32.946785 18985 raft_consensus.cc:348] T 24dcc20832804229a2664873bf6ddaa7 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:32.947036 18985 raft_consensus.cc:374] T 24dcc20832804229a2664873bf6ddaa7 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:32.947131 18985 raft_consensus.cc:729] T 24dcc20832804229a2664873bf6ddaa7 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:32.947381 18985 consensus_queue.cc:260] T 24dcc20832804229a2664873bf6ddaa7 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:32.948033 18985 ts_tablet_manager.cc:1398] T 24dcc20832804229a2664873bf6ddaa7 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240430 20:11:32.948357 18985 tablet_bootstrap.cc:492] T 4e69ca12bd10494fb013614741bc166c P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:32.950420 18985 tablet_bootstrap.cc:654] T 4e69ca12bd10494fb013614741bc166c P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:32.959298 18985 tablet_bootstrap.cc:492] T 4e69ca12bd10494fb013614741bc166c P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:32.959503 18985 ts_tablet_manager.cc:1367] T 4e69ca12bd10494fb013614741bc166c P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.003s
I20240430 20:11:32.960317 18985 raft_consensus.cc:348] T 4e69ca12bd10494fb013614741bc166c P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:32.960572 18985 raft_consensus.cc:374] T 4e69ca12bd10494fb013614741bc166c P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:32.960656 18985 raft_consensus.cc:729] T 4e69ca12bd10494fb013614741bc166c P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:32.960915 18985 consensus_queue.cc:260] T 4e69ca12bd10494fb013614741bc166c P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:32.961441 18985 ts_tablet_manager.cc:1398] T 4e69ca12bd10494fb013614741bc166c P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240430 20:11:32.961727 18985 tablet_bootstrap.cc:492] T 255c13ad8fae4ea4aa8e7b7aabb7602e P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:32.968497 18985 tablet_bootstrap.cc:654] T 255c13ad8fae4ea4aa8e7b7aabb7602e P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:32.974848 18985 tablet_bootstrap.cc:492] T 255c13ad8fae4ea4aa8e7b7aabb7602e P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:32.975621 18985 ts_tablet_manager.cc:1367] T 255c13ad8fae4ea4aa8e7b7aabb7602e P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.002s
I20240430 20:11:32.976433 18985 raft_consensus.cc:348] T 255c13ad8fae4ea4aa8e7b7aabb7602e P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:32.976660 18985 raft_consensus.cc:374] T 255c13ad8fae4ea4aa8e7b7aabb7602e P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:32.976754 18985 raft_consensus.cc:729] T 255c13ad8fae4ea4aa8e7b7aabb7602e P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:32.976984 18985 consensus_queue.cc:260] T 255c13ad8fae4ea4aa8e7b7aabb7602e P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:32.979374 18985 ts_tablet_manager.cc:1398] T 255c13ad8fae4ea4aa8e7b7aabb7602e P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240430 20:11:32.980367 18985 tablet_bootstrap.cc:492] T fd40967ab4a04001819e16c06a6c4300 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:32.982200 18985 tablet_bootstrap.cc:654] T fd40967ab4a04001819e16c06a6c4300 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
W20240430 20:11:32.993119 19051 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 99203cc522de424184c6f97453f166fd consensus state has no leader
I20240430 20:11:32.996939 19048 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_fa64149f685c4784bda92ae4f578dd92"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:32.997336 19028 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 12c6e131fa904d23865b8673b58a7e33 consensus state has no leader
W20240430 20:11:32.997660 19048 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fa64149f685c4784bda92ae4f578dd92 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:32.997757 19028 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_8028554bc993419e80dbf7d526b63bf4" } modify_external_catalogs: true
I20240430 20:11:32.997944 19028 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_8028554bc993419e80dbf7d526b63bf4" } modify_external_catalogs: true
I20240430 20:11:32.999711 19049 catalog_manager.cc:6029] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: default.Table_e34613952dcb46618c3bc7cecb31fdaf [id=8f1ad1f8d0224b3daa5d84dc6916227b] alter complete (version 1)
I20240430 20:11:33.001401 19049 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_067c3b36dd2e4cfd828ab8ff6bbcbd2e" } modify_external_catalogs: true
I20240430 20:11:33.001606 19049 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_067c3b36dd2e4cfd828ab8ff6bbcbd2e" } modify_external_catalogs: true
I20240430 20:11:33.002261 19024 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_e68dc52988f24f1788517de94d7d2a0e" } new_table_name: "default.Table_ff91abfadbc746d6b01fdbfecd16fac5" modify_external_catalogs: true
I20240430 20:11:33.003146 19023 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_3471b34c1c0d4d829bb27a8520eff502"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:33.003705 19023 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3471b34c1c0d4d829bb27a8520eff502 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:33.005578 19033 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_211cbad17eba4c48ab02ad45543616ca"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:33.006196 19033 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_211cbad17eba4c48ab02ad45543616ca in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:33.007905 19051 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_e431f94220aa44ecba799a54dfb612ae" } new_table_name: "default.Table_3ec50a6d9cd84592becff23d2cfa0c03" modify_external_catalogs: true
I20240430 20:11:33.019726 18988 tablet_bootstrap.cc:492] T 75ebf051383e42529dcb5f7dd0770a4b P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:33.021690 18988 tablet_bootstrap.cc:654] T 75ebf051383e42529dcb5f7dd0770a4b P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:33.028447 19026 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_4a09e48b08fa4c4387262a5b92b2a42d" } modify_external_catalogs: true
I20240430 20:11:33.028734 19026 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_4a09e48b08fa4c4387262a5b92b2a42d" } modify_external_catalogs: true
I20240430 20:11:33.032163 19029 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_36adeb442f1245aba00549559bb329f3" } modify_external_catalogs: true
I20240430 20:11:33.032420 19029 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_36adeb442f1245aba00549559bb329f3" } modify_external_catalogs: true
I20240430 20:11:33.034715 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 60) took 1773 ms. Trace:
I20240430 20:11:33.034866 19489 rpcz_store.cc:276] 0430 20:11:31.261053 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:31.261354 (+ 301us) service_pool.cc:225] Handling call
0430 20:11:32.154511 (+893157us) ts_tablet_manager.cc:1120] Deleting tablet 29974dc2c7dd45b99b6cc0712378fc30
0430 20:11:32.154523 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:32.598692 (+444169us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:32.850057 (+251365us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:33.034662 (+184605us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":890765}
I20240430 20:11:33.035324 19489 tablet_replica.cc:330] T bbe9a2429a434f3f95d6e7f314fcfe4c P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:33.035507 19027 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_01e4bb745b8749a28f6182742fd38168" } modify_external_catalogs: true
I20240430 20:11:33.035568 19489 raft_consensus.cc:2229] T bbe9a2429a434f3f95d6e7f314fcfe4c P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Raft consensus shutting down.
I20240430 20:11:33.035730 19027 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_01e4bb745b8749a28f6182742fd38168" } modify_external_catalogs: true
I20240430 20:11:33.036165 19001 catalog_manager.cc:4830] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): tablet 29974dc2c7dd45b99b6cc0712378fc30 (table default.Table_28b63364f56b484eb48763e662dda489 [id=cc553a8f7052423b9724a3b28c63592d]) successfully deleted
I20240430 20:11:33.043615 18988 tablet_bootstrap.cc:492] T 75ebf051383e42529dcb5f7dd0770a4b P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:33.043903 18988 ts_tablet_manager.cc:1367] T 75ebf051383e42529dcb5f7dd0770a4b P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.024s user 0.000s sys 0.009s
I20240430 20:11:33.044975 18988 raft_consensus.cc:348] T 75ebf051383e42529dcb5f7dd0770a4b P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:33.045455 18988 raft_consensus.cc:374] T 75ebf051383e42529dcb5f7dd0770a4b P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:33.045589 18988 raft_consensus.cc:729] T 75ebf051383e42529dcb5f7dd0770a4b P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:33.046008 19028 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: Sending DeleteTablet for 3 replicas of tablet c2b019f529a54228b804437bb025e31d
I20240430 20:11:33.045979 18988 consensus_queue.cc:260] T 75ebf051383e42529dcb5f7dd0770a4b P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:33.046784 19024 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_aed0b9eeaac74c98944426712706be6b" } new_table_name: "default.Table_a2e85efc6b454524b0b122102ec07002" modify_external_catalogs: true
I20240430 20:11:33.048897 19048 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_81547db3074240c98cdd95ed3b2dd813"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240430 20:11:33.050170 18446 tablet_service.cc:1514] Processing DeleteTablet for tablet c2b019f529a54228b804437bb025e31d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:32 UTC) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:33.058241 18988 ts_tablet_manager.cc:1398] T 75ebf051383e42529dcb5f7dd0770a4b P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.014s user 0.001s sys 0.001s
I20240430 20:11:33.058322 19049 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: Sending DeleteTablet for 3 replicas of tablet 246fb63a5e9842df8263800e28a5935c
I20240430 20:11:33.059211 18724 tablet_service.cc:1514] Processing DeleteTablet for tablet 246fb63a5e9842df8263800e28a5935c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:32 UTC) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:33.059742 18446 tablet_service.cc:1514] Processing DeleteTablet for tablet 246fb63a5e9842df8263800e28a5935c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:32 UTC) from {username='slave'} at 127.0.0.1:41680
W20240430 20:11:33.060031 19048 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_81547db3074240c98cdd95ed3b2dd813 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:33.062140 19027 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet 1244a99afffb4d9bb0adba64ae91a5fb
I20240430 20:11:33.064970 18988 tablet_bootstrap.cc:492] T 24dcc20832804229a2664873bf6ddaa7 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:33.041270 19645 raft_consensus.cc:504] T 12c6e131fa904d23865b8673b58a7e33 P 5e1c6397f09747f091534faf93542710 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:33.067039 18988 tablet_bootstrap.cc:654] T 24dcc20832804229a2664873bf6ddaa7 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:33.077031 18988 tablet_bootstrap.cc:492] T 24dcc20832804229a2664873bf6ddaa7 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:33.078722 18988 ts_tablet_manager.cc:1367] T 24dcc20832804229a2664873bf6ddaa7 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.000s
I20240430 20:11:33.082026 18988 raft_consensus.cc:348] T 24dcc20832804229a2664873bf6ddaa7 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:33.083693 18988 raft_consensus.cc:374] T 24dcc20832804229a2664873bf6ddaa7 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:33.084110 18988 raft_consensus.cc:729] T 24dcc20832804229a2664873bf6ddaa7 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
W20240430 20:11:33.084846 18685 kernel_stack_watchdog.cc:198] Thread 18699 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:308 for 113ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7f30eb118980 (unknown) at ??:0
@ 0x90ef77 __sanitizer::internal_sched_yield() at ??:?
@ 0x87462d __interceptor_pthread_create at ??:?
@ 0x7f30e7d86d11 kudu::Thread::StartThread() at ??:0
@ 0xa4d187 kudu::Thread::Create() at ??:?
@ 0x7f30e7da77ab kudu::ThreadPool::CreateThread() at ??:0
@ 0x7f30e7da33ad kudu::ThreadPool::DoSubmit() at ??:0
@ 0x7f30e7da184f kudu::ThreadPoolToken::Submit() at ??:0
@ 0x7f30f35d619b kudu::consensus::RaftConsensus::ElectionCallback() at ??:0
@ 0x7f30f35e0db4 kudu::consensus::RaftConsensus::StartElection()::$_3::operator()() at ??:0
@ 0x7f30f35e0ac2 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f30f3544c9e std::function<>::operator()() at ??:0
@ 0x7f30f353e5b3 kudu::consensus::LeaderElection::CheckForDecision() at ??:0
@ 0x7f30f353f782 kudu::consensus::LeaderElection::VoteResponseRpcCallback() at ??:0
@ 0x7f30f354212a kudu::consensus::LeaderElection::Run()::$_0::operator()() at ??:0
@ 0x7f30f3541eee std::_Function_handler<>::_M_invoke() at ??:0
I20240430 20:11:33.084740 18988 consensus_queue.cc:260] T 24dcc20832804229a2664873bf6ddaa7 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:33.086231 18988 ts_tablet_manager.cc:1398] T 24dcc20832804229a2664873bf6ddaa7 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240430 20:11:33.086776 18988 tablet_bootstrap.cc:492] T c9e7660eccc448dbb99d135f82bfd261 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:33.088708 18988 tablet_bootstrap.cc:654] T c9e7660eccc448dbb99d135f82bfd261 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:33.098520 18623 raft_consensus.cc:1226] T 12c6e131fa904d23865b8673b58a7e33 P 5e1c6397f09747f091534faf93542710 [term 2 FOLLOWER]: Rejecting Update request from peer 3c51c33d5616482cafb68e26688820f4 for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20240430 20:11:33.099027 18590 tablet_service.cc:1514] Processing DeleteTablet for tablet 246fb63a5e9842df8263800e28a5935c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:32 UTC) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:33.100306 19727 consensus_queue.cc:1046] T 12c6e131fa904d23865b8673b58a7e33 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:33.100620 19727 raft_consensus.cc:3032] T 12c6e131fa904d23865b8673b58a7e33 P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Stepping down as leader of term 1
I20240430 20:11:33.100721 19727 raft_consensus.cc:729] T 12c6e131fa904d23865b8673b58a7e33 P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Running, Role: LEADER
I20240430 20:11:33.101012 19727 consensus_queue.cc:260] T 12c6e131fa904d23865b8673b58a7e33 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:33.101428 19727 raft_consensus.cc:3037] T 12c6e131fa904d23865b8673b58a7e33 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Advancing to term 2
I20240430 20:11:33.107009 18622 raft_consensus.cc:2454] T 01436e5b6232494babbb414342f03090 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:33.107601 18696 leader_election.cc:302] T 01436e5b6232494babbb414342f03090 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:33.107983 19674 raft_consensus.cc:2781] T 01436e5b6232494babbb414342f03090 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:33.108153 19674 raft_consensus.cc:686] T 01436e5b6232494babbb414342f03090 P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Becoming Leader. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Running, Role: LEADER
I20240430 20:11:33.108500 19674 consensus_queue.cc:237] T 01436e5b6232494babbb414342f03090 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:33.120118 19027 master_service.cc:946] ReplaceTablet: received request to replace tablet 50486644ec234933b9a0b45ee39a6685 from {username='slave'} at 127.0.0.1:51664
I20240430 20:11:33.126751 18988 tablet_bootstrap.cc:492] T c9e7660eccc448dbb99d135f82bfd261 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:33.126966 18988 ts_tablet_manager.cc:1367] T c9e7660eccc448dbb99d135f82bfd261 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.040s user 0.011s sys 0.005s
I20240430 20:11:33.127938 18988 raft_consensus.cc:348] T c9e7660eccc448dbb99d135f82bfd261 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:33.128216 18988 raft_consensus.cc:374] T c9e7660eccc448dbb99d135f82bfd261 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:33.128300 18988 raft_consensus.cc:729] T c9e7660eccc448dbb99d135f82bfd261 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:33.128573 18988 consensus_queue.cc:260] T c9e7660eccc448dbb99d135f82bfd261 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:33.129245 18988 ts_tablet_manager.cc:1398] T c9e7660eccc448dbb99d135f82bfd261 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240430 20:11:33.129886 18988 tablet_bootstrap.cc:492] T b72fe1f0713b4076a81a3fc1c74f7fa7 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:33.131958 18988 tablet_bootstrap.cc:654] T b72fe1f0713b4076a81a3fc1c74f7fa7 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:33.132246 19624 tablet.cc:1722] T c66c142419ca4e1399ecd1c8d1e62a59 P 5e1c6397f09747f091534faf93542710: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240430 20:11:33.134317 19645 leader_election.cc:288] T 12c6e131fa904d23865b8673b58a7e33 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 2 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:33.134712 18477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12c6e131fa904d23865b8673b58a7e33" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:33.134924 18477 raft_consensus.cc:3037] T 12c6e131fa904d23865b8673b58a7e33 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Advancing to term 2
I20240430 20:11:33.159541 19645 raft_consensus.cc:482] T 26ed12f498644d15bf45e884dcad172e P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:33.159768 19645 raft_consensus.cc:504] T 26ed12f498644d15bf45e884dcad172e P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
W20240430 20:11:33.164659 19707 kernel_stack_watchdog.cc:198] Thread 19519 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:637 for 400ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7f6d26e7d980 (unknown) at ??:0
@ 0x90ef77 __sanitizer::internal_sched_yield() at ??:?
@ 0x87462d __interceptor_pthread_create at ??:?
@ 0x7f6d23aebd11 kudu::Thread::StartThread() at ??:0
@ 0xa4d187 kudu::Thread::Create() at ??:?
@ 0x7f6d2c42b559 kudu::cloud::InstanceDetector::Detect() at ??:0
@ 0x7f6d2cac35b7 kudu::server::ServerBase::WallClockJumpDetectionNeeded() at ??:0
@ 0x7f6d2cabe90d kudu::server::ServerBase::Init() at ??:0
@ 0x7f6d2e0ced45 kudu::kserver::KuduServer::Init() at ??:0
@ 0x7f6d37033cca kudu::master::Master::Init() at ??:0
@ 0x7f6d370957d9 kudu::master::RunMasterServer() at ??:0
@ 0xa7a0b7 kudu::tools::(anonymous namespace)::MasterRun() at ??:?
@ 0x98d981 std::_Function_handler<>::_M_invoke() at ??:?
@ 0x7f6d379bcb79 std::function<>::operator()() at ??:0
@ 0x7f6d379b8655 kudu::tools::Action::Run() at ??:0
@ 0xb970f4 kudu::tools::DispatchCommand() at ??:?
I20240430 20:11:33.176101 19033 master_service.cc:946] ReplaceTablet: received request to replace tablet 0b4d9e860d7c45fd91d7a566605c9283 from {username='slave'} at 127.0.0.1:51664
I20240430 20:11:33.177506 18480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26ed12f498644d15bf45e884dcad172e" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:33.177803 18480 raft_consensus.cc:2454] T 26ed12f498644d15bf45e884dcad172e P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 0.
W20240430 20:11:33.183686 18603 consensus_meta.cc:287] T de1dfe6c82d34a4d8d02e466c55327ff P 5e1c6397f09747f091534faf93542710: Time spent flushing consensus metadata: real 0.531s user 0.006s sys 0.000s
I20240430 20:11:33.186844 18734 tablet_service.cc:1514] Processing DeleteTablet for tablet c2b019f529a54228b804437bb025e31d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:32 UTC) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:33.189898 18558 leader_election.cc:302] T 26ed12f498644d15bf45e884dcad172e P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:33.191051 19474 raft_consensus.cc:2781] T 26ed12f498644d15bf45e884dcad172e P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:33.191192 19474 raft_consensus.cc:482] T 26ed12f498644d15bf45e884dcad172e P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:33.191287 19474 raft_consensus.cc:3037] T 26ed12f498644d15bf45e884dcad172e P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
W20240430 20:11:33.191571 18583 consensus_meta.cc:287] T a0baee91f5b44945b595d0e234af018e P 5e1c6397f09747f091534faf93542710: Time spent flushing consensus metadata: real 0.594s user 0.006s sys 0.000s
I20240430 20:11:33.192435 19645 leader_election.cc:288] T 26ed12f498644d15bf45e884dcad172e P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:33.193519 18588 tablet_service.cc:1514] Processing DeleteTablet for tablet c2b019f529a54228b804437bb025e31d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:32 UTC) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:33.195173 19023 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_97a0991bf9284215b482c532e9dfc113"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:33.195842 19023 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_97a0991bf9284215b482c532e9dfc113 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:33.204736 18446 tablet_service.cc:1467] Processing CreateTablet for tablet 331c031d44854d679a27d10d9151d0ea (DEFAULT_TABLE table=default.Table_999c55e7bef74339b7b3ab1142599447 [id=61c8a5b99ef34a0baeeb343fef7cfd05]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:33.208003 18983 tablet_bootstrap.cc:492] T e3e74e76ab2b4fe492ffcc1838656fd1 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:33.208148 18983 ts_tablet_manager.cc:1367] T e3e74e76ab2b4fe492ffcc1838656fd1 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.723s user 0.029s sys 0.005s
I20240430 20:11:33.208988 18983 raft_consensus.cc:348] T e3e74e76ab2b4fe492ffcc1838656fd1 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:33.209255 18983 raft_consensus.cc:374] T e3e74e76ab2b4fe492ffcc1838656fd1 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:33.215306 18734 tablet_service.cc:1467] Processing CreateTablet for tablet 6d1d2193fd424c91b143f5256df5fcd0 (DEFAULT_TABLE table=default.Table_dd819f96762949c786839dda00487cc2 [id=e1e2e2a5b0b740cba09110991dd28193]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:33.215866 18734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d1d2193fd424c91b143f5256df5fcd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:33.239456 19048 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_575286687a4e4020a315d78080c2e28e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:33.240371 19048 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_575286687a4e4020a315d78080c2e28e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:33.251250 19700 raft_consensus.cc:482] T b72fe1f0713b4076a81a3fc1c74f7fa7 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:33.251487 19700 raft_consensus.cc:504] T b72fe1f0713b4076a81a3fc1c74f7fa7 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:33.252209 19700 leader_election.cc:288] T b72fe1f0713b4076a81a3fc1c74f7fa7 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:33.256851 18587 tablet_service.cc:1467] Processing CreateTablet for tablet 6d1d2193fd424c91b143f5256df5fcd0 (DEFAULT_TABLE table=default.Table_dd819f96762949c786839dda00487cc2 [id=e1e2e2a5b0b740cba09110991dd28193]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:33.256850 18448 tablet_service.cc:1467] Processing CreateTablet for tablet 6d1d2193fd424c91b143f5256df5fcd0 (DEFAULT_TABLE table=default.Table_dd819f96762949c786839dda00487cc2 [id=e1e2e2a5b0b740cba09110991dd28193]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:33.260505 19581 raft_consensus.cc:482] T dd9b893d48684cfea7c242b93ecba01f P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:33.260603 18622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b72fe1f0713b4076a81a3fc1c74f7fa7" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:33.260707 19581 raft_consensus.cc:504] T dd9b893d48684cfea7c242b93ecba01f P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:33.260877 18622 raft_consensus.cc:2454] T b72fe1f0713b4076a81a3fc1c74f7fa7 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 0.
I20240430 20:11:33.261583 19581 leader_election.cc:288] T dd9b893d48684cfea7c242b93ecba01f P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:33.261857 18622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd9b893d48684cfea7c242b93ecba01f" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:33.262092 19700 raft_consensus.cc:482] T b688e03ea71940c69bda5520b62c1fc3 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:33.262144 18622 raft_consensus.cc:2379] T dd9b893d48684cfea7c242b93ecba01f P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c51c33d5616482cafb68e26688820f4 in current term 1: Already voted for candidate 2efb542aa55e439b99ccb98d1c5c884a in this term.
I20240430 20:11:33.262238 19700 raft_consensus.cc:504] T b688e03ea71940c69bda5520b62c1fc3 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:33.263011 19700 leader_election.cc:288] T b688e03ea71940c69bda5520b62c1fc3 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:33.263792 18622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b688e03ea71940c69bda5520b62c1fc3" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:33.264031 18622 raft_consensus.cc:2454] T b688e03ea71940c69bda5520b62c1fc3 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 0.
W20240430 20:11:33.266495 18699 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.575s user 0.070s sys 0.113s
W20240430 20:11:33.266976 18699 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.576s user 0.070s sys 0.113s
W20240430 20:11:33.267220 18699 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.177.1:41217, user_credentials={real_user=slave}} blocked reactor thread for 667296us
I20240430 20:11:33.268651 18475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eed6338ad3d64859af087ee15e1ec798" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:33.268954 18475 raft_consensus.cc:3037] T eed6338ad3d64859af087ee15e1ec798 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:33.269168 18478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01436e5b6232494babbb414342f03090" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:33.269778 18483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3e74e76ab2b4fe492ffcc1838656fd1" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:33.270040 18483 raft_consensus.cc:3037] T e3e74e76ab2b4fe492ffcc1838656fd1 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
W20240430 20:11:33.271078 18557 leader_election.cc:341] T 255c13ad8fae4ea4aa8e7b7aabb7602e P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:33.271229 18557 leader_election.cc:302] T 255c13ad8fae4ea4aa8e7b7aabb7602e P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4
W20240430 20:11:33.271490 18557 raft_consensus.cc:2670] T 255c13ad8fae4ea4aa8e7b7aabb7602e P 5e1c6397f09747f091534faf93542710: unable to run election callback: Service unavailable: Thread pool token was shut down
I20240430 20:11:33.275082 19180 raft_consensus.cc:482] T 75ebf051383e42529dcb5f7dd0770a4b P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:33.275270 19180 raft_consensus.cc:504] T 75ebf051383e42529dcb5f7dd0770a4b P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:33.276219 19180 leader_election.cc:288] T 75ebf051383e42529dcb5f7dd0770a4b P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:33.278635 19028 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet acd23f0a63d5448aaa7ed78422e67bad (table default.Table_972f13be4dca4cd19b9a04534dc59f51 [id=69f37db4de634a3185613e40013e0ed7]). Expected version 1 got 0
I20240430 20:11:33.278926 19028 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 2e79df8744e74b5991a79ce978ff0202 (table default.Table_276bbdc7cfe542d1a2936485a2aad1ec [id=f146a15e6e8a4acbb4c734c1c863da57]). Expected version 1 got 0
I20240430 20:11:33.279111 19028 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 24dcc20832804229a2664873bf6ddaa7 (table default.Table_be2ebe77f859489f9cd3e912cb1edc8b [id=2d44ec94131148048021b2ee6366ba30]). Expected version 2 got 0
I20240430 20:11:33.279321 19028 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet de1dfe6c82d34a4d8d02e466c55327ff (table default.Table_e431f94220aa44ecba799a54dfb612ae [id=1e785b6c159b4312bd1b064271f55a49]). Expected version 1 got 0
I20240430 20:11:33.279639 19028 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet aec9c09cc0454acab6006bb3e10982b7 (table default.Table_ff91abfadbc746d6b01fdbfecd16fac5 [id=26d4860fb33047b7b812ebd4309f2e28]). Expected version 2 got 1
I20240430 20:11:33.283107 19048 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_1755296c0a63478699d9b8387f606a8e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:33.283823 19048 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1755296c0a63478699d9b8387f606a8e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:33.293982 19048 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_85bed92bc6b94df3bd1f89734050f578" } new_table_name: "default.Table_58f35a2120874472978ca0e5f816e86c" modify_external_catalogs: true
I20240430 20:11:33.297442 18415 leader_election.cc:302] T b72fe1f0713b4076a81a3fc1c74f7fa7 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:33.297816 19700 raft_consensus.cc:2781] T b72fe1f0713b4076a81a3fc1c74f7fa7 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:33.297935 19700 raft_consensus.cc:482] T b72fe1f0713b4076a81a3fc1c74f7fa7 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:33.298039 19700 raft_consensus.cc:3037] T b72fe1f0713b4076a81a3fc1c74f7fa7 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:33.299186 18472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75ebf051383e42529dcb5f7dd0770a4b" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:33.299547 18472 raft_consensus.cc:2454] T 75ebf051383e42529dcb5f7dd0770a4b P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 0.
I20240430 20:11:33.299655 18415 leader_election.cc:302] T b688e03ea71940c69bda5520b62c1fc3 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:33.300539 18558 leader_election.cc:302] T 75ebf051383e42529dcb5f7dd0770a4b P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:33.300997 19645 raft_consensus.cc:2781] T 75ebf051383e42529dcb5f7dd0770a4b P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:33.301142 19645 raft_consensus.cc:482] T 75ebf051383e42529dcb5f7dd0770a4b P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:33.301257 19645 raft_consensus.cc:3037] T 75ebf051383e42529dcb5f7dd0770a4b P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:33.325119 18450 tablet_service.cc:1467] Processing CreateTablet for tablet 78ba48dadef548708153878810cc46f8 (DEFAULT_TABLE table=default.Table_78678179d193432ebe9e2484951c2102 [id=29769be2b9ec4ecba9de5c0a65692f85]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:33.326200 18588 tablet_service.cc:1467] Processing CreateTablet for tablet 53da4b3855344b6d92211af767195dc8 (DEFAULT_TABLE table=default.Table_07993f6164104b888058e1b35c9f4ed7 [id=f2e6cac20f6a4261bb60f2e40f210491]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:33.326596 18599 tablet_service.cc:1467] Processing CreateTablet for tablet 78ba48dadef548708153878810cc46f8 (DEFAULT_TABLE table=default.Table_78678179d193432ebe9e2484951c2102 [id=29769be2b9ec4ecba9de5c0a65692f85]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:33.326929 18463 tablet_service.cc:1467] Processing CreateTablet for tablet 53da4b3855344b6d92211af767195dc8 (DEFAULT_TABLE table=default.Table_07993f6164104b888058e1b35c9f4ed7 [id=f2e6cac20f6a4261bb60f2e40f210491]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:33.328500 18583 tablet_service.cc:1467] Processing CreateTablet for tablet fdf53061efd74fa29a9d8208d1fd210e (DEFAULT_TABLE table=default.Table_f4b025d97b144efa9449e830a6f44126 [id=7fd6f8576f4c4cbc8ffcdb4ab95e15dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:33.329056 18449 tablet_service.cc:1467] Processing CreateTablet for tablet fdf53061efd74fa29a9d8208d1fd210e (DEFAULT_TABLE table=default.Table_f4b025d97b144efa9449e830a6f44126 [id=7fd6f8576f4c4cbc8ffcdb4ab95e15dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:33.339066 19753 tablet.cc:1722] T c66c142419ca4e1399ecd1c8d1e62a59 P 2efb542aa55e439b99ccb98d1c5c884a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20240430 20:11:33.340941 19048 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 488d0e609b224674932f890939664df2 has no consensus state
I20240430 20:11:33.341519 19048 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_78678179d193432ebe9e2484951c2102" } new_table_name: "default.Table_b9983e101a6547649eb44ee04b5e1147" modify_external_catalogs: true
I20240430 20:11:33.209347 18983 raft_consensus.cc:729] T e3e74e76ab2b4fe492ffcc1838656fd1 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:33.344754 18983 consensus_queue.cc:260] T e3e74e76ab2b4fe492ffcc1838656fd1 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:33.353320 18983 ts_tablet_manager.cc:1398] T e3e74e76ab2b4fe492ffcc1838656fd1 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.145s user 0.001s sys 0.001s
I20240430 20:11:33.378809 19617 raft_consensus.cc:2781] T b688e03ea71940c69bda5520b62c1fc3 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:33.380712 18729 tablet_service.cc:1467] Processing CreateTablet for tablet 53da4b3855344b6d92211af767195dc8 (DEFAULT_TABLE table=default.Table_07993f6164104b888058e1b35c9f4ed7 [id=f2e6cac20f6a4261bb60f2e40f210491]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:33.380793 18724 tablet_service.cc:1467] Processing CreateTablet for tablet 78ba48dadef548708153878810cc46f8 (DEFAULT_TABLE table=default.Table_78678179d193432ebe9e2484951c2102 [id=29769be2b9ec4ecba9de5c0a65692f85]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:33.381165 18729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53da4b3855344b6d92211af767195dc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:33.382922 18621 raft_consensus.cc:2454] T eed6338ad3d64859af087ee15e1ec798 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:33.383476 19471 ts_tablet_manager.cc:1888] T 255c13ad8fae4ea4aa8e7b7aabb7602e P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:33.383607 19471 log.cc:1196] T 255c13ad8fae4ea4aa8e7b7aabb7602e P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/255c13ad8fae4ea4aa8e7b7aabb7602e
I20240430 20:11:33.383949 19471 ts_tablet_manager.cc:1909] T 255c13ad8fae4ea4aa8e7b7aabb7602e P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
W20240430 20:11:33.384511 18595 consensus_meta.cc:287] T f5eaf2413e1d4e69b56fae61f768f0ce P 5e1c6397f09747f091534faf93542710: Time spent flushing consensus metadata: real 0.510s user 0.003s sys 0.001s
I20240430 20:11:33.384764 19726 consensus_queue.cc:1035] T 1555924b5f1444398f12ee82c3cb5b13 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20240430 20:11:33.387557 18583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdf53061efd74fa29a9d8208d1fd210e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:33.389492 18595 tablet_service.cc:1467] Processing CreateTablet for tablet 130007acc8ba4e709c3eb85fabe75903 (DEFAULT_TABLE table=default.Table_c6d5c8ecce28428790b07d228879a0e4 [id=36f478a20a1f4a53a52fc29ade62b7f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:33.389679 18983 tablet_bootstrap.cc:492] T b76a4105a2974fe280e759349037d027 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
W20240430 20:11:33.390650 18593 consensus_meta.cc:287] T 22fef8bb6b034db098c3372ae992f7a7 P 5e1c6397f09747f091534faf93542710: Time spent flushing consensus metadata: real 0.511s user 0.004s sys 0.001s
I20240430 20:11:33.391394 18587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d1d2193fd424c91b143f5256df5fcd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:33.391907 18588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53da4b3855344b6d92211af767195dc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:33.392392 18599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78ba48dadef548708153878810cc46f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:33.279850 19028 catalog_manager.cc:5484] T 294644d7a16b4ca2bbb6d106bcb8c7ff P 3c51c33d5616482cafb68e26688820f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c51c33d5616482cafb68e26688820f4 (127.17.177.3). New cstate: current_term: 1 leader_uuid: "3c51c33d5616482cafb68e26688820f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } health_report { overall_health: UNKNOWN } } }
W20240430 20:11:33.397495 18601 consensus_meta.cc:287] T 04b3dff86d5c48a09d03da58bd48f360 P 5e1c6397f09747f091534faf93542710: Time spent flushing consensus metadata: real 0.644s user 0.000s sys 0.004s
I20240430 20:11:33.399469 19028 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 294644d7a16b4ca2bbb6d106bcb8c7ff (table default.Table_190b2d86515a44898fa38a49517b9df1 [id=97d8a36ddfae4bb5b99c04a7a08544af]). Expected version 2 got 0
I20240430 20:11:33.403458 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 54) took 2412 ms. Trace:
I20240430 20:11:33.403569 19471 rpcz_store.cc:276] 0430 20:11:30.990825 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:30.992064 (+ 1239us) service_pool.cc:225] Handling call
0430 20:11:32.689647 (+1697583us) ts_tablet_manager.cc:1120] Deleting tablet 255c13ad8fae4ea4aa8e7b7aabb7602e
0430 20:11:32.689656 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:32.952470 (+262814us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:33.383359 (+430889us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:33.403390 (+ 20031us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1695711}
I20240430 20:11:33.404069 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 62) took 2412 ms. Trace:
I20240430 20:11:33.404166 19471 rpcz_store.cc:276] 0430 20:11:30.991480 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:31.001498 (+ 10018us) service_pool.cc:225] Handling call
0430 20:11:33.403870 (+2402372us) ts_tablet_manager.cc:1120] Deleting tablet a9e0659564894b0e8cb7f600bb07a085
0430 20:11:33.403880 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:33.404002 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2402078}
I20240430 20:11:33.404459 19471 tablet_replica.cc:330] T a1aeee1b02214975a376cfd36957afe8 P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:33.404662 19471 raft_consensus.cc:2229] T a1aeee1b02214975a376cfd36957afe8 P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Raft consensus shutting down.
I20240430 20:11:33.404994 19471 raft_consensus.cc:2258] T a1aeee1b02214975a376cfd36957afe8 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:33.405195 18738 tablet_service.cc:1467] Processing CreateTablet for tablet fdf53061efd74fa29a9d8208d1fd210e (DEFAULT_TABLE table=default.Table_f4b025d97b144efa9449e830a6f44126 [id=7fd6f8576f4c4cbc8ffcdb4ab95e15dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:33.405594 19471 ts_tablet_manager.cc:1875] T a1aeee1b02214975a376cfd36957afe8 P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:33.406714 18595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 130007acc8ba4e709c3eb85fabe75903. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:33.406234 18999 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet 255c13ad8fae4ea4aa8e7b7aabb7602e (table default.Table_a5ab8fc47d4e4d128ba4370b3249d8e5 [id=1238beeef04b476b8f4dc5af2fb68012]) successfully deleted
W20240430 20:11:33.406996 18999 catalog_manager.cc:4810] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): delete failed for tablet a9e0659564894b0e8cb7f600bb07a085 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a9e0659564894b0e8cb7f600bb07a085 already in progress: creating tablet
I20240430 20:11:33.415701 18724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78ba48dadef548708153878810cc46f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:33.418933 18696 leader_election.cc:302] T eed6338ad3d64859af087ee15e1ec798 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:33.419298 19726 raft_consensus.cc:2781] T eed6338ad3d64859af087ee15e1ec798 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:33.419554 18726 tablet_service.cc:1467] Processing CreateTablet for tablet 885d27d4f8df48dd996d5b309b783f3b (DEFAULT_TABLE table=default.Table_47779624a66e46468c8e9e35a8e9487f [id=8ce80df659a146c9b7d5fdb3059902cd]), partition=RANGE (key) PARTITION UNBOUNDED
W20240430 20:11:33.420007 18744 consensus_meta.cc:287] T b22aaedb0a6c4fd9ad0ea8fa019001e1 P 3c51c33d5616482cafb68e26688820f4: Time spent flushing consensus metadata: real 0.531s user 0.003s sys 0.003s
I20240430 20:11:33.422785 18732 tablet_service.cc:1467] Processing CreateTablet for tablet 130007acc8ba4e709c3eb85fabe75903 (DEFAULT_TABLE table=default.Table_c6d5c8ecce28428790b07d228879a0e4 [id=36f478a20a1f4a53a52fc29ade62b7f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:33.423182 19726 raft_consensus.cc:686] T eed6338ad3d64859af087ee15e1ec798 P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Becoming Leader. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Running, Role: LEADER
I20240430 20:11:33.424695 18738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdf53061efd74fa29a9d8208d1fd210e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:33.405936 19028 catalog_manager.cc:5484] T b76a4105a2974fe280e759349037d027 P 3c51c33d5616482cafb68e26688820f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c51c33d5616482cafb68e26688820f4 (127.17.177.3). New cstate: current_term: 1 leader_uuid: "3c51c33d5616482cafb68e26688820f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } health_report { overall_health: UNKNOWN } } }
W20240430 20:11:33.425959 19000 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 488d0e609b224674932f890939664df2 (table default.Table_58f35a2120874472978ca0e5f816e86c [id=ff18001c6d814afc8c1768d0c84a6bac], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 488d0e609b224674932f890939664df2 has no consensus state
I20240430 20:11:33.426154 19028 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet c66c142419ca4e1399ecd1c8d1e62a59 (table default.Table_0443359dbeff437fb6cffb0db33cbb30 [id=93c44abdf6b147228224388398cd7aeb]). Expected version 2 got 0
I20240430 20:11:33.426362 19726 consensus_queue.cc:237] T eed6338ad3d64859af087ee15e1ec798 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:33.427196 18732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 130007acc8ba4e709c3eb85fabe75903. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:33.427857 18759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a1e50f99c5d4b44bdbb51b58a92bc75" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
W20240430 20:11:33.427943 19723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240430 20:11:33.430439 19028 catalog_manager.cc:5484] T 12c6e131fa904d23865b8673b58a7e33 P 3c51c33d5616482cafb68e26688820f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c51c33d5616482cafb68e26688820f4 (127.17.177.3). New cstate: current_term: 1 leader_uuid: "3c51c33d5616482cafb68e26688820f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } health_report { overall_health: UNKNOWN } } }
I20240430 20:11:33.438630 19474 raft_consensus.cc:504] T 26ed12f498644d15bf45e884dcad172e P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:33.443081 19028 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 12c6e131fa904d23865b8673b58a7e33 (table default.Table_a9dc8fc6a4aa46dc914d0e327a4de554 [id=19f4e6decedf4ee59fccc0a24a05cedd]). Expected version 2 got 0
I20240430 20:11:33.448709 19474 leader_election.cc:288] T 26ed12f498644d15bf45e884dcad172e P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:33.455076 18621 raft_consensus.cc:3037] T a05e32a33aa7488eb287cc6314c09c44 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:33.457464 19028 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 66dd767bd3f8494f94061fb89e5e5680 (table default.Table_9c1e6182953140bb9cd594f08458c7d9 [id=5227206f44d04fa89b4842b899f33afb]). Expected version 1 got 0
I20240430 20:11:33.458796 19028 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 6e754af64cb64fd2a082adae7d5f24da (table default.Table_208b12b8e9bd4f3e8bc9c6d422ada649 [id=1df4e95702cd46f59ab21b101049a9ed]). Expected version 1 got 0
I20240430 20:11:33.460005 18726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 885d27d4f8df48dd996d5b309b783f3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:33.461437 19028 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 32c823d1700949ab8b22adcd83b7bfd1 (table default.Table_e8c73c89623d4b49b8d7f0a01d729bdd [id=5cd10682ce52414cb5a103e4433a6a0d]). Expected version 1 got 0
I20240430 20:11:33.462536 19028 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 34130f2820064fa0898ef1f9d7de1c6d (table default.Table_4537f891b01048ceb51202abc3600911 [id=5a1c4a7be2fc465fac89a50854eb3322]). Expected version 1 got 0
I20240430 20:11:33.463549 19028 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 99203cc522de424184c6f97453f166fd (table default.Table_cb2a4b9ceaf4431facfff2c9bcf94c5d [id=3f8690a437ae49a3beb009fa7899d5d0]). Expected version 1 got 0
I20240430 20:11:33.464233 18762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b72fe1f0713b4076a81a3fc1c74f7fa7" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
W20240430 20:11:33.471345 19519 thread.cc:636] Azure (cloud detector) Time spent creating pthread: real 0.853s user 0.096s sys 0.173s
W20240430 20:11:33.471575 19519 thread.cc:602] Azure (cloud detector) Time spent starting thread: real 0.853s user 0.096s sys 0.173s
I20240430 20:11:33.479476 18599 tablet_service.cc:1467] Processing CreateTablet for tablet 885d27d4f8df48dd996d5b309b783f3b (DEFAULT_TABLE table=default.Table_47779624a66e46468c8e9e35a8e9487f [id=8ce80df659a146c9b7d5fdb3059902cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:33.480636 18764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b688e03ea71940c69bda5520b62c1fc3" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:33.481048 18600 tablet_service.cc:1467] Processing CreateTablet for tablet c091701480c4412faabad3d071d85d03 (DEFAULT_TABLE table=default.Table_3a35534b6ef2416b968740d28b156dd0 [id=f3c08d1b91be4bf58242ca535ff28897]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:33.481185 19727 raft_consensus.cc:482] T 75ebf051383e42529dcb5f7dd0770a4b P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:33.482812 18983 tablet_bootstrap.cc:654] T b76a4105a2974fe280e759349037d027 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:33.485424 19617 raft_consensus.cc:482] T b688e03ea71940c69bda5520b62c1fc3 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:33.485589 19617 raft_consensus.cc:3037] T b688e03ea71940c69bda5520b62c1fc3 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:33.485898 19186 raft_consensus.cc:482] T fcc9698abda4491b98e1e879589913a8 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:33.486030 19186 raft_consensus.cc:504] T fcc9698abda4491b98e1e879589913a8 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:33.486778 19186 leader_election.cc:288] T fcc9698abda4491b98e1e879589913a8 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:33.487900 18759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acd23f0a63d5448aaa7ed78422e67bad" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
W20240430 20:11:33.488279 18415 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.177.2:46027, user_credentials={real_user=slave}} blocked reactor thread for 79546.9us
I20240430 20:11:33.488294 19727 raft_consensus.cc:504] T 75ebf051383e42529dcb5f7dd0770a4b P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:33.488926 18448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d1d2193fd424c91b143f5256df5fcd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:33.489127 19727 leader_election.cc:288] T 75ebf051383e42529dcb5f7dd0770a4b P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:33.489981 18623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75ebf051383e42529dcb5f7dd0770a4b" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:33.490836 18446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 331c031d44854d679a27d10d9151d0ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:33.490885 18764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b72fe1f0713b4076a81a3fc1c74f7fa7" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
W20240430 20:11:33.491241 19673 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.754s user 0.000s sys 0.014s
W20240430 20:11:33.491346 19673 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.804s user 0.000s sys 0.014s
I20240430 20:11:33.495499 19489 raft_consensus.cc:2258] T bbe9a2429a434f3f95d6e7f314fcfe4c P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:33.496243 19489 ts_tablet_manager.cc:1875] T bbe9a2429a434f3f95d6e7f314fcfe4c P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:33.498355 18618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcc9698abda4491b98e1e879589913a8" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:33.498700 18618 raft_consensus.cc:2454] T fcc9698abda4491b98e1e879589913a8 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 0.
I20240430 20:11:33.508522 18755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12c6e131fa904d23865b8673b58a7e33" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:33.508723 18762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75ebf051383e42529dcb5f7dd0770a4b" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:33.563442 19519 server_base.cc:1034] running on GCE node
W20240430 20:11:33.566614 19773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240430 20:11:33.511358 18557 leader_election.cc:341] T 13c1107ea4d04eb78a74d3f658c72fb3 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Not found: Tablet not found: 13c1107ea4d04eb78a74d3f658c72fb3
I20240430 20:11:33.798240 18557 leader_election.cc:302] T 13c1107ea4d04eb78a74d3f658c72fb3 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:33.514385 19028 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 2da900b6bbfa4fa99fe8494fd2e68da8 (table default.Table_19a91a7f17ff456ebb145e8dd89c0940 [id=67283dd3995a44c6a7a7bc8722e4f955]). Expected version 1 got 0
I20240430 20:11:33.798954 19028 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 46251c08277e498da04a41d009310c62 (table default.Table_4e56923d440944978f0aacc140a0dd8e [id=b1312a6cd1a54003b1db727a2a9d125d]). Expected version 1 got 0
I20240430 20:11:33.799579 19028 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet e87fb2f0361d4aa3ada27bbdfa88e94c (table default.Table_2f122c3c1c5949108231e55f6326e934 [id=02a19d9d05264236b35244d4e9998d36]). Expected version 1 got 0
I20240430 20:11:33.799775 19028 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet d61c279d20b0471a967942525ceb4fcb (table default.Table_1e2196b2ecb846caa42407f194800886 [id=a95fed1bc3fa4865bf9a35dfc51b659c]). Expected version 1 got 0
I20240430 20:11:33.799945 19028 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet fbdeb4714b8c4b93ab72822c43329aae (table default.Table_c12ca6938d424fbfa2e1df5cdb00fdbc [id=0fb93f36bc17408ea56bc4cafcfd8580]). Expected version 1 got 0
I20240430 20:11:33.800001 19756 raft_consensus.cc:2726] T 13c1107ea4d04eb78a74d3f658c72fb3 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240430 20:11:33.553328 19048 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 78ba48dadef548708153878810cc46f8 consensus state has no leader
I20240430 20:11:33.800419 19028 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 53a899fa5a424a0d99f1b47925e8c7ad (table default.Table_ffb3192f443d437ab63f56f628e596ad [id=608890834da843fa9af0cfc0d548029e]). Expected version 2 got 0
I20240430 20:11:33.800619 19028 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 76d6e9d547ca4a848eae852944b1d117 (table default.Table_104ae015924a48fd8d27220792f07470 [id=ff86c0357c164dd0991098c416087773]). Expected version 2 got 0
I20240430 20:11:33.800848 19028 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet fd40967ab4a04001819e16c06a6c4300 (table default.Table_4130caf534004edfb413056d7f6cfd52 [id=da9ad991953a40c4b77b31317597ad1d]). Expected version 4 got 0
I20240430 20:11:33.800710 19048 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_d5e6a23db079462bb59fa312d8c07b7f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240430 20:11:33.801144 19028 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet a59ef6c545dd41a28265a4c1b8ce10e0 (table default.Table_07e40af8f1c546048975c8edf88170d7 [id=86df5d161d8641c89070e7eed9a39360]). Expected version 2 got 0
W20240430 20:11:33.801424 19048 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d5e6a23db079462bb59fa312d8c07b7f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:33.801575 19028 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet c9e7660eccc448dbb99d135f82bfd261 (table default.Table_685a26737bb046fb85ced15df7ae3423 [id=51d177e8498848b38c43e3f8d11f092c]). Expected version 1 got 0
I20240430 20:11:33.801908 19028 catalog_manager.cc:5484] T a05e32a33aa7488eb287cc6314c09c44 P 3c51c33d5616482cafb68e26688820f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c51c33d5616482cafb68e26688820f4 (127.17.177.3). New cstate: current_term: 1 leader_uuid: "3c51c33d5616482cafb68e26688820f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } health_report { overall_health: UNKNOWN } } }
I20240430 20:11:33.802384 19028 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet a05e32a33aa7488eb287cc6314c09c44 (table default.Table_2d1cfc78363a47aa8dba7ef7d80f8c9a [id=955d833ff8f14bf6957abadf9ef669e2]). Expected version 2 got 0
I20240430 20:11:33.803774 19519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240430 20:11:33.515424 18600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c091701480c4412faabad3d071d85d03. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:33.559137 18983 tablet_bootstrap.cc:492] T b76a4105a2974fe280e759349037d027 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:33.619719 19471 ts_tablet_manager.cc:1888] T a1aeee1b02214975a376cfd36957afe8 P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:33.805945 19471 log.cc:1196] T a1aeee1b02214975a376cfd36957afe8 P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/a1aeee1b02214975a376cfd36957afe8
I20240430 20:11:33.805985 18983 ts_tablet_manager.cc:1367] T b76a4105a2974fe280e759349037d027 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.417s user 0.006s sys 0.004s
I20240430 20:11:33.806546 19471 ts_tablet_manager.cc:1909] T a1aeee1b02214975a376cfd36957afe8 P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:33.648739 19474 raft_consensus.cc:482] T e3e74e76ab2b4fe492ffcc1838656fd1 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:33.806766 19474 raft_consensus.cc:504] T e3e74e76ab2b4fe492ffcc1838656fd1 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:33.811645 19029 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: Sending DeleteTablet for 3 replicas of tablet a745334c12c545c1ba825c7552c60359
I20240430 20:11:33.853216 19519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240430 20:11:33.521173 18477 raft_consensus.cc:2454] T 12c6e131fa904d23865b8673b58a7e33 P 2efb542aa55e439b99ccb98d1c5c884a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 2.
I20240430 20:11:33.869927 19519 hybrid_clock.cc:648] HybridClock initialized: now 1714507893869686 us; error 176 us; skew 500 ppm
W20240430 20:11:33.521214 18457 consensus_meta.cc:287] T c6e3d09402574f1e8fe9e4160c70a3ee P 2efb542aa55e439b99ccb98d1c5c884a: Time spent flushing consensus metadata: real 0.840s user 0.005s sys 0.000s
I20240430 20:11:33.812619 19029 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_18385e32857b4c8bb1e56a85c590153a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240430 20:11:34.067123 19519 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240430 20:11:34.067134 18457 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41680 (request call id 139) took 1553 ms. Trace:
I20240430 20:11:34.067272 18457 rpcz_store.cc:276] 0430 20:11:32.513617 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:32.528967 (+ 15350us) service_pool.cc:225] Handling call
0430 20:11:32.529602 (+ 635us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0430 20:11:32.529670 (+ 68us) ts_tablet_manager.cc:655] Creating new metadata...
0430 20:11:32.555797 (+ 26127us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:34.066927 (+1511130us) tablet_replica.cc:205] Creating consensus instance
0430 20:11:34.067095 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240430 20:11:34.067679 18480 raft_consensus.cc:1226] T 12c6e131fa904d23865b8673b58a7e33 P 2efb542aa55e439b99ccb98d1c5c884a [term 2 FOLLOWER]: Rejecting Update request from peer 3c51c33d5616482cafb68e26688820f4 for earlier term 1. Current term is 2. Ops: [1.1-1.1]
W20240430 20:11:34.067692 19029 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_18385e32857b4c8bb1e56a85c590153a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:33.522279 18450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78ba48dadef548708153878810cc46f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:33.522511 19495 ts_tablet_manager.cc:1888] T 15674aeb56b74210a4a2809608fd4457 P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:34.068361 19495 log.cc:1196] T 15674aeb56b74210a4a2809608fd4457 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/15674aeb56b74210a4a2809608fd4457
W20240430 20:11:33.536129 18985 thread.cc:636] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.553s user 0.039s sys 0.078s
W20240430 20:11:34.068696 18985 thread.cc:602] log-alloc [worker] (thread pool) Time spent starting thread: real 1.086s user 0.039s sys 0.078s
I20240430 20:11:34.068861 19495 ts_tablet_manager.cc:1909] T 15674aeb56b74210a4a2809608fd4457 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
W20240430 20:11:33.820748 19024 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 50486644ec234933b9a0b45ee39a6685 consensus state has no leader
I20240430 20:11:34.069146 19024 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:51664 (request call id 131) took 1140 ms. Trace:
I20240430 20:11:34.069254 19024 rpcz_store.cc:276] 0430 20:11:32.928726 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:33.046541 (+117815us) service_pool.cc:225] Handling call
0430 20:11:33.046979 (+ 438us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0430 20:11:33.047189 (+ 210us) catalog_manager.cc:3613] Apply alter schema
0430 20:11:33.047365 (+ 176us) catalog_manager.cc:3636] Acquired catalog manager lock
0430 20:11:33.047460 (+ 95us) catalog_manager.cc:3783] Updating metadata on disk
0430 20:11:33.238414 (+190954us) catalog_manager.cc:3814] Committing alterations to in-memory state
0430 20:11:34.069074 (+830660us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:33.048508 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:33.048598 (+ 90us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:33.048602 (+ 4us) write_op.cc:427] Acquired schema lock
0430 20:11:33.048604 (+ 2us) tablet.cc:660] Decoding operations
0430 20:11:33.048698 (+ 94us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:33.048708 (+ 10us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:33.048713 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0430 20:11:33.048765 (+ 52us) tablet.cc:699] Row locks acquired
0430 20:11:33.048767 (+ 2us) write_op.cc:260] PREPARE: finished
0430 20:11:33.048801 (+ 34us) write_op.cc:270] Start()
0430 20:11:33.048828 (+ 27us) write_op.cc:276] Timestamp: P: 1714507893048744 usec, L: 0
0430 20:11:33.048831 (+ 3us) op_driver.cc:331] REPLICATION: starting
0430 20:11:33.049038 (+ 207us) log.cc:842] Serialized 405 byte log entry
0430 20:11:33.220689 (+171651us) op_driver.cc:444] REPLICATION: finished
0430 20:11:33.227409 (+ 6720us) write_op.cc:301] APPLY: starting
0430 20:11:33.227430 (+ 21us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:33.227479 (+ 49us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:33.227516 (+ 37us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:33.227577 (+ 61us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:33.227621 (+ 44us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0430 20:11:33.227632 (+ 11us) write_op.cc:312] APPLY: finished
0430 20:11:33.227733 (+ 101us) log.cc:842] Serialized 23 byte log entry
0430 20:11:33.227799 (+ 66us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:33.227822 (+ 23us) write_op.cc:446] Released schema lock
0430 20:11:33.233235 (+ 5413us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:33.233324 (+ 89us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:33.233333 (+ 9us) write_op.cc:446] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":6650,"apply.run_cpu_time_us":610,"apply.run_wall_time_us":5986,"num_ops":1,"prepare.queue_time_us":568,"prepare.run_cpu_time_us":612,"prepare.run_wall_time_us":612,"raft.queue_time_us":61,"raft.run_cpu_time_us":3,"raft.run_wall_time_us":4,"replication_time_us":171827}]]}
I20240430 20:11:34.069751 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 61) took 2960 ms. Trace:
I20240430 20:11:33.509027 18762 raft_consensus.cc:2454] T 75ebf051383e42529dcb5f7dd0770a4b P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 0.
I20240430 20:11:34.069873 19495 rpcz_store.cc:276] 0430 20:11:31.109698 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:31.109749 (+ 51us) service_pool.cc:225] Handling call
0430 20:11:32.895974 (+1786225us) ts_tablet_manager.cc:1120] Deleting tablet 15674aeb56b74210a4a2809608fd4457
0430 20:11:32.895982 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:32.953792 (+ 57810us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:33.381923 (+428131us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:34.069709 (+687786us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1785975}
I20240430 20:11:33.806840 18983 raft_consensus.cc:348] T b76a4105a2974fe280e759349037d027 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:34.070087 19024 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_3471b34c1c0d4d829bb27a8520eff502" } modify_external_catalogs: true
I20240430 20:11:34.070253 19495 tablet_replica.cc:330] T 4e69ca12bd10494fb013614741bc166c P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:34.070261 18983 raft_consensus.cc:374] T b76a4105a2974fe280e759349037d027 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:34.070288 19024 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_3471b34c1c0d4d829bb27a8520eff502" } modify_external_catalogs: true
I20240430 20:11:34.070366 18983 raft_consensus.cc:729] T b76a4105a2974fe280e759349037d027 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:33.554411 19761 raft_consensus.cc:482] T 4e69ca12bd10494fb013614741bc166c P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:34.070730 18983 consensus_queue.cc:260] T b76a4105a2974fe280e759349037d027 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:34.070708 19761 raft_consensus.cc:504] T 4e69ca12bd10494fb013614741bc166c P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:34.071316 18983 ts_tablet_manager.cc:1398] T b76a4105a2974fe280e759349037d027 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.265s user 0.001s sys 0.000s
I20240430 20:11:33.821054 19026 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: Sending DeleteTablet for 3 replicas of tablet 06d28b6f48a34b17b0d7fb3102df2f4c
I20240430 20:11:34.071522 19761 leader_election.cc:288] T 4e69ca12bd10494fb013614741bc166c P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:33.565325 19186 raft_consensus.cc:482] T 24dcc20832804229a2664873bf6ddaa7 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:34.071672 18983 tablet_bootstrap.cc:492] T f5546f9edeea44d7a6c5cb954f5157ba P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
W20240430 20:11:33.777097 18544 kernel_stack_watchdog.cc:198] Thread 18557 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:308 for 179ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7ffa8ed33980 (unknown) at ??:0
@ 0x7ffa87b55255 __write at ??:0
@ 0x7ffa87ad028d _IO_file_write at ??:0
@ 0x7ffa87ad0bff _IO_file_xsputn at ??:0
@ 0x7ffa87ac4a57 fwrite at ??:0
@ 0x8793e7 __interceptor_fwrite.part.56 at ??:?
@ 0x7ffa88cb78a0 google::LogMessage::SendToLog() at ??:0
@ 0x7ffa88cb37cc google::LogMessage::Flush() at ??:0
@ 0x7ffa88cb3a11 google::LogMessage::~LogMessage() at ??:0
@ 0x7ffa9715a5d3 kudu::consensus::LeaderElection::VoteResponseRpcCallback() at ??:0
@ 0x7ffa9715d12a kudu::consensus::LeaderElection::Run()::$_0::operator()() at ??:0
@ 0x7ffa9715ceee std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7ffa925f38cf kudu::rpc::Proxy::AsyncRequest()::$_1::operator()() at ??:0
@ 0x7ffa925f34ae std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
I20240430 20:11:34.071820 19186 raft_consensus.cc:504] T 24dcc20832804229a2664873bf6ddaa7 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:34.072074 19026 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:51664 (request call id 128) took 1255 ms. Trace:
I20240430 20:11:33.509044 18755 raft_consensus.cc:2396] T 12c6e131fa904d23865b8673b58a7e33 P 3c51c33d5616482cafb68e26688820f4 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 5e1c6397f09747f091534faf93542710 for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
W20240430 20:11:33.811002 18557 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.177.3:44867, user_credentials={real_user=slave}} blocked reactor thread for 299648us
I20240430 20:11:33.510777 18759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e69ca12bd10494fb013614741bc166c" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:34.072399 19186 leader_election.cc:288] T 24dcc20832804229a2664873bf6ddaa7 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:33.570196 19774 raft_consensus.cc:482] T 255c13ad8fae4ea4aa8e7b7aabb7602e P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:34.072229 19026 rpcz_store.cc:276] 0430 20:11:32.816676 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:33.028182 (+211506us) service_pool.cc:225] Handling call
0430 20:11:33.028869 (+ 687us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0430 20:11:33.030038 (+ 1169us) catalog_manager.cc:2749] Modifying in-memory table state
0430 20:11:33.030125 (+ 87us) catalog_manager.cc:2758] Locking tablets
0430 20:11:33.030296 (+ 171us) catalog_manager.cc:2772] Removing table and tablets from system table
0430 20:11:33.070738 (+ 40442us) catalog_manager.cc:2790] Removing table from by-name map
0430 20:11:33.074622 (+ 3884us) catalog_manager.cc:2806] Committing in-memory state
0430 20:11:33.812204 (+737582us) catalog_manager.cc:2810] Aborting table tasks
0430 20:11:34.072029 (+259825us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:33.043214 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:33.043310 (+ 96us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:33.043316 (+ 6us) write_op.cc:427] Acquired schema lock
0430 20:11:33.043318 (+ 2us) tablet.cc:660] Decoding operations
0430 20:11:33.043428 (+ 110us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:33.043438 (+ 10us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:33.043442 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
0430 20:11:33.043551 (+ 109us) tablet.cc:699] Row locks acquired
0430 20:11:33.043554 (+ 3us) write_op.cc:260] PREPARE: finished
0430 20:11:33.043593 (+ 39us) write_op.cc:270] Start()
0430 20:11:33.043625 (+ 32us) write_op.cc:276] Timestamp: P: 1714507893043535 usec, L: 0
0430 20:11:33.043629 (+ 4us) op_driver.cc:331] REPLICATION: starting
0430 20:11:33.043817 (+ 188us) log.cc:842] Serialized 728 byte log entry
0430 20:11:33.069878 (+ 26061us) op_driver.cc:444] REPLICATION: finished
0430 20:11:33.069994 (+ 116us) write_op.cc:301] APPLY: starting
0430 20:11:33.070016 (+ 22us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:33.070075 (+ 59us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:33.070118 (+ 43us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:33.070203 (+ 85us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:33.070261 (+ 58us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0430 20:11:33.070296 (+ 35us) write_op.cc:312] APPLY: finished
0430 20:11:33.070395 (+ 99us) log.cc:842] Serialized 29 byte log entry
0430 20:11:33.070474 (+ 79us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:33.070497 (+ 23us) write_op.cc:446] Released schema lock
0430 20:11:33.070507 (+ 10us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:33.070580 (+ 73us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:33.070602 (+ 22us) write_op.cc:446] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":40,"apply.run_cpu_time_us":677,"apply.run_wall_time_us":677,"num_ops":2,"prepare.queue_time_us":2108,"prepare.run_cpu_time_us":716,"prepare.run_wall_time_us":715,"raft.queue_time_us":121,"raft.run_cpu_time_us":9,"raft.run_wall_time_us":9,"replication_time_us":26211}]]}
I20240430 20:11:34.072609 19774 raft_consensus.cc:504] T 255c13ad8fae4ea4aa8e7b7aabb7602e P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:34.073155 19026 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_2a72e434cca94803ae3e8dfcd2e47235" } new_table_name: "default.Table_656db97355f64d7c8aa29b1352dfe9c8" modify_external_catalogs: true
I20240430 20:11:34.073343 19774 leader_election.cc:288] T 255c13ad8fae4ea4aa8e7b7aabb7602e P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:33.511132 18754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26ed12f498644d15bf45e884dcad172e" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:33.571940 19778 raft_consensus.cc:482] T 802336602d8f4ac4bd7e0872d6f3dae9 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:34.073796 19778 raft_consensus.cc:504] T 802336602d8f4ac4bd7e0872d6f3dae9 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:34.074535 19778 leader_election.cc:288] T 802336602d8f4ac4bd7e0872d6f3dae9 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:33.511555 18752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26ed12f498644d15bf45e884dcad172e" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:33.575003 18472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd9b893d48684cfea7c242b93ecba01f" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:34.075261 18985 tablet_bootstrap.cc:492] T fd40967ab4a04001819e16c06a6c4300 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:34.075347 18463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53da4b3855344b6d92211af767195dc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:33.825408 19048 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_408be016c50f4d0fa19189bbaddab089"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240430 20:11:34.075434 18985 ts_tablet_manager.cc:1367] T fd40967ab4a04001819e16c06a6c4300 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 1.095s user 0.042s sys 0.078s
W20240430 20:11:34.076081 19048 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_408be016c50f4d0fa19189bbaddab089 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:34.076222 18985 raft_consensus.cc:348] T fd40967ab4a04001819e16c06a6c4300 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:34.076457 18985 raft_consensus.cc:374] T fd40967ab4a04001819e16c06a6c4300 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:34.076572 18985 raft_consensus.cc:729] T fd40967ab4a04001819e16c06a6c4300 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:34.076577 19495 raft_consensus.cc:2229] T 4e69ca12bd10494fb013614741bc166c P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:34.076953 19495 raft_consensus.cc:2258] T 4e69ca12bd10494fb013614741bc166c P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:34.076828 18985 consensus_queue.cc:260] T fd40967ab4a04001819e16c06a6c4300 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:33.807511 19474 leader_election.cc:288] T e3e74e76ab2b4fe492ffcc1838656fd1 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:33.710752 19645 raft_consensus.cc:504] T 75ebf051383e42529dcb5f7dd0770a4b P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:34.077471 18985 ts_tablet_manager.cc:1398] T fd40967ab4a04001819e16c06a6c4300 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240430 20:11:34.077553 19495 ts_tablet_manager.cc:1875] T 4e69ca12bd10494fb013614741bc166c P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:34.077700 19645 leader_election.cc:288] T 75ebf051383e42529dcb5f7dd0770a4b P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:33.511730 18761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53a899fa5a424a0d99f1b47925e8c7ad" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:34.077971 18761 raft_consensus.cc:2454] T 53a899fa5a424a0d99f1b47925e8c7ad P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 0.
I20240430 20:11:33.847047 19033 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: Sending DeleteTablet for 3 replicas of tablet 0b4d9e860d7c45fd91d7a566605c9283
I20240430 20:11:34.078619 19033 catalog_manager.cc:6554] ReplaceTablet: tablet 0b4d9e860d7c45fd91d7a566605c9283 deleted and replaced by tablet 12795670f21447a8964fe1efc6e92398
I20240430 20:11:34.079063 19033 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_0fcc91de96a946018c72ba03c71bcc05"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:34.079623 19033 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0fcc91de96a946018c72ba03c71bcc05 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:34.079957 18985 tablet_bootstrap.cc:492] T e6eac92dfa5f4456bc49571878f0f8dd P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:33.808698 18599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 885d27d4f8df48dd996d5b309b783f3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:33.579200 18481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75ebf051383e42529dcb5f7dd0770a4b" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:34.081262 18481 raft_consensus.cc:2454] T 75ebf051383e42529dcb5f7dd0770a4b P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:33.580233 18479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26ed12f498644d15bf45e884dcad172e" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:34.081555 18479 raft_consensus.cc:3037] T 26ed12f498644d15bf45e884dcad172e P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:33.822292 18583 tablet_service.cc:1514] Processing DeleteTablet for tablet a745334c12c545c1ba825c7552c60359 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:33 UTC) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:33.517782 18764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12c6e131fa904d23865b8673b58a7e33" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:34.086971 18764 raft_consensus.cc:2396] T 12c6e131fa904d23865b8673b58a7e33 P 3c51c33d5616482cafb68e26688820f4 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e1c6397f09747f091534faf93542710 for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20240430 20:11:34.087119 18983 tablet_bootstrap.cc:654] T f5546f9edeea44d7a6c5cb954f5157ba P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:33.853811 19027 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: Sending DeleteTablet for 3 replicas of tablet 50486644ec234933b9a0b45ee39a6685
I20240430 20:11:33.874850 19023 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet acd23f0a63d5448aaa7ed78422e67bad (table default.Table_972f13be4dca4cd19b9a04534dc59f51 [id=69f37db4de634a3185613e40013e0ed7]). Expected version 1 got 0
W20240430 20:11:34.015821 18999 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.177.62:36441: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240430 20:11:34.087577 19023 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 53a899fa5a424a0d99f1b47925e8c7ad (table default.Table_ffb3192f443d437ab63f56f628e596ad [id=608890834da843fa9af0cfc0d548029e]). Expected version 2 got 0
I20240430 20:11:34.087775 19027 catalog_manager.cc:6554] ReplaceTablet: tablet 50486644ec234933b9a0b45ee39a6685 deleted and replaced by tablet ed0c37a71ede4bfc9eb5ad1a4365f25e
I20240430 20:11:34.087926 19023 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 24dcc20832804229a2664873bf6ddaa7 (table default.Table_be2ebe77f859489f9cd3e912cb1edc8b [id=2d44ec94131148048021b2ee6366ba30]). Expected version 2 got 0
W20240430 20:11:34.088259 18557 consensus_peers.cc:485] T 20d166756819406ba85044f4fee9c4ce P 5e1c6397f09747f091534faf93542710 -> Peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Couldn't send request to peer 3c51c33d5616482cafb68e26688820f4. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240430 20:11:34.088361 19027 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_83d417131d35409582550b7b8ce4a141"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:34.088519 18557 leader_election.cc:341] T b72fe1f0713b4076a81a3fc1c74f7fa7 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:34.088645 18557 leader_election.cc:302] T b72fe1f0713b4076a81a3fc1c74f7fa7 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4
W20240430 20:11:34.088915 19027 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_83d417131d35409582550b7b8ce4a141 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:34.089763 19756 raft_consensus.cc:2726] T b72fe1f0713b4076a81a3fc1c74f7fa7 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:33.829355 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 50) took 2838 ms. Trace:
I20240430 20:11:34.090334 19471 rpcz_store.cc:276] 0430 20:11:30.990367 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:30.990449 (+ 82us) service_pool.cc:225] Handling call
0430 20:11:33.404380 (+2413931us) ts_tablet_manager.cc:1120] Deleting tablet a1aeee1b02214975a376cfd36957afe8
0430 20:11:33.404389 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:33.512772 (+108383us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:33.619496 (+106724us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:33.829295 (+209799us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2399696}
I20240430 20:11:34.090791 19471 tablet_replica.cc:330] T 34ab40d8d91e401db299ddee7175ba28 P 5e1c6397f09747f091534faf93542710: stopping tablet replica
W20240430 20:11:34.090883 19051 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet de1dfe6c82d34a4d8d02e466c55327ff consensus state has no leader
I20240430 20:11:34.091104 19471 raft_consensus.cc:2229] T 34ab40d8d91e401db299ddee7175ba28 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:34.091105 19051 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:51664 (request call id 124) took 1311 ms. Trace:
I20240430 20:11:34.091310 19471 raft_consensus.cc:2258] T 34ab40d8d91e401db299ddee7175ba28 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:34.092031 19471 ts_tablet_manager.cc:1875] T 34ab40d8d91e401db299ddee7175ba28 P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:34.091246 19051 rpcz_store.cc:276] 0430 20:11:32.779492 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:32.993465 (+213973us) service_pool.cc:225] Handling call
0430 20:11:33.007821 (+ 14356us) spinlock_profiling.cc:243] Waited 14.1 ms on lock 0x616000040654. stack: 00007feb16c74756 00007feb16c74510 00007feb15ca748d 0000000000955804 00000000009557cd 00007feb257b315d 00007feb257b3253 00007feb224e5d41 00007feb2a0d134d 00007feb2a25ff46 00007feb21049c26 00007feb21049955 00007feb1d97c04e 00007feb1d97af2e 00007feb1d981405 00007feb1d981d2f
0430 20:11:33.008115 (+ 294us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0430 20:11:33.008376 (+ 261us) catalog_manager.cc:3613] Apply alter schema
0430 20:11:33.008582 (+ 206us) catalog_manager.cc:3636] Acquired catalog manager lock
0430 20:11:33.008656 (+ 74us) catalog_manager.cc:3783] Updating metadata on disk
0430 20:11:33.056365 (+ 47709us) catalog_manager.cc:3814] Committing alterations to in-memory state
0430 20:11:34.091055 (+1034690us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:33.009217 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:33.009335 (+ 118us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:33.009342 (+ 7us) write_op.cc:427] Acquired schema lock
0430 20:11:33.009346 (+ 4us) tablet.cc:660] Decoding operations
0430 20:11:33.009445 (+ 99us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:33.009457 (+ 12us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:33.009465 (+ 8us) tablet.cc:683] Acquiring locks for 1 operations
0430 20:11:33.009528 (+ 63us) tablet.cc:699] Row locks acquired
0430 20:11:33.009533 (+ 5us) write_op.cc:260] PREPARE: finished
0430 20:11:33.009566 (+ 33us) write_op.cc:270] Start()
0430 20:11:33.009598 (+ 32us) write_op.cc:276] Timestamp: P: 1714507893009512 usec, L: 0
0430 20:11:33.009603 (+ 5us) op_driver.cc:331] REPLICATION: starting
0430 20:11:33.009803 (+ 200us) log.cc:842] Serialized 405 byte log entry
0430 20:11:33.045193 (+ 35390us) op_driver.cc:444] REPLICATION: finished
0430 20:11:33.047358 (+ 2165us) write_op.cc:301] APPLY: starting
0430 20:11:33.047370 (+ 12us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:33.047396 (+ 26us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:33.047444 (+ 48us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:33.047481 (+ 37us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:33.047505 (+ 24us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0430 20:11:33.047513 (+ 8us) write_op.cc:312] APPLY: finished
0430 20:11:33.047567 (+ 54us) log.cc:842] Serialized 23 byte log entry
0430 20:11:33.047604 (+ 37us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:33.047615 (+ 11us) write_op.cc:446] Released schema lock
0430 20:11:33.047623 (+ 8us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:33.047666 (+ 43us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:33.047669 (+ 3us) write_op.cc:446] Released schema lock
Metrics: {"spinlock_wait_cycles":32329088,"child_traces":[["op",{"apply.queue_time_us":2136,"apply.run_cpu_time_us":353,"apply.run_wall_time_us":352,"num_ops":1,"prepare.queue_time_us":44,"prepare.run_cpu_time_us":696,"prepare.run_wall_time_us":696,"raft.queue_time_us":375,"raft.run_cpu_time_us":7,"raft.run_wall_time_us":7,"replication_time_us":35555}]]}
I20240430 20:11:34.092509 19051 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_64d14df4e12146ecb85239e7ca03cf6e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:34.093269 19051 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_64d14df4e12146ecb85239e7ca03cf6e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:33.707816 19617 raft_consensus.cc:504] T b688e03ea71940c69bda5520b62c1fc3 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:34.095311 19617 leader_election.cc:288] T b688e03ea71940c69bda5520b62c1fc3 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Requested vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:33.877024 18600 tablet_service.cc:1467] Processing CreateTablet for tablet df92eb85261d47d88cbf16d29d9ad30e (DEFAULT_TABLE table=default.Table_1755296c0a63478699d9b8387f606a8e [id=8322f4f67ad546849d72aca9a5e985cf]), partition=RANGE (key) PARTITION UNBOUNDED
W20240430 20:11:34.095777 19049 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.17.177.2:41749 (request call id 94) took 950 ms (client timeout 1000 ms). Trace:
I20240430 20:11:34.095880 18600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df92eb85261d47d88cbf16d29d9ad30e. 1 dirs total, 0 dirs full, 0 dirs failed
W20240430 20:11:34.095937 19049 rpcz_store.cc:269] 0430 20:11:33.145234 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:33.145574 (+ 340us) service_pool.cc:225] Handling call
0430 20:11:34.095715 (+950141us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":2}
I20240430 20:11:34.096508 19049 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_0443359dbeff437fb6cffb0db33cbb30" } modify_external_catalogs: true
I20240430 20:11:34.096737 19049 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_0443359dbeff437fb6cffb0db33cbb30" } modify_external_catalogs: true
I20240430 20:11:33.519222 18744 tablet_service.cc:1467] Processing CreateTablet for tablet c091701480c4412faabad3d071d85d03 (DEFAULT_TABLE table=default.Table_3a35534b6ef2416b968740d28b156dd0 [id=f3c08d1b91be4bf58242ca535ff28897]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:33.878958 18588 tablet_service.cc:1467] Processing CreateTablet for tablet e4cffa295d8c401aa8415ce7e8f56526 (DEFAULT_TABLE table=default.Table_81547db3074240c98cdd95ed3b2dd813 [id=e0ea901452334b36a8493baed478cc1a]), partition=RANGE (key) PARTITION UNBOUNDED
W20240430 20:11:34.099484 19023 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.17.177.1:50353 (request call id 85) took 887 ms (client timeout 1000 ms). Trace:
W20240430 20:11:34.099620 19023 rpcz_store.cc:269] 0430 20:11:33.211784 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:33.229653 (+ 17869us) service_pool.cc:225] Handling call
0430 20:11:34.099402 (+869749us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":15}
I20240430 20:11:34.099833 18588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4cffa295d8c401aa8415ce7e8f56526. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:34.100178 19023 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_0034641cfa4d42a68ff4341e87fcb396" } modify_external_catalogs: true
I20240430 20:11:34.100400 19023 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_0034641cfa4d42a68ff4341e87fcb396" } modify_external_catalogs: true
W20240430 20:11:33.708685 19751 log.cc:924] Time spent T c66c142419ca4e1399ecd1c8d1e62a59 P 2efb542aa55e439b99ccb98d1c5c884a: Append to log took a long time: real 0.083s user 0.000s sys 0.000s
I20240430 20:11:33.879547 18595 tablet_service.cc:1467] Processing CreateTablet for tablet d7c1fc7e81364fb890bbc13d7fb6848b (DEFAULT_TABLE table=default.Table_fa64149f685c4784bda92ae4f578dd92 [id=3d6e1fcb9c0f403394d8b436d100b9d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:34.103119 18595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7c1fc7e81364fb890bbc13d7fb6848b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:33.519646 18760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a05e32a33aa7488eb287cc6314c09c44" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:33.881449 18587 tablet_service.cc:1467] Processing CreateTablet for tablet 9f0b341a31224bf1be525b84b9500304 (DEFAULT_TABLE table=default.Table_5b6a4bb3832e497888af8e8db7b96c12 [id=883dc5b0bfa349e9947d4c5dfe84a970]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:34.107030 18985 tablet_bootstrap.cc:654] T e6eac92dfa5f4456bc49571878f0f8dd P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:34.107476 18587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f0b341a31224bf1be525b84b9500304. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:34.110829 18449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdf53061efd74fa29a9d8208d1fd210e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:33.718562 18475 raft_consensus.cc:2454] T eed6338ad3d64859af087ee15e1ec798 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:33.887023 18589 tablet_service.cc:1467] Processing CreateTablet for tablet b1792c160af346ffb4f047c5d4435c10 (DEFAULT_TABLE table=default.Table_3127eaa465c84dcab3eb9935d15667ce [id=e4cf2d34f4b747afa5b8a512f211c01f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:33.887755 18586 tablet_service.cc:1467] Processing CreateTablet for tablet fda942d50b014f3d9589e1fb909fc2fc (DEFAULT_TABLE table=default.Table_68528229d5a94981a7fc0677d48003d1 [id=25ac2d71caf74e5ba14e3e84b84c9c60]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:33.888283 18601 tablet_service.cc:1467] Processing CreateTablet for tablet 8e91920e7e36418d8c27d1f0bfd9b981 (DEFAULT_TABLE table=default.Table_20f3b06af23d476f8ebc0471d08f8659 [id=f393bcc9669c4ea985a2d40deb161f0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:33.891542 18597 tablet_service.cc:1467] Processing CreateTablet for tablet 488d0e609b224674932f890939664df2 (DEFAULT_TABLE table=default.Table_58f35a2120874472978ca0e5f816e86c [id=ff18001c6d814afc8c1768d0c84a6bac]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:33.892076 18593 tablet_service.cc:1467] Processing CreateTablet for tablet b285e3eb16d148779c74e057458d8461 (DEFAULT_TABLE table=default.Table_575286687a4e4020a315d78080c2e28e [id=d1655c507b434f129ad7ae0b04365e51]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:33.892752 18592 tablet_service.cc:1467] Processing CreateTablet for tablet 6db55bb510b24b0cae897936211e15f0 (DEFAULT_TABLE table=default.Table_3471b34c1c0d4d829bb27a8520eff502 [id=f3498e7e44974f65bbd2e92c6c8c119f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:33.892753 18603 tablet_service.cc:1467] Processing CreateTablet for tablet 50e1c55a1a0a484984ad5cbb78eec2ed (DEFAULT_TABLE table=default.Table_211cbad17eba4c48ab02ad45543616ca [id=a693b863f7f34539aa46c3f006aca925]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:33.893255 18602 tablet_service.cc:1467] Processing CreateTablet for tablet 90a694bd5db649afa50f3c4e752939c1 (DEFAULT_TABLE table=default.Table_97a0991bf9284215b482c532e9dfc113 [id=7e3ac0868c0a4cc581427f121f305323]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:34.111685 18589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1792c160af346ffb4f047c5d4435c10. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:33.893715 18590 tablet_service.cc:1467] Processing CreateTablet for tablet 70247a4c910946c0afc24ce5ce0e85d6 (DEFAULT_TABLE table=default.Table_e20b13b4103849b9bc5e5f96f9b911e4 [id=9fd8ccb805024968b9ad8ab060d5db3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:33.894325 18591 tablet_service.cc:1467] Processing CreateTablet for tablet dad16ae02b8b4e29a724d8eec67bb4b8 (DEFAULT_TABLE table=default.Table_8a6b5152d20b4890918aedb454764ee9 [id=121444fdfd2642a38cdbd3a6c411c4fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:34.112179 18597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 488d0e609b224674932f890939664df2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:33.522240 19673 consensus_queue.cc:1035] T b76a4105a2974fe280e759349037d027 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:33.887398 18585 tablet_service.cc:1467] Processing CreateTablet for tablet 1f4e860484784aa9a2d28af2e9fb611c (DEFAULT_TABLE table=default.Table_45d50fc81325443097375ae6e65a1633 [id=9c861883de1d49c78146d65e9d4dd39f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:34.115528 18585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f4e860484784aa9a2d28af2e9fb611c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:34.118119 18593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b285e3eb16d148779c74e057458d8461. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:33.808317 18473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3e74e76ab2b4fe492ffcc1838656fd1" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
W20240430 20:11:33.613706 19766 log.cc:924] Time spent T eed6338ad3d64859af087ee15e1ec798 P 3c51c33d5616482cafb68e26688820f4: Append to log took a long time: real 0.082s user 0.000s sys 0.000s
W20240430 20:11:33.817647 18483 consensus_meta.cc:287] T e3e74e76ab2b4fe492ffcc1838656fd1 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent flushing consensus metadata: real 0.547s user 0.003s sys 0.000s
I20240430 20:11:34.120254 18483 raft_consensus.cc:2454] T e3e74e76ab2b4fe492ffcc1838656fd1 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:33.635391 19726 consensus_queue.cc:1035] T 294644d7a16b4ca2bbb6d106bcb8c7ff P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240430 20:11:33.920416 19700 consensus_meta.cc:287] T b72fe1f0713b4076a81a3fc1c74f7fa7 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent flushing consensus metadata: real 0.622s user 0.000s sys 0.004s
I20240430 20:11:33.521509 18415 leader_election.cc:302] T fcc9698abda4491b98e1e879589913a8 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
W20240430 20:11:34.008054 18404 kernel_stack_watchdog.cc:198] Thread 18415 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:308 for 443ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7f9c67949980 (unknown) at ??:0
@ 0x7f9c679439c6 __GI___pthread_rwlock_wrlock at ??:0
@ 0x7f9c618c9904 google::LogMessage::Flush() at ??:0
@ 0x7f9c618c9a11 google::LogMessage::~LogMessage() at ??:0
@ 0x7f9c6fd6f3f0 kudu::consensus::LeaderElection::CheckForDecision() at ??:0
@ 0x7f9c6fd70782 kudu::consensus::LeaderElection::VoteResponseRpcCallback() at ??:0
@ 0x7f9c6fd7312a kudu::consensus::LeaderElection::Run()::$_0::operator()() at ??:0
@ 0x7f9c6fd72eee std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7f9c6b2098cf kudu::rpc::Proxy::AsyncRequest()::$_1::operator()() at ??:0
@ 0x7f9c6b2094ae std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7f9c6b1f54e0 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7f9c6b1f5a2c kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7f9c6b19251a kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7f9c6b190f45 kudu::rpc::Connection::ReadHandler() at ??:0
W20240430 20:11:34.121237 18415 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.177.2:46027, user_credentials={real_user=slave}} blocked reactor thread for 620921us
I20240430 20:11:34.121032 19700 raft_consensus.cc:504] T b72fe1f0713b4076a81a3fc1c74f7fa7 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
W20240430 20:11:33.578758 18417 leader_election.cc:341] T 2a1e50f99c5d4b44bdbb51b58a92bc75 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:34.124997 18592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6db55bb510b24b0cae897936211e15f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:34.129930 18603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50e1c55a1a0a484984ad5cbb78eec2ed. 1 dirs total, 0 dirs full, 0 dirs failed
W20240430 20:11:33.521724 18699 leader_election.cc:341] T 01436e5b6232494babbb414342f03090 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:34.134876 18602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90a694bd5db649afa50f3c4e752939c1. 1 dirs total, 0 dirs full, 0 dirs failed
W20240430 20:11:33.708868 19771 log.cc:924] Time spent T 1555924b5f1444398f12ee82c3cb5b13 P 3c51c33d5616482cafb68e26688820f4: Append to log took a long time: real 0.179s user 0.000s sys 0.000s
I20240430 20:11:34.122185 18417 leader_election.cc:302] T 2a1e50f99c5d4b44bdbb51b58a92bc75 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a; no voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710
W20240430 20:11:34.199043 18699 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.177.1:41217, user_credentials={real_user=slave}} blocked reactor thread for 677295us
W20240430 20:11:34.199380 18417 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.177.3:44867, user_credentials={real_user=slave}} blocked reactor thread for 620583us
I20240430 20:11:33.718003 18988 tablet_bootstrap.cc:492] T b72fe1f0713b4076a81a3fc1c74f7fa7 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
W20240430 20:11:34.199864 18417 consensus_peers.cc:485] T 29974dc2c7dd45b99b6cc0712378fc30 P 2efb542aa55e439b99ccb98d1c5c884a -> Peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Couldn't send request to peer 3c51c33d5616482cafb68e26688820f4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 29974dc2c7dd45b99b6cc0712378fc30. This is attempt 1: this message will repeat every 5th retry.
I20240430 20:11:34.200044 18988 ts_tablet_manager.cc:1367] T b72fe1f0713b4076a81a3fc1c74f7fa7 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 1.070s user 0.067s sys 0.154s
W20240430 20:11:34.200146 18417 leader_election.cc:341] T b72fe1f0713b4076a81a3fc1c74f7fa7 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
W20240430 20:11:34.200731 18417 leader_election.cc:341] T b688e03ea71940c69bda5520b62c1fc3 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
W20240430 20:11:34.201066 18417 leader_election.cc:341] T acd23f0a63d5448aaa7ed78422e67bad P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:34.200987 18988 raft_consensus.cc:348] T b72fe1f0713b4076a81a3fc1c74f7fa7 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:34.201220 18988 raft_consensus.cc:374] T b72fe1f0713b4076a81a3fc1c74f7fa7 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:34.201200 18417 leader_election.cc:302] T acd23f0a63d5448aaa7ed78422e67bad P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a; no voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710
I20240430 20:11:34.201716 19519 webserver.cc:460] Webserver started at http://127.17.177.62:33023/ using document root <none> and password file <none>
I20240430 20:11:34.135093 18623 raft_consensus.cc:2379] T 75ebf051383e42529dcb5f7dd0770a4b P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c51c33d5616482cafb68e26688820f4 in current term 1: Already voted for candidate 5e1c6397f09747f091534faf93542710 in this term.
I20240430 20:11:34.144290 18558 leader_election.cc:302] T 12c6e131fa904d23865b8673b58a7e33 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:34.150847 18983 tablet_bootstrap.cc:492] T f5546f9edeea44d7a6c5cb954f5157ba P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:34.165066 18618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e69ca12bd10494fb013614741bc166c" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:34.191322 18622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24dcc20832804229a2664873bf6ddaa7" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:34.224112 19519 fs_manager.cc:359] Metadata directory not provided
I20240430 20:11:34.224330 19519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240430 20:11:34.192625 18615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b688e03ea71940c69bda5520b62c1fc3" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:34.187229 18616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "802336602d8f4ac4bd7e0872d6f3dae9" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:34.196141 18613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b72fe1f0713b4076a81a3fc1c74f7fa7" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:34.248507 18613 raft_consensus.cc:3037] T b72fe1f0713b4076a81a3fc1c74f7fa7 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:34.258688 18983 ts_tablet_manager.cc:1367] T f5546f9edeea44d7a6c5cb954f5157ba P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.152s user 0.002s sys 0.009s
I20240430 20:11:34.258713 18615 raft_consensus.cc:3037] T b688e03ea71940c69bda5520b62c1fc3 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:34.259871 18590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70247a4c910946c0afc24ce5ce0e85d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:34.264021 18618 raft_consensus.cc:2454] T 4e69ca12bd10494fb013614741bc166c P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 0.
W20240430 20:11:34.264576 18558 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.177.1:41217, user_credentials={real_user=slave}} blocked reactor thread for 79752.1us
I20240430 20:11:34.328195 19519 fs_manager.cc:709] Time spent opening directory manager: real 0.100s user 0.008s sys 0.003s
I20240430 20:11:34.266258 18616 raft_consensus.cc:2454] T 802336602d8f4ac4bd7e0872d6f3dae9 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 0.
I20240430 20:11:34.270579 19815 raft_consensus.cc:2781] T 12c6e131fa904d23865b8673b58a7e33 P 5e1c6397f09747f091534faf93542710 [term 2 FOLLOWER]: Leader election won for term 2
I20240430 20:11:34.440905 19815 raft_consensus.cc:686] T 12c6e131fa904d23865b8673b58a7e33 P 5e1c6397f09747f091534faf93542710 [term 2 LEADER]: Becoming Leader. State: Replica: 5e1c6397f09747f091534faf93542710, State: Running, Role: LEADER
I20240430 20:11:34.441285 19815 consensus_queue.cc:237] T 12c6e131fa904d23865b8673b58a7e33 P 5e1c6397f09747f091534faf93542710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:34.276348 18618 raft_consensus.cc:3037] T b76a4105a2974fe280e759349037d027 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:34.165844 19048 master_service.cc:946] ReplaceTablet: received request to replace tablet 680553d4d0bc4fcb839235c1832757e1 from {username='slave'} at 127.0.0.1:51664
I20240430 20:11:34.122377 19617 raft_consensus.cc:2781] T fcc9698abda4491b98e1e879589913a8 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:34.444227 19617 raft_consensus.cc:482] T fcc9698abda4491b98e1e879589913a8 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:34.444401 19617 raft_consensus.cc:3037] T fcc9698abda4491b98e1e879589913a8 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:34.278251 18586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fda942d50b014f3d9589e1fb909fc2fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:34.290833 18983 raft_consensus.cc:348] T f5546f9edeea44d7a6c5cb954f5157ba P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:34.445469 18983 raft_consensus.cc:374] T f5546f9edeea44d7a6c5cb954f5157ba P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:34.445605 18983 raft_consensus.cc:729] T f5546f9edeea44d7a6c5cb954f5157ba P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:34.445871 18983 consensus_queue.cc:260] T f5546f9edeea44d7a6c5cb954f5157ba P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:34.201314 18988 raft_consensus.cc:729] T b72fe1f0713b4076a81a3fc1c74f7fa7 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:34.446848 18988 consensus_queue.cc:260] T b72fe1f0713b4076a81a3fc1c74f7fa7 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:34.447104 18601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e91920e7e36418d8c27d1f0bfd9b981. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:34.447422 18988 ts_tablet_manager.cc:1398] T b72fe1f0713b4076a81a3fc1c74f7fa7 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.247s user 0.001s sys 0.001s
I20240430 20:11:34.447762 18988 tablet_bootstrap.cc:492] T fcc9698abda4491b98e1e879589913a8 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:34.167189 19027 master_service.cc:946] ReplaceTablet: received request to replace tablet 44fa6917a85c46f3aaa363df46ea586b from {username='slave'} at 127.0.0.1:51664
I20240430 20:11:34.448597 19027 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:51664 (request call id 151) took 1095 ms. Trace:
I20240430 20:11:34.448711 19027 rpcz_store.cc:276] 0430 20:11:33.352909 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:34.167116 (+814207us) service_pool.cc:225] Handling call
0430 20:11:34.448533 (+281417us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240430 20:11:34.123713 18479 raft_consensus.cc:2454] T 26ed12f498644d15bf45e884dcad172e P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
I20240430 20:11:34.321455 19836 tablet.cc:1722] T aec9c09cc0454acab6006bb3e10982b7 P 5e1c6397f09747f091534faf93542710: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240430 20:11:34.173624 19049 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: Sending DeleteTablet for 3 replicas of tablet c66c142419ca4e1399ecd1c8d1e62a59
I20240430 20:11:34.143393 19774 raft_consensus.cc:482] T f77086a5675f44e99e8a867b694df4b8 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:34.449908 19049 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:51664 (request call id 148) took 1189 ms. Trace:
I20240430 20:11:34.450052 19774 raft_consensus.cc:504] T f77086a5675f44e99e8a867b694df4b8 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:34.450047 19049 rpcz_store.cc:276] 0430 20:11:33.260545 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:34.096255 (+835710us) service_pool.cc:225] Handling call
0430 20:11:34.096862 (+ 607us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0430 20:11:34.097033 (+ 171us) catalog_manager.cc:2749] Modifying in-memory table state
0430 20:11:34.097116 (+ 83us) catalog_manager.cc:2758] Locking tablets
0430 20:11:34.097290 (+ 174us) catalog_manager.cc:2772] Removing table and tablets from system table
0430 20:11:34.173392 (+ 76102us) catalog_manager.cc:2790] Removing table from by-name map
0430 20:11:34.173490 (+ 98us) catalog_manager.cc:2806] Committing in-memory state
0430 20:11:34.173539 (+ 49us) catalog_manager.cc:2810] Aborting table tasks
0430 20:11:34.449859 (+276320us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:34.108571 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:34.108757 (+ 186us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:34.108767 (+ 10us) write_op.cc:427] Acquired schema lock
0430 20:11:34.108773 (+ 6us) tablet.cc:660] Decoding operations
0430 20:11:34.108949 (+ 176us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:34.108967 (+ 18us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:34.108975 (+ 8us) tablet.cc:683] Acquiring locks for 2 operations
0430 20:11:34.109160 (+ 185us) tablet.cc:699] Row locks acquired
0430 20:11:34.109165 (+ 5us) write_op.cc:260] PREPARE: finished
0430 20:11:34.109214 (+ 49us) write_op.cc:270] Start()
0430 20:11:34.109265 (+ 51us) write_op.cc:276] Timestamp: P: 1714507894109151 usec, L: 0
0430 20:11:34.109272 (+ 7us) op_driver.cc:331] REPLICATION: starting
0430 20:11:34.109555 (+ 283us) log.cc:842] Serialized 793 byte log entry
0430 20:11:34.172500 (+ 62945us) op_driver.cc:444] REPLICATION: finished
0430 20:11:34.172788 (+ 288us) write_op.cc:301] APPLY: starting
0430 20:11:34.172804 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:34.172840 (+ 36us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:34.172874 (+ 34us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:34.172945 (+ 71us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:34.172975 (+ 30us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0430 20:11:34.172983 (+ 8us) write_op.cc:312] APPLY: finished
0430 20:11:34.173062 (+ 79us) log.cc:842] Serialized 29 byte log entry
0430 20:11:34.173282 (+ 220us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:34.173301 (+ 19us) write_op.cc:446] Released schema lock
0430 20:11:34.173308 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:34.173742 (+ 434us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:34.173748 (+ 6us) write_op.cc:446] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":230,"apply.run_cpu_time_us":492,"apply.run_wall_time_us":1015,"num_ops":2,"prepare.queue_time_us":10297,"prepare.run_cpu_time_us":1095,"prepare.run_wall_time_us":1576,"replication_time_us":63200}]]}
I20240430 20:11:34.450786 19774 leader_election.cc:288] T f77086a5675f44e99e8a867b694df4b8 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
W20240430 20:11:33.795744 18685 kernel_stack_watchdog.cc:198] Thread 18699 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:308 for 165ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7f30eb118980 (unknown) at ??:0
@ 0x7f30eb1129c6 __GI___pthread_rwlock_wrlock at ??:0
@ 0x7f30e5098904 google::LogMessage::Flush() at ??:0
@ 0x7f30e5098a11 google::LogMessage::~LogMessage() at ??:0
@ 0x7f30f353f5d3 kudu::consensus::LeaderElection::VoteResponseRpcCallback() at ??:0
@ 0x7f30f354212a kudu::consensus::LeaderElection::Run()::$_0::operator()() at ??:0
@ 0x7f30f3541eee std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7f30ee9d88cf kudu::rpc::Proxy::AsyncRequest()::$_1::operator()() at ??:0
@ 0x7f30ee9d84ae std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7f30ee9c44e0 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7f30ee9c4a2c kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7f30ee96151a kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7f30ee95ff45 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7f30ee9722c5 ev::base<>::method_thunk<>() at ??:0
I20240430 20:11:34.339632 18613 raft_consensus.cc:2454] T b72fe1f0713b4076a81a3fc1c74f7fa7 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 1.
I20240430 20:11:34.450990 19049 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_3be3b20c86ea424fbc173dd32e01f90c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240430 20:11:34.144918 18483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75ebf051383e42529dcb5f7dd0770a4b" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:34.451442 18483 raft_consensus.cc:3037] T 75ebf051383e42529dcb5f7dd0770a4b P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
W20240430 20:11:34.451594 19049 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3be3b20c86ea424fbc173dd32e01f90c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:34.451619 18558 leader_election.cc:302] T 26ed12f498644d15bf45e884dcad172e P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
W20240430 20:11:34.195515 18415 leader_election.cc:341] T 255c13ad8fae4ea4aa8e7b7aabb7602e P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Not found: Tablet not found: 255c13ad8fae4ea4aa8e7b7aabb7602e
W20240430 20:11:34.452584 18415 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.177.2:46027, user_credentials={real_user=slave}} blocked reactor thread for 257464us
I20240430 20:11:34.452822 19756 raft_consensus.cc:2781] T 26ed12f498644d15bf45e884dcad172e P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:34.452986 19756 raft_consensus.cc:686] T 26ed12f498644d15bf45e884dcad172e P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Becoming Leader. State: Replica: 5e1c6397f09747f091534faf93542710, State: Running, Role: LEADER
I20240430 20:11:34.174656 19023 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: Sending DeleteTablet for 3 replicas of tablet 84568f90b9c54cf292099808d4cbf5dc
W20240430 20:11:34.453145 18415 leader_election.cc:341] T 24dcc20832804229a2664873bf6ddaa7 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:34.453446 18415 leader_election.cc:302] T 4e69ca12bd10494fb013614741bc166c P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:34.453348 19756 consensus_queue.cc:237] T 26ed12f498644d15bf45e884dcad172e P 5e1c6397f09747f091534faf93542710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:34.453574 19023 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:51664 (request call id 149) took 1189 ms. Trace:
W20240430 20:11:34.453616 18415 raft_consensus.cc:2670] T 4e69ca12bd10494fb013614741bc166c P 2efb542aa55e439b99ccb98d1c5c884a: unable to run election callback: Service unavailable: Thread pool token was shut down
I20240430 20:11:33.817909 18734 tablet_service.cc:1514] Processing DeleteTablet for tablet a745334c12c545c1ba825c7552c60359 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:33 UTC) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:34.453696 19023 rpcz_store.cc:276] 0430 20:11:33.264321 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:34.099900 (+835579us) service_pool.cc:225] Handling call
0430 20:11:34.100537 (+ 637us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0430 20:11:34.100725 (+ 188us) catalog_manager.cc:2749] Modifying in-memory table state
0430 20:11:34.100800 (+ 75us) catalog_manager.cc:2758] Locking tablets
0430 20:11:34.101020 (+ 220us) catalog_manager.cc:2772] Removing table and tablets from system table
0430 20:11:34.174390 (+ 73370us) catalog_manager.cc:2790] Removing table from by-name map
0430 20:11:34.174500 (+ 110us) catalog_manager.cc:2806] Committing in-memory state
0430 20:11:34.174557 (+ 57us) catalog_manager.cc:2810] Aborting table tasks
0430 20:11:34.453534 (+278977us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:34.110181 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:34.110342 (+ 161us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:34.110350 (+ 8us) write_op.cc:427] Acquired schema lock
0430 20:11:34.110354 (+ 4us) tablet.cc:660] Decoding operations
0430 20:11:34.110509 (+ 155us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:34.110524 (+ 15us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:34.110532 (+ 8us) tablet.cc:683] Acquiring locks for 2 operations
0430 20:11:34.111510 (+ 978us) tablet.cc:699] Row locks acquired
0430 20:11:34.111515 (+ 5us) write_op.cc:260] PREPARE: finished
0430 20:11:34.111556 (+ 41us) write_op.cc:270] Start()
0430 20:11:34.111600 (+ 44us) write_op.cc:276] Timestamp: P: 1714507894111494 usec, L: 0
0430 20:11:34.111604 (+ 4us) op_driver.cc:331] REPLICATION: starting
0430 20:11:34.111792 (+ 188us) log.cc:842] Serialized 728 byte log entry
0430 20:11:34.172580 (+ 60788us) op_driver.cc:444] REPLICATION: finished
0430 20:11:34.173842 (+ 1262us) write_op.cc:301] APPLY: starting
0430 20:11:34.173855 (+ 13us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:34.173888 (+ 33us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:34.173912 (+ 24us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:34.173973 (+ 61us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:34.174001 (+ 28us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0430 20:11:34.174008 (+ 7us) write_op.cc:312] APPLY: finished
0430 20:11:34.174064 (+ 56us) log.cc:842] Serialized 29 byte log entry
0430 20:11:34.174275 (+ 211us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:34.174293 (+ 18us) write_op.cc:446] Released schema lock
0430 20:11:34.174302 (+ 9us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:34.181661 (+ 7359us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:34.181667 (+ 6us) write_op.cc:446] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1223,"apply.run_cpu_time_us":449,"apply.run_wall_time_us":7878,"num_ops":2,"prepare.queue_time_us":8306,"prepare.run_cpu_time_us":883,"prepare.run_wall_time_us":2052,"replication_time_us":60957}]]}
I20240430 20:11:34.353262 19844 tablet.cc:1722] T 294644d7a16b4ca2bbb6d106bcb8c7ff P 5e1c6397f09747f091534faf93542710: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240430 20:11:34.454501 19023 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_1e2196b2ecb846caa42407f194800886" } new_table_name: "default.Table_1177828cad1f4a24bd397619dad96170" modify_external_catalogs: true
W20240430 20:11:34.454632 18734 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:59830: Illegal state: replica 3c51c33d5616482cafb68e26688820f4 is not leader of this config: current role FOLLOWER
I20240430 20:11:34.204149 18699 leader_election.cc:302] T dd9b893d48684cfea7c242b93ecba01f P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710
I20240430 20:11:34.455322 18734 tablet_service.cc:1467] Processing CreateTablet for tablet 3e5225f1f5f544ecbf8dfa470af3daca (DEFAULT_TABLE table=default.Table_d5e6a23db079462bb59fa312d8c07b7f [id=abf3765074eb4d7d9d2bcc4ab721e6bd]), partition=RANGE (key) PARTITION UNBOUNDED
W20240430 20:11:34.455509 18699 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.177.1:41217, user_credentials={real_user=slave}} blocked reactor thread for 251386us
I20240430 20:11:34.455762 18699 leader_election.cc:302] T 75ebf051383e42529dcb5f7dd0770a4b P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters: 5e1c6397f09747f091534faf93542710
I20240430 20:11:34.161870 19825 tablet.cc:1722] T aec9c09cc0454acab6006bb3e10982b7 P 2efb542aa55e439b99ccb98d1c5c884a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240430 20:11:34.456267 18699 leader_election.cc:302] T e3e74e76ab2b4fe492ffcc1838656fd1 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters: 5e1c6397f09747f091534faf93542710
I20240430 20:11:34.176506 19033 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_2dbff4cecdff4d2dbbb0004c88377ce7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240430 20:11:34.162818 19700 leader_election.cc:288] T b72fe1f0713b4076a81a3fc1c74f7fa7 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Requested vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
W20240430 20:11:34.456905 19033 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2dbff4cecdff4d2dbbb0004c88377ce7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:34.456919 19700 raft_consensus.cc:482] T 93d158de64994f3c9fa45a1250a4f533 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5e1c6397f09747f091534faf93542710)
I20240430 20:11:34.457084 19700 raft_consensus.cc:504] T 93d158de64994f3c9fa45a1250a4f533 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:34.457734 19700 leader_election.cc:288] T 93d158de64994f3c9fa45a1250a4f533 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:33.850819 19727 raft_consensus.cc:482] T 24dcc20832804229a2664873bf6ddaa7 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:34.458003 19727 raft_consensus.cc:504] T 24dcc20832804229a2664873bf6ddaa7 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:34.458669 19727 leader_election.cc:288] T 24dcc20832804229a2664873bf6ddaa7 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:34.458810 19727 raft_consensus.cc:3037] T dd9b893d48684cfea7c242b93ecba01f P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:34.164006 19186 consensus_queue.cc:1035] T dd9b893d48684cfea7c242b93ecba01f P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:33.823431 19581 consensus_queue.cc:1035] T a05e32a33aa7488eb287cc6314c09c44 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:34.460028 19581 raft_consensus.cc:2781] T 75ebf051383e42529dcb5f7dd0770a4b P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:34.460127 19581 raft_consensus.cc:482] T 75ebf051383e42529dcb5f7dd0770a4b P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:34.460223 19581 raft_consensus.cc:3037] T 75ebf051383e42529dcb5f7dd0770a4b P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:34.460183 18620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24dcc20832804229a2664873bf6ddaa7" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
W20240430 20:11:34.460917 18696 leader_election.cc:341] T 24dcc20832804229a2664873bf6ddaa7 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:34.461553 19726 raft_consensus.cc:2781] T e3e74e76ab2b4fe492ffcc1838656fd1 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:34.461691 19726 raft_consensus.cc:686] T e3e74e76ab2b4fe492ffcc1838656fd1 P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Becoming Leader. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Running, Role: LEADER
I20240430 20:11:34.461963 19726 consensus_queue.cc:237] T e3e74e76ab2b4fe492ffcc1838656fd1 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:34.446555 18983 ts_tablet_manager.cc:1398] T f5546f9edeea44d7a6c5cb954f5157ba P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.188s user 0.001s sys 0.001s
I20240430 20:11:34.202188 19761 raft_consensus.cc:2726] T 2a1e50f99c5d4b44bdbb51b58a92bc75 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:34.202582 18450 tablet_service.cc:1467] Processing CreateTablet for tablet 885d27d4f8df48dd996d5b309b783f3b (DEFAULT_TABLE table=default.Table_47779624a66e46468c8e9e35a8e9487f [id=8ce80df659a146c9b7d5fdb3059902cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:34.462916 18983 tablet_bootstrap.cc:492] T acd23f0a63d5448aaa7ed78422e67bad P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:34.463143 18450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 885d27d4f8df48dd996d5b309b783f3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:33.876132 18724 tablet_service.cc:1467] Processing CreateTablet for tablet df92eb85261d47d88cbf16d29d9ad30e (DEFAULT_TABLE table=default.Table_1755296c0a63478699d9b8387f606a8e [id=8322f4f67ad546849d72aca9a5e985cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:33.878921 18726 tablet_service.cc:1467] Processing CreateTablet for tablet d7c1fc7e81364fb890bbc13d7fb6848b (DEFAULT_TABLE table=default.Table_fa64149f685c4784bda92ae4f578dd92 [id=3d6e1fcb9c0f403394d8b436d100b9d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:33.880513 18732 tablet_service.cc:1467] Processing CreateTablet for tablet 9f0b341a31224bf1be525b84b9500304 (DEFAULT_TABLE table=default.Table_5b6a4bb3832e497888af8e8db7b96c12 [id=883dc5b0bfa349e9947d4c5dfe84a970]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:34.178617 19051 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_027891085c5747d58ddb32ad51a453ba"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240430 20:11:33.882078 18729 tablet_service.cc:1467] Processing CreateTablet for tablet b1792c160af346ffb4f047c5d4435c10 (DEFAULT_TABLE table=default.Table_3127eaa465c84dcab3eb9935d15667ce [id=e4cf2d34f4b747afa5b8a512f211c01f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:33.882753 18739 tablet_service.cc:1467] Processing CreateTablet for tablet 1f4e860484784aa9a2d28af2e9fb611c (DEFAULT_TABLE table=default.Table_45d50fc81325443097375ae6e65a1633 [id=9c861883de1d49c78146d65e9d4dd39f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:33.883764 18735 tablet_service.cc:1467] Processing CreateTablet for tablet fda942d50b014f3d9589e1fb909fc2fc (DEFAULT_TABLE table=default.Table_68528229d5a94981a7fc0677d48003d1 [id=25ac2d71caf74e5ba14e3e84b84c9c60]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:33.884119 18741 tablet_service.cc:1467] Processing CreateTablet for tablet 8e91920e7e36418d8c27d1f0bfd9b981 (DEFAULT_TABLE table=default.Table_20f3b06af23d476f8ebc0471d08f8659 [id=f393bcc9669c4ea985a2d40deb161f0c]), partition=RANGE (key) PARTITION UNBOUNDED
W20240430 20:11:34.464362 19051 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_027891085c5747d58ddb32ad51a453ba in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:33.885147 18736 tablet_service.cc:1467] Processing CreateTablet for tablet 488d0e609b224674932f890939664df2 (DEFAULT_TABLE table=default.Table_58f35a2120874472978ca0e5f816e86c [id=ff18001c6d814afc8c1768d0c84a6bac]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:34.464601 18983 tablet_bootstrap.cc:654] T acd23f0a63d5448aaa7ed78422e67bad P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:33.885468 18743 tablet_service.cc:1467] Processing CreateTablet for tablet b285e3eb16d148779c74e057458d8461 (DEFAULT_TABLE table=default.Table_575286687a4e4020a315d78080c2e28e [id=d1655c507b434f129ad7ae0b04365e51]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:33.886175 18742 tablet_service.cc:1467] Processing CreateTablet for tablet 6db55bb510b24b0cae897936211e15f0 (DEFAULT_TABLE table=default.Table_3471b34c1c0d4d829bb27a8520eff502 [id=f3498e7e44974f65bbd2e92c6c8c119f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:33.888952 18740 tablet_service.cc:1467] Processing CreateTablet for tablet 50e1c55a1a0a484984ad5cbb78eec2ed (DEFAULT_TABLE table=default.Table_211cbad17eba4c48ab02ad45543616ca [id=a693b863f7f34539aa46c3f006aca925]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:33.889647 18731 tablet_service.cc:1467] Processing CreateTablet for tablet 90a694bd5db649afa50f3c4e752939c1 (DEFAULT_TABLE table=default.Table_97a0991bf9284215b482c532e9dfc113 [id=7e3ac0868c0a4cc581427f121f305323]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:33.890334 18730 tablet_service.cc:1467] Processing CreateTablet for tablet 70247a4c910946c0afc24ce5ce0e85d6 (DEFAULT_TABLE table=default.Table_e20b13b4103849b9bc5e5f96f9b911e4 [id=9fd8ccb805024968b9ad8ab060d5db3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:33.891005 18733 tablet_service.cc:1467] Processing CreateTablet for tablet dad16ae02b8b4e29a724d8eec67bb4b8 (DEFAULT_TABLE table=default.Table_8a6b5152d20b4890918aedb454764ee9 [id=121444fdfd2642a38cdbd3a6c411c4fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:33.946346 19803 raft_consensus.cc:482] T c9e7660eccc448dbb99d135f82bfd261 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:34.203218 19778 raft_consensus.cc:2726] T acd23f0a63d5448aaa7ed78422e67bad P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240430 20:11:34.181531 19028 catalog_manager.cc:5676] Failed to send AlterTable RPC for tablet 66dd767bd3f8494f94061fb89e5e5680 (table default.Table_9c1e6182953140bb9cd594f08458c7d9 [id=5227206f44d04fa89b4842b899f33afb], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 66dd767bd3f8494f94061fb89e5e5680 consensus state has no leader
I20240430 20:11:34.186515 19029 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_67d51131549440cc99629dda924693f7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:34.779848 19029 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_67d51131549440cc99629dda924693f7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240430 20:11:34.779990 19028 catalog_manager.cc:5676] Failed to send AlterTable RPC for tablet fbdeb4714b8c4b93ab72822c43329aae (table default.Table_c12ca6938d424fbfa2e1df5cdb00fdbc [id=0fb93f36bc17408ea56bc4cafcfd8580], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet fbdeb4714b8c4b93ab72822c43329aae consensus state has no leader
I20240430 20:11:34.503011 19866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240430 20:11:34.191076 19024 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: Sending DeleteTablet for 3 replicas of tablet 6db55bb510b24b0cae897936211e15f0
I20240430 20:11:34.453027 19474 raft_consensus.cc:482] T f5546f9edeea44d7a6c5cb954f5157ba P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:34.781438 19024 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:51664 (request call id 143) took 1529 ms. Trace:
W20240430 20:11:34.196882 19026 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c6e3d09402574f1e8fe9e4160c70a3ee consensus state has no leader
W20240430 20:11:34.781637 19028 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.17.177.3:42157 (request call id 81) took 1739 ms (client timeout 1000 ms). Trace:
I20240430 20:11:34.781479 19474 raft_consensus.cc:504] T f5546f9edeea44d7a6c5cb954f5157ba P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:34.781807 19026 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:51664 (request call id 144) took 1529 ms. Trace:
I20240430 20:11:34.782320 19474 leader_election.cc:288] T f5546f9edeea44d7a6c5cb954f5157ba P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:34.781929 19026 rpcz_store.cc:276] 0430 20:11:33.252582 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:34.072969 (+820387us) service_pool.cc:225] Handling call
0430 20:11:34.073353 (+ 384us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0430 20:11:34.073636 (+ 283us) catalog_manager.cc:3613] Apply alter schema
0430 20:11:34.073854 (+ 218us) catalog_manager.cc:3636] Acquired catalog manager lock
0430 20:11:34.073947 (+ 93us) catalog_manager.cc:3783] Updating metadata on disk
0430 20:11:34.196561 (+122614us) catalog_manager.cc:3814] Committing alterations to in-memory state
0430 20:11:34.781756 (+585195us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:34.083443 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:34.083537 (+ 94us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:34.083542 (+ 5us) write_op.cc:427] Acquired schema lock
0430 20:11:34.083545 (+ 3us) tablet.cc:660] Decoding operations
0430 20:11:34.083644 (+ 99us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:34.083653 (+ 9us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:34.083657 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0430 20:11:34.083701 (+ 44us) tablet.cc:699] Row locks acquired
0430 20:11:34.083702 (+ 1us) write_op.cc:260] PREPARE: finished
0430 20:11:34.083729 (+ 27us) write_op.cc:270] Start()
0430 20:11:34.083754 (+ 25us) write_op.cc:276] Timestamp: P: 1714507894083674 usec, L: 0
0430 20:11:34.083758 (+ 4us) op_driver.cc:331] REPLICATION: starting
0430 20:11:34.083900 (+ 142us) log.cc:842] Serialized 405 byte log entry
0430 20:11:34.160935 (+ 77035us) op_driver.cc:444] REPLICATION: finished
0430 20:11:34.161830 (+ 895us) write_op.cc:301] APPLY: starting
0430 20:11:34.161840 (+ 10us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:34.161872 (+ 32us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:34.161904 (+ 32us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:34.161942 (+ 38us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:34.161978 (+ 36us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0430 20:11:34.161985 (+ 7us) write_op.cc:312] APPLY: finished
0430 20:11:34.162037 (+ 52us) log.cc:842] Serialized 23 byte log entry
0430 20:11:34.162089 (+ 52us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:34.162107 (+ 18us) write_op.cc:446] Released schema lock
0430 20:11:34.162114 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:34.162175 (+ 61us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:34.162181 (+ 6us) write_op.cc:446] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":864,"apply.run_cpu_time_us":394,"apply.run_wall_time_us":393,"num_ops":1,"prepare.queue_time_us":8967,"prepare.run_cpu_time_us":522,"prepare.run_wall_time_us":521,"replication_time_us":77158,"thread_start_us":3259,"threads_started":1}]]}
I20240430 20:11:34.205014 18457 tablet_service.cc:1467] Processing CreateTablet for tablet c091701480c4412faabad3d071d85d03 (DEFAULT_TABLE table=default.Table_3a35534b6ef2416b968740d28b156dd0 [id=f3c08d1b91be4bf58242ca535ff28897]), partition=RANGE (key) PARTITION UNBOUNDED
W20240430 20:11:34.781752 19028 rpcz_store.cc:269] 0430 20:11:33.042704 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:33.046541 (+ 3837us) service_pool.cc:225] Handling call
0430 20:11:34.781598 (+1735057us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:33.830526 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:33.830762 (+ 236us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:33.830769 (+ 7us) write_op.cc:427] Acquired schema lock
0430 20:11:33.830772 (+ 3us) tablet.cc:660] Decoding operations
0430 20:11:33.830946 (+ 174us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:33.830958 (+ 12us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:33.830962 (+ 4us) tablet.cc:683] Acquiring locks for 4 operations
0430 20:11:33.831051 (+ 89us) tablet.cc:699] Row locks acquired
0430 20:11:33.831052 (+ 1us) write_op.cc:260] PREPARE: finished
0430 20:11:33.831086 (+ 34us) write_op.cc:270] Start()
0430 20:11:33.831116 (+ 30us) write_op.cc:276] Timestamp: P: 1714507893831053 usec, L: 0
0430 20:11:33.831118 (+ 2us) op_driver.cc:331] REPLICATION: starting
0430 20:11:33.831291 (+ 173us) log.cc:842] Serialized 1662 byte log entry
0430 20:11:33.835505 (+ 4214us) op_driver.cc:444] REPLICATION: finished
0430 20:11:33.835700 (+ 195us) write_op.cc:301] APPLY: starting
0430 20:11:33.835719 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:33.835795 (+ 76us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:33.835826 (+ 31us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:33.835941 (+ 115us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:33.835990 (+ 49us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=4
0430 20:11:33.835998 (+ 8us) write_op.cc:312] APPLY: finished
0430 20:11:33.836120 (+ 122us) log.cc:842] Serialized 41 byte log entry
0430 20:11:33.836189 (+ 69us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:33.836226 (+ 37us) write_op.cc:446] Released schema lock
0430 20:11:33.836234 (+ 8us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:33.836293 (+ 59us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:33.836296 (+ 3us) write_op.cc:446] Released schema lock
Metrics: {"reported_tablets":111,"child_traces":[["op",{"apply.queue_time_us":122,"apply.run_cpu_time_us":673,"apply.run_wall_time_us":671,"num_ops":4,"prepare.queue_time_us":54,"prepare.run_cpu_time_us":856,"prepare.run_wall_time_us":885,"raft.queue_time_us":477,"raft.run_cpu_time_us":374,"raft.run_wall_time_us":374,"replication_time_us":4353}]]}
I20240430 20:11:34.244735 19004 catalog_manager.cc:4830] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): tablet 15674aeb56b74210a4a2809608fd4457 (table default.Table_7974c4f995c047469ccc69cc0d0521f3 [id=efc0011648254789a4f3b41a1fffdf4f]) successfully deleted
W20240430 20:11:34.322002 18999 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.177.62:36441, user_credentials={real_user=slave}} blocked reactor thread for 275220us
I20240430 20:11:34.782883 18457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c091701480c4412faabad3d071d85d03. 1 dirs total, 0 dirs full, 0 dirs failed
W20240430 20:11:34.782899 19004 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.17.177.1:41217, user_credentials={real_user=slave}} blocked reactor thread for 538121us
I20240430 20:11:34.784787 19519 fs_manager.cc:725] Time spent opening block manager: real 0.342s user 0.005s sys 0.008s
I20240430 20:11:34.785017 19519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/master-0/data,/tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/master-0/wal
uuid: "88b3f98793354c7aaa5c106e8ec31ed8"
format_stamp: "Formatted at 2024-04-30 20:11:17 on dist-test-slave-54l7"
I20240430 20:11:34.465719 19803 raft_consensus.cc:504] T c9e7660eccc448dbb99d135f82bfd261 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:34.786273 19519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240430 20:11:34.787245 19803 leader_election.cc:288] T c9e7660eccc448dbb99d135f82bfd261 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:34.788329 18472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5546f9edeea44d7a6c5cb954f5157ba" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:34.789467 18601 tablet_service.cc:1514] Processing DeleteTablet for tablet 50486644ec234933b9a0b45ee39a6685 with delete_type TABLET_DATA_DELETED (replaced by tablet ed0c37a71ede4bfc9eb5ad1a4365f25e) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:34.789744 18602 tablet_service.cc:1514] Processing DeleteTablet for tablet c66c142419ca4e1399ecd1c8d1e62a59 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:34 UTC) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:34.436466 18615 raft_consensus.cc:2454] T b688e03ea71940c69bda5520b62c1fc3 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 1.
I20240430 20:11:34.205631 18446 tablet_service.cc:1514] Processing DeleteTablet for tablet a745334c12c545c1ba825c7552c60359 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:33 UTC) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:34.790246 18601 tablet_service.cc:1467] Processing CreateTablet for tablet 3e5225f1f5f544ecbf8dfa470af3daca (DEFAULT_TABLE table=default.Table_d5e6a23db079462bb59fa312d8c07b7f [id=abf3765074eb4d7d9d2bcc4ab721e6bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:34.790402 18446 tablet_service.cc:1514] Processing DeleteTablet for tablet 50486644ec234933b9a0b45ee39a6685 with delete_type TABLET_DATA_DELETED (replaced by tablet ed0c37a71ede4bfc9eb5ad1a4365f25e) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:34.790683 18595 tablet_service.cc:1467] Processing CreateTablet for tablet ed0c37a71ede4bfc9eb5ad1a4365f25e (DEFAULT_TABLE table=default.Table_a2e85efc6b454524b0b122102ec07002 [id=86d795a82e674d859104741ebd04f4e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:34.790809 18446 tablet_service.cc:1467] Processing CreateTablet for tablet 3e5225f1f5f544ecbf8dfa470af3daca (DEFAULT_TABLE table=default.Table_d5e6a23db079462bb59fa312d8c07b7f [id=abf3765074eb4d7d9d2bcc4ab721e6bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:34.790923 18590 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b4d9e860d7c45fd91d7a566605c9283 with delete_type TABLET_DATA_DELETED (replaced by tablet 12795670f21447a8964fe1efc6e92398) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:34.206310 18448 tablet_service.cc:1467] Processing CreateTablet for tablet df92eb85261d47d88cbf16d29d9ad30e (DEFAULT_TABLE table=default.Table_1755296c0a63478699d9b8387f606a8e [id=8322f4f67ad546849d72aca9a5e985cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:34.791105 18602 tablet_service.cc:1467] Processing CreateTablet for tablet 12795670f21447a8964fe1efc6e92398 (DEFAULT_TABLE table=default.Table_421069f84cb844368066c93102a8faf8 [id=d5cea90cbc2c48df9d8b80da1eccc949]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:34.791226 18446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e5225f1f5f544ecbf8dfa470af3daca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:34.207209 18459 tablet_service.cc:1467] Processing CreateTablet for tablet e4cffa295d8c401aa8415ce7e8f56526 (DEFAULT_TABLE table=default.Table_81547db3074240c98cdd95ed3b2dd813 [id=e0ea901452334b36a8493baed478cc1a]), partition=RANGE (key) PARTITION UNBOUNDED
W20240430 20:11:34.471050 19000 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 488d0e609b224674932f890939664df2 (table default.Table_58f35a2120874472978ca0e5f816e86c [id=ff18001c6d814afc8c1768d0c84a6bac], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 488d0e609b224674932f890939664df2 consensus state has no leader
I20240430 20:11:34.791596 18459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4cffa295d8c401aa8415ce7e8f56526. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:34.140645 18725 tablet_service.cc:1514] Processing DeleteTablet for tablet 06d28b6f48a34b17b0d7fb3102df2f4c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:33 UTC) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:34.792822 18725 tablet_service.cc:1467] Processing CreateTablet for tablet ed0c37a71ede4bfc9eb5ad1a4365f25e (DEFAULT_TABLE table=default.Table_a2e85efc6b454524b0b122102ec07002 [id=86d795a82e674d859104741ebd04f4e8]), partition=RANGE (key) PARTITION UNBOUNDED
W20240430 20:11:34.510030 18557 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.177.3:44867, user_credentials={real_user=slave}} blocked reactor thread for 52109us
I20240430 20:11:34.480072 19027 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 294644d7a16b4ca2bbb6d106bcb8c7ff (table default.Table_190b2d86515a44898fa38a49517b9df1 [id=97d8a36ddfae4bb5b99c04a7a08544af]). Expected version 2 got 0
I20240430 20:11:34.207996 18455 tablet_service.cc:1467] Processing CreateTablet for tablet d7c1fc7e81364fb890bbc13d7fb6848b (DEFAULT_TABLE table=default.Table_fa64149f685c4784bda92ae4f578dd92 [id=3d6e1fcb9c0f403394d8b436d100b9d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:34.795436 18455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7c1fc7e81364fb890bbc13d7fb6848b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:34.140893 18737 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b4d9e860d7c45fd91d7a566605c9283 with delete_type TABLET_DATA_DELETED (replaced by tablet 12795670f21447a8964fe1efc6e92398) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:34.796219 18737 tablet_service.cc:1467] Processing CreateTablet for tablet 12795670f21447a8964fe1efc6e92398 (DEFAULT_TABLE table=default.Table_421069f84cb844368066c93102a8faf8 [id=d5cea90cbc2c48df9d8b80da1eccc949]), partition=RANGE (key) PARTITION UNBOUNDED
W20240430 20:11:34.469383 19001 catalog_manager.cc:4909] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): alter failed for tablet 12c6e131fa904d23865b8673b58a7e33 (table default.Table_a9dc8fc6a4aa46dc914d0e327a4de554 [id=19f4e6decedf4ee59fccc0a24a05cedd]): Illegal state: replica 3c51c33d5616482cafb68e26688820f4 is not leader of this config: current role FOLLOWER
W20240430 20:11:34.796674 19001 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.AlterSchema -> {remote=127.17.177.3:44867, user_credentials={real_user=slave}} blocked reactor thread for 327313us
W20240430 20:11:34.693046 18963 kernel_stack_watchdog.cc:198] Thread 19001 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:308 for 113ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7feb1a029980 (unknown) at ??:0
@ 0x7feb1a0239c6 __GI___pthread_rwlock_wrlock at ??:0
@ 0x7feb13fa9904 google::LogMessage::Flush() at ??:0
@ 0x7feb13fa9a11 google::LogMessage::~LogMessage() at ??:0
@ 0x7feb2a1521b0 kudu::master::AsyncAlterTable::HandleResponse() at ??:0
@ 0x7feb2a0a1b09 kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7feb2a153d60 _ZZN4kudu6master15AsyncAlterTable11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7feb2a153b7e _ZNSt17_Function_handlerIFvvEZN4kudu6master15AsyncAlterTable11SendRequestEiEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7feb1d8e98cf kudu::rpc::Proxy::AsyncRequest()::$_1::operator()() at ??:0
@ 0x7feb1d8e94ae std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7feb1d8d54e0 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7feb1d8d5a2c kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7feb1d87251a kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7feb1d870f45 kudu::rpc::Connection::ReadHandler() at ??:0
I20240430 20:11:34.511144 18983 tablet_bootstrap.cc:492] T acd23f0a63d5448aaa7ed78422e67bad P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:34.798831 18983 ts_tablet_manager.cc:1367] T acd23f0a63d5448aaa7ed78422e67bad P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.336s user 0.009s sys 0.002s
W20240430 20:11:34.799510 18557 leader_election.cc:341] T 4e69ca12bd10494fb013614741bc166c P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:34.799665 18557 leader_election.cc:302] T 4e69ca12bd10494fb013614741bc166c P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:34.799844 18983 raft_consensus.cc:348] T acd23f0a63d5448aaa7ed78422e67bad P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:34.800136 18983 raft_consensus.cc:374] T acd23f0a63d5448aaa7ed78422e67bad P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20240430 20:11:34.800196 18557 leader_election.cc:341] T 26ed12f498644d15bf45e884dcad172e P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:34.800232 18983 raft_consensus.cc:729] T acd23f0a63d5448aaa7ed78422e67bad P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
W20240430 20:11:34.800472 18557 leader_election.cc:341] T 26ed12f498644d15bf45e884dcad172e P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:34.800781 18557 leader_election.cc:302] T 53a899fa5a424a0d99f1b47925e8c7ad P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a
I20240430 20:11:34.801373 18557 leader_election.cc:302] T a05e32a33aa7488eb287cc6314c09c44 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:34.800530 18983 consensus_queue.cc:260] T acd23f0a63d5448aaa7ed78422e67bad P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:34.213691 18456 tablet_service.cc:1467] Processing CreateTablet for tablet 9f0b341a31224bf1be525b84b9500304 (DEFAULT_TABLE table=default.Table_5b6a4bb3832e497888af8e8db7b96c12 [id=883dc5b0bfa349e9947d4c5dfe84a970]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:34.802117 18456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f0b341a31224bf1be525b84b9500304. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:34.608088 18591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dad16ae02b8b4e29a724d8eec67bb4b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:35.255239 19519 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240430 20:11:35.441520 19519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240430 20:11:35.441856 19519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240430 20:11:34.781567 19024 rpcz_store.cc:276] 0430 20:11:33.252394 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:34.069892 (+817498us) service_pool.cc:225] Handling call
0430 20:11:34.070398 (+ 506us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0430 20:11:34.070550 (+ 152us) catalog_manager.cc:2749] Modifying in-memory table state
0430 20:11:34.070635 (+ 85us) catalog_manager.cc:2758] Locking tablets
0430 20:11:34.070778 (+ 143us) catalog_manager.cc:2772] Removing table and tablets from system table
0430 20:11:34.185914 (+115136us) catalog_manager.cc:2790] Removing table from by-name map
0430 20:11:34.186064 (+ 150us) catalog_manager.cc:2806] Committing in-memory state
0430 20:11:34.186120 (+ 56us) catalog_manager.cc:2810] Aborting table tasks
0430 20:11:34.781392 (+595272us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:34.082697 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:34.082803 (+ 106us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:34.082811 (+ 8us) write_op.cc:427] Acquired schema lock
0430 20:11:34.082816 (+ 5us) tablet.cc:660] Decoding operations
0430 20:11:34.082942 (+ 126us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:34.082953 (+ 11us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:34.082965 (+ 12us) tablet.cc:683] Acquiring locks for 2 operations
0430 20:11:34.083056 (+ 91us) tablet.cc:699] Row locks acquired
0430 20:11:34.083059 (+ 3us) write_op.cc:260] PREPARE: finished
0430 20:11:34.083099 (+ 40us) write_op.cc:270] Start()
0430 20:11:34.083133 (+ 34us) write_op.cc:276] Timestamp: P: 1714507894083041 usec, L: 0
0430 20:11:34.083137 (+ 4us) op_driver.cc:331] REPLICATION: starting
0430 20:11:34.083326 (+ 189us) log.cc:842] Serialized 728 byte log entry
0430 20:11:34.160855 (+ 77529us) op_driver.cc:444] REPLICATION: finished
0430 20:11:34.161194 (+ 339us) write_op.cc:301] APPLY: starting
0430 20:11:34.161216 (+ 22us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:34.161280 (+ 64us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:34.161316 (+ 36us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:34.161414 (+ 98us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:34.161460 (+ 46us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0430 20:11:34.161471 (+ 11us) write_op.cc:312] APPLY: finished
0430 20:11:34.161569 (+ 98us) log.cc:842] Serialized 29 byte log entry
0430 20:11:34.161645 (+ 76us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:34.161668 (+ 23us) write_op.cc:446] Released schema lock
0430 20:11:34.161676 (+ 8us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:34.161741 (+ 65us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:34.161745 (+ 4us) write_op.cc:446] Released schema lock
Metrics: {"mutex_wait_us":4841,"child_traces":[["op",{"apply.queue_time_us":272,"apply.run_cpu_time_us":608,"apply.run_wall_time_us":607,"num_ops":2,"prepare.queue_time_us":11190,"prepare.run_cpu_time_us":717,"prepare.run_wall_time_us":718,"replication_time_us":77683}]]}
I20240430 20:11:34.141062 18728 tablet_service.cc:1514] Processing DeleteTablet for tablet 50486644ec234933b9a0b45ee39a6685 with delete_type TABLET_DATA_DELETED (replaced by tablet ed0c37a71ede4bfc9eb5ad1a4365f25e) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:35.443224 18728 tablet_service.cc:1467] Processing CreateTablet for tablet f3711610610542128c4a84e17bce4a3e (DEFAULT_TABLE table=default.Table_64d14df4e12146ecb85239e7ca03cf6e [id=c376119401934812aef62a3a18eeb368]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:34.227401 18454 tablet_service.cc:1467] Processing CreateTablet for tablet b1792c160af346ffb4f047c5d4435c10 (DEFAULT_TABLE table=default.Table_3127eaa465c84dcab3eb9935d15667ce [id=e4cf2d34f4b747afa5b8a512f211c01f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:34.805668 18602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12795670f21447a8964fe1efc6e92398. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:35.444245 18601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e5225f1f5f544ecbf8dfa470af3daca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:34.809944 19026 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_4537f891b01048ceb51202abc3600911" } new_table_name: "default.Table_343ee22ed6554f328aa58881079e1905" modify_external_catalogs: true
I20240430 20:11:34.143137 18744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c091701480c4412faabad3d071d85d03. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:35.444571 18595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed0c37a71ede4bfc9eb5ad1a4365f25e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:35.444864 18729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1792c160af346ffb4f047c5d4435c10. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:35.445350 18735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fda942d50b014f3d9589e1fb909fc2fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:34.227823 18462 tablet_service.cc:1467] Processing CreateTablet for tablet 6db55bb510b24b0cae897936211e15f0 (DEFAULT_TABLE table=default.Table_3471b34c1c0d4d829bb27a8520eff502 [id=f3498e7e44974f65bbd2e92c6c8c119f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:34.227885 18452 tablet_service.cc:1467] Processing CreateTablet for tablet fda942d50b014f3d9589e1fb909fc2fc (DEFAULT_TABLE table=default.Table_68528229d5a94981a7fc0677d48003d1 [id=25ac2d71caf74e5ba14e3e84b84c9c60]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:34.201936 19725 consensus_queue.cc:1035] T 1555924b5f1444398f12ee82c3cb5b13 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:34.636243 19471 ts_tablet_manager.cc:1888] T 34ab40d8d91e401db299ddee7175ba28 P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:35.445866 19471 log.cc:1196] T 34ab40d8d91e401db299ddee7175ba28 P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/34ab40d8d91e401db299ddee7175ba28
I20240430 20:11:34.227928 18453 tablet_service.cc:1467] Processing CreateTablet for tablet 50e1c55a1a0a484984ad5cbb78eec2ed (DEFAULT_TABLE table=default.Table_211cbad17eba4c48ab02ad45543616ca [id=a693b863f7f34539aa46c3f006aca925]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:33.718247 19489 ts_tablet_manager.cc:1888] T bbe9a2429a434f3f95d6e7f314fcfe4c P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:35.446233 19489 log.cc:1196] T bbe9a2429a434f3f95d6e7f314fcfe4c P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/bbe9a2429a434f3f95d6e7f314fcfe4c
I20240430 20:11:35.446305 19471 ts_tablet_manager.cc:1909] T 34ab40d8d91e401db299ddee7175ba28 P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:35.446746 19489 ts_tablet_manager.cc:1909] T bbe9a2429a434f3f95d6e7f314fcfe4c P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:35.446738 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 64) took 4435 ms. Trace:
I20240430 20:11:35.446837 19471 rpcz_store.cc:276] 0430 20:11:31.011067 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:31.011352 (+ 285us) service_pool.cc:225] Handling call
0430 20:11:34.090705 (+3079353us) ts_tablet_manager.cc:1120] Deleting tablet 34ab40d8d91e401db299ddee7175ba28
0430 20:11:34.090720 (+ 15us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:34.339051 (+248331us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:34.636083 (+297032us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:35.446666 (+810583us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3078798}
I20240430 20:11:35.447270 19471 tablet_replica.cc:330] T cace1398492f4d968dcb6899385d527c P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:35.447475 19471 raft_consensus.cc:2229] T cace1398492f4d968dcb6899385d527c P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:35.447625 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 61) took 4184 ms. Trace:
I20240430 20:11:35.447695 19471 raft_consensus.cc:2258] T cace1398492f4d968dcb6899385d527c P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:35.447723 19489 rpcz_store.cc:276] 0430 20:11:31.263199 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:31.263246 (+ 47us) service_pool.cc:225] Handling call
0430 20:11:33.035249 (+1772003us) ts_tablet_manager.cc:1120] Deleting tablet bbe9a2429a434f3f95d6e7f314fcfe4c
0430 20:11:33.035260 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:33.619979 (+584719us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:33.718106 (+ 98127us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:35.447555 (+1729449us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1770674}
I20240430 20:11:35.448113 19489 tablet_replica.cc:330] T 34ab40d8d91e401db299ddee7175ba28 P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:34.228228 18458 tablet_service.cc:1467] Processing CreateTablet for tablet 8e91920e7e36418d8c27d1f0bfd9b981 (DEFAULT_TABLE table=default.Table_20f3b06af23d476f8ebc0471d08f8659 [id=f393bcc9669c4ea985a2d40deb161f0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:34.348680 19843 tablet.cc:1722] T 294644d7a16b4ca2bbb6d106bcb8c7ff P 3c51c33d5616482cafb68e26688820f4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240430 20:11:35.448276 19471 ts_tablet_manager.cc:1875] T cace1398492f4d968dcb6899385d527c P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:34.228574 18447 tablet_service.cc:1467] Processing CreateTablet for tablet 488d0e609b224674932f890939664df2 (DEFAULT_TABLE table=default.Table_58f35a2120874472978ca0e5f816e86c [id=ff18001c6d814afc8c1768d0c84a6bac]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:34.352684 19673 consensus_queue.cc:1035] T 01436e5b6232494babbb414342f03090 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:34.897434 19048 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: Sending DeleteTablet for 3 replicas of tablet 680553d4d0bc4fcb839235c1832757e1
I20240430 20:11:35.449712 19048 catalog_manager.cc:6554] ReplaceTablet: tablet 680553d4d0bc4fcb839235c1832757e1 deleted and replaced by tablet 9c73d0a6bc414d3b9900570848d19a51
I20240430 20:11:35.449764 19001 catalog_manager.cc:4830] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): tablet bbe9a2429a434f3f95d6e7f314fcfe4c (table default.Table_3acf40c6d58741d4a8e94ca96d642249 [id=71226c1fc71242a6aad13625c9ca3c23]) successfully deleted
I20240430 20:11:35.449857 19048 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:51664 (request call id 150) took 2126 ms. Trace:
I20240430 20:11:35.449927 18741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e91920e7e36418d8c27d1f0bfd9b981. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:34.228931 18445 tablet_service.cc:1467] Processing CreateTablet for tablet b285e3eb16d148779c74e057458d8461 (DEFAULT_TABLE table=default.Table_575286687a4e4020a315d78080c2e28e [id=d1655c507b434f129ad7ae0b04365e51]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:33.877380 18738 tablet_service.cc:1467] Processing CreateTablet for tablet e4cffa295d8c401aa8415ce7e8f56526 (DEFAULT_TABLE table=default.Table_81547db3074240c98cdd95ed3b2dd813 [id=e0ea901452334b36a8493baed478cc1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:35.449977 19048 rpcz_store.cc:276] 0430 20:11:33.323021 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:34.165711 (+842690us) service_pool.cc:225] Handling call
0430 20:11:35.449813 (+1284102us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:34.444220 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:34.444394 (+ 174us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:34.444402 (+ 8us) write_op.cc:427] Acquired schema lock
0430 20:11:34.444405 (+ 3us) tablet.cc:660] Decoding operations
0430 20:11:34.444543 (+ 138us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:34.444560 (+ 17us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:34.444567 (+ 7us) tablet.cc:683] Acquiring locks for 2 operations
0430 20:11:34.444659 (+ 92us) tablet.cc:699] Row locks acquired
0430 20:11:34.444661 (+ 2us) write_op.cc:260] PREPARE: finished
0430 20:11:34.444696 (+ 35us) write_op.cc:270] Start()
0430 20:11:34.444735 (+ 39us) write_op.cc:276] Timestamp: P: 1714507894444649 usec, L: 0
0430 20:11:34.444739 (+ 4us) op_driver.cc:331] REPLICATION: starting
0430 20:11:34.444983 (+ 244us) log.cc:842] Serialized 659 byte log entry
0430 20:11:34.855868 (+410885us) op_driver.cc:444] REPLICATION: finished
0430 20:11:34.894950 (+ 39082us) write_op.cc:301] APPLY: starting
0430 20:11:34.894971 (+ 21us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:34.895021 (+ 50us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:34.895054 (+ 33us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:34.895196 (+ 142us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:34.895270 (+ 74us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0430 20:11:34.895282 (+ 12us) write_op.cc:312] APPLY: finished
0430 20:11:34.895364 (+ 82us) log.cc:842] Serialized 29 byte log entry
0430 20:11:34.896541 (+ 1177us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:34.896564 (+ 23us) write_op.cc:446] Released schema lock
0430 20:11:34.896573 (+ 9us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:34.897574 (+ 1001us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:34.897582 (+ 8us) write_op.cc:446] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":38974,"apply.run_cpu_time_us":702,"apply.run_wall_time_us":2701,"num_ops":2,"prepare.queue_time_us":79,"prepare.run_cpu_time_us":880,"prepare.run_wall_time_us":880,"raft.queue_time_us":67,"raft.run_cpu_time_us":260,"raft.run_wall_time_us":260,"replication_time_us":411075,"thread_start_us":43661,"threads_started":1}]]}
I20240430 20:11:35.450469 18738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4cffa295d8c401aa8415ce7e8f56526. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:34.629844 19818 raft_consensus.cc:482] T b76a4105a2974fe280e759349037d027 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3c51c33d5616482cafb68e26688820f4)
I20240430 20:11:35.451162 19818 raft_consensus.cc:504] T b76a4105a2974fe280e759349037d027 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:35.452050 19818 leader_election.cc:288] T b76a4105a2974fe280e759349037d027 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
W20240430 20:11:34.901011 19023 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d61c279d20b0471a967942525ceb4fcb consensus state has no leader
I20240430 20:11:35.452347 19023 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:51664 (request call id 156) took 1909 ms. Trace:
I20240430 20:11:35.452455 19023 rpcz_store.cc:276] 0430 20:11:33.542660 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:34.454327 (+911667us) service_pool.cc:225] Handling call
0430 20:11:34.454706 (+ 379us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0430 20:11:34.454973 (+ 267us) catalog_manager.cc:3613] Apply alter schema
0430 20:11:34.455187 (+ 214us) catalog_manager.cc:3636] Acquired catalog manager lock
0430 20:11:34.455270 (+ 83us) catalog_manager.cc:3783] Updating metadata on disk
0430 20:11:34.900720 (+445450us) catalog_manager.cc:3814] Committing alterations to in-memory state
0430 20:11:35.452294 (+551574us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:34.457956 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:34.458077 (+ 121us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:34.458085 (+ 8us) write_op.cc:427] Acquired schema lock
0430 20:11:34.458088 (+ 3us) tablet.cc:660] Decoding operations
0430 20:11:34.458222 (+ 134us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:34.458234 (+ 12us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:34.458241 (+ 7us) tablet.cc:683] Acquiring locks for 1 operations
0430 20:11:34.458314 (+ 73us) tablet.cc:699] Row locks acquired
0430 20:11:34.458316 (+ 2us) write_op.cc:260] PREPARE: finished
0430 20:11:34.458368 (+ 52us) write_op.cc:270] Start()
0430 20:11:34.458446 (+ 78us) write_op.cc:276] Timestamp: P: 1714507894458313 usec, L: 0
0430 20:11:34.458451 (+ 5us) op_driver.cc:331] REPLICATION: starting
0430 20:11:34.458664 (+ 213us) log.cc:842] Serialized 405 byte log entry
0430 20:11:34.899901 (+441237us) op_driver.cc:444] REPLICATION: finished
0430 20:11:34.900134 (+ 233us) write_op.cc:301] APPLY: starting
0430 20:11:34.900148 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:34.900185 (+ 37us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:34.900211 (+ 26us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:34.900264 (+ 53us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:34.900300 (+ 36us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0430 20:11:34.900308 (+ 8us) write_op.cc:312] APPLY: finished
0430 20:11:34.900379 (+ 71us) log.cc:842] Serialized 23 byte log entry
0430 20:11:34.900616 (+ 237us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:34.900634 (+ 18us) write_op.cc:446] Released schema lock
0430 20:11:34.900639 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:34.901151 (+ 512us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:34.901157 (+ 6us) write_op.cc:446] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":188,"apply.run_cpu_time_us":461,"apply.run_wall_time_us":1072,"num_ops":1,"prepare.queue_time_us":2242,"prepare.run_cpu_time_us":793,"prepare.run_wall_time_us":793,"replication_time_us":441426}]]}
I20240430 20:11:34.806069 18586 tablet_service.cc:1514] Processing DeleteTablet for tablet 06d28b6f48a34b17b0d7fb3102df2f4c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:33 UTC) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:35.453823 18731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90a694bd5db649afa50f3c4e752939c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:34.902137 19033 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:51664 (request call id 153) took 1448 ms. Trace:
I20240430 20:11:34.229415 19495 ts_tablet_manager.cc:1888] T 4e69ca12bd10494fb013614741bc166c P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:34.233937 18985 tablet_bootstrap.cc:492] T e6eac92dfa5f4456bc49571878f0f8dd P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:35.454234 19495 log.cc:1196] T 4e69ca12bd10494fb013614741bc166c P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/4e69ca12bd10494fb013614741bc166c
I20240430 20:11:35.454310 18985 ts_tablet_manager.cc:1367] T e6eac92dfa5f4456bc49571878f0f8dd P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 1.374s user 0.008s sys 0.025s
I20240430 20:11:35.453976 19033 rpcz_store.cc:276] 0430 20:11:33.454222 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:34.176200 (+721978us) service_pool.cc:225] Handling call
0430 20:11:34.457008 (+280808us) catalog_manager.cc:2072] Acquired catalog manager lock
0430 20:11:34.457471 (+ 463us) catalog_manager.cc:2118] Created new table and tablet info
0430 20:11:34.901839 (+444368us) catalog_manager.cc:2178] Wrote table and tablets to system table
0430 20:11:34.901957 (+ 118us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0430 20:11:34.902087 (+ 130us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:34.458792 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:34.458889 (+ 97us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:34.458895 (+ 6us) write_op.cc:427] Acquired schema lock
0430 20:11:34.458898 (+ 3us) tablet.cc:660] Decoding operations
0430 20:11:34.458977 (+ 79us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:34.458986 (+ 9us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:34.458993 (+ 7us) tablet.cc:683] Acquiring locks for 2 operations
0430 20:11:34.459063 (+ 70us) tablet.cc:699] Row locks acquired
0430 20:11:34.459066 (+ 3us) write_op.cc:260] PREPARE: finished
0430 20:11:34.459094 (+ 28us) write_op.cc:270] Start()
0430 20:11:34.459119 (+ 25us) write_op.cc:276] Timestamp: P: 1714507894459047 usec, L: 0
0430 20:11:34.459122 (+ 3us) op_driver.cc:331] REPLICATION: starting
0430 20:11:34.459261 (+ 139us) log.cc:842] Serialized 433 byte log entry
0430 20:11:34.899961 (+440700us) op_driver.cc:444] REPLICATION: finished
0430 20:11:34.901244 (+ 1283us) write_op.cc:301] APPLY: starting
0430 20:11:34.901256 (+ 12us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:34.901292 (+ 36us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:34.901326 (+ 34us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:34.901436 (+ 110us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:34.901467 (+ 31us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0430 20:11:34.901474 (+ 7us) write_op.cc:312] APPLY: finished
0430 20:11:34.901536 (+ 62us) log.cc:842] Serialized 29 byte log entry
0430 20:11:34.901745 (+ 209us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:34.901761 (+ 16us) write_op.cc:446] Released schema lock
0430 20:11:34.901767 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:34.902252 (+ 485us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:34.902257 (+ 5us) write_op.cc:446] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1251,"apply.run_cpu_time_us":479,"apply.run_wall_time_us":1055,"num_ops":2,"prepare.queue_time_us":885,"prepare.run_cpu_time_us":522,"prepare.run_wall_time_us":521,"replication_time_us":440825,"thread_start_us":6912,"threads_started":1}]]}
I20240430 20:11:34.814385 19756 raft_consensus.cc:2726] T 4e69ca12bd10494fb013614741bc166c P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:35.454715 19495 ts_tablet_manager.cc:1909] T 4e69ca12bd10494fb013614741bc166c P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:34.908491 19051 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:51664 (request call id 154) took 1424 ms. Trace:
I20240430 20:11:34.049852 18810 heartbeater.cc:503] Master 127.17.177.60:37225 was elected leader, sending a full tablet report...
I20240430 20:11:35.454916 19051 rpcz_store.cc:276] 0430 20:11:33.484521 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:34.178330 (+693809us) service_pool.cc:225] Handling call
0430 20:11:34.464485 (+286155us) catalog_manager.cc:2072] Acquired catalog manager lock
0430 20:11:34.465003 (+ 518us) catalog_manager.cc:2118] Created new table and tablet info
0430 20:11:34.908167 (+443164us) catalog_manager.cc:2178] Wrote table and tablets to system table
0430 20:11:34.908314 (+ 147us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0430 20:11:34.908442 (+ 128us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:34.469625 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:34.469785 (+ 160us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:34.469794 (+ 9us) write_op.cc:427] Acquired schema lock
0430 20:11:34.469798 (+ 4us) tablet.cc:660] Decoding operations
0430 20:11:34.469926 (+ 128us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:34.469941 (+ 15us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:34.469949 (+ 8us) tablet.cc:683] Acquiring locks for 2 operations
0430 20:11:34.470060 (+ 111us) tablet.cc:699] Row locks acquired
0430 20:11:34.470064 (+ 4us) write_op.cc:260] PREPARE: finished
0430 20:11:34.470108 (+ 44us) write_op.cc:270] Start()
0430 20:11:34.470154 (+ 46us) write_op.cc:276] Timestamp: P: 1714507894470056 usec, L: 0
0430 20:11:34.470160 (+ 6us) op_driver.cc:331] REPLICATION: starting
0430 20:11:34.470409 (+ 249us) log.cc:842] Serialized 433 byte log entry
0430 20:11:34.906981 (+436572us) op_driver.cc:444] REPLICATION: finished
0430 20:11:34.907334 (+ 353us) write_op.cc:301] APPLY: starting
0430 20:11:34.907353 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:34.907415 (+ 62us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:34.907457 (+ 42us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:34.907594 (+ 137us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:34.907636 (+ 42us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0430 20:11:34.907643 (+ 7us) write_op.cc:312] APPLY: finished
0430 20:11:34.907735 (+ 92us) log.cc:842] Serialized 29 byte log entry
0430 20:11:34.908046 (+ 311us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:34.908070 (+ 24us) write_op.cc:446] Released schema lock
0430 20:11:34.908078 (+ 8us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:34.908644 (+ 566us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:34.908651 (+ 7us) write_op.cc:446] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":289,"apply.run_cpu_time_us":647,"apply.run_wall_time_us":1372,"num_ops":2,"prepare.queue_time_us":4079,"prepare.run_cpu_time_us":884,"prepare.run_wall_time_us":884,"replication_time_us":436792}]]}
I20240430 20:11:35.455261 18985 raft_consensus.cc:348] T e6eac92dfa5f4456bc49571878f0f8dd P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:34.814656 19861 raft_consensus.cc:2781] T 53a899fa5a424a0d99f1b47925e8c7ad P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:35.455487 18985 raft_consensus.cc:374] T e6eac92dfa5f4456bc49571878f0f8dd P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:35.455575 18985 raft_consensus.cc:729] T e6eac92dfa5f4456bc49571878f0f8dd P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:35.455544 19861 raft_consensus.cc:482] T 53a899fa5a424a0d99f1b47925e8c7ad P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:34.909641 19049 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:51664 (request call id 155) took 1404 ms. Trace:
I20240430 20:11:35.455816 18985 consensus_queue.cc:260] T e6eac92dfa5f4456bc49571878f0f8dd P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:35.455727 19049 rpcz_store.cc:276] 0430 20:11:33.505857 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:34.450754 (+944897us) service_pool.cc:225] Handling call
0430 20:11:34.451700 (+ 946us) catalog_manager.cc:2072] Acquired catalog manager lock
0430 20:11:34.452451 (+ 751us) catalog_manager.cc:2118] Created new table and tablet info
0430 20:11:34.909382 (+456931us) catalog_manager.cc:2178] Wrote table and tablets to system table
0430 20:11:34.909512 (+ 130us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0430 20:11:34.909600 (+ 88us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:34.479860 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:34.480035 (+ 175us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:34.480045 (+ 10us) write_op.cc:427] Acquired schema lock
0430 20:11:34.480054 (+ 9us) tablet.cc:660] Decoding operations
0430 20:11:34.480195 (+ 141us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:34.480212 (+ 17us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:34.480220 (+ 8us) tablet.cc:683] Acquiring locks for 2 operations
0430 20:11:34.480334 (+ 114us) tablet.cc:699] Row locks acquired
0430 20:11:34.480338 (+ 4us) write_op.cc:260] PREPARE: finished
0430 20:11:34.480383 (+ 45us) write_op.cc:270] Start()
0430 20:11:34.480419 (+ 36us) write_op.cc:276] Timestamp: P: 1714507894480332 usec, L: 0
0430 20:11:34.480432 (+ 13us) op_driver.cc:331] REPLICATION: starting
0430 20:11:34.480647 (+ 215us) log.cc:842] Serialized 433 byte log entry
0430 20:11:34.907063 (+426416us) op_driver.cc:444] REPLICATION: finished
0430 20:11:34.908743 (+ 1680us) write_op.cc:301] APPLY: starting
0430 20:11:34.908755 (+ 12us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:34.908797 (+ 42us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:34.908825 (+ 28us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:34.908947 (+ 122us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:34.908984 (+ 37us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0430 20:11:34.908992 (+ 8us) write_op.cc:312] APPLY: finished
0430 20:11:34.909061 (+ 69us) log.cc:842] Serialized 29 byte log entry
0430 20:11:34.909280 (+ 219us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:34.909298 (+ 18us) write_op.cc:446] Released schema lock
0430 20:11:34.909306 (+ 8us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:34.909760 (+ 454us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:34.909768 (+ 8us) write_op.cc:446] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1645,"apply.run_cpu_time_us":523,"apply.run_wall_time_us":1069,"num_ops":2,"prepare.queue_time_us":80,"prepare.run_cpu_time_us":876,"prepare.run_wall_time_us":876,"replication_time_us":426613}]]}
I20240430 20:11:34.910727 19029 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:51664 (request call id 152) took 1457 ms. Trace:
I20240430 20:11:35.456432 18985 ts_tablet_manager.cc:1398] T e6eac92dfa5f4456bc49571878f0f8dd P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240430 20:11:34.814671 19815 raft_consensus.cc:2726] T a05e32a33aa7488eb287cc6314c09c44 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:35.456388 19029 rpcz_store.cc:276] 0430 20:11:33.453763 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:34.170513 (+716750us) service_pool.cc:225] Handling call
0430 20:11:34.186364 (+ 15851us) spinlock_profiling.cc:243] Waited 15.5 ms on lock 0x616000040654. stack: 00007feb16c74756 00007feb16c74510 00007feb15ca748d 0000000000955804 00000000009557cd 00007feb257b315d 00007feb257b3253 00007feb224e5d41 00007feb2a0d134d 00007feb2a25dcb3 00007feb210462a6 00007feb21045fd5 00007feb1d97c04e 00007feb1d97af2e 00007feb1d981405 00007feb1d981d2f
0430 20:11:34.779996 (+593632us) catalog_manager.cc:2072] Acquired catalog manager lock
0430 20:11:34.780486 (+ 490us) catalog_manager.cc:2118] Created new table and tablet info
0430 20:11:34.910466 (+129980us) catalog_manager.cc:2178] Wrote table and tablets to system table
0430 20:11:34.910581 (+ 115us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0430 20:11:34.910688 (+ 107us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:34.781898 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:34.782043 (+ 145us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:34.782050 (+ 7us) write_op.cc:427] Acquired schema lock
0430 20:11:34.782053 (+ 3us) tablet.cc:660] Decoding operations
0430 20:11:34.782154 (+ 101us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:34.782166 (+ 12us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:34.782172 (+ 6us) tablet.cc:683] Acquiring locks for 2 operations
0430 20:11:34.782281 (+ 109us) tablet.cc:699] Row locks acquired
0430 20:11:34.782284 (+ 3us) write_op.cc:260] PREPARE: finished
0430 20:11:34.782328 (+ 44us) write_op.cc:270] Start()
0430 20:11:34.782365 (+ 37us) write_op.cc:276] Timestamp: P: 1714507894782302 usec, L: 0
0430 20:11:34.782369 (+ 4us) op_driver.cc:331] REPLICATION: starting
0430 20:11:34.791147 (+ 8778us) log.cc:842] Serialized 433 byte log entry
0430 20:11:34.907115 (+115968us) op_driver.cc:444] REPLICATION: finished
0430 20:11:34.909852 (+ 2737us) write_op.cc:301] APPLY: starting
0430 20:11:34.909864 (+ 12us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:34.909898 (+ 34us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:34.909932 (+ 34us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:34.910048 (+ 116us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:34.910082 (+ 34us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0430 20:11:34.910093 (+ 11us) write_op.cc:312] APPLY: finished
0430 20:11:34.910162 (+ 69us) log.cc:842] Serialized 29 byte log entry
0430 20:11:34.910362 (+ 200us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:34.910379 (+ 17us) write_op.cc:446] Released schema lock
0430 20:11:34.910385 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:34.910867 (+ 482us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:34.910872 (+ 5us) write_op.cc:446] Released schema lock
Metrics: {"spinlock_wait_cycles":35639168,"child_traces":[["op",{"apply.queue_time_us":2702,"apply.run_cpu_time_us":513,"apply.run_wall_time_us":1063,"num_ops":2,"prepare.queue_time_us":849,"prepare.run_cpu_time_us":929,"prepare.run_wall_time_us":9368,"replication_time_us":124730,"thread_start_us":4600,"threads_started":1}]]}
I20240430 20:11:35.456852 18726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7c1fc7e81364fb890bbc13d7fb6848b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:34.815232 18983 ts_tablet_manager.cc:1398] T acd23f0a63d5448aaa7ed78422e67bad P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.016s user 0.002s sys 0.001s
W20240430 20:11:34.811055 18558 leader_election.cc:341] T f5546f9edeea44d7a6c5cb954f5157ba P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:34.241242 18461 tablet_service.cc:1467] Processing CreateTablet for tablet 1f4e860484784aa9a2d28af2e9fb611c (DEFAULT_TABLE table=default.Table_45d50fc81325443097375ae6e65a1633 [id=9c861883de1d49c78146d65e9d4dd39f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:34.541211 19727 raft_consensus.cc:2726] T dd9b893d48684cfea7c242b93ecba01f P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:34.828951 18999 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet a1aeee1b02214975a376cfd36957afe8 (table default.Table_31d8d978e96c4e12a0fbd5500c7b93cd [id=8b480956f4d146e08dcfed0d0a0e4e0d]) successfully deleted
W20240430 20:11:35.458200 18999 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.17.177.2:46027, user_credentials={real_user=slave}} blocked reactor thread for 629177us
I20240430 20:11:35.459333 18743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b285e3eb16d148779c74e057458d8461. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:35.460351 19027 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_766e3df58895427b9e5179eb555fe7e4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:35.461007 19027 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_766e3df58895427b9e5179eb555fe7e4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240430 20:11:35.131975 18544 kernel_stack_watchdog.cc:198] Thread 18558 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:308 for 158ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7ffa8ed33980 (unknown) at ??:0
@ 0x7ffa8ed2d9c6 __GI___pthread_rwlock_wrlock at ??:0
@ 0x7ffa88cb3904 google::LogMessage::Flush() at ??:0
@ 0x7ffa88cb3a11 google::LogMessage::~LogMessage() at ??:0
@ 0x7ffa9715a5d3 kudu::consensus::LeaderElection::VoteResponseRpcCallback() at ??:0
@ 0x7ffa9715d12a kudu::consensus::LeaderElection::Run()::$_0::operator()() at ??:0
@ 0x7ffa9715ceee std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7ffa925f38cf kudu::rpc::Proxy::AsyncRequest()::$_1::operator()() at ??:0
@ 0x7ffa925f34ae std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7ffa925df4e0 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7ffa925dfa2c kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7ffa9257c51a kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7ffa9257af45 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7ffa9258d2c5 ev::base<>::method_thunk<>() at ??:0
I20240430 20:11:35.167215 19474 raft_consensus.cc:482] T acd23f0a63d5448aaa7ed78422e67bad P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:34.242370 18451 tablet_service.cc:1467] Processing CreateTablet for tablet 90a694bd5db649afa50f3c4e752939c1 (DEFAULT_TABLE table=default.Table_97a0991bf9284215b482c532e9dfc113 [id=7e3ac0868c0a4cc581427f121f305323]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:34.635649 18760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3e74e76ab2b4fe492ffcc1838656fd1" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:34.639196 18764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75ebf051383e42529dcb5f7dd0770a4b" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:34.637012 19581 raft_consensus.cc:504] T 75ebf051383e42529dcb5f7dd0770a4b P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:35.470254 19581 leader_election.cc:288] T 75ebf051383e42529dcb5f7dd0770a4b P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Requested vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:34.670890 19833 consensus_queue.cc:1035] T eed6338ad3d64859af087ee15e1ec798 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240430 20:11:34.676321 18761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcc9698abda4491b98e1e879589913a8" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:34.706223 18754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e69ca12bd10494fb013614741bc166c" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:34.707669 18759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24dcc20832804229a2664873bf6ddaa7" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:35.471074 18759 raft_consensus.cc:2454] T 24dcc20832804229a2664873bf6ddaa7 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 0.
I20240430 20:11:34.709290 18762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "802336602d8f4ac4bd7e0872d6f3dae9" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:34.709738 18753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b688e03ea71940c69bda5520b62c1fc3" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:35.557006 19915 raft_consensus.cc:482] T 00000000000000000000000000000000 P 89df84ccf5394c9bb60d4e4ed872fc2e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader b632d6ef4c104a058a6db208024cab33)
I20240430 20:11:35.826774 19915 raft_consensus.cc:504] T 00000000000000000000000000000000 P 89df84ccf5394c9bb60d4e4ed872fc2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b3f98793354c7aaa5c106e8ec31ed8" member_type: VOTER last_known_addr { host: "127.17.177.62" port: 36441 } } peers { permanent_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e" member_type: VOTER last_known_addr { host: "127.17.177.61" port: 34777 } } peers { permanent_uuid: "b632d6ef4c104a058a6db208024cab33" member_type: VOTER last_known_addr { host: "127.17.177.60" port: 37225 } }
I20240430 20:11:35.827865 19915 leader_election.cc:288] T 00000000000000000000000000000000 P 89df84ccf5394c9bb60d4e4ed872fc2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 88b3f98793354c7aaa5c106e8ec31ed8 (127.17.177.62:36441), b632d6ef4c104a058a6db208024cab33 (127.17.177.60:37225)
W20240430 20:11:35.400208 18963 kernel_stack_watchdog.cc:198] Thread 18999 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:308 for 308ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7feb1a029980 (unknown) at ??:0
@ 0x7feb1a0239c6 __GI___pthread_rwlock_wrlock at ??:0
@ 0x7feb13fa9904 google::LogMessage::Flush() at ??:0
@ 0x7feb13fa9a11 google::LogMessage::~LogMessage() at ??:0
@ 0x7feb2a1491e8 kudu::master::AsyncDeleteReplica::HandleResponse() at ??:0
@ 0x7feb2a0a1b09 kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7feb2a14ce30 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7feb2a14cc4e _ZNSt17_Function_handlerIFvvEZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7feb1d8e98cf kudu::rpc::Proxy::AsyncRequest()::$_1::operator()() at ??:0
@ 0x7feb1d8e94ae std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7feb1d8d54e0 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7feb1d8d5a2c kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7feb1d87251a kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7feb1d870f45 kudu::rpc::Connection::ReadHandler() at ??:0
I20240430 20:11:35.462654 19028 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_3a588666409c4a64b13658a42a58f994"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240430 20:11:34.242789 18443 tablet_service.cc:1467] Processing CreateTablet for tablet dad16ae02b8b4e29a724d8eec67bb4b8 (DEFAULT_TABLE table=default.Table_8a6b5152d20b4890918aedb454764ee9 [id=121444fdfd2642a38cdbd3a6c411c4fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:35.254328 19890 raft_consensus.cc:482] T c9e7660eccc448dbb99d135f82bfd261 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:35.463722 19024 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_95220daaf6d44c08beff4ed6abdbea79" } modify_external_catalogs: true
W20240430 20:11:35.828708 19028 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3a588666409c4a64b13658a42a58f994 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:35.828752 19890 raft_consensus.cc:504] T c9e7660eccc448dbb99d135f82bfd261 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:35.828869 19024 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_95220daaf6d44c08beff4ed6abdbea79" } modify_external_catalogs: true
I20240430 20:11:35.829603 19890 leader_election.cc:288] T c9e7660eccc448dbb99d135f82bfd261 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:34.710317 18755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "255c13ad8fae4ea4aa8e7b7aabb7602e" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:35.829968 18755 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.17.177.1:40727 (request call id 168) took 1122 ms. Trace:
I20240430 20:11:35.830109 18755 rpcz_store.cc:276] 0430 20:11:34.708026 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:34.710200 (+ 2174us) service_pool.cc:225] Handling call
0430 20:11:35.829917 (+1119717us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240430 20:11:34.242947 18444 tablet_service.cc:1467] Processing CreateTablet for tablet 70247a4c910946c0afc24ce5ce0e85d6 (DEFAULT_TABLE table=default.Table_e20b13b4103849b9bc5e5f96f9b911e4 [id=9fd8ccb805024968b9ad8ab060d5db3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:35.469759 19861 raft_consensus.cc:3037] T 53a899fa5a424a0d99f1b47925e8c7ad P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:34.710558 18763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b72fe1f0713b4076a81a3fc1c74f7fa7" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:34.246285 19784 raft_consensus.cc:482] T 53a899fa5a424a0d99f1b47925e8c7ad P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:34.247893 18449 tablet_service.cc:1514] Processing DeleteTablet for tablet 06d28b6f48a34b17b0d7fb3102df2f4c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:33 UTC) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:34.742033 19859 raft_consensus.cc:482] T b72fe1f0713b4076a81a3fc1c74f7fa7 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:34.372084 19781 raft_consensus.cc:482] T d9e578c99058424a9b86390641f9c34f P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5e1c6397f09747f091534faf93542710)
I20240430 20:11:34.202119 18463 tablet_service.cc:1467] Processing CreateTablet for tablet 130007acc8ba4e709c3eb85fabe75903 (DEFAULT_TABLE table=default.Table_c6d5c8ecce28428790b07d228879a0e4 [id=36f478a20a1f4a53a52fc29ade62b7f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:34.536763 19825 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:41680 (request call id 152) took 1488 ms. Trace:
I20240430 20:11:34.540683 18483 raft_consensus.cc:2454] T 75ebf051383e42529dcb5f7dd0770a4b P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
W20240430 20:11:34.629532 18415 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.177.2:46027, user_credentials={real_user=slave}} blocked reactor thread for 175564us
I20240430 20:11:34.636576 19617 raft_consensus.cc:504] T fcc9698abda4491b98e1e879589913a8 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:34.643126 18450 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b4d9e860d7c45fd91d7a566605c9283 with delete_type TABLET_DATA_DELETED (replaced by tablet 12795670f21447a8964fe1efc6e92398) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:34.740458 18475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24dcc20832804229a2664873bf6ddaa7" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:34.806984 18448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df92eb85261d47d88cbf16d29d9ad30e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:35.832937 19859 raft_consensus.cc:504] T b72fe1f0713b4076a81a3fc1c74f7fa7 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:35.833757 19859 leader_election.cc:288] T b72fe1f0713b4076a81a3fc1c74f7fa7 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:35.465132 19051 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_d95c70be474e415cba6a345438fad1ba"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240430 20:11:35.834254 18763 raft_consensus.cc:3037] T b72fe1f0713b4076a81a3fc1c74f7fa7 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
W20240430 20:11:35.473341 18983 ts_tablet_manager.cc:1430] T acd23f0a63d5448aaa7ed78422e67bad P 5e1c6397f09747f091534faf93542710: Tablet startup took 658ms
W20240430 20:11:35.834699 18983 ts_tablet_manager.cc:1432] T acd23f0a63d5448aaa7ed78422e67bad P 5e1c6397f09747f091534faf93542710: Trace:
0430 20:11:26.946233 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:26.946290 (+ 57us) service_pool.cc:225] Handling call
0430 20:11:26.947217 (+ 927us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0430 20:11:26.947285 (+ 68us) ts_tablet_manager.cc:655] Creating new metadata...
0430 20:11:26.980499 (+ 33214us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:27.256882 (+276383us) tablet_replica.cc:205] Creating consensus instance
0430 20:11:27.257005 (+ 123us) inbound_call.cc:173] Queueing success response
0430 20:11:34.462765 (+7205760us) ts_tablet_manager.cc:1343] Bootstrapping tablet
0430 20:11:34.798949 (+336184us) ts_tablet_manager.cc:1399] Starting tablet replica
0430 20:11:34.799151 (+ 202us) tablet_replica.cc:259] Starting instrumentation
0430 20:11:34.799351 (+ 200us) tablet_replica.cc:278] Starting consensus
Metrics: {"raft.queue_time_us":13644,"raft.run_cpu_time_us":73,"raft.run_wall_time_us":74,"tablet-open.queue_time_us":7205784}
I20240430 20:11:35.831835 19825 rpcz_store.cc:276] 0430 20:11:33.048279 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:33.049619 (+ 1340us) service_pool.cc:225] Handling call
0430 20:11:34.536713 (+1487094us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:33.050361 (+ 0us) alter_schema_op.cc:96] PREPARE ALTER-SCHEMA: Starting
0430 20:11:33.050470 (+ 109us) alter_schema_op.cc:57] Acquiring schema lock in exclusive mode
0430 20:11:33.050480 (+ 10us) alter_schema_op.cc:59] Acquired schema lock
0430 20:11:33.050487 (+ 7us) alter_schema_op.cc:111] PREPARE ALTER-SCHEMA: finished
0430 20:11:33.050519 (+ 32us) alter_schema_op.cc:119] START. Timestamp: 1714507893050464
0430 20:11:33.050521 (+ 2us) op_driver.cc:331] REPLICATION: starting
0430 20:11:33.050696 (+ 175us) log.cc:842] Serialized 173 byte log entry
0430 20:11:34.144117 (+1093421us) op_driver.cc:444] REPLICATION: finished
0430 20:11:34.161825 (+ 17708us) alter_schema_op.cc:124] APPLY ALTER-SCHEMA: Starting
0430 20:11:34.456295 (+294470us) log.cc:842] Serialized 5 byte log entry
0430 20:11:34.535903 (+ 79608us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:34.536622 (+ 719us) log.cc:842] Serialized 14 byte log entry
0430 20:11:34.536668 (+ 46us) alter_schema_op.cc:65] Released schema lock
0430 20:11:34.536672 (+ 4us) alter_schema_op.cc:167] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":17628,"mutex_wait_us":69042,"prepare.queue_time_us":327,"prepare.run_cpu_time_us":60689,"prepare.run_wall_time_us":328699,"raft.queue_time_us":69113,"raft.run_cpu_time_us":523,"raft.run_wall_time_us":106441,"replication_time_us":1093565,"spinlock_wait_cycles":18688,"thread_start_us":346194,"threads_started":3,"wal-append.queue_time_us":324692,"wal-append.run_cpu_time_us":1244,"wal-append.run_wall_time_us":2011213}]]}
I20240430 20:11:35.831918 18475 raft_consensus.cc:2454] T 24dcc20832804229a2664873bf6ddaa7 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:35.835672 18983 tablet_bootstrap.cc:492] T e6eac92dfa5f4456bc49571878f0f8dd P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:35.835785 18475 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.17.177.3:32791 (request call id 161) took 1095 ms. Trace:
I20240430 20:11:35.835898 18475 rpcz_store.cc:276] 0430 20:11:34.740232 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:34.740320 (+ 88us) service_pool.cc:225] Handling call
0430 20:11:35.835753 (+1095433us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240430 20:11:35.836373 18463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 130007acc8ba4e709c3eb85fabe75903. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:35.837311 19617 leader_election.cc:288] T fcc9698abda4491b98e1e879589913a8 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Requested vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:35.837946 18983 tablet_bootstrap.cc:654] T e6eac92dfa5f4456bc49571878f0f8dd P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:35.837878 19784 raft_consensus.cc:504] T 53a899fa5a424a0d99f1b47925e8c7ad P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:35.832407 18415 leader_election.cc:302] T 802336602d8f4ac4bd7e0872d6f3dae9 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:34.743194 19805 consensus_queue.cc:1035] T 294644d7a16b4ca2bbb6d106bcb8c7ff P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:35.838572 19784 leader_election.cc:288] T 53a899fa5a424a0d99f1b47925e8c7ad P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:35.838776 19784 raft_consensus.cc:2781] T 802336602d8f4ac4bd7e0872d6f3dae9 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:35.831481 19781 raft_consensus.cc:504] T d9e578c99058424a9b86390641f9c34f P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:35.838887 19784 raft_consensus.cc:482] T 802336602d8f4ac4bd7e0872d6f3dae9 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:35.838985 19784 raft_consensus.cc:3037] T 802336602d8f4ac4bd7e0872d6f3dae9 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:34.826455 18459 tablet_service.cc:1467] Processing CreateTablet for tablet ed0c37a71ede4bfc9eb5ad1a4365f25e (DEFAULT_TABLE table=default.Table_a2e85efc6b454524b0b122102ec07002 [id=86d795a82e674d859104741ebd04f4e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:34.847051 18456 tablet_service.cc:1467] Processing CreateTablet for tablet 12795670f21447a8964fe1efc6e92398 (DEFAULT_TABLE table=default.Table_421069f84cb844368066c93102a8faf8 [id=d5cea90cbc2c48df9d8b80da1eccc949]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:35.466126 19033 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_20f3b06af23d476f8ebc0471d08f8659" } new_table_name: "default.Table_e860191f401c434e8cf0085ef41952b9" modify_external_catalogs: true
I20240430 20:11:35.839442 19781 leader_election.cc:288] T d9e578c99058424a9b86390641f9c34f P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:35.839586 18456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12795670f21447a8964fe1efc6e92398. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:35.474247 19474 raft_consensus.cc:504] T acd23f0a63d5448aaa7ed78422e67bad P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:35.838754 18415 leader_election.cc:302] T b72fe1f0713b4076a81a3fc1c74f7fa7 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:35.841069 19474 leader_election.cc:288] T acd23f0a63d5448aaa7ed78422e67bad P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:34.759622 19880 tablet.cc:1722] T aec9c09cc0454acab6006bb3e10982b7 P 3c51c33d5616482cafb68e26688820f4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240430 20:11:35.467813 19029 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_c06225c5a599482dab4f418fd13c593f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:35.474102 18558 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.177.1:41217, user_credentials={real_user=slave}} blocked reactor thread for 646011us
W20240430 20:11:34.741967 18417 leader_election.cc:341] T f77086a5675f44e99e8a867b694df4b8 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Not found: Tablet not found: f77086a5675f44e99e8a867b694df4b8
I20240430 20:11:35.542516 18582 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:40186 (request call id 94) took 2794 ms. Trace:
W20240430 20:11:35.842725 18417 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.177.3:44867, user_credentials={real_user=slave}} blocked reactor thread for 1.10065e+06us
I20240430 20:11:35.468765 19023 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_f323a3bead0249b5a4a3cbb07a5363b8" } new_table_name: "default.Table_ba82d0fed0264b818de8278e1d142e5d" modify_external_catalogs: true
I20240430 20:11:35.842857 18582 rpcz_store.cc:276] 0430 20:11:32.743029 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:32.743106 (+ 77us) service_pool.cc:225] Handling call
0430 20:11:32.869998 (+126892us) spinlock_profiling.cc:243] Waited 19.2 ms on lock 0x6180001d108c. stack: 00007ffa8b97e756 00007ffa8b97e510 00007ffa8a9b148d 0000000000955804 00000000009557cd 000000000095519a 00007ffa986c9d0e 00007ffa984d046f 00007ffa9aecd8c8 00007ffa956508e6 00007ffa95650615 00007ffa9268604e 00007ffa92684f2e 00007ffa9268b405 00007ffa9268bd2f 00007ffa9268bb58
0430 20:11:33.385125 (+515127us) spinlock_profiling.cc:243] Waited 452 ms on lock 0x6160000a4e54. stack: 00007ffa8b97e756 00007ffa8b97e510 00007ffa8a9b148d 0000000000955804 00000000009557cd 000000000095519a 00007ffa971bcb00 00007ffa9aecdabf 00007ffa956508e6 00007ffa95650615 00007ffa9268604e 00007ffa92684f2e 00007ffa9268b405 00007ffa9268bd2f 00007ffa9268bb58 0000000000a60bcc
0430 20:11:33.406376 (+ 21251us) spinlock_profiling.cc:243] Waited 19.4 ms on lock 0x6180000a408c. stack: 00007ffa8b97e756 00007ffa8b97e510 00007ffa8a9b148d 0000000000955804 00000000009557cd 000000000095519a 00007ffa986c9d0e 00007ffa984d046f 00007ffa9aecd8c8 00007ffa956508e6 00007ffa95650615 00007ffa9268604e 00007ffa92684f2e 00007ffa9268b405 00007ffa9268bd2f 00007ffa9268bb58
0430 20:11:33.439104 (+ 32728us) spinlock_profiling.cc:243] Waited 30.2 ms on lock 0x616000260254. stack: 00007ffa8b97e756 00007ffa8b97e510 00007ffa8a9b148d 0000000000955804 00000000009557cd 000000000095519a 00007ffa971bcb00 00007ffa9aecdabf 00007ffa956508e6 00007ffa95650615 00007ffa9268604e 00007ffa92684f2e 00007ffa9268b405 00007ffa9268bd2f 00007ffa9268bb58 0000000000a60bcc
0430 20:11:34.260487 (+821383us) spinlock_profiling.cc:243] Waited 782 ms on lock 0x616000240454. stack: 00007ffa8b97e756 00007ffa8b97e510 00007ffa8a9b148d 0000000000955804 00000000009557cd 000000000095519a 00007ffa971bcb00 00007ffa9aecdabf 00007ffa956508e6 00007ffa95650615 00007ffa9268604e 00007ffa92684f2e 00007ffa9268b405 00007ffa9268bd2f 00007ffa9268bb58 0000000000a60bcc
0430 20:11:34.629615 (+369128us) spinlock_profiling.cc:243] Waited 127 ms on lock 0x6160001b0a54. stack: 00007ffa8b97e756 00007ffa8b97e510 00007ffa8a9b148d 0000000000955804 00000000009557cd 000000000095519a 00007ffa971bcb00 00007ffa9aecdabf 00007ffa956508e6 00007ffa95650615 00007ffa9268604e 00007ffa92684f2e 00007ffa9268b405 00007ffa9268bd2f 00007ffa9268bb58 0000000000a60bcc
0430 20:11:35.537886 (+908271us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3286061952}
W20240430 20:11:35.179620 18404 kernel_stack_watchdog.cc:198] Thread 18417 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:308 for 209ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7f9c67949980 (unknown) at ??:0
@ 0x7f9c679439c6 __GI___pthread_rwlock_wrlock at ??:0
@ 0x7f9c618c9904 google::LogMessage::Flush() at ??:0
@ 0x7f9c618c9a11 google::LogMessage::~LogMessage() at ??:0
@ 0x7f9c6fd705d3 kudu::consensus::LeaderElection::VoteResponseRpcCallback() at ??:0
@ 0x7f9c6fd7312a kudu::consensus::LeaderElection::Run()::$_0::operator()() at ??:0
@ 0x7f9c6fd72eee std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7f9c6b2098cf kudu::rpc::Proxy::AsyncRequest()::$_1::operator()() at ??:0
@ 0x7f9c6b2094ae std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7f9c6b1f54e0 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7f9c6b1f5a2c kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7f9c6b19251a kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7f9c6b190f45 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7f9c6b1a32c5 ev::base<>::method_thunk<>() at ??:0
I20240430 20:11:35.466679 19778 raft_consensus.cc:482] T a05e32a33aa7488eb287cc6314c09c44 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:35.843534 19778 raft_consensus.cc:504] T a05e32a33aa7488eb287cc6314c09c44 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:35.471253 19048 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_2f122c3c1c5949108231e55f6326e934" } modify_external_catalogs: true
I20240430 20:11:35.664183 19471 ts_tablet_manager.cc:1888] T cace1398492f4d968dcb6899385d527c P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:35.565922 18591 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40218 (request call id 172) took 1671 ms. Trace:
I20240430 20:11:35.844146 19778 leader_election.cc:288] T a05e32a33aa7488eb287cc6314c09c44 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:35.844182 19471 log.cc:1196] T cace1398492f4d968dcb6899385d527c P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/cace1398492f4d968dcb6899385d527c
I20240430 20:11:35.844312 19778 raft_consensus.cc:2781] T b72fe1f0713b4076a81a3fc1c74f7fa7 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:35.844202 18591 rpcz_store.cc:276] 0430 20:11:33.893980 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:33.894061 (+ 81us) service_pool.cc:225] Handling call
0430 20:11:34.135127 (+241066us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0430 20:11:34.135180 (+ 53us) ts_tablet_manager.cc:655] Creating new metadata...
0430 20:11:35.541526 (+1406346us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:35.563822 (+ 22296us) tablet_replica.cc:205] Creating consensus instance
0430 20:11:35.565182 (+ 1360us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240430 20:11:35.844457 19778 raft_consensus.cc:686] T b72fe1f0713b4076a81a3fc1c74f7fa7 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Becoming Leader. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Running, Role: LEADER
I20240430 20:11:35.832882 18483 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.17.177.2:33361 (request call id 170) took 1687 ms. Trace:
I20240430 20:11:35.844691 19471 ts_tablet_manager.cc:1909] T cace1398492f4d968dcb6899385d527c P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:35.844671 18483 rpcz_store.cc:276] 0430 20:11:34.144643 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:34.144714 (+ 71us) service_pool.cc:225] Handling call
0430 20:11:35.831776 (+1687062us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240430 20:11:35.472615 19049 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_8e605def35724c5c945b61107596d81a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240430 20:11:35.844749 19778 consensus_queue.cc:237] T b72fe1f0713b4076a81a3fc1c74f7fa7 P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
W20240430 20:11:35.486788 19004 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet d61c279d20b0471a967942525ceb4fcb (table default.Table_1177828cad1f4a24bd397619dad96170 [id=a95fed1bc3fa4865bf9a35dfc51b659c], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet d61c279d20b0471a967942525ceb4fcb consensus state has no leader
W20240430 20:11:35.329202 18685 kernel_stack_watchdog.cc:198] Thread 18699 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:308 for 466ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7f30eb118980 (unknown) at ??:0
@ 0x7f30e6d96a61 sys_futex() at ??:0
@ 0x7f30e6d968df base::internal::SpinLockDelay() at ??:0
@ 0x7f30e6d963ae base::SpinLock::SlowLock() at ??:0
@ 0x955226 base::SpinLock::Lock() at ??:?
@ 0x9551ee kudu::simple_spinlock::lock() at ??:?
@ 0x7f30f68ac741 std::unique_lock<>::lock() at ??:0
@ 0x7f30f68a1fbf std::unique_lock<>::unique_lock() at ??:0
@ 0x7f30f35003a2 kudu::consensus::PeerMessageQueue::UpdatePeerStatus() at ??:0
@ 0x7f30f34d4ad2 kudu::consensus::Peer::ProcessResponse() at ??:0
@ 0x7f30f34db061 kudu::consensus::Peer::SendNextRequest()::$_3::operator()() at ??:0
@ 0x7f30f34dae6e std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7f30ee9d88cf kudu::rpc::Proxy::AsyncRequest()::$_1::operator()() at ??:0
@ 0x7f30ee9d84ae std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
I20240430 20:11:35.466985 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 63) took 4356 ms. Trace:
I20240430 20:11:35.845768 18443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dad16ae02b8b4e29a724d8eec67bb4b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:35.845731 19495 rpcz_store.cc:276] 0430 20:11:31.110583 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:31.110650 (+ 67us) service_pool.cc:225] Handling call
0430 20:11:34.070179 (+2959529us) ts_tablet_manager.cc:1120] Deleting tablet 4e69ca12bd10494fb013614741bc166c
0430 20:11:34.070190 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:34.076820 (+ 6630us) spinlock_profiling.cc:243] Waited 6.05 ms on lock 0x616000240454. stack: 00007f9c64594756 00007f9c64594510 00007f9c635c748d 0000000000955804 00000000009557cd 000000000095519a 00007f9c6fe02a00 00007f9c710dc853 00007f9c73b6f4bc 00007f9c73ba1814 00007f9c73b82b00 00007f9c73b8288d 0000000000a60bcc 00007f9c645dab74 00007f9c645de19b 00007f9c645de018
0430 20:11:34.122535 (+ 45715us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:34.229260 (+106725us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:35.466933 (+1237673us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":13911424,"tablet-delete.queue_time_us":2959308}
I20240430 20:11:35.846148 19495 tablet_replica.cc:330] T 34ab40d8d91e401db299ddee7175ba28 P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
W20240430 20:11:35.465523 18699 consensus_peers.cc:485] T 01436e5b6232494babbb414342f03090 P 3c51c33d5616482cafb68e26688820f4 -> Peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Couldn't send request to peer 2efb542aa55e439b99ccb98d1c5c884a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20240430 20:11:35.846271 18699 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.177.1:41217, user_credentials={real_user=slave}} blocked reactor thread for 1.10265e+06us
I20240430 20:11:35.846361 19495 raft_consensus.cc:2229] T 34ab40d8d91e401db299ddee7175ba28 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:35.846567 19495 raft_consensus.cc:2258] T 34ab40d8d91e401db299ddee7175ba28 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:35.846801 18699 leader_election.cc:302] T 24dcc20832804229a2664873bf6ddaa7 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters: 5e1c6397f09747f091534faf93542710
I20240430 20:11:35.847186 19495 ts_tablet_manager.cc:1875] T 34ab40d8d91e401db299ddee7175ba28 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:35.848958 19581 consensus_queue.cc:1035] T a05e32a33aa7488eb287cc6314c09c44 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:35.472332 19726 consensus_queue.cc:1035] T eed6338ad3d64859af087ee15e1ec798 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240430 20:11:35.477171 18985 tablet_bootstrap.cc:492] T 76d6e9d547ca4a848eae852944b1d117 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:35.849606 19581 raft_consensus.cc:2781] T 24dcc20832804229a2664873bf6ddaa7 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:35.849704 19581 raft_consensus.cc:482] T 24dcc20832804229a2664873bf6ddaa7 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:35.849807 19581 raft_consensus.cc:3037] T 24dcc20832804229a2664873bf6ddaa7 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:35.462421 18730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70247a4c910946c0afc24ce5ce0e85d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:35.850335 18459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed0c37a71ede4bfc9eb5ad1a4365f25e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:35.851111 18985 tablet_bootstrap.cc:654] T 76d6e9d547ca4a848eae852944b1d117 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:35.854640 18759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcc9698abda4491b98e1e879589913a8" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:35.472656 18764 raft_consensus.cc:2379] T 75ebf051383e42529dcb5f7dd0770a4b P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 5e1c6397f09747f091534faf93542710 in current term 1: Already voted for candidate 3c51c33d5616482cafb68e26688820f4 in this term.
I20240430 20:11:35.512701 18726 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59830 (request call id 165) took 1633 ms. Trace:
I20240430 20:11:35.512841 18744 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59830 (request call id 161) took 2031 ms. Trace:
I20240430 20:11:35.855342 18764 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.17.177.2:37173 (request call id 159) took 1217 ms. Trace:
I20240430 20:11:35.550956 19923 raft_consensus.cc:482] T e6eac92dfa5f4456bc49571878f0f8dd P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:35.855473 18744 rpcz_store.cc:276] 0430 20:11:33.507059 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:33.518872 (+ 11813us) service_pool.cc:225] Handling call
0430 20:11:34.098784 (+579912us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0430 20:11:34.098828 (+ 44us) ts_tablet_manager.cc:655] Creating new metadata...
0430 20:11:35.464331 (+1365503us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:35.509629 (+ 45298us) tablet_replica.cc:205] Creating consensus instance
0430 20:11:35.511851 (+ 2222us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":329088}
I20240430 20:11:35.855553 19923 raft_consensus.cc:504] T e6eac92dfa5f4456bc49571878f0f8dd P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:35.856025 18744 tablet_service.cc:1467] Processing CreateTablet for tablet 1701a15b115741aea13f8327e3d26b34 (DEFAULT_TABLE table=default.Table_0fcc91de96a946018c72ba03c71bcc05 [id=7127dbd368864f7ba80a755549fe7ede]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:35.855305 18726 rpcz_store.cc:276] 0430 20:11:33.878635 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:33.878711 (+ 76us) service_pool.cc:225] Handling call
0430 20:11:34.463694 (+584983us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0430 20:11:34.463727 (+ 33us) ts_tablet_manager.cc:655] Creating new metadata...
0430 20:11:35.466150 (+1002423us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:35.510318 (+ 44168us) tablet_replica.cc:205] Creating consensus instance
0430 20:11:35.511932 (+ 1614us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240430 20:11:35.856328 19923 leader_election.cc:288] T e6eac92dfa5f4456bc49571878f0f8dd P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:35.856633 18726 tablet_service.cc:1467] Processing CreateTablet for tablet 558e71ec370d46ca8fbdcec5b57eb2ec (DEFAULT_TABLE table=default.Table_408be016c50f4d0fa19189bbaddab089 [id=6b7bd5a6c20e4b43aa62365627503d5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:35.510478 18729 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59830 (request call id 167) took 1627 ms. Trace:
I20240430 20:11:35.857050 18729 rpcz_store.cc:276] 0430 20:11:33.881818 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:33.881864 (+ 46us) service_pool.cc:225] Handling call
0430 20:11:34.464036 (+582172us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0430 20:11:34.464055 (+ 19us) ts_tablet_manager.cc:655] Creating new metadata...
0430 20:11:35.448252 (+984197us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:35.507538 (+ 59286us) tablet_replica.cc:205] Creating consensus instance
0430 20:11:35.509604 (+ 2066us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240430 20:11:35.857491 18729 tablet_service.cc:1467] Processing CreateTablet for tablet f90a5c5818c54390a2ae323fed7ebe57 (DEFAULT_TABLE table=default.Table_18385e32857b4c8bb1e56a85c590153a [id=38a7d7d4f9e64c07b784d9393192ac94]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:35.564304 18738 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59830 (request call id 164) took 1686 ms. Trace:
I20240430 20:11:35.857865 18738 rpcz_store.cc:276] 0430 20:11:33.877106 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:33.877156 (+ 50us) service_pool.cc:225] Handling call
0430 20:11:35.450277 (+1573121us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0430 20:11:35.450302 (+ 25us) ts_tablet_manager.cc:655] Creating new metadata...
0430 20:11:35.453588 (+ 3286us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:35.562547 (+108959us) tablet_replica.cc:205] Creating consensus instance
0430 20:11:35.563770 (+ 1223us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240430 20:11:35.858274 18738 tablet_service.cc:1467] Processing CreateTablet for tablet a445fdfde01c4371a5e6b3daea5b00ab (DEFAULT_TABLE table=default.Table_83d417131d35409582550b7b8ce4a141 [id=8893c7824ec64fc1961e7c6851278a50]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:35.598309 18731 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59830 (request call id 175) took 1708 ms. Trace:
I20240430 20:11:35.858975 18731 rpcz_store.cc:276] 0430 20:11:33.889379 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:33.889425 (+ 46us) service_pool.cc:225] Handling call
0430 20:11:34.465261 (+575836us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0430 20:11:34.465278 (+ 17us) ts_tablet_manager.cc:655] Creating new metadata...
0430 20:11:35.480304 (+1015026us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:35.596747 (+116443us) tablet_replica.cc:205] Creating consensus instance
0430 20:11:35.598199 (+ 1452us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240430 20:11:35.859058 18611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b72fe1f0713b4076a81a3fc1c74f7fa7" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:35.859239 18731 tablet_service.cc:1514] Processing DeleteTablet for tablet c66c142419ca4e1399ecd1c8d1e62a59 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:34 UTC) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:35.859401 18611 raft_consensus.cc:2379] T b72fe1f0713b4076a81a3fc1c74f7fa7 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c51c33d5616482cafb68e26688820f4 in current term 1: Already voted for candidate 2efb542aa55e439b99ccb98d1c5c884a in this term.
I20240430 20:11:35.859436 18731 tablet_service.cc:1514] Processing DeleteTablet for tablet 84568f90b9c54cf292099808d4cbf5dc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:34 UTC) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:35.859629 18731 tablet_service.cc:1514] Processing DeleteTablet for tablet 29974dc2c7dd45b99b6cc0712378fc30 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:26 UTC) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:35.859784 18731 tablet_service.cc:1514] Processing DeleteTablet for tablet 6db55bb510b24b0cae897936211e15f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:34 UTC) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:35.860109 18616 raft_consensus.cc:3037] T e3e74e76ab2b4fe492ffcc1838656fd1 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:35.860122 18478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75ebf051383e42529dcb5f7dd0770a4b" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
W20240430 20:11:35.860265 18731 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:59830: Illegal state: replica 3c51c33d5616482cafb68e26688820f4 is not leader of this config: current role FOLLOWER
I20240430 20:11:35.860430 18731 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:59830 (request call id 195) took 1061 ms. Trace:
I20240430 20:11:35.860966 18478 raft_consensus.cc:2379] T 75ebf051383e42529dcb5f7dd0770a4b P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 3c51c33d5616482cafb68e26688820f4 in current term 1: Already voted for candidate 5e1c6397f09747f091534faf93542710 in this term.
I20240430 20:11:35.860980 18731 rpcz_store.cc:276] 0430 20:11:34.798761 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:35.859874 (+1061113us) service_pool.cc:225] Handling call
0430 20:11:35.860399 (+ 525us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:35.860209 (+ 0us) op_driver.cc:372] HandleFailure(Illegal state: replica 3c51c33d5616482cafb68e26688820f4 is not leader of this config: current role FOLLOWER)
0430 20:11:35.860217 (+ 8us) alter_schema_op.cc:152] AlterSchemaCommitCallback: op aborted
Metrics: {"child_traces":[["op",{}]]}
I20240430 20:11:35.861321 18482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b72fe1f0713b4076a81a3fc1c74f7fa7" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:35.861608 18731 tablet_service.cc:1467] Processing CreateTablet for tablet 47e98935eb21449abd3d4d24327778d5 (DEFAULT_TABLE table=default.Table_67d51131549440cc99629dda924693f7 [id=842d2b7e31cb4e7793d88103997a88d8]), partition=RANGE (key) PARTITION UNBOUNDED
W20240430 20:11:35.861938 19051 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d95c70be474e415cba6a345438fad1ba in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:35.611588 18743 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59830 (request call id 172) took 1726 ms. Trace:
W20240430 20:11:35.862668 19029 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c06225c5a599482dab4f418fd13c593f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:35.862871 18999 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet 34ab40d8d91e401db299ddee7175ba28 (table default.Table_0b4b99c1c03a4e779046063159ce6861 [id=43d7bdc84bab4c54bac58dd1dd0f74ac]) successfully deleted
I20240430 20:11:35.842990 18618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9e7660eccc448dbb99d135f82bfd261" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:35.891690 18557 leader_election.cc:302] T e3e74e76ab2b4fe492ffcc1838656fd1 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4
W20240430 20:11:35.892729 18557 consensus_peers.cc:485] T 26ed12f498644d15bf45e884dcad172e P 5e1c6397f09747f091534faf93542710 -> Peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Couldn't send request to peer 3c51c33d5616482cafb68e26688820f4. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240430 20:11:35.893918 18617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75ebf051383e42529dcb5f7dd0770a4b" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:35.893925 18618 raft_consensus.cc:2454] T c9e7660eccc448dbb99d135f82bfd261 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:35.894227 18617 raft_consensus.cc:2379] T 75ebf051383e42529dcb5f7dd0770a4b P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 3c51c33d5616482cafb68e26688820f4 in current term 1: Already voted for candidate 5e1c6397f09747f091534faf93542710 in this term.
I20240430 20:11:35.898694 19861 raft_consensus.cc:504] T 53a899fa5a424a0d99f1b47925e8c7ad P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:35.933039 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 65) took 4856 ms. Trace:
I20240430 20:11:35.940979 18591 tablet_service.cc:1467] Processing CreateTablet for tablet f3711610610542128c4a84e17bce4a3e (DEFAULT_TABLE table=default.Table_64d14df4e12146ecb85239e7ca03cf6e [id=c376119401934812aef62a3a18eeb368]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:35.941335 18602 tablet_service.cc:1467] Processing CreateTablet for tablet f90a5c5818c54390a2ae323fed7ebe57 (DEFAULT_TABLE table=default.Table_18385e32857b4c8bb1e56a85c590153a [id=38a7d7d4f9e64c07b784d9393192ac94]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:35.941558 18591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3711610610542128c4a84e17bce4a3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:35.941797 18602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f90a5c5818c54390a2ae323fed7ebe57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:35.941972 18595 tablet_service.cc:1467] Processing CreateTablet for tablet 1701a15b115741aea13f8327e3d26b34 (DEFAULT_TABLE table=default.Table_0fcc91de96a946018c72ba03c71bcc05 [id=7127dbd368864f7ba80a755549fe7ede]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:36.163022 20011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.177.62:36441 every 8 connection(s)
I20240430 20:11:35.862608 18743 rpcz_store.cc:276] 0430 20:11:33.884881 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:33.885297 (+ 416us) service_pool.cc:225] Handling call
0430 20:11:34.464788 (+579491us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0430 20:11:34.464841 (+ 53us) ts_tablet_manager.cc:655] Creating new metadata...
0430 20:11:35.484008 (+1019167us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:35.611297 (+127289us) tablet_replica.cc:205] Creating consensus instance
0430 20:11:35.611522 (+ 225us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240430 20:11:35.865603 18444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70247a4c910946c0afc24ce5ce0e85d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:35.864246 19048 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_2f122c3c1c5949108231e55f6326e934" } modify_external_catalogs: true
I20240430 20:11:36.163527 19519 rpc_server.cc:307] RPC server started. Bound to: 127.17.177.62:36441
I20240430 20:11:36.935261 18743 tablet_service.cc:1467] Processing CreateTablet for tablet 0368d37dd9e2452982fb81f539c6805b (DEFAULT_TABLE table=default.Table_3be3b20c86ea424fbc173dd32e01f90c [id=4330626e342a48699ea939add5c91b11]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:35.664328 19843 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:59830 (request call id 178) took 1660 ms. Trace:
W20240430 20:11:35.864529 19049 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8e605def35724c5c945b61107596d81a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:36.935699 19843 rpcz_store.cc:276] 0430 20:11:34.003751 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:34.003875 (+ 124us) service_pool.cc:225] Handling call
0430 20:11:35.664262 (+1660387us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:34.005076 (+ 0us) alter_schema_op.cc:96] PREPARE ALTER-SCHEMA: Starting
0430 20:11:34.005482 (+ 406us) alter_schema_op.cc:57] Acquiring schema lock in exclusive mode
0430 20:11:34.005492 (+ 10us) alter_schema_op.cc:59] Acquired schema lock
0430 20:11:34.005529 (+ 37us) alter_schema_op.cc:111] PREPARE ALTER-SCHEMA: finished
0430 20:11:34.005607 (+ 78us) alter_schema_op.cc:119] START. Timestamp: 1714507894005524
0430 20:11:34.005610 (+ 3us) op_driver.cc:331] REPLICATION: starting
0430 20:11:34.124149 (+118539us) spinlock_profiling.cc:243] Waited 118 ms on lock 0x6160002adfb8. stack: 00007f30e7d63756 00007f30e7d63510 00007f30e6d9648d 0000000000955804 00000000009557cd 000000000095519a 00007f30f3504b35 00007f30f35ccbdb 00007f30f35a8758 00007f30f35a8005 00007f30f4910b02 00007f30f490f629 00007f30f491753b 00007f30f49173b8 0000000000a60bcc 00007f30e7da9b74
0430 20:11:34.124415 (+ 266us) log.cc:842] Serialized 173 byte log entry
0430 20:11:34.347713 (+223298us) op_driver.cc:444] REPLICATION: finished
0430 20:11:34.348577 (+ 864us) alter_schema_op.cc:124] APPLY ALTER-SCHEMA: Starting
0430 20:11:35.448672 (+1100095us) log.cc:842] Serialized 5 byte log entry
0430 20:11:35.663674 (+215002us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:35.664177 (+ 503us) log.cc:842] Serialized 14 byte log entry
0430 20:11:35.664222 (+ 45us) alter_schema_op.cc:65] Released schema lock
0430 20:11:35.664226 (+ 4us) alter_schema_op.cc:167] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":649,"prepare.queue_time_us":610,"prepare.run_cpu_time_us":20269,"prepare.run_wall_time_us":285410,"replication_time_us":342074,"spinlock_wait_cycles":271055360,"thread_start_us":282380,"threads_started":4,"wal-append.queue_time_us":211895,"wal-append.run_cpu_time_us":1326,"wal-append.run_wall_time_us":2204968}]]}
I20240430 20:11:35.869714 18475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9e7660eccc448dbb99d135f82bfd261" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:35.612612 18741 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59830 (request call id 170) took 1729 ms. Trace:
I20240430 20:11:36.947162 18116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNG3HqZ/build/asan/bin/kudu as pid 19519
I20240430 20:11:35.942461 18595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1701a15b115741aea13f8327e3d26b34. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:36.936501 18741 rpcz_store.cc:276] 0430 20:11:33.883520 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:33.883903 (+ 383us) service_pool.cc:225] Handling call
0430 20:11:34.464505 (+580602us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0430 20:11:34.464525 (+ 20us) ts_tablet_manager.cc:655] Creating new metadata...
0430 20:11:35.453197 (+988672us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:35.612405 (+159208us) tablet_replica.cc:205] Creating consensus instance
0430 20:11:35.612546 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240430 20:11:35.865970 18733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dad16ae02b8b4e29a724d8eec67bb4b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:37.005254 18741 tablet_service.cc:1467] Processing CreateTablet for tablet 2a08e36c39824abf98b9b6214c8b84cb (DEFAULT_TABLE table=default.Table_027891085c5747d58ddb32ad51a453ba [id=45042e2e54a14f558c9968969d3618c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:37.005699 18741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a08e36c39824abf98b9b6214c8b84cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:36.060935 19074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 486 } } ignore_live_leader: false dest_uuid: "b632d6ef4c104a058a6db208024cab33" is_pre_election: true
W20240430 20:11:37.006429 19074 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51648 (request call id 1) took 1027 ms (client timeout 531 ms). Trace:
I20240430 20:11:35.880036 18481 raft_consensus.cc:1261] T 12c6e131fa904d23865b8673b58a7e33 P 2efb542aa55e439b99ccb98d1c5c884a [term 2 FOLLOWER]: Refusing update from remote peer 5e1c6397f09747f091534faf93542710: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20240430 20:11:37.006860 18481 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.17.177.2:33361 (request call id 182) took 1127 ms. Trace:
I20240430 20:11:37.006968 18481 rpcz_store.cc:276] 0430 20:11:35.879545 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:35.879800 (+ 255us) service_pool.cc:225] Handling call
0430 20:11:35.879893 (+ 93us) raft_consensus.cc:1457] Updating replica for 0 ops
0430 20:11:37.006685 (+1126792us) raft_consensus.cc:1683] Filling consensus response to leader.
0430 20:11:37.006795 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240430 20:11:36.085119 19026 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 34130f2820064fa0898ef1f9d7de1c6d consensus state has no leader
I20240430 20:11:37.007555 19026 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:51664 (request call id 157) took 2996 ms. Trace:
I20240430 20:11:37.007658 19026 rpcz_store.cc:276] 0430 20:11:34.011092 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:34.782534 (+771442us) service_pool.cc:225] Handling call
0430 20:11:34.809834 (+ 27300us) spinlock_profiling.cc:243] Waited 26.9 ms on lock 0x616000040654. stack: 00007feb16c74756 00007feb16c74510 00007feb15ca748d 0000000000955804 00000000009557cd 00007feb257b315d 00007feb257b3253 00007feb224e5d41 00007feb2a0d134d 00007feb2a25ff46 00007feb21049c26 00007feb21049955 00007feb1d97c04e 00007feb1d97af2e 00007feb1d981405 00007feb1d981d2f
0430 20:11:35.457669 (+647835us) spinlock_profiling.cc:243] Waited 13 ms on lock 0x6170000091e0. stack: 00007feb16c74756 00007feb16c74510 00007feb15ca748d 0000000000955804 00000000009557cd 000000000095519a 00007feb16c6bff7 00007feb2a06b5c6 00007feb2a2600a9 00007feb21049c26 00007feb21049955 00007feb1d97c04e 00007feb1d97af2e 00007feb1d981405 00007feb1d981d2f 00007feb1d981b58
0430 20:11:35.457694 (+ 25us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0430 20:11:35.457981 (+ 287us) catalog_manager.cc:3613] Apply alter schema
0430 20:11:35.458204 (+ 223us) catalog_manager.cc:3636] Acquired catalog manager lock
0430 20:11:35.458315 (+ 111us) catalog_manager.cc:3783] Updating metadata on disk
0430 20:11:36.084789 (+626474us) catalog_manager.cc:3814] Committing alterations to in-memory state
0430 20:11:37.007517 (+922728us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:35.515973 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:35.516363 (+ 390us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:35.516417 (+ 54us) write_op.cc:427] Acquired schema lock
0430 20:11:35.516422 (+ 5us) tablet.cc:660] Decoding operations
0430 20:11:35.516796 (+ 374us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:35.516867 (+ 71us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:35.516875 (+ 8us) tablet.cc:683] Acquiring locks for 1 operations
0430 20:11:35.517169 (+ 294us) tablet.cc:699] Row locks acquired
0430 20:11:35.517173 (+ 4us) write_op.cc:260] PREPARE: finished
0430 20:11:35.517313 (+ 140us) write_op.cc:270] Start()
0430 20:11:35.517404 (+ 91us) write_op.cc:276] Timestamp: P: 1714507895517233 usec, L: 0
0430 20:11:35.517464 (+ 60us) op_driver.cc:331] REPLICATION: starting
0430 20:11:35.518299 (+ 835us) log.cc:842] Serialized 405 byte log entry
0430 20:11:36.004767 (+486468us) op_driver.cc:444] REPLICATION: finished
0430 20:11:36.081791 (+ 77024us) write_op.cc:301] APPLY: starting
0430 20:11:36.081940 (+ 149us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:36.082208 (+ 268us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:36.082576 (+ 368us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:36.083438 (+ 862us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:36.083534 (+ 96us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0430 20:11:36.083595 (+ 61us) write_op.cc:312] APPLY: finished
0430 20:11:36.083968 (+ 373us) log.cc:842] Serialized 23 byte log entry
0430 20:11:36.084484 (+ 516us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:36.084555 (+ 71us) write_op.cc:446] Released schema lock
0430 20:11:36.084564 (+ 9us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:36.085330 (+ 766us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:36.085336 (+ 6us) write_op.cc:446] Released schema lock
Metrics: {"spinlock_wait_cycles":91601152,"child_traces":[["op",{"apply.queue_time_us":76638,"apply.run_cpu_time_us":1258,"apply.run_wall_time_us":3838,"num_ops":1,"prepare.queue_time_us":55956,"prepare.run_cpu_time_us":32055,"prepare.run_wall_time_us":138184,"raft.queue_time_us":41954,"raft.run_cpu_time_us":20,"raft.run_wall_time_us":20,"replication_time_us":487266,"thread_start_us":265938,"threads_started":3}]]}
I20240430 20:11:35.888188 18472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9e7660eccc448dbb99d135f82bfd261" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:37.009364 18472 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.17.177.2:33361 (request call id 190) took 1121 ms. Trace:
I20240430 20:11:37.009482 18472 rpcz_store.cc:276] 0430 20:11:35.887744 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:35.888075 (+ 331us) service_pool.cc:225] Handling call
0430 20:11:37.009328 (+1121253us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240430 20:11:36.133299 19004 catalog_manager.cc:4830] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): tablet 4e69ca12bd10494fb013614741bc166c (table default.Table_0a397bb1e5ea4cbe87a088c7fc84e9c1 [id=9abcfa9eaaa84cd2b679fd29ec6e5ab7]) successfully deleted
W20240430 20:11:37.009786 19004 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.17.177.1:41217, user_credentials={real_user=slave}} blocked reactor thread for 876457us
I20240430 20:11:35.942881 19861 leader_election.cc:288] T 53a899fa5a424a0d99f1b47925e8c7ad P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:35.867847 18764 rpcz_store.cc:276] 0430 20:11:34.638666 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:34.639058 (+ 392us) service_pool.cc:225] Handling call
0430 20:11:35.855255 (+1216197us) spinlock_profiling.cc:243] Waited 2.96 ms on lock 0x616000210454. stack: 00007f30e7d63756 00007f30e7d63510 00007f30e6d9648d 0000000000955804 00000000009557cd 000000000095519a 00007f30f35c26b5 00007f30f7293c64 00007f30f178e9b6 00007f30f178e6e5 00007f30eea6b04e 00007f30eea69f2e 00007f30eea70405 00007f30eea70d2f 00007f30eea70b58 0000000000a60bcc
0430 20:11:35.855295 (+ 40us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6813440}
I20240430 20:11:35.888931 18473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acd23f0a63d5448aaa7ed78422e67bad" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:35.872368 18754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53a899fa5a424a0d99f1b47925e8c7ad" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:35.948791 18590 tablet_service.cc:1514] Processing DeleteTablet for tablet c66c142419ca4e1399ecd1c8d1e62a59 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:34 UTC) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:37.010859 18754 raft_consensus.cc:2454] T 53a899fa5a424a0d99f1b47925e8c7ad P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 0.
I20240430 20:11:37.010859 18473 raft_consensus.cc:2454] T acd23f0a63d5448aaa7ed78422e67bad P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 0.
I20240430 20:11:35.949122 18585 tablet_service.cc:1514] Processing DeleteTablet for tablet 84568f90b9c54cf292099808d4cbf5dc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:34 UTC) from {username='slave'} at 127.0.0.1:40218
W20240430 20:11:36.223927 19033 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8e91920e7e36418d8c27d1f0bfd9b981 consensus state has no leader
I20240430 20:11:36.936942 19519 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/master-0/data/info.pb
I20240430 20:11:37.447960 19033 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:51664 (request call id 162) took 1987 ms. Trace:
W20240430 20:11:35.981604 18211 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.177.62:36441: connect: Connection refused (error 111) [suppressed 5 similar messages]
I20240430 20:11:37.448163 19033 rpcz_store.cc:276] 0430 20:11:35.460293 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:35.465977 (+ 5684us) service_pool.cc:225] Handling call
0430 20:11:35.839390 (+373413us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0430 20:11:35.839642 (+ 252us) catalog_manager.cc:3613] Apply alter schema
0430 20:11:35.839821 (+ 179us) catalog_manager.cc:3636] Acquired catalog manager lock
0430 20:11:35.839915 (+ 94us) catalog_manager.cc:3783] Updating metadata on disk
0430 20:11:36.223608 (+383693us) catalog_manager.cc:3814] Committing alterations to in-memory state
0430 20:11:37.447895 (+1224287us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:35.844316 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:35.844454 (+ 138us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:35.844463 (+ 9us) write_op.cc:427] Acquired schema lock
0430 20:11:35.844465 (+ 2us) tablet.cc:660] Decoding operations
0430 20:11:35.844587 (+ 122us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:35.844602 (+ 15us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:35.844608 (+ 6us) tablet.cc:683] Acquiring locks for 1 operations
0430 20:11:35.844682 (+ 74us) tablet.cc:699] Row locks acquired
0430 20:11:35.844687 (+ 5us) write_op.cc:260] PREPARE: finished
0430 20:11:35.844728 (+ 41us) write_op.cc:270] Start()
0430 20:11:35.844768 (+ 40us) write_op.cc:276] Timestamp: P: 1714507895844699 usec, L: 0
0430 20:11:35.844773 (+ 5us) op_driver.cc:331] REPLICATION: starting
0430 20:11:35.845001 (+ 228us) log.cc:842] Serialized 405 byte log entry
0430 20:11:36.222436 (+377435us) op_driver.cc:444] REPLICATION: finished
0430 20:11:36.222565 (+ 129us) write_op.cc:301] APPLY: starting
0430 20:11:36.222610 (+ 45us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:36.222667 (+ 57us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:36.223079 (+ 412us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:36.223158 (+ 79us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:36.223209 (+ 51us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0430 20:11:36.223221 (+ 12us) write_op.cc:312] APPLY: finished
0430 20:11:36.223336 (+ 115us) log.cc:842] Serialized 23 byte log entry
0430 20:11:36.223412 (+ 76us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:36.223449 (+ 37us) write_op.cc:446] Released schema lock
0430 20:11:36.223457 (+ 8us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:36.223540 (+ 83us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:36.223543 (+ 3us) write_op.cc:446] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":54,"apply.run_cpu_time_us":685,"apply.run_wall_time_us":1035,"num_ops":1,"prepare.queue_time_us":3839,"prepare.run_cpu_time_us":797,"prepare.run_wall_time_us":797,"raft.queue_time_us":871,"raft.run_cpu_time_us":6,"raft.run_wall_time_us":8,"replication_time_us":377628}]]}
W20240430 20:11:37.031323 18200 kernel_stack_watchdog.cc:198] Thread 18211 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:308 for 136ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7ff49b19c980 (unknown) at ??:0
@ 0x7ff493fbe255 __write at ??:0
@ 0x7ff493f3928d _IO_file_write at ??:0
@ 0x7ff493f39bff _IO_file_xsputn at ??:0
@ 0x7ff493f2da57 fwrite at ??:0
@ 0x8793e7 __interceptor_fwrite.part.56 at ??:?
@ 0x7ff4951208a0 google::LogMessage::SendToLog() at ??:0
@ 0x7ff49511c7cc google::LogMessage::Flush() at ??:0
@ 0x7ff49511ca11 google::LogMessage::~LogMessage() at ??:0
@ 0x7ff49ea5cbf4 kudu::rpc::Proxy::AsyncRequest()::$_1::operator()() at ??:0
@ 0x7ff49ea5c4ae std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7ff49ea484e0 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7ff49ea49365 kudu::rpc::OutboundCall::SetFailed() at ??:0
@ 0x7ff49e9debb3 kudu::rpc::Connection::Shutdown() at ??:0
@ 0x7ff49ea66e51 kudu::rpc::ReactorThread::DestroyConnection() at ??:0
I20240430 20:11:37.449028 19033 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_891f568fffb349c097f751981343f2b1" } new_table_name: "default.Table_4440df38610340758047115845b42b95" modify_external_catalogs: true
I20240430 20:11:35.872968 19489 raft_consensus.cc:2229] T 34ab40d8d91e401db299ddee7175ba28 P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Raft consensus shutting down.
I20240430 20:11:37.011075 18595 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40218 (request call id 182) took 1077 ms. Trace:
I20240430 20:11:37.449891 18595 rpcz_store.cc:276] 0430 20:11:35.933714 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:35.933779 (+ 65us) service_pool.cc:225] Handling call
0430 20:11:35.942199 (+ 8420us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0430 20:11:35.942225 (+ 26us) ts_tablet_manager.cc:655] Creating new metadata...
0430 20:11:37.007913 (+1065688us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:37.010463 (+ 2550us) tablet_replica.cc:205] Creating consensus instance
0430 20:11:37.011005 (+ 542us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240430 20:11:37.011070 18473 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.17.177.2:33361 (request call id 191) took 1122 ms. Trace:
I20240430 20:11:37.450342 18473 rpcz_store.cc:276] 0430 20:11:35.888483 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:35.888810 (+ 327us) service_pool.cc:225] Handling call
0430 20:11:37.011028 (+1122218us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240430 20:11:36.224239 19023 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3ec3993042f5498b818bdf32d5087928 consensus state has no leader
I20240430 20:11:35.950323 19471 rpcz_store.cc:276] 0430 20:11:31.011187 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:31.019054 (+ 7867us) service_pool.cc:225] Handling call
0430 20:11:35.447187 (+4428133us) ts_tablet_manager.cc:1120] Deleting tablet cace1398492f4d968dcb6899385d527c
0430 20:11:35.447197 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:35.564881 (+117684us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:35.664030 (+ 99149us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:35.867399 (+203369us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4427834}
I20240430 20:11:36.276348 19051 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_ff91abfadbc746d6b01fdbfecd16fac5" } modify_external_catalogs: true
I20240430 20:11:35.951277 18586 tablet_service.cc:1467] Processing CreateTablet for tablet a445fdfde01c4371a5e6b3daea5b00ab (DEFAULT_TABLE table=default.Table_83d417131d35409582550b7b8ce4a141 [id=8893c7824ec64fc1961e7c6851278a50]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:37.450822 19023 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:51664 (request call id 165) took 1990 ms. Trace:
I20240430 20:11:37.450914 19471 tablet_replica.cc:330] T 4e69ca12bd10494fb013614741bc166c P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:37.451182 19471 raft_consensus.cc:2229] T 4e69ca12bd10494fb013614741bc166c P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:37.451231 18586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a445fdfde01c4371a5e6b3daea5b00ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:37.451365 19471 raft_consensus.cc:2258] T 4e69ca12bd10494fb013614741bc166c P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:37.450968 19023 rpcz_store.cc:276] 0430 20:11:35.460571 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:35.468538 (+ 7967us) service_pool.cc:225] Handling call
0430 20:11:35.842926 (+374388us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0430 20:11:35.843154 (+ 228us) catalog_manager.cc:3613] Apply alter schema
0430 20:11:35.843339 (+ 185us) catalog_manager.cc:3636] Acquired catalog manager lock
0430 20:11:35.843438 (+ 99us) catalog_manager.cc:3783] Updating metadata on disk
0430 20:11:36.224051 (+380613us) catalog_manager.cc:3814] Committing alterations to in-memory state
0430 20:11:37.450753 (+1226702us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:35.845136 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:35.845213 (+ 77us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:35.845218 (+ 5us) write_op.cc:427] Acquired schema lock
0430 20:11:35.845220 (+ 2us) tablet.cc:660] Decoding operations
0430 20:11:35.845301 (+ 81us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:35.845309 (+ 8us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:35.845313 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0430 20:11:35.845362 (+ 49us) tablet.cc:699] Row locks acquired
0430 20:11:35.845364 (+ 2us) write_op.cc:260] PREPARE: finished
0430 20:11:35.845402 (+ 38us) write_op.cc:270] Start()
0430 20:11:35.845428 (+ 26us) write_op.cc:276] Timestamp: P: 1714507895845367 usec, L: 0
0430 20:11:35.845432 (+ 4us) op_driver.cc:331] REPLICATION: starting
0430 20:11:35.845572 (+ 140us) log.cc:842] Serialized 405 byte log entry
0430 20:11:36.222728 (+377156us) op_driver.cc:444] REPLICATION: finished
0430 20:11:36.223641 (+ 913us) write_op.cc:301] APPLY: starting
0430 20:11:36.223655 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:36.223687 (+ 32us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:36.223714 (+ 27us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:36.223756 (+ 42us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:36.223789 (+ 33us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0430 20:11:36.223799 (+ 10us) write_op.cc:312] APPLY: finished
0430 20:11:36.223860 (+ 61us) log.cc:842] Serialized 23 byte log entry
0430 20:11:36.223925 (+ 65us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:36.223938 (+ 13us) write_op.cc:446] Released schema lock
0430 20:11:36.223944 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:36.223988 (+ 44us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:36.223991 (+ 3us) write_op.cc:446] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":877,"apply.run_cpu_time_us":391,"apply.run_wall_time_us":390,"num_ops":1,"prepare.queue_time_us":1172,"prepare.run_cpu_time_us":516,"prepare.run_wall_time_us":516,"raft.queue_time_us":362,"raft.run_cpu_time_us":3,"raft.run_wall_time_us":3,"replication_time_us":377275}]]}
I20240430 20:11:37.451766 19023 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_190b2d86515a44898fa38a49517b9df1" } modify_external_catalogs: true
I20240430 20:11:37.452010 19471 ts_tablet_manager.cc:1875] T 4e69ca12bd10494fb013614741bc166c P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:37.451992 19023 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_190b2d86515a44898fa38a49517b9df1" } modify_external_catalogs: true
I20240430 20:11:36.276347 19028 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_5d0995eff94c404fa97b08aa190f4a7f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:37.453084 19028 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5d0995eff94c404fa97b08aa190f4a7f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:35.951771 18601 tablet_service.cc:1467] Processing CreateTablet for tablet 558e71ec370d46ca8fbdcec5b57eb2ec (DEFAULT_TABLE table=default.Table_408be016c50f4d0fa19189bbaddab089 [id=6b7bd5a6c20e4b43aa62365627503d5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:35.954505 19818 raft_consensus.cc:2726] T e3e74e76ab2b4fe492ffcc1838656fd1 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:35.903328 19784 raft_consensus.cc:504] T 802336602d8f4ac4bd7e0872d6f3dae9 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:37.454635 18601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 558e71ec370d46ca8fbdcec5b57eb2ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:37.455576 19784 leader_election.cc:288] T 802336602d8f4ac4bd7e0872d6f3dae9 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Requested vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:35.931983 18443 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41680 (request call id 178) took 1689 ms. Trace:
I20240430 20:11:35.962467 18456 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41680 (request call id 184) took 1679 ms. Trace:
I20240430 20:11:37.456454 18456 rpcz_store.cc:276] 0430 20:11:34.243444 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:34.846649 (+603205us) service_pool.cc:225] Handling call
0430 20:11:35.839284 (+992635us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0430 20:11:35.839322 (+ 38us) ts_tablet_manager.cc:655] Creating new metadata...
0430 20:11:35.866143 (+ 26821us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:35.888125 (+ 21982us) tablet_replica.cc:205] Creating consensus instance
0430 20:11:35.923315 (+ 35190us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8832}
I20240430 20:11:37.456902 18456 tablet_service.cc:1514] Processing DeleteTablet for tablet c66c142419ca4e1399ecd1c8d1e62a59 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:34 UTC) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:37.457341 18456 tablet_service.cc:1514] Processing DeleteTablet for tablet 84568f90b9c54cf292099808d4cbf5dc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:34 UTC) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:37.457566 18456 tablet_service.cc:1514] Processing DeleteTablet for tablet 6db55bb510b24b0cae897936211e15f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:34 UTC) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:37.457981 18456 tablet_service.cc:1467] Processing CreateTablet for tablet 47e98935eb21449abd3d4d24327778d5 (DEFAULT_TABLE table=default.Table_67d51131549440cc99629dda924693f7 [id=842d2b7e31cb4e7793d88103997a88d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:35.964195 18459 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41680 (request call id 183) took 1720 ms. Trace:
I20240430 20:11:36.019646 18448 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41680 (request call id 164) took 1811 ms. Trace:
I20240430 20:11:36.073058 18985 tablet_bootstrap.cc:492] T 76d6e9d547ca4a848eae852944b1d117 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:37.458279 18459 rpcz_store.cc:276] 0430 20:11:34.243304 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:34.825824 (+582520us) service_pool.cc:225] Handling call
0430 20:11:35.850078 (+1024254us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0430 20:11:35.850124 (+ 46us) ts_tablet_manager.cc:655] Creating new metadata...
0430 20:11:35.854207 (+ 4083us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:35.963609 (+109402us) tablet_replica.cc:205] Creating consensus instance
0430 20:11:35.963807 (+ 198us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5248}
I20240430 20:11:35.874656 18753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5546f9edeea44d7a6c5cb954f5157ba" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:37.458642 18985 ts_tablet_manager.cc:1367] T 76d6e9d547ca4a848eae852944b1d117 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 1.982s user 0.005s sys 0.003s
I20240430 20:11:37.458865 18753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.17.177.2:37173 (request call id 165) took 1585 ms. Trace:
I20240430 20:11:37.459000 18753 rpcz_store.cc:276] 0430 20:11:35.873254 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:35.874544 (+ 1290us) service_pool.cc:225] Handling call
0430 20:11:37.458792 (+1584248us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240430 20:11:37.458973 18459 tablet_service.cc:1467] Processing CreateTablet for tablet 0368d37dd9e2452982fb81f539c6805b (DEFAULT_TABLE table=default.Table_3be3b20c86ea424fbc173dd32e01f90c [id=4330626e342a48699ea939add5c91b11]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:37.456375 18443 rpcz_store.cc:276] 0430 20:11:34.242292 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:34.242383 (+ 91us) service_pool.cc:225] Handling call
0430 20:11:35.828453 (+1586070us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0430 20:11:35.828490 (+ 37us) ts_tablet_manager.cc:655] Creating new metadata...
0430 20:11:35.861623 (+ 33133us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:35.931725 (+ 70102us) tablet_replica.cc:205] Creating consensus instance
0430 20:11:35.931912 (+ 187us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240430 20:11:37.459610 18985 raft_consensus.cc:348] T 76d6e9d547ca4a848eae852944b1d117 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:37.459705 18443 tablet_service.cc:1467] Processing CreateTablet for tablet 2a08e36c39824abf98b9b6214c8b84cb (DEFAULT_TABLE table=default.Table_027891085c5747d58ddb32ad51a453ba [id=45042e2e54a14f558c9968969d3618c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:37.459879 18985 raft_consensus.cc:374] T 76d6e9d547ca4a848eae852944b1d117 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:37.458492 18448 rpcz_store.cc:276] 0430 20:11:34.205886 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:34.206018 (+ 132us) service_pool.cc:225] Handling call
0430 20:11:34.791104 (+585086us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0430 20:11:34.791128 (+ 24us) ts_tablet_manager.cc:655] Creating new metadata...
0430 20:11:35.907510 (+1116382us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:36.017102 (+109592us) tablet_replica.cc:205] Creating consensus instance
0430 20:11:36.017303 (+ 201us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240430 20:11:37.459983 18985 raft_consensus.cc:729] T 76d6e9d547ca4a848eae852944b1d117 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:37.460263 18985 consensus_queue.cc:260] T 76d6e9d547ca4a848eae852944b1d117 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:36.094461 18450 tablet_service.cc:1467] Processing CreateTablet for tablet f3711610610542128c4a84e17bce4a3e (DEFAULT_TABLE table=default.Table_64d14df4e12146ecb85239e7ca03cf6e [id=c376119401934812aef62a3a18eeb368]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:36.098917 18449 tablet_service.cc:1467] Processing CreateTablet for tablet 1701a15b115741aea13f8327e3d26b34 (DEFAULT_TABLE table=default.Table_0fcc91de96a946018c72ba03c71bcc05 [id=7127dbd368864f7ba80a755549fe7ede]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:36.114825 18446 tablet_service.cc:1467] Processing CreateTablet for tablet 558e71ec370d46ca8fbdcec5b57eb2ec (DEFAULT_TABLE table=default.Table_408be016c50f4d0fa19189bbaddab089 [id=6b7bd5a6c20e4b43aa62365627503d5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:36.115360 18457 tablet_service.cc:1467] Processing CreateTablet for tablet f90a5c5818c54390a2ae323fed7ebe57 (DEFAULT_TABLE table=default.Table_18385e32857b4c8bb1e56a85c590153a [id=38a7d7d4f9e64c07b784d9393192ac94]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:36.118954 18455 tablet_service.cc:1467] Processing CreateTablet for tablet a445fdfde01c4371a5e6b3daea5b00ab (DEFAULT_TABLE table=default.Table_83d417131d35409582550b7b8ce4a141 [id=8893c7824ec64fc1961e7c6851278a50]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:36.124223 19825 tablet.cc:1722] T 294644d7a16b4ca2bbb6d106bcb8c7ff P 2efb542aa55e439b99ccb98d1c5c884a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240430 20:11:37.460881 18985 ts_tablet_manager.cc:1398] T 76d6e9d547ca4a848eae852944b1d117 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240430 20:11:36.277221 19029 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_408be016c50f4d0fa19189bbaddab089" } modify_external_catalogs: true
I20240430 20:11:36.295408 19027 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet a05e32a33aa7488eb287cc6314c09c44 (table default.Table_2d1cfc78363a47aa8dba7ef7d80f8c9a [id=955d833ff8f14bf6957abadf9ef669e2]). Expected version 2 got 0
I20240430 20:11:37.461112 19029 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_408be016c50f4d0fa19189bbaddab089" } modify_external_catalogs: true
I20240430 20:11:37.461345 19027 catalog_manager.cc:5484] T 12c6e131fa904d23865b8673b58a7e33 P 5e1c6397f09747f091534faf93542710 reported cstate change: term changed from 1 to 2, leader changed from 3c51c33d5616482cafb68e26688820f4 (127.17.177.3) to 5e1c6397f09747f091534faf93542710 (127.17.177.2). New cstate: current_term: 2 leader_uuid: "5e1c6397f09747f091534faf93542710" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } health_report { overall_health: UNKNOWN } } }
I20240430 20:11:37.461959 19027 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 12c6e131fa904d23865b8673b58a7e33 (table default.Table_a9dc8fc6a4aa46dc914d0e327a4de554 [id=19f4e6decedf4ee59fccc0a24a05cedd]). Expected version 2 got 0
W20240430 20:11:36.134874 18417 leader_election.cc:341] T 93d158de64994f3c9fa45a1250a4f533 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Not found: Tablet not found: 93d158de64994f3c9fa45a1250a4f533
W20240430 20:11:37.462342 18417 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.177.3:44867, user_credentials={real_user=slave}} blocked reactor thread for 1.32739e+06us
W20240430 20:11:35.875118 18415 leader_election.cc:341] T f77086a5675f44e99e8a867b694df4b8 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Not found: Tablet not found: f77086a5675f44e99e8a867b694df4b8
I20240430 20:11:37.462961 18415 leader_election.cc:302] T f77086a5675f44e99e8a867b694df4b8 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a; no voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710
I20240430 20:11:36.936746 18475 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.17.177.3:32791 (request call id 165) took 1080 ms. Trace:
I20240430 20:11:37.463299 18475 rpcz_store.cc:276] 0430 20:11:35.856444 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:35.869588 (+ 13144us) service_pool.cc:225] Handling call
0430 20:11:36.936686 (+1067098us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240430 20:11:35.967511 18983 tablet_bootstrap.cc:492] T e6eac92dfa5f4456bc49571878f0f8dd P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:37.464494 18983 ts_tablet_manager.cc:1367] T e6eac92dfa5f4456bc49571878f0f8dd P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 1.629s user 0.000s sys 0.004s
I20240430 20:11:37.465449 18983 raft_consensus.cc:348] T e6eac92dfa5f4456bc49571878f0f8dd P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:37.465708 18983 raft_consensus.cc:374] T e6eac92dfa5f4456bc49571878f0f8dd P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:37.465803 18983 raft_consensus.cc:729] T e6eac92dfa5f4456bc49571878f0f8dd P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:37.466086 18983 consensus_queue.cc:260] T e6eac92dfa5f4456bc49571878f0f8dd P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:36.936892 18461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f4e860484784aa9a2d28af2e9fb611c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:36.942889 18444 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41680 (request call id 177) took 2700 ms. Trace:
I20240430 20:11:36.073145 18463 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41680 (request call id 160) took 1821 ms. Trace:
I20240430 20:11:35.877336 18752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b76a4105a2974fe280e759349037d027" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:37.466795 18451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90a694bd5db649afa50f3c4e752939c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:37.466652 18444 rpcz_store.cc:276] 0430 20:11:34.242114 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:34.242532 (+ 418us) service_pool.cc:225] Handling call
0430 20:11:35.830508 (+1587976us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0430 20:11:35.830546 (+ 38us) ts_tablet_manager.cc:655] Creating new metadata...
0430 20:11:36.938972 (+1108426us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:36.942497 (+ 3525us) tablet_replica.cc:205] Creating consensus instance
0430 20:11:36.942809 (+ 312us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240430 20:11:37.467017 18752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.17.177.2:37173 (request call id 175) took 1589 ms. Trace:
I20240430 20:11:37.467114 18752 rpcz_store.cc:276] 0430 20:11:35.877043 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:35.877207 (+ 164us) service_pool.cc:225] Handling call
0430 20:11:37.466955 (+1589748us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240430 20:11:37.467232 18444 tablet_service.cc:1467] Processing CreateTablet for tablet 9c73d0a6bc414d3b9900570848d19a51 (DEFAULT_TABLE table=default.Table_49a012a7c4c8457fadb0a8f0fc9654c6 [id=eecb2f8eeef547c5ab1abe6a2430e592]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:35.968281 18588 tablet_service.cc:1467] Processing CreateTablet for tablet 47e98935eb21449abd3d4d24327778d5 (DEFAULT_TABLE table=default.Table_67d51131549440cc99629dda924693f7 [id=842d2b7e31cb4e7793d88103997a88d8]), partition=RANGE (key) PARTITION UNBOUNDED
W20240430 20:11:36.533918 18999 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 99203cc522de424184c6f97453f166fd (table default.Table_cb2a4b9ceaf4431facfff2c9bcf94c5d [id=3f8690a437ae49a3beb009fa7899d5d0], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 99203cc522de424184c6f97453f166fd consensus state has no leader
I20240430 20:11:37.467644 18444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c73d0a6bc414d3b9900570848d19a51. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:37.467864 18588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47e98935eb21449abd3d4d24327778d5. 1 dirs total, 0 dirs full, 0 dirs failed
W20240430 20:11:36.991849 18404 kernel_stack_watchdog.cc:198] Thread 18415 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:308 for 983ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7f9c67949980 (unknown) at ??:0
@ 0x7f9c679439c6 __GI___pthread_rwlock_wrlock at ??:0
@ 0x7f9c618c9904 google::LogMessage::Flush() at ??:0
@ 0x7f9c618c9a11 google::LogMessage::~LogMessage() at ??:0
@ 0x7f9c6fd705d3 kudu::consensus::LeaderElection::VoteResponseRpcCallback() at ??:0
@ 0x7f9c6fd7312a kudu::consensus::LeaderElection::Run()::$_0::operator()() at ??:0
@ 0x7f9c6fd72eee std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7f9c6b2098cf kudu::rpc::Proxy::AsyncRequest()::$_1::operator()() at ??:0
@ 0x7f9c6b2094ae std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7f9c6b1f54e0 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7f9c6b1f5a2c kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7f9c6b19251a kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7f9c6b190f45 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7f9c6b1a32c5 ev::base<>::method_thunk<>() at ??:0
I20240430 20:11:35.894829 18480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b76a4105a2974fe280e759349037d027" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:37.470810 18462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6db55bb510b24b0cae897936211e15f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:35.968955 18587 tablet_service.cc:1467] Processing CreateTablet for tablet 0368d37dd9e2452982fb81f539c6805b (DEFAULT_TABLE table=default.Table_3be3b20c86ea424fbc173dd32e01f90c [id=4330626e342a48699ea939add5c91b11]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:37.471035 18480 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.17.177.2:33361 (request call id 188) took 1587 ms. Trace:
I20240430 20:11:37.471226 18587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0368d37dd9e2452982fb81f539c6805b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:37.471177 18480 rpcz_store.cc:276] 0430 20:11:35.886650 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:35.894686 (+ 8036us) service_pool.cc:225] Handling call
0430 20:11:37.470958 (+1576272us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240430 20:11:35.877502 18762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acd23f0a63d5448aaa7ed78422e67bad" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:37.471376 18443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a08e36c39824abf98b9b6214c8b84cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:37.471575 18762 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.17.177.2:37173 (request call id 177) took 1594 ms. Trace:
I20240430 20:11:37.471675 18762 rpcz_store.cc:276] 0430 20:11:35.877345 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:35.877386 (+ 41us) service_pool.cc:225] Handling call
0430 20:11:37.471507 (+1594121us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240430 20:11:35.869194 19001 catalog_manager.cc:4909] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): alter failed for tablet 12c6e131fa904d23865b8673b58a7e33 (table default.Table_a9dc8fc6a4aa46dc914d0e327a4de554 [id=19f4e6decedf4ee59fccc0a24a05cedd]): Illegal state: replica 3c51c33d5616482cafb68e26688820f4 is not leader of this config: current role FOLLOWER
W20240430 20:11:37.471974 19001 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.AlterSchema -> {remote=127.17.177.3:44867, user_credentials={real_user=slave}} blocked reactor thread for 1.60263e+06us
I20240430 20:11:37.011262 19955 raft_consensus.cc:482] T 12c6e131fa904d23865b8673b58a7e33 P 2efb542aa55e439b99ccb98d1c5c884a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:37.473188 19955 raft_consensus.cc:504] T 12c6e131fa904d23865b8673b58a7e33 P 2efb542aa55e439b99ccb98d1c5c884a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:37.474089 19955 leader_election.cc:288] T 12c6e131fa904d23865b8673b58a7e33 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:37.474253 19955 raft_consensus.cc:2726] T f77086a5675f44e99e8a867b694df4b8 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:37.266463 20025 raft_consensus.cc:482] T c66c142419ca4e1399ecd1c8d1e62a59 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5e1c6397f09747f091534faf93542710)
I20240430 20:11:35.998996 18600 tablet_service.cc:1467] Processing CreateTablet for tablet 9c73d0a6bc414d3b9900570848d19a51 (DEFAULT_TABLE table=default.Table_49a012a7c4c8457fadb0a8f0fc9654c6 [id=eecb2f8eeef547c5ab1abe6a2430e592]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:37.474902 18600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c73d0a6bc414d3b9900570848d19a51. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:35.877727 18760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9e7660eccc448dbb99d135f82bfd261" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:37.478801 18760 raft_consensus.cc:2454] T c9e7660eccc448dbb99d135f82bfd261 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 0.
I20240430 20:11:37.479058 18760 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.17.177.2:37173 (request call id 176) took 1601 ms. Trace:
I20240430 20:11:37.479146 18760 rpcz_store.cc:276] 0430 20:11:35.877200 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:35.877610 (+ 410us) service_pool.cc:225] Handling call
0430 20:11:37.478983 (+1601373us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240430 20:11:36.816982 18963 kernel_stack_watchdog.cc:198] Thread 19001 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:308 for 793ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7feb1a029980 (unknown) at ??:0
@ 0x7feb1a0239c6 __GI___pthread_rwlock_wrlock at ??:0
@ 0x7feb13fa9904 google::LogMessage::Flush() at ??:0
@ 0x7feb13fa9a11 google::LogMessage::~LogMessage() at ??:0
@ 0x7feb2a1521b0 kudu::master::AsyncAlterTable::HandleResponse() at ??:0
@ 0x7feb2a0a1b09 kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7feb2a153d60 _ZZN4kudu6master15AsyncAlterTable11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7feb2a153b7e _ZNSt17_Function_handlerIFvvEZN4kudu6master15AsyncAlterTable11SendRequestEiEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7feb1d8e98cf kudu::rpc::Proxy::AsyncRequest()::$_1::operator()() at ??:0
@ 0x7feb1d8e94ae std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7feb1d8d54e0 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7feb1d8d5a2c kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7feb1d87251a kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7feb1d870f45 kudu::rpc::Connection::ReadHandler() at ??:0
I20240430 20:11:37.460317 18448 tablet_service.cc:1467] Processing CreateTablet for tablet e900f62a5aa143e791c7539660b60cd9 (DEFAULT_TABLE table=default.Table_2dbff4cecdff4d2dbbb0004c88377ce7 [id=ebf546ade5f140f19fff7c5841bfee30]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:35.999223 18592 tablet_service.cc:1467] Processing CreateTablet for tablet 2a08e36c39824abf98b9b6214c8b84cb (DEFAULT_TABLE table=default.Table_027891085c5747d58ddb32ad51a453ba [id=45042e2e54a14f558c9968969d3618c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:37.481206 18592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a08e36c39824abf98b9b6214c8b84cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:35.879913 18755 raft_consensus.cc:1261] T 12c6e131fa904d23865b8673b58a7e33 P 3c51c33d5616482cafb68e26688820f4 [term 2 FOLLOWER]: Refusing update from remote peer 5e1c6397f09747f091534faf93542710: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 1. (term mismatch)
I20240430 20:11:35.886528 18763 raft_consensus.cc:2454] T b72fe1f0713b4076a81a3fc1c74f7fa7 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 1.
I20240430 20:11:37.485361 18755 pending_rounds.cc:76] T 12c6e131fa904d23865b8673b58a7e33 P 3c51c33d5616482cafb68e26688820f4: Aborting all ops after (but not including) 0
I20240430 20:11:37.485497 18755 pending_rounds.cc:97] T 12c6e131fa904d23865b8673b58a7e33 P 3c51c33d5616482cafb68e26688820f4: Aborting uncommitted NO_OP operation due to leader change: 1.1
W20240430 20:11:37.485641 18763 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.17.177.1:40727 (request call id 171) took 2775 ms (client timeout 3243 ms). Trace:
I20240430 20:11:37.485669 18755 raft_consensus.cc:2866] T 12c6e131fa904d23865b8673b58a7e33 P 3c51c33d5616482cafb68e26688820f4 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20240430 20:11:37.485749 18763 rpcz_store.cc:269] 0430 20:11:34.710246 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:34.710408 (+ 162us) service_pool.cc:225] Handling call
0430 20:11:37.485479 (+2775071us) spinlock_profiling.cc:243] Waited 2.89 ms on lock 0x6160000c4c54. stack: 00007f30e7d63756 00007f30e7d63510 00007f30e6d9648d 0000000000955804 00000000009557cd 000000000095519a 00007f30f35c26b5 00007f30f7293c64 00007f30f178e9b6 00007f30f178e6e5 00007f30eea6b04e 00007f30eea69f2e 00007f30eea70405 00007f30eea70d2f 00007f30eea70b58 0000000000a60bcc
0430 20:11:37.485536 (+ 57us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6648192}
I20240430 20:11:35.911983 19581 raft_consensus.cc:504] T 24dcc20832804229a2664873bf6ddaa7 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:37.485965 19805 raft_consensus.cc:482] T 12c6e131fa904d23865b8673b58a7e33 P 3c51c33d5616482cafb68e26688820f4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240430 20:11:37.006548 19074 rpcz_store.cc:269] 0430 20:11:35.979407 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:35.979512 (+ 105us) service_pool.cc:225] Handling call
0430 20:11:37.006385 (+1026873us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240430 20:11:37.486218 19805 raft_consensus.cc:504] T 12c6e131fa904d23865b8673b58a7e33 P 3c51c33d5616482cafb68e26688820f4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:36.000377 18593 tablet_service.cc:1467] Processing CreateTablet for tablet e900f62a5aa143e791c7539660b60cd9 (DEFAULT_TABLE table=default.Table_2dbff4cecdff4d2dbbb0004c88377ce7 [id=ebf546ade5f140f19fff7c5841bfee30]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:37.486838 18593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e900f62a5aa143e791c7539660b60cd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:37.486882 19581 leader_election.cc:288] T 24dcc20832804229a2664873bf6ddaa7 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Requested vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:35.929548 18730 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59830 (request call id 176) took 2039 ms. Trace:
I20240430 20:11:37.487035 19805 leader_election.cc:288] T 12c6e131fa904d23865b8673b58a7e33 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:36.210032 19880 tablet.cc:1722] T c66c142419ca4e1399ecd1c8d1e62a59 P 3c51c33d5616482cafb68e26688820f4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240430 20:11:37.487071 18730 rpcz_store.cc:276] 0430 20:11:33.890038 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:33.890082 (+ 44us) service_pool.cc:225] Handling call
0430 20:11:34.465396 (+575314us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0430 20:11:34.465414 (+ 18us) ts_tablet_manager.cc:655] Creating new metadata...
0430 20:11:35.853443 (+1388029us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:35.929231 (+ 75788us) tablet_replica.cc:205] Creating consensus instance
0430 20:11:35.929444 (+ 213us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240430 20:11:36.011576 18603 tablet_service.cc:1514] Processing DeleteTablet for tablet 680553d4d0bc4fcb839235c1832757e1 with delete_type TABLET_DATA_DELETED (replaced by tablet 9c73d0a6bc414d3b9900570848d19a51) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:36.254722 19859 raft_consensus.cc:482] T 0fe1247349344863a6273a769ce8ddf6 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5e1c6397f09747f091534faf93542710)
W20240430 20:11:36.030314 18557 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.177.3:44867, user_credentials={real_user=slave}} blocked reactor thread for 136770us
I20240430 20:11:36.014932 18558 leader_election.cc:302] T 75ebf051383e42529dcb5f7dd0770a4b P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:36.542374 19968 raft_consensus.cc:482] T 47c37876694c4a4abb1091c631dab7a5 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:37.488406 19968 raft_consensus.cc:504] T 47c37876694c4a4abb1091c631dab7a5 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:37.489538 19968 leader_election.cc:288] T 47c37876694c4a4abb1091c631dab7a5 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:35.949959 18589 tablet_service.cc:1514] Processing DeleteTablet for tablet 6db55bb510b24b0cae897936211e15f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:34 UTC) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:37.044600 18591 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40218 (request call id 181) took 1145 ms. Trace:
I20240430 20:11:37.490197 18591 rpcz_store.cc:276] 0430 20:11:35.898502 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:35.898556 (+ 54us) service_pool.cc:225] Handling call
0430 20:11:35.941282 (+ 42726us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0430 20:11:35.941321 (+ 39us) ts_tablet_manager.cc:655] Creating new metadata...
0430 20:11:37.013927 (+1072606us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:37.043888 (+ 29961us) tablet_replica.cc:205] Creating consensus instance
0430 20:11:37.044094 (+ 206us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240430 20:11:37.053484 19048 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: Sending DeleteTablet for 3 replicas of tablet e87fb2f0361d4aa3ada27bbdfa88e94c
I20240430 20:11:37.134846 19861 raft_consensus.cc:482] T 724e51ff05ec4baebc2bb55e05c8eddc P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2efb542aa55e439b99ccb98d1c5c884a)
I20240430 20:11:37.490731 19861 raft_consensus.cc:504] T 724e51ff05ec4baebc2bb55e05c8eddc P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:37.491173 19048 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:51664 (request call id 166) took 2030 ms. Trace:
I20240430 20:11:37.491355 19861 leader_election.cc:288] T 724e51ff05ec4baebc2bb55e05c8eddc P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:37.134886 20024 raft_consensus.cc:482] T 77285afcafe04a829e485998bbbe00f0 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2efb542aa55e439b99ccb98d1c5c884a)
I20240430 20:11:37.240255 20038 raft_consensus.cc:482] T aec9c09cc0454acab6006bb3e10982b7 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2efb542aa55e439b99ccb98d1c5c884a)
W20240430 20:11:37.248467 18544 kernel_stack_watchdog.cc:198] Thread 18558 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:308 for 291ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7ffa8ed33980 (unknown) at ??:0
@ 0x7ffa8ed2d9c6 __GI___pthread_rwlock_wrlock at ??:0
@ 0x7ffa88cb3904 google::LogMessage::Flush() at ??:0
@ 0x7ffa88cb3a11 google::LogMessage::~LogMessage() at ??:0
@ 0x7ffa971593f0 kudu::consensus::LeaderElection::CheckForDecision() at ??:0
@ 0x7ffa9715a782 kudu::consensus::LeaderElection::VoteResponseRpcCallback() at ??:0
@ 0x7ffa9715d12a kudu::consensus::LeaderElection::Run()::$_0::operator()() at ??:0
@ 0x7ffa9715ceee std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7ffa925f38cf kudu::rpc::Proxy::AsyncRequest()::$_1::operator()() at ??:0
@ 0x7ffa925f34ae std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7ffa925df4e0 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7ffa925dfa2c kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7ffa9257c51a kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7ffa9257af45 kudu::rpc::Connection::ReadHandler() at ??:0
I20240430 20:11:37.279078 20040 raft_consensus.cc:482] T 6e826459c3f4437da81c9e6e09f0c22f P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2efb542aa55e439b99ccb98d1c5c884a)
I20240430 20:11:37.407181 20047 raft_consensus.cc:482] T b72fe1f0713b4076a81a3fc1c74f7fa7 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:37.491272 19048 rpcz_store.cc:276] 0430 20:11:35.460636 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:35.471057 (+ 10421us) service_pool.cc:225] Handling call
0430 20:11:36.934712 (+1463655us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0430 20:11:36.934918 (+ 206us) catalog_manager.cc:2749] Modifying in-memory table state
0430 20:11:36.935020 (+ 102us) catalog_manager.cc:2758] Locking tablets
0430 20:11:36.935161 (+ 141us) catalog_manager.cc:2772] Removing table and tablets from system table
0430 20:11:37.053245 (+118084us) catalog_manager.cc:2790] Removing table from by-name map
0430 20:11:37.053351 (+ 106us) catalog_manager.cc:2806] Committing in-memory state
0430 20:11:37.053414 (+ 63us) catalog_manager.cc:2810] Aborting table tasks
0430 20:11:37.491107 (+437693us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:36.982566 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:36.982753 (+ 187us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:36.982762 (+ 9us) write_op.cc:427] Acquired schema lock
0430 20:11:36.982765 (+ 3us) tablet.cc:660] Decoding operations
0430 20:11:36.982903 (+ 138us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:36.982918 (+ 15us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:36.982923 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0430 20:11:36.983056 (+ 133us) tablet.cc:699] Row locks acquired
0430 20:11:36.983058 (+ 2us) write_op.cc:260] PREPARE: finished
0430 20:11:36.983104 (+ 46us) write_op.cc:270] Start()
0430 20:11:36.983144 (+ 40us) write_op.cc:276] Timestamp: P: 1714507896983063 usec, L: 0
0430 20:11:36.983149 (+ 5us) op_driver.cc:331] REPLICATION: starting
0430 20:11:36.983411 (+ 262us) log.cc:842] Serialized 759 byte log entry
0430 20:11:36.994197 (+ 10786us) op_driver.cc:444] REPLICATION: finished
0430 20:11:37.045667 (+ 51470us) write_op.cc:301] APPLY: starting
0430 20:11:37.045821 (+ 154us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:37.046664 (+ 843us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:37.047113 (+ 449us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:37.048780 (+ 1667us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:37.049174 (+ 394us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0430 20:11:37.049385 (+ 211us) write_op.cc:312] APPLY: finished
0430 20:11:37.051283 (+ 1898us) log.cc:842] Serialized 29 byte log entry
0430 20:11:37.052698 (+ 1415us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:37.052858 (+ 160us) write_op.cc:446] Released schema lock
0430 20:11:37.052870 (+ 12us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:37.053474 (+ 604us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:37.053482 (+ 8us) write_op.cc:446] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":51203,"apply.run_cpu_time_us":1473,"apply.run_wall_time_us":8204,"num_ops":2,"prepare.queue_time_us":46576,"prepare.run_cpu_time_us":1355,"prepare.run_wall_time_us":53095,"raft.queue_time_us":866,"raft.run_cpu_time_us":360,"raft.run_wall_time_us":574,"replication_time_us":11015,"thread_start_us":148877,"threads_started":3}]]}
I20240430 20:11:37.492043 19861 raft_consensus.cc:2781] T 75ebf051383e42529dcb5f7dd0770a4b P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:37.492374 19048 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_8c7f3f5e74f64d409fbecfce64acf648" } new_table_name: "default.Table_01450a1e661d450a8cbe185e16e8250c" modify_external_catalogs: true
I20240430 20:11:37.493566 18730 tablet_service.cc:1467] Processing CreateTablet for tablet e900f62a5aa143e791c7539660b60cd9 (DEFAULT_TABLE table=default.Table_2dbff4cecdff4d2dbbb0004c88377ce7 [id=ebf546ade5f140f19fff7c5841bfee30]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:36.471449 20019 raft_consensus.cc:482] T 15674aeb56b74210a4a2809608fd4457 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2efb542aa55e439b99ccb98d1c5c884a)
I20240430 20:11:37.494547 20019 raft_consensus.cc:504] T 15674aeb56b74210a4a2809608fd4457 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
W20240430 20:11:35.888095 18699 consensus_peers.cc:485] T 34ab40d8d91e401db299ddee7175ba28 P 3c51c33d5616482cafb68e26688820f4 -> Peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Couldn't send request to peer 2efb542aa55e439b99ccb98d1c5c884a. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240430 20:11:37.232198 19049 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:51664 (request call id 163) took 1771 ms. Trace:
I20240430 20:11:37.495595 19049 rpcz_store.cc:276] 0430 20:11:35.460391 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:35.466454 (+ 6063us) service_pool.cc:225] Handling call
0430 20:11:36.936152 (+1469698us) catalog_manager.cc:2072] Acquired catalog manager lock
0430 20:11:36.936639 (+ 487us) catalog_manager.cc:2118] Created new table and tablet info
0430 20:11:37.231889 (+295250us) catalog_manager.cc:2178] Wrote table and tablets to system table
0430 20:11:37.232047 (+ 158us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0430 20:11:37.232154 (+ 107us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:37.035732 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:37.035923 (+ 191us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:37.035933 (+ 10us) write_op.cc:427] Acquired schema lock
0430 20:11:37.035938 (+ 5us) tablet.cc:660] Decoding operations
0430 20:11:37.036061 (+ 123us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:37.036077 (+ 16us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:37.036087 (+ 10us) tablet.cc:683] Acquiring locks for 2 operations
0430 20:11:37.036200 (+ 113us) tablet.cc:699] Row locks acquired
0430 20:11:37.036204 (+ 4us) write_op.cc:260] PREPARE: finished
0430 20:11:37.036251 (+ 47us) write_op.cc:270] Start()
0430 20:11:37.036294 (+ 43us) write_op.cc:276] Timestamp: P: 1714507897036203 usec, L: 0
0430 20:11:37.036298 (+ 4us) op_driver.cc:331] REPLICATION: starting
0430 20:11:37.049813 (+ 13515us) log.cc:842] Serialized 433 byte log entry
0430 20:11:37.050823 (+ 1010us) spinlock_profiling.cc:243] Waited 13.2 ms on lock 0x616000040654. stack: 00007feb16c74756 00007feb16c74510 00007feb15ca748d 0000000000955804 00000000009557cd 000000000095519a 00007feb224b9162 00007feb23821b02 00007feb23820629 00007feb2382853b 00007feb238283b8 0000000000a60bcc 00007feb16cbab74 00007feb16cbe19b 00007feb16cbe018 0000000000a60bcc
0430 20:11:37.230933 (+180110us) op_driver.cc:444] REPLICATION: finished
0430 20:11:37.231149 (+ 216us) write_op.cc:301] APPLY: starting
0430 20:11:37.231169 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:37.231230 (+ 61us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:37.231232 (+ 2us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:37.231397 (+ 165us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:37.231444 (+ 47us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0430 20:11:37.231456 (+ 12us) write_op.cc:312] APPLY: finished
0430 20:11:37.231560 (+ 104us) log.cc:842] Serialized 29 byte log entry
0430 20:11:37.231631 (+ 71us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:37.231656 (+ 25us) write_op.cc:446] Released schema lock
0430 20:11:37.231665 (+ 9us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:37.231729 (+ 64us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:37.231733 (+ 4us) write_op.cc:446] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":138,"apply.run_cpu_time_us":654,"apply.run_wall_time_us":653,"num_ops":2,"prepare.queue_time_us":98627,"prepare.run_cpu_time_us":1511,"prepare.run_wall_time_us":15174,"raft.queue_time_us":69,"raft.run_cpu_time_us":5,"raft.run_wall_time_us":5,"replication_time_us":194595,"spinlock_wait_cycles":30427392}]]}
I20240430 20:11:37.496811 19049 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_b7c7025fbf2e4039aadd143d021bfef5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:37.738802 18211 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.177.62:36441, user_credentials={real_user=slave}} blocked reactor thread for 1.75806e+06us
I20240430 20:11:38.113579 20053 sys_catalog.cc:263] Verifying existing consensus state
I20240430 20:11:37.475277 18454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1792c160af346ffb4f047c5d4435c10. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:38.390964 18453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50e1c55a1a0a484984ad5cbb78eec2ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:38.391250 18446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 558e71ec370d46ca8fbdcec5b57eb2ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:38.391474 18457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f90a5c5818c54390a2ae323fed7ebe57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:38.391686 18456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47e98935eb21449abd3d4d24327778d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:37.545825 18983 ts_tablet_manager.cc:1398] T e6eac92dfa5f4456bc49571878f0f8dd P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.049s user 0.001s sys 0.001s
W20240430 20:11:38.392028 18983 ts_tablet_manager.cc:1430] T e6eac92dfa5f4456bc49571878f0f8dd P 5e1c6397f09747f091534faf93542710: Tablet startup took 927ms
W20240430 20:11:38.392138 18983 ts_tablet_manager.cc:1432] T e6eac92dfa5f4456bc49571878f0f8dd P 5e1c6397f09747f091534faf93542710: Trace:
0430 20:11:27.196351 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:27.196401 (+ 50us) service_pool.cc:225] Handling call
0430 20:11:27.203886 (+ 7485us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0430 20:11:27.203925 (+ 39us) ts_tablet_manager.cc:655] Creating new metadata...
0430 20:11:27.212918 (+ 8993us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:27.262232 (+ 49314us) tablet_replica.cc:205] Creating consensus instance
0430 20:11:27.262403 (+ 171us) inbound_call.cc:173] Queueing success response
0430 20:11:35.835456 (+8573053us) ts_tablet_manager.cc:1343] Bootstrapping tablet
0430 20:11:37.464592 (+1629136us) ts_tablet_manager.cc:1399] Starting tablet replica
0430 20:11:37.464850 (+ 258us) tablet_replica.cc:259] Starting instrumentation
0430 20:11:37.465071 (+ 221us) tablet_replica.cc:278] Starting consensus
Metrics: {"raft.queue_time_us":52,"raft.run_cpu_time_us":66,"raft.run_wall_time_us":68,"tablet-open.queue_time_us":8573077}
I20240430 20:11:38.392472 18447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 488d0e609b224674932f890939664df2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:38.392728 18983 tablet_bootstrap.cc:492] T 24dcc20832804229a2664873bf6ddaa7 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:38.394397 18983 tablet_bootstrap.cc:654] T 24dcc20832804229a2664873bf6ddaa7 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
W20240430 20:11:36.788702 18685 kernel_stack_watchdog.cc:198] Thread 18699 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:308 for 725ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7f30eb118980 (unknown) at ??:0
@ 0x7f30eb1129c6 __GI___pthread_rwlock_wrlock at ??:0
@ 0x7f30e5098904 google::LogMessage::Flush() at ??:0
@ 0x7f30e5098a11 google::LogMessage::~LogMessage() at ??:0
@ 0x7f30f34d59ee kudu::consensus::Peer::ProcessResponseErrorUnlocked() at ??:0
@ 0x7f30f34d4adf kudu::consensus::Peer::ProcessResponse() at ??:0
@ 0x7f30f34db061 kudu::consensus::Peer::SendNextRequest()::$_3::operator()() at ??:0
@ 0x7f30f34dae6e std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7f30ee9d88cf kudu::rpc::Proxy::AsyncRequest()::$_1::operator()() at ??:0
@ 0x7f30ee9d84ae std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7f30ee9c44e0 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7f30ee9c4a2c kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7f30ee96151a kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7f30ee95ff45 kudu::rpc::Connection::ReadHandler() at ??:0
I20240430 20:11:38.394709 18449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1701a15b115741aea13f8327e3d26b34. 1 dirs total, 0 dirs full, 0 dirs failed
W20240430 20:11:37.497562 19049 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b7c7025fbf2e4039aadd143d021bfef5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:38.394943 18458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e91920e7e36418d8c27d1f0bfd9b981. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:38.395180 18445 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b285e3eb16d148779c74e057458d8461. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:38.395391 18455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a445fdfde01c4371a5e6b3daea5b00ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:38.395604 18459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0368d37dd9e2452982fb81f539c6805b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:38.396153 18448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e900f62a5aa143e791c7539660b60cd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:37.544656 20038 raft_consensus.cc:504] T aec9c09cc0454acab6006bb3e10982b7 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:38.399292 20038 leader_election.cc:288] T aec9c09cc0454acab6006bb3e10982b7 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:37.011709 18754 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.17.177.1:40727 (request call id 182) took 1161 ms. Trace:
I20240430 20:11:38.399502 18754 rpcz_store.cc:276] 0430 20:11:35.849213 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:35.872215 (+ 23002us) service_pool.cc:225] Handling call
0430 20:11:37.011022 (+1138807us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240430 20:11:37.511570 18699 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.177.1:41217, user_credentials={real_user=slave}} blocked reactor thread for 1.60736e+06us
I20240430 20:11:37.450829 19051 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_ff91abfadbc746d6b01fdbfecd16fac5" } modify_external_catalogs: true
I20240430 20:11:37.544874 20024 raft_consensus.cc:504] T 77285afcafe04a829e485998bbbe00f0 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:38.401268 20024 leader_election.cc:288] T 77285afcafe04a829e485998bbbe00f0 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:38.402252 19495 ts_tablet_manager.cc:1888] T 34ab40d8d91e401db299ddee7175ba28 P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:38.402388 19495 log.cc:1196] T 34ab40d8d91e401db299ddee7175ba28 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/34ab40d8d91e401db299ddee7175ba28
I20240430 20:11:37.012146 18729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f90a5c5818c54390a2ae323fed7ebe57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:37.014019 18741 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59830 (request call id 198) took 2074 ms. Trace:
I20240430 20:11:38.403762 18741 rpcz_store.cc:276] 0430 20:11:34.938307 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:37.004939 (+2066632us) service_pool.cc:225] Handling call
0430 20:11:37.005490 (+ 551us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0430 20:11:37.005527 (+ 37us) ts_tablet_manager.cc:655] Creating new metadata...
0430 20:11:37.008792 (+ 3265us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:37.012359 (+ 3567us) tablet_replica.cc:205] Creating consensus instance
0430 20:11:37.012468 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240430 20:11:38.404469 18731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47e98935eb21449abd3d4d24327778d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:38.404428 18741 tablet_service.cc:1467] Processing CreateTablet for tablet 9c73d0a6bc414d3b9900570848d19a51 (DEFAULT_TABLE table=default.Table_49a012a7c4c8457fadb0a8f0fc9654c6 [id=eecb2f8eeef547c5ab1abe6a2430e592]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:37.510113 19029 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: Sending DeleteTablet for 3 replicas of tablet 558e71ec370d46ca8fbdcec5b57eb2ec
I20240430 20:11:38.405253 19029 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:51664 (request call id 168) took 2944 ms. Trace:
I20240430 20:11:38.405355 19029 rpcz_store.cc:276] 0430 20:11:35.460820 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:36.224902 (+764082us) service_pool.cc:225] Handling call
0430 20:11:36.277156 (+ 52254us) spinlock_profiling.cc:243] Waited 51.5 ms on lock 0x616000040654. stack: 00007feb16c74756 00007feb16c74510 00007feb15ca748d 0000000000955804 00000000009557cd 00007feb257b315d 00007feb257b3253 00007feb224e5d41 00007feb2a0d134d 00007feb2a25eca6 00007feb21047f66 00007feb21047c95 00007feb1d97c04e 00007feb1d97af2e 00007feb1d981405 00007feb1d981d2f
0430 20:11:37.461278 (+1184122us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0430 20:11:37.461448 (+ 170us) catalog_manager.cc:2749] Modifying in-memory table state
0430 20:11:37.461516 (+ 68us) catalog_manager.cc:2758] Locking tablets
0430 20:11:37.461667 (+ 151us) catalog_manager.cc:2772] Removing table and tablets from system table
0430 20:11:37.509844 (+ 48177us) catalog_manager.cc:2790] Removing table from by-name map
0430 20:11:37.509975 (+ 131us) catalog_manager.cc:2806] Committing in-memory state
0430 20:11:37.510038 (+ 63us) catalog_manager.cc:2810] Aborting table tasks
0430 20:11:38.405193 (+895155us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:37.463407 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:37.463496 (+ 89us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:37.463505 (+ 9us) write_op.cc:427] Acquired schema lock
0430 20:11:37.463510 (+ 5us) tablet.cc:660] Decoding operations
0430 20:11:37.463649 (+ 139us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:37.463663 (+ 14us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:37.463670 (+ 7us) tablet.cc:683] Acquiring locks for 2 operations
0430 20:11:37.463786 (+ 116us) tablet.cc:699] Row locks acquired
0430 20:11:37.463789 (+ 3us) write_op.cc:260] PREPARE: finished
0430 20:11:37.463816 (+ 27us) write_op.cc:270] Start()
0430 20:11:37.463840 (+ 24us) write_op.cc:276] Timestamp: P: 1714507897463794 usec, L: 0
0430 20:11:37.463844 (+ 4us) op_driver.cc:331] REPLICATION: starting
0430 20:11:37.464048 (+ 204us) log.cc:842] Serialized 728 byte log entry
0430 20:11:37.498226 (+ 34178us) op_driver.cc:444] REPLICATION: finished
0430 20:11:37.499298 (+ 1072us) write_op.cc:301] APPLY: starting
0430 20:11:37.499311 (+ 13us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:37.499362 (+ 51us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:37.499400 (+ 38us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:37.499472 (+ 72us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:37.499510 (+ 38us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0430 20:11:37.499519 (+ 9us) write_op.cc:312] APPLY: finished
0430 20:11:37.499579 (+ 60us) log.cc:842] Serialized 29 byte log entry
0430 20:11:37.499632 (+ 53us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:37.499651 (+ 19us) write_op.cc:446] Released schema lock
0430 20:11:37.499656 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:37.499707 (+ 51us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:37.499713 (+ 6us) write_op.cc:446] Released schema lock
Metrics: {"spinlock_wait_cycles":118452608,"child_traces":[["op",{"apply.queue_time_us":1029,"apply.run_cpu_time_us":460,"apply.run_wall_time_us":460,"num_ops":2,"prepare.queue_time_us":984,"prepare.run_cpu_time_us":19601,"prepare.run_wall_time_us":218140,"raft.queue_time_us":6523,"raft.run_cpu_time_us":5,"raft.run_wall_time_us":6,"replication_time_us":34364,"thread_start_us":254975,"threads_started":2}]]}
I20240430 20:11:37.546272 20047 raft_consensus.cc:504] T b72fe1f0713b4076a81a3fc1c74f7fa7 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:37.014045 18733 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59830 (request call id 177) took 3121 ms. Trace:
I20240430 20:11:37.014430 18988 tablet_bootstrap.cc:654] T fcc9698abda4491b98e1e879589913a8 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:38.406877 18742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6db55bb510b24b0cae897936211e15f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:38.407155 18730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e900f62a5aa143e791c7539660b60cd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:38.407182 20047 leader_election.cc:288] T b72fe1f0713b4076a81a3fc1c74f7fa7 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:38.407392 18740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50e1c55a1a0a484984ad5cbb78eec2ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:38.407593 18738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a445fdfde01c4371a5e6b3daea5b00ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:35.613111 18735 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59830 (request call id 169) took 1729 ms. Trace:
I20240430 20:11:37.511143 19028 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:51664 (request call id 167) took 2050 ms. Trace:
I20240430 20:11:38.408033 18736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 488d0e609b224674932f890939664df2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:38.408169 18452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fda942d50b014f3d9589e1fb909fc2fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:38.408088 19028 rpcz_store.cc:276] 0430 20:11:35.460732 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:36.167059 (+706327us) service_pool.cc:225] Handling call
0430 20:11:36.276198 (+109139us) spinlock_profiling.cc:243] Waited 109 ms on lock 0x616000040654. stack: 00007feb16c74756 00007feb16c74510 00007feb15ca748d 0000000000955804 00000000009557cd 00007feb257b315d 00007feb257b3253 00007feb224e5d41 00007feb2a0d134d 00007feb2a25dcb3 00007feb210462a6 00007feb21045fd5 00007feb1d97c04e 00007feb1d97af2e 00007feb1d981405 00007feb1d981d2f
0430 20:11:37.453198 (+1177000us) catalog_manager.cc:2072] Acquired catalog manager lock
0430 20:11:37.453684 (+ 486us) catalog_manager.cc:2118] Created new table and tablet info
0430 20:11:37.510806 (+ 57122us) catalog_manager.cc:2178] Wrote table and tablets to system table
0430 20:11:37.510940 (+ 134us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0430 20:11:37.511092 (+ 152us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:37.462490 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:37.462622 (+ 132us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:37.462629 (+ 7us) write_op.cc:427] Acquired schema lock
0430 20:11:37.462632 (+ 3us) tablet.cc:660] Decoding operations
0430 20:11:37.462757 (+ 125us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:37.462784 (+ 27us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:37.462790 (+ 6us) tablet.cc:683] Acquiring locks for 2 operations
0430 20:11:37.462923 (+ 133us) tablet.cc:699] Row locks acquired
0430 20:11:37.462927 (+ 4us) write_op.cc:260] PREPARE: finished
0430 20:11:37.462972 (+ 45us) write_op.cc:270] Start()
0430 20:11:37.463017 (+ 45us) write_op.cc:276] Timestamp: P: 1714507897462944 usec, L: 0
0430 20:11:37.463021 (+ 4us) op_driver.cc:331] REPLICATION: starting
0430 20:11:37.463252 (+ 231us) log.cc:842] Serialized 433 byte log entry
0430 20:11:37.498139 (+ 34887us) op_driver.cc:444] REPLICATION: finished
0430 20:11:37.498559 (+ 420us) write_op.cc:301] APPLY: starting
0430 20:11:37.498583 (+ 24us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:37.498669 (+ 86us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:37.498702 (+ 33us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:37.498839 (+ 137us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:37.498890 (+ 51us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0430 20:11:37.498898 (+ 8us) write_op.cc:312] APPLY: finished
0430 20:11:37.499004 (+ 106us) log.cc:842] Serialized 29 byte log entry
0430 20:11:37.499084 (+ 80us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:37.499111 (+ 27us) write_op.cc:446] Released schema lock
0430 20:11:37.499121 (+ 10us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:37.499193 (+ 72us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:37.499198 (+ 5us) write_op.cc:446] Released schema lock
Metrics: {"spinlock_wait_cycles":250238720,"child_traces":[["op",{"apply.queue_time_us":350,"apply.run_cpu_time_us":702,"apply.run_wall_time_us":702,"num_ops":2,"prepare.queue_time_us":8293,"prepare.run_cpu_time_us":884,"prepare.run_wall_time_us":885,"raft.queue_time_us":7577,"raft.run_cpu_time_us":392,"raft.run_wall_time_us":404,"replication_time_us":35084}]]}
I20240430 20:11:38.408941 19028 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_2582fc0f97c1470d8b6b21cd1240697e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:38.409530 19028 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2582fc0f97c1470d8b6b21cd1240697e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:37.503527 19859 raft_consensus.cc:504] T 0fe1247349344863a6273a769ce8ddf6 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:37.554291 19861 raft_consensus.cc:686] T 75ebf051383e42529dcb5f7dd0770a4b P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Becoming Leader. State: Replica: 5e1c6397f09747f091534faf93542710, State: Running, Role: LEADER
I20240430 20:11:38.410813 18743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0368d37dd9e2452982fb81f539c6805b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:38.410924 19861 consensus_queue.cc:237] T 75ebf051383e42529dcb5f7dd0770a4b P 5e1c6397f09747f091534faf93542710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:38.411204 19859 leader_election.cc:288] T 0fe1247349344863a6273a769ce8ddf6 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:37.546912 20040 raft_consensus.cc:504] T 6e826459c3f4437da81c9e6e09f0c22f P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:37.486007 18755 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.17.177.2:37173 (request call id 173) took 1609 ms. Trace:
W20240430 20:11:37.539121 18999 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 2da900b6bbfa4fa99fe8494fd2e68da8 (table default.Table_19a91a7f17ff456ebb145e8dd89c0940 [id=67283dd3995a44c6a7a7bc8722e4f955], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 2da900b6bbfa4fa99fe8494fd2e68da8 consensus state has no leader
I20240430 20:11:37.503561 20019 leader_election.cc:288] T 15674aeb56b74210a4a2809608fd4457 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:37.512804 19489 raft_consensus.cc:2258] T 34ab40d8d91e401db299ddee7175ba28 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:38.412557 20040 leader_election.cc:288] T 6e826459c3f4437da81c9e6e09f0c22f P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:38.408033 18735 rpcz_store.cc:276] 0430 20:11:33.883396 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:33.883569 (+ 173us) service_pool.cc:225] Handling call
0430 20:11:34.464340 (+580771us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0430 20:11:34.464359 (+ 19us) ts_tablet_manager.cc:655] Creating new metadata...
0430 20:11:35.448714 (+984355us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:35.612951 (+164237us) tablet_replica.cc:205] Creating consensus instance
0430 20:11:35.613064 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240430 20:11:38.413148 19489 ts_tablet_manager.cc:1875] T 34ab40d8d91e401db299ddee7175ba28 P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:37.697597 19026 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 12c6e131fa904d23865b8673b58a7e33 (table default.Table_a9dc8fc6a4aa46dc914d0e327a4de554 [id=19f4e6decedf4ee59fccc0a24a05cedd]). Expected version 2 got 0
I20240430 20:11:38.406852 18733 rpcz_store.cc:276] 0430 20:11:33.890749 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:33.890794 (+ 45us) service_pool.cc:225] Handling call
0430 20:11:34.465543 (+574749us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0430 20:11:34.465559 (+ 16us) ts_tablet_manager.cc:655] Creating new metadata...
0430 20:11:37.008166 (+2542607us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:37.011795 (+ 3629us) tablet_replica.cc:205] Creating consensus instance
0430 20:11:37.011920 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240430 20:11:37.587213 18578 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:40186 (request call id 97) took 2546 ms. Trace:
I20240430 20:11:38.412091 18728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3711610610542128c4a84e17bce4a3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:38.413604 18578 rpcz_store.cc:276] 0430 20:11:35.030543 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:35.030672 (+ 129us) service_pool.cc:225] Handling call
0430 20:11:35.937283 (+906611us) spinlock_profiling.cc:243] Waited 289 ms on lock 0x6160000c6454. stack: 00007ffa8b97e756 00007ffa8b97e510 00007ffa8a9b148d 0000000000955804 00000000009557cd 000000000095519a 00007ffa971bcb00 00007ffa9aecdabf 00007ffa956508e6 00007ffa95650615 00007ffa9268604e 00007ffa92684f2e 00007ffa9268b405 00007ffa9268bd2f 00007ffa9268bb58 0000000000a60bcc
0430 20:11:37.455890 (+1518607us) spinlock_profiling.cc:243] Waited 239 ms on lock 0x616000190f54. stack: 00007ffa8b97e756 00007ffa8b97e510 00007ffa8a9b148d 0000000000955804 00000000009557cd 000000000095519a 00007ffa971bcb00 00007ffa9aecdabf 00007ffa956508e6 00007ffa95650615 00007ffa9268604e 00007ffa92684f2e 00007ffa9268b405 00007ffa9268bd2f 00007ffa9268bb58 0000000000a60bcc
0430 20:11:37.488973 (+ 33083us) spinlock_profiling.cc:243] Waited 32.6 ms on lock 0x6160000d3054. stack: 00007ffa8b97e756 00007ffa8b97e510 00007ffa8a9b148d 0000000000955804 00000000009557cd 000000000095519a 00007ffa971bcb00 00007ffa9aecdabf 00007ffa956508e6 00007ffa95650615 00007ffa9268604e 00007ffa92684f2e 00007ffa9268b405 00007ffa9268bd2f 00007ffa9268bb58 0000000000a60bcc
0430 20:11:37.576883 (+ 87910us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1287952640}
I20240430 20:11:37.705595 19805 raft_consensus.cc:482] T cace1398492f4d968dcb6899385d527c P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2efb542aa55e439b99ccb98d1c5c884a)
I20240430 20:11:38.413712 19026 catalog_manager.cc:5484] T 01436e5b6232494babbb414342f03090 P 5e1c6397f09747f091534faf93542710 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c51c33d5616482cafb68e26688820f4 (127.17.177.3). New cstate: current_term: 1 leader_uuid: "3c51c33d5616482cafb68e26688820f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } }
I20240430 20:11:38.414022 19805 raft_consensus.cc:504] T cace1398492f4d968dcb6899385d527c P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:38.414237 19026 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet acd23f0a63d5448aaa7ed78422e67bad (table default.Table_972f13be4dca4cd19b9a04534dc59f51 [id=69f37db4de634a3185613e40013e0ed7]). Expected version 1 got 0
I20240430 20:11:38.414443 19026 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet a05e32a33aa7488eb287cc6314c09c44 (table default.Table_2d1cfc78363a47aa8dba7ef7d80f8c9a [id=955d833ff8f14bf6957abadf9ef669e2]). Expected version 2 got 0
I20240430 20:11:38.414724 19805 leader_election.cc:288] T cace1398492f4d968dcb6899385d527c P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
W20240430 20:11:37.719112 19048 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet acf3b6fead41483a82c43dc0286fab7c consensus state has no leader
I20240430 20:11:37.615653 18586 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40218 (request call id 185) took 1639 ms. Trace:
I20240430 20:11:38.415131 18586 rpcz_store.cc:276] 0430 20:11:35.948517 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:35.948582 (+ 65us) service_pool.cc:225] Handling call
0430 20:11:37.450929 (+1502347us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0430 20:11:37.450962 (+ 33us) ts_tablet_manager.cc:655] Creating new metadata...
0430 20:11:37.456247 (+ 5285us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:37.588064 (+131817us) tablet_replica.cc:205] Creating consensus instance
0430 20:11:37.588237 (+ 173us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240430 20:11:38.415228 19048 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:51664 (request call id 172) took 2749 ms. Trace:
W20240430 20:11:37.617367 18557 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.177.3:44867, user_credentials={real_user=slave}} blocked reactor thread for 118226us
I20240430 20:11:38.470903 20053 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: Bootstrap starting.
I20240430 20:11:38.415316 19048 rpcz_store.cc:276] 0430 20:11:35.665882 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:37.492122 (+1826240us) service_pool.cc:225] Handling call
0430 20:11:37.492657 (+ 535us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0430 20:11:37.492920 (+ 263us) catalog_manager.cc:3613] Apply alter schema
0430 20:11:37.493175 (+ 255us) catalog_manager.cc:3636] Acquired catalog manager lock
0430 20:11:37.493270 (+ 95us) catalog_manager.cc:3783] Updating metadata on disk
0430 20:11:37.718831 (+225561us) catalog_manager.cc:3814] Committing alterations to in-memory state
0430 20:11:38.415175 (+696344us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:37.691456 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:37.694197 (+ 2741us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:37.694209 (+ 12us) write_op.cc:427] Acquired schema lock
0430 20:11:37.694431 (+ 222us) tablet.cc:660] Decoding operations
0430 20:11:37.698485 (+ 4054us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:37.698918 (+ 433us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:37.698927 (+ 9us) tablet.cc:683] Acquiring locks for 1 operations
0430 20:11:37.701097 (+ 2170us) tablet.cc:699] Row locks acquired
0430 20:11:37.701407 (+ 310us) write_op.cc:260] PREPARE: finished
0430 20:11:37.702715 (+ 1308us) write_op.cc:270] Start()
0430 20:11:37.704242 (+ 1527us) write_op.cc:276] Timestamp: P: 1714507897702021 usec, L: 0
0430 20:11:37.704254 (+ 12us) op_driver.cc:331] REPLICATION: starting
0430 20:11:37.710490 (+ 6236us) log.cc:842] Serialized 405 byte log entry
0430 20:11:37.718154 (+ 7664us) op_driver.cc:444] REPLICATION: finished
0430 20:11:37.718242 (+ 88us) write_op.cc:301] APPLY: starting
0430 20:11:37.718260 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:37.718306 (+ 46us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:37.718386 (+ 80us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:37.718440 (+ 54us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:37.718478 (+ 38us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0430 20:11:37.718486 (+ 8us) write_op.cc:312] APPLY: finished
0430 20:11:37.718584 (+ 98us) log.cc:842] Serialized 23 byte log entry
0430 20:11:37.718669 (+ 85us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:37.718691 (+ 22us) write_op.cc:446] Released schema lock
0430 20:11:37.718698 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:37.718759 (+ 61us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:37.718763 (+ 4us) write_op.cc:446] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":35,"apply.run_cpu_time_us":569,"apply.run_wall_time_us":569,"num_ops":1,"prepare.queue_time_us":196102,"prepare.run_cpu_time_us":2477,"prepare.run_wall_time_us":22308,"replication_time_us":13870}]]}
I20240430 20:11:38.415864 18737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12795670f21447a8964fe1efc6e92398. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:37.618717 19471 ts_tablet_manager.cc:1888] T 4e69ca12bd10494fb013614741bc166c P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:38.480620 19471 log.cc:1196] T 4e69ca12bd10494fb013614741bc166c P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/4e69ca12bd10494fb013614741bc166c
I20240430 20:11:38.480659 19048 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_17cb787c6a4c4c9fa720ce8c4e326605"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240430 20:11:38.481056 19471 ts_tablet_manager.cc:1909] T 4e69ca12bd10494fb013614741bc166c P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
W20240430 20:11:38.481300 19048 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_17cb787c6a4c4c9fa720ce8c4e326605 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:38.481437 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 66) took 7460 ms. Trace:
I20240430 20:11:38.481527 19471 rpcz_store.cc:276] 0430 20:11:31.021169 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:31.025292 (+ 4123us) service_pool.cc:225] Handling call
0430 20:11:37.450819 (+6425527us) ts_tablet_manager.cc:1120] Deleting tablet 4e69ca12bd10494fb013614741bc166c
0430 20:11:37.450834 (+ 15us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:37.484059 (+ 33225us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:37.618082 (+134023us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:38.481379 (+863297us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6425234}
I20240430 20:11:38.481957 19471 tablet_replica.cc:330] T f5236574672b477a9026e91adcaf70ec P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:38.482147 19471 raft_consensus.cc:2229] T f5236574672b477a9026e91adcaf70ec P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:38.482352 19471 raft_consensus.cc:2258] T f5236574672b477a9026e91adcaf70ec P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:38.415921 19024 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: Sending DeleteTablet for 3 replicas of tablet b76a4105a2974fe280e759349037d027
I20240430 20:11:38.482977 19471 ts_tablet_manager.cc:1875] T f5236574672b477a9026e91adcaf70ec P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:38.482957 19024 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:51664 (request call id 158) took 3023 ms. Trace:
I20240430 20:11:38.483055 19024 rpcz_store.cc:276] 0430 20:11:35.459887 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:35.463546 (+ 3659us) service_pool.cc:225] Handling call
0430 20:11:35.829129 (+365583us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0430 20:11:35.829542 (+ 413us) catalog_manager.cc:2749] Modifying in-memory table state
0430 20:11:35.829631 (+ 89us) catalog_manager.cc:2758] Locking tablets
0430 20:11:35.829910 (+ 279us) catalog_manager.cc:2772] Removing table and tablets from system table
0430 20:11:36.294767 (+464857us) catalog_manager.cc:2790] Removing table from by-name map
0430 20:11:36.294881 (+ 114us) catalog_manager.cc:2806] Committing in-memory state
0430 20:11:38.415879 (+2120998us) catalog_manager.cc:2810] Aborting table tasks
0430 20:11:38.482899 (+ 67020us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:35.870117 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:35.870250 (+ 133us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:35.870257 (+ 7us) write_op.cc:427] Acquired schema lock
0430 20:11:35.870262 (+ 5us) tablet.cc:660] Decoding operations
0430 20:11:35.871519 (+ 1257us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:35.871531 (+ 12us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:35.871536 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0430 20:11:35.871688 (+ 152us) tablet.cc:699] Row locks acquired
0430 20:11:35.871691 (+ 3us) write_op.cc:260] PREPARE: finished
0430 20:11:35.871731 (+ 40us) write_op.cc:270] Start()
0430 20:11:35.871773 (+ 42us) write_op.cc:276] Timestamp: P: 1714507895871699 usec, L: 0
0430 20:11:35.871779 (+ 6us) op_driver.cc:331] REPLICATION: starting
0430 20:11:35.871996 (+ 217us) log.cc:842] Serialized 762 byte log entry
0430 20:11:36.265043 (+393047us) op_driver.cc:444] REPLICATION: finished
0430 20:11:36.266333 (+ 1290us) write_op.cc:301] APPLY: starting
0430 20:11:36.266343 (+ 10us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:36.266374 (+ 31us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:36.266404 (+ 30us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:36.266468 (+ 64us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:36.266507 (+ 39us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0430 20:11:36.266518 (+ 11us) write_op.cc:312] APPLY: finished
0430 20:11:36.266583 (+ 65us) log.cc:842] Serialized 29 byte log entry
0430 20:11:36.286931 (+ 20348us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:36.294625 (+ 7694us) write_op.cc:446] Released schema lock
0430 20:11:36.294637 (+ 12us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:36.294729 (+ 92us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:36.294736 (+ 7us) write_op.cc:446] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1256,"apply.run_cpu_time_us":2061,"apply.run_wall_time_us":28464,"num_ops":2,"prepare.queue_time_us":4258,"prepare.run_cpu_time_us":929,"prepare.run_wall_time_us":5128,"raft.queue_time_us":3071,"raft.run_cpu_time_us":2,"raft.run_wall_time_us":3,"replication_time_us":393247}]]}
I20240430 20:11:38.483955 19024 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_5d51defcfdfd4d7f8e7d341c5daf0cdf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:38.484499 19024 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5d51defcfdfd4d7f8e7d341c5daf0cdf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:38.412027 18755 rpcz_store.cc:276] 0430 20:11:35.876830 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:35.879639 (+ 2809us) service_pool.cc:225] Handling call
0430 20:11:35.879770 (+ 131us) raft_consensus.cc:1457] Updating replica for 0 ops
0430 20:11:37.485832 (+1606062us) raft_consensus.cc:1683] Filling consensus response to leader.
0430 20:11:37.485962 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240430 20:11:37.743257 19033 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 22fef8bb6b034db098c3372ae992f7a7 consensus state has no leader
I20240430 20:11:38.485770 19033 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:51664 (request call id 170) took 3024 ms. Trace:
I20240430 20:11:38.486152 18737 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59830 (request call id 185) took 4250 ms. Trace:
I20240430 20:11:38.486258 18737 rpcz_store.cc:276] 0430 20:11:34.235982 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:34.795898 (+559916us) service_pool.cc:225] Handling call
0430 20:11:34.796441 (+ 543us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0430 20:11:34.796468 (+ 27us) ts_tablet_manager.cc:655] Creating new metadata...
0430 20:11:38.483214 (+3686746us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:38.485889 (+ 2675us) tablet_replica.cc:205] Creating consensus instance
0430 20:11:38.486081 (+ 192us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240430 20:11:38.485881 19033 rpcz_store.cc:276] 0430 20:11:35.460998 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:37.448783 (+1987785us) service_pool.cc:225] Handling call
0430 20:11:37.449247 (+ 464us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0430 20:11:37.449606 (+ 359us) catalog_manager.cc:3613] Apply alter schema
0430 20:11:37.449972 (+ 366us) catalog_manager.cc:3636] Acquired catalog manager lock
0430 20:11:37.450073 (+ 101us) catalog_manager.cc:3783] Updating metadata on disk
0430 20:11:37.742446 (+292373us) catalog_manager.cc:3814] Committing alterations to in-memory state
0430 20:11:38.485731 (+743285us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:37.714743 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:37.720556 (+ 5813us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:37.720562 (+ 6us) write_op.cc:427] Acquired schema lock
0430 20:11:37.720943 (+ 381us) tablet.cc:660] Decoding operations
0430 20:11:37.723579 (+ 2636us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:37.723967 (+ 388us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:37.724189 (+ 222us) tablet.cc:683] Acquiring locks for 1 operations
0430 20:11:37.726887 (+ 2698us) tablet.cc:699] Row locks acquired
0430 20:11:37.726891 (+ 4us) write_op.cc:260] PREPARE: finished
0430 20:11:37.727971 (+ 1080us) write_op.cc:270] Start()
0430 20:11:37.728911 (+ 940us) write_op.cc:276] Timestamp: P: 1714507897727650 usec, L: 0
0430 20:11:37.729210 (+ 299us) op_driver.cc:331] REPLICATION: starting
0430 20:11:37.734998 (+ 5788us) log.cc:842] Serialized 405 byte log entry
0430 20:11:37.741755 (+ 6757us) op_driver.cc:444] REPLICATION: finished
0430 20:11:37.741866 (+ 111us) write_op.cc:301] APPLY: starting
0430 20:11:37.741887 (+ 21us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:37.741943 (+ 56us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:37.741979 (+ 36us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:37.742039 (+ 60us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:37.742085 (+ 46us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0430 20:11:37.742096 (+ 11us) write_op.cc:312] APPLY: finished
0430 20:11:37.742191 (+ 95us) log.cc:842] Serialized 23 byte log entry
0430 20:11:37.742274 (+ 83us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:37.742296 (+ 22us) write_op.cc:446] Released schema lock
0430 20:11:37.742304 (+ 8us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:37.742384 (+ 80us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:37.742388 (+ 4us) write_op.cc:446] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":44,"apply.run_cpu_time_us":589,"apply.run_wall_time_us":589,"num_ops":1,"prepare.queue_time_us":199035,"prepare.run_cpu_time_us":2618,"prepare.run_wall_time_us":23267,"raft.queue_time_us":971,"raft.run_cpu_time_us":380,"raft.run_wall_time_us":984,"replication_time_us":12514}]]}
I20240430 20:11:37.474468 20025 raft_consensus.cc:504] T c66c142419ca4e1399ecd1c8d1e62a59 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:38.486567 19033 master_service.cc:946] ReplaceTablet: received request to replace tablet 24dcc20832804229a2664873bf6ddaa7 from {username='slave'} at 127.0.0.1:51664
I20240430 20:11:38.487310 20025 leader_election.cc:288] T c66c142419ca4e1399ecd1c8d1e62a59 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:37.630275 18593 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40218 (request call id 192) took 1660 ms. Trace:
I20240430 20:11:38.487493 18593 rpcz_store.cc:276] 0430 20:11:35.969337 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:35.969573 (+ 236us) service_pool.cc:225] Handling call
0430 20:11:37.486575 (+1517002us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0430 20:11:37.486631 (+ 56us) ts_tablet_manager.cc:655] Creating new metadata...
0430 20:11:37.503392 (+ 16761us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:37.629364 (+125972us) tablet_replica.cc:205] Creating consensus instance
0430 20:11:37.629572 (+ 208us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240430 20:11:37.632659 18601 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40218 (request call id 183) took 1691 ms. Trace:
I20240430 20:11:38.487768 18601 rpcz_store.cc:276] 0430 20:11:35.940810 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:35.940885 (+ 75us) service_pool.cc:225] Handling call
0430 20:11:37.454355 (+1513470us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0430 20:11:37.454390 (+ 35us) ts_tablet_manager.cc:655] Creating new metadata...
0430 20:11:37.457934 (+ 3544us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:37.631802 (+173868us) tablet_replica.cc:205] Creating consensus instance
0430 20:11:37.631964 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240430 20:11:37.641984 20060 raft_consensus.cc:482] T 13c1107ea4d04eb78a74d3f658c72fb3 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:37.466683 18463 rpcz_store.cc:276] 0430 20:11:34.201678 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:34.201762 (+ 84us) service_pool.cc:225] Handling call
0430 20:11:35.831418 (+1629656us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0430 20:11:35.831825 (+ 407us) ts_tablet_manager.cc:655] Creating new metadata...
0430 20:11:35.891892 (+ 60067us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:36.022906 (+131014us) tablet_replica.cc:205] Creating consensus instance
0430 20:11:36.023089 (+ 183us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240430 20:11:37.671685 18600 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40218 (request call id 193) took 1685 ms. Trace:
I20240430 20:11:37.503257 18444 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41680 (request call id 197) took 1382 ms. Trace:
I20240430 20:11:37.514384 18985 tablet_bootstrap.cc:492] T 01436e5b6232494babbb414342f03090 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:38.488052 18600 rpcz_store.cc:276] 0430 20:11:35.969449 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:35.969604 (+ 155us) service_pool.cc:225] Handling call
0430 20:11:37.474679 (+1505075us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0430 20:11:37.474710 (+ 31us) ts_tablet_manager.cc:655] Creating new metadata...
0430 20:11:37.478571 (+ 3861us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:37.654407 (+175836us) tablet_replica.cc:205] Creating consensus instance
0430 20:11:37.654639 (+ 232us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240430 20:11:38.488133 18444 rpcz_store.cc:276] 0430 20:11:36.121040 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:37.466930 (+1345890us) service_pool.cc:225] Handling call
0430 20:11:37.467425 (+ 495us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0430 20:11:37.467447 (+ 22us) ts_tablet_manager.cc:655] Creating new metadata...
0430 20:11:37.482467 (+ 15020us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:37.502835 (+ 20368us) tablet_replica.cc:205] Creating consensus instance
0430 20:11:37.503126 (+ 291us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240430 20:11:37.674757 18588 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40218 (request call id 189) took 1707 ms. Trace:
I20240430 20:11:38.488368 18588 rpcz_store.cc:276] 0430 20:11:35.967834 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:35.967895 (+ 61us) service_pool.cc:225] Handling call
0430 20:11:37.467650 (+1499755us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0430 20:11:37.467679 (+ 29us) ts_tablet_manager.cc:655] Creating new metadata...
0430 20:11:37.577251 (+109572us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:37.674480 (+ 97229us) tablet_replica.cc:205] Creating consensus instance
0430 20:11:37.674710 (+ 230us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240430 20:11:38.488454 18444 tablet_service.cc:1514] Processing DeleteTablet for tablet 680553d4d0bc4fcb839235c1832757e1 with delete_type TABLET_DATA_DELETED (replaced by tablet 9c73d0a6bc414d3b9900570848d19a51) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:37.677099 18592 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40218 (request call id 191) took 1674 ms. Trace:
I20240430 20:11:37.520160 18443 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41680 (request call id 195) took 1400 ms. Trace:
I20240430 20:11:38.488626 18592 rpcz_store.cc:276] 0430 20:11:35.969211 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:35.969282 (+ 71us) service_pool.cc:225] Handling call
0430 20:11:37.480948 (+1511666us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0430 20:11:37.480982 (+ 34us) ts_tablet_manager.cc:655] Creating new metadata...
0430 20:11:37.576230 (+ 95248us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:37.643502 (+ 67272us) tablet_replica.cc:205] Creating consensus instance
0430 20:11:37.643658 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240430 20:11:38.488644 18443 rpcz_store.cc:276] 0430 20:11:36.119602 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:37.459413 (+1339811us) service_pool.cc:225] Handling call
0430 20:11:37.464406 (+ 4993us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0430 20:11:37.464462 (+ 56us) ts_tablet_manager.cc:655] Creating new metadata...
0430 20:11:37.474705 (+ 10243us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:37.517112 (+ 42407us) tablet_replica.cc:205] Creating consensus instance
0430 20:11:37.519556 (+ 2444us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240430 20:11:37.684059 18558 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.177.1:41217, user_credentials={real_user=slave}} blocked reactor thread for 1.71257e+06us
I20240430 20:11:37.688000 18587 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40218 (request call id 190) took 1673 ms. Trace:
W20240430 20:11:37.528875 18415 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.177.2:46027, user_credentials={real_user=slave}} blocked reactor thread for 1.68786e+06us
I20240430 20:11:37.692772 18668 heartbeater.cc:347] Connected to a master server at 127.17.177.62:36441
I20240430 20:11:38.420138 20061 raft_consensus.cc:482] T e2d90ff17e574f88b2061b1c117d4b7d P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2efb542aa55e439b99ccb98d1c5c884a)
I20240430 20:11:38.489511 18415 leader_election.cc:302] T b688e03ea71940c69bda5520b62c1fc3 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:38.420187 20024 raft_consensus.cc:482] T fcc9698abda4491b98e1e879589913a8 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240430 20:11:37.705053 18417 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.177.3:44867, user_credentials={real_user=slave}} blocked reactor thread for 242397us
I20240430 20:11:38.099800 19784 raft_consensus.cc:482] T 75ebf051383e42529dcb5f7dd0770a4b P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:38.490473 18729 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59830 (request call id 189) took 4211 ms. Trace:
I20240430 20:11:38.490532 19784 raft_consensus.cc:504] T 75ebf051383e42529dcb5f7dd0770a4b P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:38.490582 18729 rpcz_store.cc:276] 0430 20:11:34.279375 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:35.857247 (+1577872us) service_pool.cc:225] Handling call
0430 20:11:35.872174 (+ 14927us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0430 20:11:35.872217 (+ 43us) ts_tablet_manager.cc:655] Creating new metadata...
0430 20:11:38.405906 (+2533689us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:38.490259 (+ 84353us) tablet_replica.cc:205] Creating consensus instance
0430 20:11:38.490400 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240430 20:11:38.488077 20060 raft_consensus.cc:504] T 13c1107ea4d04eb78a74d3f658c72fb3 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
W20240430 20:11:38.547291 18211 connection.cc:232] RPC call timeout handler was delayed by -2.0798738920000233s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.053100000000000008s for a response
W20240430 20:11:38.102669 18451 consensus_meta.cc:287] T 90a694bd5db649afa50f3c4e752939c1 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent flushing consensus metadata: real 0.631s user 0.026s sys 0.022s
I20240430 20:11:38.599123 18451 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41680 (request call id 176) took 4357 ms. Trace:
I20240430 20:11:38.599258 18451 rpcz_store.cc:276] 0430 20:11:34.241949 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:34.242020 (+ 71us) service_pool.cc:225] Handling call
0430 20:11:35.477426 (+1235406us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0430 20:11:35.477472 (+ 46us) ts_tablet_manager.cc:655] Creating new metadata...
0430 20:11:37.470100 (+1992628us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:38.598910 (+1128810us) tablet_replica.cc:205] Creating consensus instance
0430 20:11:38.599075 (+ 165us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240430 20:11:38.257508 20071 raft_consensus.cc:482] T 76d6e9d547ca4a848eae852944b1d117 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:38.491127 18729 tablet_service.cc:1467] Processing CreateTablet for tablet d2381047f6dc4bdf962f9383e6c41548 (DEFAULT_TABLE table=default.Table_3a588666409c4a64b13658a42a58f994 [id=053d7bb753464e968b05493053eb229a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:38.469048 18462 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41680 (request call id 174) took 4240 ms. Trace:
I20240430 20:11:38.599592 20071 raft_consensus.cc:504] T 76d6e9d547ca4a848eae852944b1d117 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:38.600230 18462 rpcz_store.cc:276] 0430 20:11:34.227538 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:34.227597 (+ 59us) service_pool.cc:225] Handling call
0430 20:11:35.445600 (+1218003us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0430 20:11:35.445642 (+ 42us) ts_tablet_manager.cc:655] Creating new metadata...
0430 20:11:37.474154 (+2028512us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:37.677818 (+203664us) tablet_replica.cc:205] Creating consensus instance
0430 20:11:38.467986 (+790168us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240430 20:11:38.413161 18735 tablet_service.cc:1514] Processing DeleteTablet for tablet 680553d4d0bc4fcb839235c1832757e1 with delete_type TABLET_DATA_DELETED (replaced by tablet 9c73d0a6bc414d3b9900570848d19a51) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:38.600914 20071 leader_election.cc:288] T 76d6e9d547ca4a848eae852944b1d117 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:38.601068 20071 raft_consensus.cc:482] T e3e74e76ab2b4fe492ffcc1838656fd1 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240430 20:11:38.428532 18733 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:59830: Illegal state: replica 3c51c33d5616482cafb68e26688820f4 is not leader of this config: current role FOLLOWER
I20240430 20:11:38.601226 20071 raft_consensus.cc:504] T e3e74e76ab2b4fe492ffcc1838656fd1 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:38.601339 18733 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:59830 (request call id 202) took 2761 ms. Trace:
I20240430 20:11:38.491559 20060 leader_election.cc:288] T 13c1107ea4d04eb78a74d3f658c72fb3 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:38.601449 18733 rpcz_store.cc:276] 0430 20:11:35.840259 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:38.413541 (+2573282us) service_pool.cc:225] Handling call
0430 20:11:38.601301 (+187760us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:38.428466 (+ 0us) op_driver.cc:372] HandleFailure(Illegal state: replica 3c51c33d5616482cafb68e26688820f4 is not leader of this config: current role FOLLOWER)
0430 20:11:38.428475 (+ 9us) alter_schema_op.cc:152] AlterSchemaCommitCallback: op aborted
Metrics: {"child_traces":[["op",{}]]}
I20240430 20:11:38.494900 20024 raft_consensus.cc:504] T fcc9698abda4491b98e1e879589913a8 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:38.445065 18983 tablet_bootstrap.cc:492] T 24dcc20832804229a2664873bf6ddaa7 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:38.601784 20071 leader_election.cc:288] T e3e74e76ab2b4fe492ffcc1838656fd1 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:38.601809 18983 ts_tablet_manager.cc:1367] T 24dcc20832804229a2664873bf6ddaa7 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.209s user 0.003s sys 0.002s
I20240430 20:11:38.494674 18587 rpcz_store.cc:276] 0430 20:11:35.968664 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:35.968715 (+ 51us) service_pool.cc:225] Handling call
0430 20:11:37.471013 (+1502298us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0430 20:11:37.471041 (+ 28us) ts_tablet_manager.cc:655] Creating new metadata...
0430 20:11:37.474281 (+ 3240us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:37.641678 (+167397us) tablet_replica.cc:205] Creating consensus instance
0430 20:11:37.641877 (+ 199us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240430 20:11:38.492215 20061 raft_consensus.cc:504] T e2d90ff17e574f88b2061b1c117d4b7d P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:38.507601 19471 ts_tablet_manager.cc:1888] T f5236574672b477a9026e91adcaf70ec P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:38.602188 18733 tablet_service.cc:1467] Processing CreateTablet for tablet c1017a50d1d646c486c0b9a75cb88ce4 (DEFAULT_TABLE table=default.Table_d95c70be474e415cba6a345438fad1ba [id=095b8b568ebc475aa3c21e0c03ef5d79]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:38.543852 18581 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:40186 (request call id 95) took 3553 ms. Trace:
W20240430 20:11:38.461666 18699 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.177.1:41217, user_credentials={real_user=slave}} blocked reactor thread for 61369.2us
I20240430 20:11:38.544015 18580 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:40186 (request call id 98) took 2845 ms. Trace:
I20240430 20:11:38.602629 18580 rpcz_store.cc:276] 0430 20:11:35.698946 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:35.699012 (+ 66us) service_pool.cc:225] Handling call
0430 20:11:35.937943 (+238931us) spinlock_profiling.cc:243] Waited 227 ms on lock 0x6160000c6454. stack: 00007ffa8b97e756 00007ffa8b97e510 00007ffa8a9b148d 0000000000955804 00000000009557cd 000000000095519a 00007ffa971bcb00 00007ffa9aecdabf 00007ffa956508e6 00007ffa95650615 00007ffa9268604e 00007ffa92684f2e 00007ffa9268b405 00007ffa9268bd2f 00007ffa9268bb58 0000000000a60bcc
0430 20:11:35.950198 (+ 12255us) spinlock_profiling.cc:243] Waited 1.08 ms on lock 0x616000152c30. stack: 00007ffa8b97e756 00007ffa8b97e510 00007ffa8a9b148d 0000000000955804 00000000009557cd 000000000095519a 00007ffa984d02f8 00007ffa9aecd8c8 00007ffa956508e6 00007ffa95650615 00007ffa9268604e 00007ffa92684f2e 00007ffa9268b405 00007ffa9268bd2f 00007ffa9268bb58 0000000000a60bcc
0430 20:11:37.112059 (+1161861us) spinlock_profiling.cc:243] Waited 1.01 ms on lock 0x6160001a0630. stack: 00007ffa8b97e756 00007ffa8b97e510 00007ffa8a9b148d 0000000000955804 00000000009557cd 000000000095519a 00007ffa984d02f8 00007ffa9aecd8c8 00007ffa956508e6 00007ffa95650615 00007ffa9268604e 00007ffa92684f2e 00007ffa9268b405 00007ffa9268bd2f 00007ffa9268bb58 0000000000a60bcc
0430 20:11:38.402746 (+1290687us) spinlock_profiling.cc:243] Waited 1.18 s on lock 0x616000170854. stack: 00007ffa8b97e756 00007ffa8b97e510 00007ffa8a9b148d 0000000000955804 00000000009557cd 000000000095519a 00007ffa971bcb00 00007ffa9aecdabf 00007ffa956508e6 00007ffa95650615 00007ffa9268604e 00007ffa92684f2e 00007ffa9268b405 00007ffa9268bd2f 00007ffa9268bb58 0000000000a60bcc
0430 20:11:38.541307 (+138561us) spinlock_profiling.cc:243] Waited 137 ms on lock 0x616000240454. stack: 00007ffa8b97e756 00007ffa8b97e510 00007ffa8a9b148d 0000000000955804 00000000009557cd 000000000095519a 00007ffa971bcb00 00007ffa9aecdabf 00007ffa956508e6 00007ffa95650615 00007ffa9268604e 00007ffa92684f2e 00007ffa9268b405 00007ffa9268bd2f 00007ffa9268bb58 0000000000a60bcc
0430 20:11:38.543952 (+ 2645us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3551169280}
I20240430 20:11:38.547624 18579 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:40186 (request call id 96) took 3554 ms. Trace:
W20240430 20:11:38.547847 18558 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.177.1:41217, user_credentials={real_user=slave}} blocked reactor thread for 58641.8us
I20240430 20:11:38.493230 18743 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59830 (request call id 197) took 3555 ms. Trace:
I20240430 20:11:38.568723 18616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcc9698abda4491b98e1e879589913a8" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:38.603365 20019 consensus_queue.cc:1035] T e3e74e76ab2b4fe492ffcc1838656fd1 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:38.573457 18610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53a899fa5a424a0d99f1b47925e8c7ad" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:38.493765 18737 tablet_service.cc:1467] Processing CreateTablet for tablet 2fc6908ecf954020a676b9d10a2b00e7 (DEFAULT_TABLE table=default.Table_766e3df58895427b9e5179eb555fe7e4 [id=5c11311047b34a3eb06cc06e8cc10223]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:38.575003 18608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a05e32a33aa7488eb287cc6314c09c44" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:38.575975 18622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6eac92dfa5f4456bc49571878f0f8dd" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:38.582196 18617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "802336602d8f4ac4bd7e0872d6f3dae9" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:38.582964 18612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12c6e131fa904d23865b8673b58a7e33" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:38.476066 18450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3711610610542128c4a84e17bce4a3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:38.603885 20024 leader_election.cc:288] T fcc9698abda4491b98e1e879589913a8 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:38.495301 18741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c73d0a6bc414d3b9900570848d19a51. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:38.510723 19495 ts_tablet_manager.cc:1909] T 34ab40d8d91e401db299ddee7175ba28 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:38.583544 18618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66c142419ca4e1399ecd1c8d1e62a59" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:38.511355 20025 raft_consensus.cc:2781] T b688e03ea71940c69bda5520b62c1fc3 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:38.604934 20025 raft_consensus.cc:686] T b688e03ea71940c69bda5520b62c1fc3 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Becoming Leader. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Running, Role: LEADER
I20240430 20:11:38.605195 20025 consensus_queue.cc:237] T b688e03ea71940c69bda5520b62c1fc3 P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:38.605444 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 59) took 7496 ms. Trace:
I20240430 20:11:38.605576 19495 rpcz_store.cc:276] 0430 20:11:31.109347 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:31.111155 (+ 1808us) service_pool.cc:225] Handling call
0430 20:11:35.846055 (+4734900us) ts_tablet_manager.cc:1120] Deleting tablet 34ab40d8d91e401db299ddee7175ba28
0430 20:11:35.846065 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:35.964951 (+118886us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:38.402101 (+2437150us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:38.605396 (+203295us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4734588}
I20240430 20:11:38.605939 19495 tablet_replica.cc:330] T c5fd0730133c472299c93f4b0830e527 P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:38.606170 19495 raft_consensus.cc:2229] T c5fd0730133c472299c93f4b0830e527 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Raft consensus shutting down.
I20240430 20:11:38.515061 19784 leader_election.cc:288] T 75ebf051383e42529dcb5f7dd0770a4b P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:38.606364 19495 raft_consensus.cc:2258] T c5fd0730133c472299c93f4b0830e527 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:38.497749 18736 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59830 (request call id 171) took 4612 ms. Trace:
I20240430 20:11:38.606669 18736 rpcz_store.cc:276] 0430 20:11:33.884752 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:33.884938 (+ 186us) service_pool.cc:225] Handling call
0430 20:11:34.464659 (+579721us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0430 20:11:34.464679 (+ 20us) ts_tablet_manager.cc:655] Creating new metadata...
0430 20:11:38.451468 (+3986789us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:38.497519 (+ 46051us) tablet_replica.cc:205] Creating consensus instance
0430 20:11:38.497682 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240430 20:11:38.606956 19495 ts_tablet_manager.cc:1875] T c5fd0730133c472299c93f4b0830e527 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:38.607196 18736 tablet_service.cc:1467] Processing CreateTablet for tablet f7d3dea34cf54fe2b949d1b7ca68577a (DEFAULT_TABLE table=default.Table_c06225c5a599482dab4f418fd13c593f [id=ea8038670f7c44a9bb20c396b54d6440]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:38.603318 18743 rpcz_store.cc:276] 0430 20:11:34.937758 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:36.934859 (+1997101us) service_pool.cc:225] Handling call
0430 20:11:36.935497 (+ 638us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0430 20:11:36.935532 (+ 35us) ts_tablet_manager.cc:655] Creating new metadata...
0430 20:11:38.431632 (+1496100us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:38.493027 (+ 61395us) tablet_replica.cc:205] Creating consensus instance
0430 20:11:38.493174 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240430 20:11:38.607782 18743 tablet_service.cc:1514] Processing DeleteTablet for tablet e87fb2f0361d4aa3ada27bbdfa88e94c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:36 UTC) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:38.608388 18743 tablet_service.cc:1514] Processing DeleteTablet for tablet 558e71ec370d46ca8fbdcec5b57eb2ec with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:37 UTC) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:38.608583 18743 tablet_service.cc:1514] Processing DeleteTablet for tablet b76a4105a2974fe280e759349037d027 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:35 UTC) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:38.515591 18458 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41680 (request call id 171) took 4288 ms. Trace:
I20240430 20:11:38.529515 18455 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41680 (request call id 189) took 2408 ms. Trace:
I20240430 20:11:38.609469 18455 rpcz_store.cc:276] 0430 20:11:36.117961 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:36.118029 (+ 68us) service_pool.cc:225] Handling call
0430 20:11:37.476027 (+1357998us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0430 20:11:37.476055 (+ 28us) ts_tablet_manager.cc:655] Creating new metadata...
0430 20:11:38.423975 (+947920us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:38.525418 (+101443us) tablet_replica.cc:205] Creating consensus instance
0430 20:11:38.525565 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240430 20:11:38.529611 18456 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41680 (request call id 193) took 2401 ms. Trace:
I20240430 20:11:38.531683 18447 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41680 (request call id 172) took 4304 ms. Trace:
I20240430 20:11:38.533577 18452 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41680 (request call id 170) took 4306 ms. Trace:
I20240430 20:11:38.609738 18456 rpcz_store.cc:276] 0430 20:11:36.119352 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:37.457663 (+1338311us) service_pool.cc:225] Handling call
0430 20:11:37.460214 (+ 2551us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0430 20:11:37.460275 (+ 61us) ts_tablet_manager.cc:655] Creating new metadata...
0430 20:11:38.465568 (+1005293us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:38.520905 (+ 55337us) tablet_replica.cc:205] Creating consensus instance
0430 20:11:38.521103 (+ 198us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240430 20:11:38.609934 18452 rpcz_store.cc:276] 0430 20:11:34.227128 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:34.227705 (+ 577us) service_pool.cc:225] Handling call
0430 20:11:35.477538 (+1249833us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0430 20:11:35.477558 (+ 20us) ts_tablet_manager.cc:655] Creating new metadata...
0430 20:11:38.466160 (+2988602us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:38.533415 (+ 67255us) tablet_replica.cc:205] Creating consensus instance
0430 20:11:38.533538 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240430 20:11:38.609344 18458 rpcz_store.cc:276] 0430 20:11:34.227236 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:34.228000 (+ 764us) service_pool.cc:225] Handling call
0430 20:11:35.464134 (+1236134us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0430 20:11:35.464181 (+ 47us) ts_tablet_manager.cc:655] Creating new metadata...
0430 20:11:38.401546 (+2937365us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:38.510019 (+108473us) tablet_replica.cc:205] Creating consensus instance
0430 20:11:38.515537 (+ 5518us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240430 20:11:38.534622 18453 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41680 (request call id 175) took 4297 ms. Trace:
I20240430 20:11:38.610152 18453 rpcz_store.cc:276] 0430 20:11:34.227664 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:34.227710 (+ 46us) service_pool.cc:225] Handling call
0430 20:11:35.463929 (+1236219us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0430 20:11:35.463995 (+ 66us) ts_tablet_manager.cc:655] Creating new metadata...
0430 20:11:38.517126 (+3053131us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:38.525453 (+ 8327us) tablet_replica.cc:205] Creating consensus instance
0430 20:11:38.525592 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240430 20:11:38.536145 18445 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41680 (request call id 173) took 4308 ms. Trace:
I20240430 20:11:38.610361 18445 rpcz_store.cc:276] 0430 20:11:34.227439 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:34.228714 (+ 1275us) service_pool.cc:225] Handling call
0430 20:11:35.464367 (+1235653us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0430 20:11:35.464433 (+ 66us) ts_tablet_manager.cc:655] Creating new metadata...
0430 20:11:38.520435 (+3056002us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:38.535948 (+ 15513us) tablet_replica.cc:205] Creating consensus instance
0430 20:11:38.536101 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240430 20:11:38.537675 18457 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41680 (request call id 188) took 2430 ms. Trace:
I20240430 20:11:38.541872 18448 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41680 (request call id 196) took 2422 ms. Trace:
I20240430 20:11:38.610623 18457 rpcz_store.cc:276] 0430 20:11:36.106972 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:36.107052 (+ 80us) service_pool.cc:225] Handling call
0430 20:11:37.475923 (+1368871us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0430 20:11:37.475955 (+ 32us) ts_tablet_manager.cc:655] Creating new metadata...
0430 20:11:38.427725 (+951770us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:38.537498 (+109773us) tablet_replica.cc:205] Creating consensus instance
0430 20:11:38.537632 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240430 20:11:38.610648 18448 rpcz_store.cc:276] 0430 20:11:36.119704 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:37.460084 (+1340380us) service_pool.cc:225] Handling call
0430 20:11:37.480572 (+ 20488us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0430 20:11:37.480660 (+ 88us) ts_tablet_manager.cc:655] Creating new metadata...
0430 20:11:38.468533 (+987873us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:38.541694 (+ 73161us) tablet_replica.cc:205] Creating consensus instance
0430 20:11:38.541821 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240430 20:11:38.544947 18461 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41680 (request call id 169) took 4317 ms. Trace:
I20240430 20:11:38.610847 18461 rpcz_store.cc:276] 0430 20:11:34.226987 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:34.227040 (+ 53us) service_pool.cc:225] Handling call
0430 20:11:35.477249 (+1250209us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0430 20:11:35.477309 (+ 60us) ts_tablet_manager.cc:655] Creating new metadata...
0430 20:11:38.529016 (+3051707us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:38.544739 (+ 15723us) tablet_replica.cc:205] Creating consensus instance
0430 20:11:38.544878 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240430 20:11:38.549221 18454 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41680 (request call id 168) took 4322 ms. Trace:
I20240430 20:11:38.556131 18459 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41680 (request call id 194) took 2431 ms. Trace:
I20240430 20:11:38.609791 18447 rpcz_store.cc:276] 0430 20:11:34.227312 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:34.228372 (+ 1060us) service_pool.cc:225] Handling call
0430 20:11:35.464265 (+1235893us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0430 20:11:35.464296 (+ 31us) ts_tablet_manager.cc:655] Creating new metadata...
0430 20:11:38.403477 (+2939181us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:38.530727 (+127250us) tablet_replica.cc:205] Creating consensus instance
0430 20:11:38.531643 (+ 916us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240430 20:11:38.611131 18459 rpcz_store.cc:276] 0430 20:11:36.119448 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:37.458558 (+1339110us) service_pool.cc:225] Handling call
0430 20:11:37.464740 (+ 6182us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0430 20:11:37.464793 (+ 53us) ts_tablet_manager.cc:655] Creating new metadata...
0430 20:11:38.464433 (+999640us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:38.551187 (+ 86754us) tablet_replica.cc:205] Creating consensus instance
0430 20:11:38.551326 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240430 20:11:38.566484 18446 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41680 (request call id 187) took 2463 ms. Trace:
I20240430 20:11:38.611089 18454 rpcz_store.cc:276] 0430 20:11:34.226832 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:34.227002 (+ 170us) service_pool.cc:225] Handling call
0430 20:11:35.443786 (+1216784us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0430 20:11:35.443819 (+ 33us) ts_tablet_manager.cc:655] Creating new metadata...
0430 20:11:38.541390 (+3097571us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:38.548691 (+ 7301us) tablet_replica.cc:205] Creating consensus instance
0430 20:11:38.549137 (+ 446us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240430 20:11:38.611383 18446 rpcz_store.cc:276] 0430 20:11:36.098162 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:36.098238 (+ 76us) service_pool.cc:225] Handling call
0430 20:11:37.475816 (+1377578us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0430 20:11:37.475847 (+ 31us) ts_tablet_manager.cc:655] Creating new metadata...
0430 20:11:38.535844 (+1059997us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:38.561079 (+ 25235us) tablet_replica.cc:205] Creating consensus instance
0430 20:11:38.561253 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240430 20:11:38.567015 18449 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41680 (request call id 186) took 2470 ms. Trace:
I20240430 20:11:38.611574 18449 rpcz_store.cc:276] 0430 20:11:36.096472 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:36.096567 (+ 95us) service_pool.cc:225] Handling call
0430 20:11:37.475719 (+1379152us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0430 20:11:37.475749 (+ 30us) ts_tablet_manager.cc:655] Creating new metadata...
0430 20:11:38.517202 (+1041453us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:38.566693 (+ 49491us) tablet_replica.cc:205] Creating consensus instance
0430 20:11:38.566865 (+ 172us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240430 20:11:38.582433 18417 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.177.3:44867, user_credentials={real_user=slave}} blocked reactor thread for 91803.8us
W20240430 20:11:38.612035 18417 leader_election.cc:341] T fcc9698abda4491b98e1e879589913a8 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:38.612089 19784 consensus_queue.cc:1035] T dd9b893d48684cfea7c242b93ecba01f P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240430 20:11:38.612387 18417 leader_election.cc:341] T 4e69ca12bd10494fb013614741bc166c P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:38.613078 18417 leader_election.cc:302] T 24dcc20832804229a2664873bf6ddaa7 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters: 5e1c6397f09747f091534faf93542710
I20240430 20:11:38.613359 19784 raft_consensus.cc:2781] T 24dcc20832804229a2664873bf6ddaa7 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:38.583755 20100 raft_consensus.cc:482] T e5111d7607dd42188a0176cf5be2ec05 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:38.613477 19784 raft_consensus.cc:482] T 24dcc20832804229a2664873bf6ddaa7 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:38.613607 19784 raft_consensus.cc:3037] T 24dcc20832804229a2664873bf6ddaa7 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:38.613605 20100 raft_consensus.cc:504] T e5111d7607dd42188a0176cf5be2ec05 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
W20240430 20:11:38.614216 18558 leader_election.cc:341] T c9e7660eccc448dbb99d135f82bfd261 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:38.614416 18610 raft_consensus.cc:2379] T 53a899fa5a424a0d99f1b47925e8c7ad P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2efb542aa55e439b99ccb98d1c5c884a in current term 1: Already voted for candidate 5e1c6397f09747f091534faf93542710 in this term.
I20240430 20:11:38.614753 19471 log.cc:1196] T f5236574672b477a9026e91adcaf70ec P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/f5236574672b477a9026e91adcaf70ec
I20240430 20:11:38.614944 18744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1701a15b115741aea13f8327e3d26b34. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:38.615206 18739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f4e860484784aa9a2d28af2e9fb611c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:38.615453 18734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e5225f1f5f544ecbf8dfa470af3daca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:38.615661 18732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f0b341a31224bf1be525b84b9500304. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:38.617120 18616 raft_consensus.cc:3037] T fcc9698abda4491b98e1e879589913a8 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:38.615001 18581 rpcz_store.cc:276] 0430 20:11:34.989936 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:34.990007 (+ 71us) service_pool.cc:225] Handling call
0430 20:11:35.936929 (+946922us) spinlock_profiling.cc:243] Waited 397 ms on lock 0x6160000c6454. stack: 00007ffa8b97e756 00007ffa8b97e510 00007ffa8a9b148d 0000000000955804 00000000009557cd 000000000095519a 00007ffa971bcb00 00007ffa9aecdabf 00007ffa956508e6 00007ffa95650615 00007ffa9268604e 00007ffa92684f2e 00007ffa9268b405 00007ffa9268bd2f 00007ffa9268bb58 0000000000a60bcc
0430 20:11:37.546386 (+1609457us) spinlock_profiling.cc:243] Waited 481 ms on lock 0x616000190f54. stack: 00007ffa8b97e756 00007ffa8b97e510 00007ffa8a9b148d 0000000000955804 00000000009557cd 000000000095519a 00007ffa971bcb00 00007ffa9aecdabf 00007ffa956508e6 00007ffa95650615 00007ffa9268604e 00007ffa92684f2e 00007ffa9268b405 00007ffa9268bd2f 00007ffa9268bb58 0000000000a60bcc
0430 20:11:38.541802 (+995416us) spinlock_profiling.cc:243] Waited 987 ms on lock 0x616000240454. stack: 00007ffa8b97e756 00007ffa8b97e510 00007ffa8a9b148d 0000000000955804 00000000009557cd 000000000095519a 00007ffa971bcb00 00007ffa9aecdabf 00007ffa956508e6 00007ffa95650615 00007ffa9268604e 00007ffa92684f2e 00007ffa9268b405 00007ffa9268bd2f 00007ffa9268bb58 0000000000a60bcc
0430 20:11:38.543782 (+ 1980us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4289029120}
I20240430 20:11:38.618929 18737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fc6908ecf954020a676b9d10a2b00e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:38.614507 18558 leader_election.cc:302] T acd23f0a63d5448aaa7ed78422e67bad P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:38.619266 18725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed0c37a71ede4bfc9eb5ad1a4365f25e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:38.619468 18617 raft_consensus.cc:3037] T 802336602d8f4ac4bd7e0872d6f3dae9 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:38.619627 20047 consensus_queue.cc:1035] T 26ed12f498644d15bf45e884dcad172e P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.016s
I20240430 20:11:38.619799 18729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2381047f6dc4bdf962f9383e6c41548. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:38.619952 18622 raft_consensus.cc:2454] T e6eac92dfa5f4456bc49571878f0f8dd P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 0.
I20240430 20:11:38.618808 20061 leader_election.cc:288] T e2d90ff17e574f88b2061b1c117d4b7d P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:38.622351 18741 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59830 (request call id 200) took 3682 ms. Trace:
I20240430 20:11:38.622500 18741 rpcz_store.cc:276] 0430 20:11:34.939680 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:38.404110 (+3464430us) service_pool.cc:225] Handling call
0430 20:11:38.420627 (+ 16517us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0430 20:11:38.420685 (+ 58us) ts_tablet_manager.cc:655] Creating new metadata...
0430 20:11:38.607428 (+186743us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:38.622113 (+ 14685us) tablet_replica.cc:205] Creating consensus instance
0430 20:11:38.622274 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240430 20:11:38.623318 20060 consensus_queue.cc:1035] T 12c6e131fa904d23865b8673b58a7e33 P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.009s
I20240430 20:11:38.623692 18983 raft_consensus.cc:348] T 24dcc20832804229a2664873bf6ddaa7 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:38.618885 18579 rpcz_store.cc:276] 0430 20:11:34.993062 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:34.993132 (+ 70us) service_pool.cc:225] Handling call
0430 20:11:35.937546 (+944414us) spinlock_profiling.cc:243] Waited 379 ms on lock 0x6160000c6454. stack: 00007ffa8b97e756 00007ffa8b97e510 00007ffa8a9b148d 0000000000955804 00000000009557cd 000000000095519a 00007ffa971bcb00 00007ffa9aecdabf 00007ffa956508e6 00007ffa95650615 00007ffa9268604e 00007ffa92684f2e 00007ffa9268b405 00007ffa9268bd2f 00007ffa9268bb58 0000000000a60bcc
0430 20:11:35.950053 (+ 12507us) spinlock_profiling.cc:243] Waited 2.68 ms on lock 0x616000152c30. stack: 00007ffa8b97e756 00007ffa8b97e510 00007ffa8a9b148d 0000000000955804 00000000009557cd 000000000095519a 00007ffa984d02f8 00007ffa9aecd8c8 00007ffa956508e6 00007ffa95650615 00007ffa9268604e 00007ffa92684f2e 00007ffa9268b405 00007ffa9268bd2f 00007ffa9268bb58 0000000000a60bcc
0430 20:11:38.404881 (+2454828us) spinlock_profiling.cc:243] Waited 1.2 s on lock 0x616000170854. stack: 00007ffa8b97e756 00007ffa8b97e510 00007ffa8a9b148d 0000000000955804 00000000009557cd 000000000095519a 00007ffa971bcb00 00007ffa9aecdabf 00007ffa956508e6 00007ffa95650615 00007ffa9268604e 00007ffa92684f2e 00007ffa9268b405 00007ffa9268bd2f 00007ffa9268bb58 0000000000a60bcc
0430 20:11:38.544801 (+139920us) spinlock_profiling.cc:243] Waited 138 ms on lock 0x616000240454. stack: 00007ffa8b97e756 00007ffa8b97e510 00007ffa8a9b148d 0000000000955804 00000000009557cd 000000000095519a 00007ffa971bcb00 00007ffa9aecdabf 00007ffa956508e6 00007ffa95650615 00007ffa9268604e 00007ffa92684f2e 00007ffa9268b405 00007ffa9268bd2f 00007ffa9268bb58 0000000000a60bcc
0430 20:11:38.547552 (+ 2751us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3957428608}
I20240430 20:11:38.624321 20047 raft_consensus.cc:2781] T acd23f0a63d5448aaa7ed78422e67bad P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:38.624428 20047 raft_consensus.cc:482] T acd23f0a63d5448aaa7ed78422e67bad P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:38.624531 20047 raft_consensus.cc:3037] T acd23f0a63d5448aaa7ed78422e67bad P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:38.626142 18983 raft_consensus.cc:374] T 24dcc20832804229a2664873bf6ddaa7 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:38.626252 18983 raft_consensus.cc:729] T 24dcc20832804229a2664873bf6ddaa7 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:38.626926 18983 consensus_queue.cc:260] T 24dcc20832804229a2664873bf6ddaa7 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:38.627633 19489 ts_tablet_manager.cc:1888] T 34ab40d8d91e401db299ddee7175ba28 P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:38.627787 19489 log.cc:1196] T 34ab40d8d91e401db299ddee7175ba28 P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/34ab40d8d91e401db299ddee7175ba28
I20240430 20:11:38.628260 19489 ts_tablet_manager.cc:1909] T 34ab40d8d91e401db299ddee7175ba28 P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:38.629257 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 62) took 7365 ms. Trace:
I20240430 20:11:38.594965 20103 raft_consensus.cc:482] T fd40967ab4a04001819e16c06a6c4300 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:38.629415 19489 rpcz_store.cc:276] 0430 20:11:31.263955 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:31.264042 (+ 87us) service_pool.cc:225] Handling call
0430 20:11:35.448020 (+4183978us) ts_tablet_manager.cc:1120] Deleting tablet 34ab40d8d91e401db299ddee7175ba28
0430 20:11:35.448029 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:37.511993 (+2063964us) spinlock_profiling.cc:243] Waited 45.9 ms on lock 0x61700008bfd0. stack: 00007f30e7d63756 00007f30e7d63510 00007f30e6d9648d 0000000000955804 00000000009557cd 000000000095519a 00007f30f34d6e21 00007f30f356449c 00007f30f35d1ae3 00007f30f48ab853 00007f30f733e4bc 00007f30f7370814 00007f30f7351b00 00007f30f735188d 0000000000a60bcc 00007f30e7da9b74
0430 20:11:38.497179 (+985186us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:38.625630 (+128451us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:38.629194 (+ 3564us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":424579,"spinlock_wait_cycles":105582976,"tablet-delete.queue_time_us":4183022}
I20240430 20:11:38.630620 19471 ts_tablet_manager.cc:1909] T f5236574672b477a9026e91adcaf70ec P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:38.631016 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 69) took 7593 ms. Trace:
I20240430 20:11:38.631435 19471 rpcz_store.cc:276] 0430 20:11:31.037741 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:31.037820 (+ 79us) service_pool.cc:225] Handling call
0430 20:11:38.481858 (+7444038us) ts_tablet_manager.cc:1120] Deleting tablet f5236574672b477a9026e91adcaf70ec
0430 20:11:38.481871 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:38.486018 (+ 4147us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:38.507457 (+ 21439us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:38.630970 (+123513us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7443584}
I20240430 20:11:38.631850 19471 tablet_replica.cc:330] T eed6338ad3d64859af087ee15e1ec798 P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:38.632297 19471 raft_consensus.cc:2229] T eed6338ad3d64859af087ee15e1ec798 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:38.632369 18736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7d3dea34cf54fe2b949d1b7ca68577a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:38.633599 18617 raft_consensus.cc:2454] T 802336602d8f4ac4bd7e0872d6f3dae9 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 1.
I20240430 20:11:38.634572 19471 pending_rounds.cc:62] T eed6338ad3d64859af087ee15e1ec798 P 5e1c6397f09747f091534faf93542710: Trying to abort 1 pending ops.
I20240430 20:11:38.634881 19471 pending_rounds.cc:68] T eed6338ad3d64859af087ee15e1ec798 P 5e1c6397f09747f091534faf93542710: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7022624331487547392 op_type: NO_OP noop_request { }
I20240430 20:11:38.635035 19471 raft_consensus.cc:2866] T eed6338ad3d64859af087ee15e1ec798 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20240430 20:11:38.635403 19471 raft_consensus.cc:2258] T eed6338ad3d64859af087ee15e1ec798 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:38.635931 18733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1017a50d1d646c486c0b9a75cb88ce4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:38.636070 19471 ts_tablet_manager.cc:1875] T eed6338ad3d64859af087ee15e1ec798 P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:38.637631 18983 ts_tablet_manager.cc:1398] T 24dcc20832804229a2664873bf6ddaa7 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.031s user 0.002s sys 0.001s
I20240430 20:11:38.634568 18616 raft_consensus.cc:2454] T fcc9698abda4491b98e1e879589913a8 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 1.
I20240430 20:11:38.638643 18983 tablet_bootstrap.cc:492] T 2a1e50f99c5d4b44bdbb51b58a92bc75 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:38.638978 19489 tablet_replica.cc:330] T cace1398492f4d968dcb6899385d527c P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:38.640421 20047 raft_consensus.cc:504] T acd23f0a63d5448aaa7ed78422e67bad P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:38.641491 20047 leader_election.cc:288] T acd23f0a63d5448aaa7ed78422e67bad P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:38.641882 18610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12c6e131fa904d23865b8673b58a7e33" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:38.642714 18617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24dcc20832804229a2664873bf6ddaa7" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:38.642944 18617 raft_consensus.cc:3037] T 24dcc20832804229a2664873bf6ddaa7 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:38.643332 18587 tablet_service.cc:1467] Processing CreateTablet for tablet c1017a50d1d646c486c0b9a75cb88ce4 (DEFAULT_TABLE table=default.Table_d95c70be474e415cba6a345438fad1ba [id=095b8b568ebc475aa3c21e0c03ef5d79]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:38.643780 18587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1017a50d1d646c486c0b9a75cb88ce4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:38.644112 18983 tablet_bootstrap.cc:654] T 2a1e50f99c5d4b44bdbb51b58a92bc75 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:38.646502 18617 raft_consensus.cc:2454] T 24dcc20832804229a2664873bf6ddaa7 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:38.647279 18592 tablet_service.cc:1467] Processing CreateTablet for tablet 2fc6908ecf954020a676b9d10a2b00e7 (DEFAULT_TABLE table=default.Table_766e3df58895427b9e5179eb555fe7e4 [id=5c11311047b34a3eb06cc06e8cc10223]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:38.647796 18592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fc6908ecf954020a676b9d10a2b00e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:38.651283 18588 tablet_service.cc:1467] Processing CreateTablet for tablet f7d3dea34cf54fe2b949d1b7ca68577a (DEFAULT_TABLE table=default.Table_c06225c5a599482dab4f418fd13c593f [id=ea8038670f7c44a9bb20c396b54d6440]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:38.651751 18588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7d3dea34cf54fe2b949d1b7ca68577a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:38.653283 19489 raft_consensus.cc:2229] T cace1398492f4d968dcb6899385d527c P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:38.655165 18600 tablet_service.cc:1467] Processing CreateTablet for tablet d2381047f6dc4bdf962f9383e6c41548 (DEFAULT_TABLE table=default.Table_3a588666409c4a64b13658a42a58f994 [id=053d7bb753464e968b05493053eb229a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:38.655611 18600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2381047f6dc4bdf962f9383e6c41548. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:38.659143 18601 tablet_service.cc:1514] Processing DeleteTablet for tablet e87fb2f0361d4aa3ada27bbdfa88e94c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:36 UTC) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:38.659305 18586 tablet_service.cc:1514] Processing DeleteTablet for tablet b76a4105a2974fe280e759349037d027 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:35 UTC) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:38.659489 18593 tablet_service.cc:1514] Processing DeleteTablet for tablet 558e71ec370d46ca8fbdcec5b57eb2ec with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:37 UTC) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:38.504516 18738 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59830 (request call id 190) took 4223 ms. Trace:
I20240430 20:11:38.427814 18731 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59830 (request call id 196) took 3492 ms. Trace:
I20240430 20:11:38.531841 18988 tablet_bootstrap.cc:492] T fcc9698abda4491b98e1e879589913a8 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
W20240430 20:11:38.551038 18696 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.17.177.2:46027 timed out after 3.746s (SENT)
I20240430 20:11:38.554876 18740 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59830 (request call id 174) took 4666 ms. Trace:
I20240430 20:11:38.571549 18742 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59830 (request call id 173) took 4685 ms. Trace:
I20240430 20:11:38.441552 18728 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59830 (request call id 186) took 4164 ms. Trace:
I20240430 20:11:38.678992 19489 raft_consensus.cc:2258] T cace1398492f4d968dcb6899385d527c P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240430 20:11:38.613483 18417 leader_election.cc:341] T 802336602d8f4ac4bd7e0872d6f3dae9 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:38.679689 20103 raft_consensus.cc:504] T fd40967ab4a04001819e16c06a6c4300 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
W20240430 20:11:38.683542 19001 catalog_manager.cc:4909] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): alter failed for tablet 12c6e131fa904d23865b8673b58a7e33 (table default.Table_a9dc8fc6a4aa46dc914d0e327a4de554 [id=19f4e6decedf4ee59fccc0a24a05cedd]): Illegal state: replica 3c51c33d5616482cafb68e26688820f4 is not leader of this config: current role FOLLOWER
W20240430 20:11:38.684598 18417 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.177.3:44867, user_credentials={real_user=slave}} blocked reactor thread for 66967.7us
I20240430 20:11:38.686051 19784 raft_consensus.cc:504] T 24dcc20832804229a2664873bf6ddaa7 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:38.686393 18728 rpcz_store.cc:276] 0430 20:11:34.277358 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:35.442785 (+1165427us) service_pool.cc:225] Handling call
0430 20:11:35.443484 (+ 699us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0430 20:11:35.443520 (+ 36us) ts_tablet_manager.cc:655] Creating new metadata...
0430 20:11:38.416807 (+2973287us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:38.441268 (+ 24461us) tablet_replica.cc:205] Creating consensus instance
0430 20:11:38.441460 (+ 192us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240430 20:11:38.686693 18726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 558e71ec370d46ca8fbdcec5b57eb2ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:38.686808 18473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53a899fa5a424a0d99f1b47925e8c7ad" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:38.691191 18999 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet cace1398492f4d968dcb6899385d527c (table default.Table_a8d5ac73c6d047698d87b3d1ba34e66f [id=9c80193cde2346c982cf84ae9612e2f7]) successfully deleted
I20240430 20:11:38.691779 18999 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet 4e69ca12bd10494fb013614741bc166c (table default.Table_0a397bb1e5ea4cbe87a088c7fc84e9c1 [id=9abcfa9eaaa84cd2b679fd29ec6e5ab7]) successfully deleted
I20240430 20:11:38.697279 18731 rpcz_store.cc:276] 0430 20:11:34.935286 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:35.861420 (+926134us) service_pool.cc:225] Handling call
0430 20:11:35.872540 (+ 11120us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0430 20:11:35.872572 (+ 32us) ts_tablet_manager.cc:655] Creating new metadata...
0430 20:11:38.422841 (+2550269us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:38.427169 (+ 4328us) tablet_replica.cc:205] Creating consensus instance
0430 20:11:38.427685 (+ 516us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240430 20:11:38.685216 18738 rpcz_store.cc:276] 0430 20:11:34.281083 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:35.858065 (+1576982us) service_pool.cc:225] Handling call
0430 20:11:35.872331 (+ 14266us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0430 20:11:35.872382 (+ 51us) ts_tablet_manager.cc:655] Creating new metadata...
0430 20:11:38.501227 (+2628845us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:38.504285 (+ 3058us) tablet_replica.cc:205] Creating consensus instance
0430 20:11:38.504448 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240430 20:11:38.699122 18724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df92eb85261d47d88cbf16d29d9ad30e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:38.702615 18988 ts_tablet_manager.cc:1367] T fcc9698abda4491b98e1e879589913a8 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 4.213s user 0.081s sys 0.111s
I20240430 20:11:38.704911 18988 raft_consensus.cc:348] T fcc9698abda4491b98e1e879589913a8 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:38.686707 20100 leader_election.cc:288] T e5111d7607dd42188a0176cf5be2ec05 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
W20240430 20:11:38.686343 19001 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.AlterSchema -> {remote=127.17.177.3:44867, user_credentials={real_user=slave}} blocked reactor thread for 81288.1us
I20240430 20:11:38.705888 18739 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59830 (request call id 168) took 4823 ms. Trace:
I20240430 20:11:38.706353 18472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "724e51ff05ec4baebc2bb55e05c8eddc" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:38.707527 18740 rpcz_store.cc:276] 0430 20:11:33.888702 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:33.888753 (+ 51us) service_pool.cc:225] Handling call
0430 20:11:34.465117 (+576364us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0430 20:11:34.465132 (+ 15us) ts_tablet_manager.cc:655] Creating new metadata...
0430 20:11:38.484472 (+4019340us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:38.554285 (+ 69813us) tablet_replica.cc:205] Creating consensus instance
0430 20:11:38.554741 (+ 456us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240430 20:11:38.651298 19471 ts_tablet_manager.cc:1888] T eed6338ad3d64859af087ee15e1ec798 P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:38.708632 19471 log.cc:1196] T eed6338ad3d64859af087ee15e1ec798 P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/eed6338ad3d64859af087ee15e1ec798
I20240430 20:11:38.709815 18473 raft_consensus.cc:3037] T 53a899fa5a424a0d99f1b47925e8c7ad P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:38.710332 19471 ts_tablet_manager.cc:1909] T eed6338ad3d64859af087ee15e1ec798 P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:38.711148 19784 leader_election.cc:288] T 24dcc20832804229a2664873bf6ddaa7 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Requested vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:38.711529 19489 ts_tablet_manager.cc:1875] T cace1398492f4d968dcb6899385d527c P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:38.710248 18742 rpcz_store.cc:276] 0430 20:11:33.885942 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:33.885990 (+ 48us) service_pool.cc:225] Handling call
0430 20:11:34.464988 (+578998us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0430 20:11:34.465004 (+ 16us) ts_tablet_manager.cc:655] Creating new metadata...
0430 20:11:38.494428 (+4029424us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:38.570959 (+ 76531us) tablet_replica.cc:205] Creating consensus instance
0430 20:11:38.571457 (+ 498us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240430 20:11:38.712245 20103 leader_election.cc:288] T fd40967ab4a04001819e16c06a6c4300 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
W20240430 20:11:38.712491 18417 leader_election.cc:341] T b688e03ea71940c69bda5520b62c1fc3 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
W20240430 20:11:38.713027 19001 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 34130f2820064fa0898ef1f9d7de1c6d (table default.Table_343ee22ed6554f328aa58881079e1905 [id=5a1c4a7be2fc465fac89a50854eb3322], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 34130f2820064fa0898ef1f9d7de1c6d consensus state has no leader
I20240430 20:11:38.715350 19001 catalog_manager.cc:4830] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): tablet 34ab40d8d91e401db299ddee7175ba28 (table default.Table_0b4b99c1c03a4e779046063159ce6861 [id=43d7bdc84bab4c54bac58dd1dd0f74ac]) successfully deleted
W20240430 20:11:38.715639 18558 leader_election.cc:341] T 47c37876694c4a4abb1091c631dab7a5 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Not found: Tablet not found: 47c37876694c4a4abb1091c631dab7a5
I20240430 20:11:38.716333 18999 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet f5236574672b477a9026e91adcaf70ec (table default.Table_b8388e076be44f45ab6ed2ab3bce74a8 [id=00dd4c11a1cc4018bb900da95768bb88]) successfully deleted
W20240430 20:11:38.716657 18558 leader_election.cc:341] T 13c1107ea4d04eb78a74d3f658c72fb3 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Not found: Tablet not found: 13c1107ea4d04eb78a74d3f658c72fb3
W20240430 20:11:38.718935 18557 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.17.177.3:44867 timed out after 3.026s (SENT)
I20240430 20:11:38.722716 18988 raft_consensus.cc:374] T fcc9698abda4491b98e1e879589913a8 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:38.722854 18988 raft_consensus.cc:729] T fcc9698abda4491b98e1e879589913a8 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:38.723114 18988 consensus_queue.cc:260] T fcc9698abda4491b98e1e879589913a8 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:38.723698 18988 ts_tablet_manager.cc:1398] T fcc9698abda4491b98e1e879589913a8 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.021s user 0.001s sys 0.001s
I20240430 20:11:38.724217 18739 rpcz_store.cc:276] 0430 20:11:33.882459 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:33.882520 (+ 61us) service_pool.cc:225] Handling call
0430 20:11:34.464180 (+581660us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0430 20:11:34.464196 (+ 16us) ts_tablet_manager.cc:655] Creating new metadata...
0430 20:11:38.625660 (+4161464us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:38.705693 (+ 80033us) tablet_replica.cc:205] Creating consensus instance
0430 20:11:38.705844 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240430 20:11:38.724527 18417 leader_election.cc:341] T 255c13ad8fae4ea4aa8e7b7aabb7602e P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:38.724475 18482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acd23f0a63d5448aaa7ed78422e67bad" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:38.725628 18732 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59830 (request call id 166) took 4845 ms. Trace:
I20240430 20:11:38.725761 18732 rpcz_store.cc:276] 0430 20:11:33.880241 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:33.880289 (+ 48us) service_pool.cc:225] Handling call
0430 20:11:34.463875 (+583586us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0430 20:11:34.463902 (+ 27us) ts_tablet_manager.cc:655] Creating new metadata...
0430 20:11:38.648247 (+4184345us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:38.724045 (+ 75798us) tablet_replica.cc:205] Creating consensus instance
0430 20:11:38.725555 (+ 1510us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240430 20:11:38.726837 18988 tablet_bootstrap.cc:492] T 26ed12f498644d15bf45e884dcad172e P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:38.730836 18737 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59830 (request call id 205) took 1222 ms. Trace:
I20240430 20:11:38.731009 18417 leader_election.cc:302] T 255c13ad8fae4ea4aa8e7b7aabb7602e P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a; no voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710
I20240430 20:11:38.724498 18472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aec9c09cc0454acab6006bb3e10982b7" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:38.733484 18985 tablet_bootstrap.cc:654] T 01436e5b6232494babbb414342f03090 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:38.734356 20103 raft_consensus.cc:2726] T 255c13ad8fae4ea4aa8e7b7aabb7602e P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:38.734725 18614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75ebf051383e42529dcb5f7dd0770a4b" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:38.734750 18617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76d6e9d547ca4a848eae852944b1d117" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:38.733645 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 70) took 7695 ms. Trace:
I20240430 20:11:38.736166 19471 rpcz_store.cc:276] 0430 20:11:31.037897 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:31.037994 (+ 97us) service_pool.cc:225] Handling call
0430 20:11:38.631777 (+7593783us) ts_tablet_manager.cc:1120] Deleting tablet eed6338ad3d64859af087ee15e1ec798
0430 20:11:38.631790 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:38.647191 (+ 15401us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:38.651141 (+ 3950us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:38.733434 (+ 82293us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7593473}
I20240430 20:11:38.738090 18999 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet eed6338ad3d64859af087ee15e1ec798 (table default.Table_1dc3c3ef217d4f8b9e8fdb60180c5d0f [id=bc29c76ef4a448a2863c278309b548e8]) successfully deleted
W20240430 20:11:38.738181 18417 leader_election.cc:341] T fcc9698abda4491b98e1e879589913a8 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:38.738412 19471 tablet_replica.cc:330] T 13c1107ea4d04eb78a74d3f658c72fb3 P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:38.738692 18483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2d90ff17e574f88b2061b1c117d4b7d" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:38.739323 19784 raft_consensus.cc:482] T 372b135dd01f44f98cd713ca8857ba61 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3c51c33d5616482cafb68e26688820f4)
I20240430 20:11:38.740123 18475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77285afcafe04a829e485998bbbe00f0" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:38.740532 18482 raft_consensus.cc:3037] T acd23f0a63d5448aaa7ed78422e67bad P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:38.733343 18737 rpcz_store.cc:276] 0430 20:11:37.480159 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:38.486473 (+1006314us) service_pool.cc:225] Handling call
0430 20:11:38.603700 (+117227us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0430 20:11:38.603727 (+ 27us) ts_tablet_manager.cc:655] Creating new metadata...
0430 20:11:38.629374 (+ 25647us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:38.702098 (+ 72724us) tablet_replica.cc:205] Creating consensus instance
0430 20:11:38.702285 (+ 187us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240430 20:11:38.739766 19471 raft_consensus.cc:2229] T 13c1107ea4d04eb78a74d3f658c72fb3 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:38.741587 19471 raft_consensus.cc:2258] T 13c1107ea4d04eb78a74d3f658c72fb3 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:38.742653 18480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b72fe1f0713b4076a81a3fc1c74f7fa7" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:38.743225 18481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e826459c3f4437da81c9e6e09f0c22f" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:38.743921 18474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcc9698abda4491b98e1e879589913a8" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:38.744231 18474 raft_consensus.cc:2379] T fcc9698abda4491b98e1e879589913a8 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e1c6397f09747f091534faf93542710 in current term 1: Already voted for candidate 2efb542aa55e439b99ccb98d1c5c884a in this term.
I20240430 20:11:38.745654 18450 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41680 (request call id 185) took 2646 ms. Trace:
I20240430 20:11:38.745790 18450 rpcz_store.cc:276] 0430 20:11:36.093764 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:36.094055 (+ 291us) service_pool.cc:225] Handling call
0430 20:11:37.475572 (+1381517us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0430 20:11:37.475619 (+ 47us) ts_tablet_manager.cc:655] Creating new metadata...
0430 20:11:38.613622 (+1138003us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:38.738987 (+125365us) tablet_replica.cc:205] Creating consensus instance
0430 20:11:38.739368 (+ 381us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240430 20:11:38.745925 18473 raft_consensus.cc:2454] T 53a899fa5a424a0d99f1b47925e8c7ad P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
I20240430 20:11:38.746788 18729 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59830 (request call id 207) took 1264 ms. Trace:
I20240430 20:11:38.746909 18729 rpcz_store.cc:276] 0430 20:11:37.480371 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:38.490829 (+1010458us) service_pool.cc:225] Handling call
0430 20:11:38.599663 (+108834us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0430 20:11:38.599694 (+ 31us) ts_tablet_manager.cc:655] Creating new metadata...
0430 20:11:38.653752 (+ 54058us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:38.744307 (+ 90555us) tablet_replica.cc:205] Creating consensus instance
0430 20:11:38.744476 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240430 20:11:38.746774 19784 raft_consensus.cc:504] T 372b135dd01f44f98cd713ca8857ba61 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:38.747582 19784 leader_election.cc:288] T 372b135dd01f44f98cd713ca8857ba61 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
W20240430 20:11:38.749259 18417 leader_election.cc:341] T d9e578c99058424a9b86390641f9c34f P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Not found: Tablet not found: d9e578c99058424a9b86390641f9c34f
W20240430 20:11:38.750684 18699 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.177.1:41217, user_credentials={real_user=slave}} blocked reactor thread for 137851us
I20240430 20:11:38.750788 19004 catalog_manager.cc:4830] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): tablet 34ab40d8d91e401db299ddee7175ba28 (table default.Table_0b4b99c1c03a4e779046063159ce6861 [id=43d7bdc84bab4c54bac58dd1dd0f74ac]) successfully deleted
I20240430 20:11:38.753609 18482 raft_consensus.cc:2454] T acd23f0a63d5448aaa7ed78422e67bad P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
I20240430 20:11:38.754133 19471 ts_tablet_manager.cc:1875] T 13c1107ea4d04eb78a74d3f658c72fb3 P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:38.755985 18558 leader_election.cc:302] T 53a899fa5a424a0d99f1b47925e8c7ad P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:38.756389 20047 raft_consensus.cc:2781] T 53a899fa5a424a0d99f1b47925e8c7ad P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:38.756570 20047 raft_consensus.cc:686] T 53a899fa5a424a0d99f1b47925e8c7ad P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Becoming Leader. State: Replica: 5e1c6397f09747f091534faf93542710, State: Running, Role: LEADER
I20240430 20:11:38.756676 18983 tablet_bootstrap.cc:492] T 2a1e50f99c5d4b44bdbb51b58a92bc75 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:38.756862 20047 consensus_queue.cc:237] T 53a899fa5a424a0d99f1b47925e8c7ad P 5e1c6397f09747f091534faf93542710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:38.757273 18558 leader_election.cc:302] T acd23f0a63d5448aaa7ed78422e67bad P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:38.757561 20060 raft_consensus.cc:2781] T acd23f0a63d5448aaa7ed78422e67bad P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:38.757697 20060 raft_consensus.cc:686] T acd23f0a63d5448aaa7ed78422e67bad P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Becoming Leader. State: Replica: 5e1c6397f09747f091534faf93542710, State: Running, Role: LEADER
I20240430 20:11:38.757966 20060 consensus_queue.cc:237] T acd23f0a63d5448aaa7ed78422e67bad P 5e1c6397f09747f091534faf93542710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
W20240430 20:11:38.759063 18417 connection.cc:232] RPC call timeout handler was delayed by -1.5872285110000348s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.32430000000000003s for a response
I20240430 20:11:38.759949 18449 tablet_service.cc:1467] Processing CreateTablet for tablet 2fc6908ecf954020a676b9d10a2b00e7 (DEFAULT_TABLE table=default.Table_766e3df58895427b9e5179eb555fe7e4 [id=5c11311047b34a3eb06cc06e8cc10223]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:38.760365 18454 tablet_service.cc:1467] Processing CreateTablet for tablet d2381047f6dc4bdf962f9383e6c41548 (DEFAULT_TABLE table=default.Table_3a588666409c4a64b13658a42a58f994 [id=053d7bb753464e968b05493053eb229a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:38.760545 18449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fc6908ecf954020a676b9d10a2b00e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:38.760833 18454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2381047f6dc4bdf962f9383e6c41548. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:38.761233 18446 tablet_service.cc:1467] Processing CreateTablet for tablet f7d3dea34cf54fe2b949d1b7ca68577a (DEFAULT_TABLE table=default.Table_c06225c5a599482dab4f418fd13c593f [id=ea8038670f7c44a9bb20c396b54d6440]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:38.761652 18446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7d3dea34cf54fe2b949d1b7ca68577a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:38.763098 18447 tablet_service.cc:1514] Processing DeleteTablet for tablet 558e71ec370d46ca8fbdcec5b57eb2ec with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:37 UTC) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:38.765664 18450 tablet_service.cc:1467] Processing CreateTablet for tablet c1017a50d1d646c486c0b9a75cb88ce4 (DEFAULT_TABLE table=default.Table_d95c70be474e415cba6a345438fad1ba [id=095b8b568ebc475aa3c21e0c03ef5d79]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:38.766075 18450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1017a50d1d646c486c0b9a75cb88ce4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:38.775091 18459 tablet_service.cc:1514] Processing DeleteTablet for tablet e87fb2f0361d4aa3ada27bbdfa88e94c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:36 UTC) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:38.776391 18616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24dcc20832804229a2664873bf6ddaa7" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:38.776741 18616 raft_consensus.cc:2379] T 24dcc20832804229a2664873bf6ddaa7 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 2efb542aa55e439b99ccb98d1c5c884a in current term 1: Already voted for candidate 3c51c33d5616482cafb68e26688820f4 in this term.
I20240430 20:11:38.777033 18612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd40967ab4a04001819e16c06a6c4300" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:38.777412 18617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3e74e76ab2b4fe492ffcc1838656fd1" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:38.777665 18617 raft_consensus.cc:2454] T e3e74e76ab2b4fe492ffcc1838656fd1 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 1.
W20240430 20:11:38.782909 18699 leader_election.cc:341] T c9e7660eccc448dbb99d135f82bfd261 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:38.784003 18447 tablet_service.cc:1514] Processing DeleteTablet for tablet b76a4105a2974fe280e759349037d027 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:35 UTC) from {username='slave'} at 127.0.0.1:41680
W20240430 20:11:38.784428 18415 leader_election.cc:341] T 93d158de64994f3c9fa45a1250a4f533 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Not found: Tablet not found: 93d158de64994f3c9fa45a1250a4f533
I20240430 20:11:38.784945 18988 tablet_bootstrap.cc:654] T 26ed12f498644d15bf45e884dcad172e P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:38.788398 18733 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59830 (request call id 204) took 1308 ms. Trace:
I20240430 20:11:38.756860 18983 ts_tablet_manager.cc:1367] T 2a1e50f99c5d4b44bdbb51b58a92bc75 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.119s user 0.016s sys 0.016s
I20240430 20:11:38.794278 18983 raft_consensus.cc:348] T 2a1e50f99c5d4b44bdbb51b58a92bc75 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:38.794646 18983 raft_consensus.cc:374] T 2a1e50f99c5d4b44bdbb51b58a92bc75 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:38.794899 18983 raft_consensus.cc:729] T 2a1e50f99c5d4b44bdbb51b58a92bc75 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:38.795403 18983 consensus_queue.cc:260] T 2a1e50f99c5d4b44bdbb51b58a92bc75 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:38.799386 19049 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:51664 (request call id 173) took 3133 ms. Trace:
I20240430 20:11:38.799546 19049 rpcz_store.cc:276] 0430 20:11:35.665960 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:37.496540 (+1830580us) service_pool.cc:225] Handling call
0430 20:11:38.394940 (+898400us) catalog_manager.cc:2072] Acquired catalog manager lock
0430 20:11:38.395499 (+ 559us) catalog_manager.cc:2118] Created new table and tablet info
0430 20:11:38.799119 (+403620us) catalog_manager.cc:2178] Wrote table and tablets to system table
0430 20:11:38.799243 (+ 124us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0430 20:11:38.799345 (+ 102us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:38.532660 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:38.532861 (+ 201us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:38.532869 (+ 8us) write_op.cc:427] Acquired schema lock
0430 20:11:38.532872 (+ 3us) tablet.cc:660] Decoding operations
0430 20:11:38.533046 (+ 174us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:38.533061 (+ 15us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:38.533066 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0430 20:11:38.533199 (+ 133us) tablet.cc:699] Row locks acquired
0430 20:11:38.533201 (+ 2us) write_op.cc:260] PREPARE: finished
0430 20:11:38.533271 (+ 70us) write_op.cc:270] Start()
0430 20:11:38.533339 (+ 68us) write_op.cc:276] Timestamp: P: 1714507898533219 usec, L: 0
0430 20:11:38.533369 (+ 30us) op_driver.cc:331] REPLICATION: starting
0430 20:11:38.533739 (+ 370us) log.cc:842] Serialized 433 byte log entry
0430 20:11:38.739547 (+205808us) op_driver.cc:444] REPLICATION: finished
0430 20:11:38.798514 (+ 58967us) write_op.cc:301] APPLY: starting
0430 20:11:38.798534 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:38.798576 (+ 42us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:38.798644 (+ 68us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:38.798806 (+ 162us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:38.798842 (+ 36us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0430 20:11:38.798850 (+ 8us) write_op.cc:312] APPLY: finished
0430 20:11:38.798927 (+ 77us) log.cc:842] Serialized 29 byte log entry
0430 20:11:38.799001 (+ 74us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:38.799025 (+ 24us) write_op.cc:446] Released schema lock
0430 20:11:38.799032 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":58872,"num_ops":2,"prepare.queue_time_us":136537,"prepare.run_cpu_time_us":1069,"prepare.run_wall_time_us":1313,"raft.queue_time_us":340,"raft.run_cpu_time_us":240,"raft.run_wall_time_us":241,"replication_time_us":206142,"thread_start_us":167849,"threads_started":2}]]}
I20240430 20:11:38.802487 18415 leader_election.cc:302] T 93d158de64994f3c9fa45a1250a4f533 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a; no voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710
I20240430 20:11:38.804344 18736 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59830 (request call id 206) took 1312 ms. Trace:
I20240430 20:11:38.804503 18736 rpcz_store.cc:276] 0430 20:11:37.480256 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:38.606877 (+1126621us) service_pool.cc:225] Handling call
0430 20:11:38.607394 (+ 517us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0430 20:11:38.607419 (+ 25us) ts_tablet_manager.cc:655] Creating new metadata...
0430 20:11:38.711724 (+104305us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:38.792116 (+ 80392us) tablet_replica.cc:205] Creating consensus instance
0430 20:11:38.792257 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240430 20:11:38.805366 18733 rpcz_store.cc:276] 0430 20:11:37.480070 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:38.601886 (+1121816us) service_pool.cc:225] Handling call
0430 20:11:38.602364 (+ 478us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0430 20:11:38.602392 (+ 28us) ts_tablet_manager.cc:655] Creating new metadata...
0430 20:11:38.748989 (+146597us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:38.788124 (+ 39135us) tablet_replica.cc:205] Creating consensus instance
0430 20:11:38.788271 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240430 20:11:38.811798 19784 raft_consensus.cc:2726] T 93d158de64994f3c9fa45a1250a4f533 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240430 20:11:38.816396 18417 leader_election.cc:302] T 53a899fa5a424a0d99f1b47925e8c7ad P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters:
I20240430 20:11:38.819901 18473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12c6e131fa904d23865b8673b58a7e33" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:38.821511 18482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24dcc20832804229a2664873bf6ddaa7" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:38.822038 18482 raft_consensus.cc:2379] T 24dcc20832804229a2664873bf6ddaa7 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 3c51c33d5616482cafb68e26688820f4 in current term 1: Already voted for candidate 2efb542aa55e439b99ccb98d1c5c884a in this term.
I20240430 20:11:38.822268 19784 raft_consensus.cc:2741] T 53a899fa5a424a0d99f1b47925e8c7ad P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
W20240430 20:11:38.824000 18415 consensus_peers.cc:485] T f5236574672b477a9026e91adcaf70ec P 2efb542aa55e439b99ccb98d1c5c884a -> Peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Couldn't send request to peer 5e1c6397f09747f091534faf93542710. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240430 20:11:38.826301 18726 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59830 (request call id 188) took 4527 ms. Trace:
W20240430 20:11:38.830348 18415 leader_election.cc:341] T d9e578c99058424a9b86390641f9c34f P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Not found: Tablet not found: d9e578c99058424a9b86390641f9c34f
I20240430 20:11:38.830536 18415 leader_election.cc:302] T d9e578c99058424a9b86390641f9c34f P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a; no voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710
I20240430 20:11:38.830857 19784 raft_consensus.cc:2726] T d9e578c99058424a9b86390641f9c34f P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240430 20:11:38.831449 18415 consensus_peers.cc:485] T cace1398492f4d968dcb6899385d527c P 2efb542aa55e439b99ccb98d1c5c884a -> Peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Couldn't send request to peer 5e1c6397f09747f091534faf93542710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cace1398492f4d968dcb6899385d527c. This is attempt 1: this message will repeat every 5th retry.
I20240430 20:11:38.832772 18473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cace1398492f4d968dcb6899385d527c" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:38.836128 18415 leader_election.cc:302] T e6eac92dfa5f4456bc49571878f0f8dd P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:38.836512 19784 raft_consensus.cc:2781] T e6eac92dfa5f4456bc49571878f0f8dd P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:38.836649 19784 raft_consensus.cc:482] T e6eac92dfa5f4456bc49571878f0f8dd P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:38.836752 19784 raft_consensus.cc:3037] T e6eac92dfa5f4456bc49571878f0f8dd P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:38.839957 19028 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:51664 (request call id 174) took 3173 ms. Trace:
I20240430 20:11:38.840108 19028 rpcz_store.cc:276] 0430 20:11:35.666034 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:38.408706 (+2742672us) service_pool.cc:225] Handling call
0430 20:11:38.409640 (+ 934us) catalog_manager.cc:2072] Acquired catalog manager lock
0430 20:11:38.410145 (+ 505us) catalog_manager.cc:2118] Created new table and tablet info
0430 20:11:38.839622 (+429477us) catalog_manager.cc:2178] Wrote table and tablets to system table
0430 20:11:38.839786 (+ 164us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0430 20:11:38.839910 (+ 124us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:38.535511 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:38.535673 (+ 162us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:38.535679 (+ 6us) write_op.cc:427] Acquired schema lock
0430 20:11:38.535682 (+ 3us) tablet.cc:660] Decoding operations
0430 20:11:38.535899 (+ 217us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:38.535908 (+ 9us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:38.535950 (+ 42us) tablet.cc:683] Acquiring locks for 2 operations
0430 20:11:38.536103 (+ 153us) tablet.cc:699] Row locks acquired
0430 20:11:38.536105 (+ 2us) write_op.cc:260] PREPARE: finished
0430 20:11:38.536176 (+ 71us) write_op.cc:270] Start()
0430 20:11:38.536248 (+ 72us) write_op.cc:276] Timestamp: P: 1714507898536127 usec, L: 0
0430 20:11:38.536255 (+ 7us) op_driver.cc:331] REPLICATION: starting
0430 20:11:38.567003 (+ 30748us) log.cc:842] Serialized 433 byte log entry
0430 20:11:38.813384 (+246381us) op_driver.cc:444] REPLICATION: finished
0430 20:11:38.837100 (+ 23716us) write_op.cc:301] APPLY: starting
0430 20:11:38.837119 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:38.837168 (+ 49us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:38.837221 (+ 53us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:38.837400 (+ 179us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:38.837440 (+ 40us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0430 20:11:38.837448 (+ 8us) write_op.cc:312] APPLY: finished
0430 20:11:38.837529 (+ 81us) log.cc:842] Serialized 29 byte log entry
0430 20:11:38.837594 (+ 65us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:38.837616 (+ 22us) write_op.cc:446] Released schema lock
0430 20:11:38.837623 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:38.837677 (+ 54us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:38.837680 (+ 3us) write_op.cc:446] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":23640,"apply.run_cpu_time_us":637,"apply.run_wall_time_us":637,"num_ops":2,"prepare.queue_time_us":103469,"prepare.run_cpu_time_us":1149,"prepare.run_wall_time_us":34148,"replication_time_us":277109,"threads_started":1}]]}
I20240430 20:11:38.843694 19784 raft_consensus.cc:504] T e6eac92dfa5f4456bc49571878f0f8dd P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:38.844723 19784 leader_election.cc:288] T e6eac92dfa5f4456bc49571878f0f8dd P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Requested vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
W20240430 20:11:38.847682 18557 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.177.3:44867, user_credentials={real_user=slave}} blocked reactor thread for 128826us
I20240430 20:11:38.848822 18557 leader_election.cc:302] T b76a4105a2974fe280e759349037d027 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:38.849251 20061 raft_consensus.cc:2726] T b76a4105a2974fe280e759349037d027 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240430 20:11:38.849458 18557 leader_election.cc:341] T acd23f0a63d5448aaa7ed78422e67bad P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:38.849833 18557 leader_election.cc:302] T c9e7660eccc448dbb99d135f82bfd261 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a
I20240430 20:11:38.850153 20061 raft_consensus.cc:2781] T c9e7660eccc448dbb99d135f82bfd261 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:38.850266 20061 raft_consensus.cc:482] T c9e7660eccc448dbb99d135f82bfd261 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:38.850390 20061 raft_consensus.cc:3037] T c9e7660eccc448dbb99d135f82bfd261 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:38.850836 20038 consensus_queue.cc:1035] T 12c6e131fa904d23865b8673b58a7e33 P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:38.851034 18726 rpcz_store.cc:276] 0430 20:11:34.279036 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:35.856390 (+1577354us) service_pool.cc:225] Handling call
0430 20:11:35.870622 (+ 14232us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0430 20:11:35.870671 (+ 49us) ts_tablet_manager.cc:655] Creating new metadata...
0430 20:11:38.771179 (+2900508us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:38.806606 (+ 35427us) tablet_replica.cc:205] Creating consensus instance
0430 20:11:38.806777 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240430 20:11:38.853994 20061 raft_consensus.cc:504] T c9e7660eccc448dbb99d135f82bfd261 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:38.855158 20061 leader_election.cc:288] T c9e7660eccc448dbb99d135f82bfd261 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:38.858881 19805 raft_consensus.cc:482] T e2d90ff17e574f88b2061b1c117d4b7d P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2efb542aa55e439b99ccb98d1c5c884a)
I20240430 20:11:38.862686 20107 raft_consensus.cc:482] T aec9c09cc0454acab6006bb3e10982b7 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2efb542aa55e439b99ccb98d1c5c884a)
I20240430 20:11:38.862824 19495 ts_tablet_manager.cc:1888] T c5fd0730133c472299c93f4b0830e527 P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:38.862746 20019 raft_consensus.cc:482] T 77285afcafe04a829e485998bbbe00f0 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2efb542aa55e439b99ccb98d1c5c884a)
I20240430 20:11:38.862898 20107 raft_consensus.cc:504] T aec9c09cc0454acab6006bb3e10982b7 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:38.863076 20019 raft_consensus.cc:504] T 77285afcafe04a829e485998bbbe00f0 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:38.862800 19805 raft_consensus.cc:504] T e2d90ff17e574f88b2061b1c117d4b7d P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:38.863734 20107 leader_election.cc:288] T aec9c09cc0454acab6006bb3e10982b7 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:38.864625 19495 log.cc:1196] T c5fd0730133c472299c93f4b0830e527 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/c5fd0730133c472299c93f4b0830e527
I20240430 20:11:38.866003 20061 raft_consensus.cc:482] T 7dfdecba959543a795dfcd3eeda29e58 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3c51c33d5616482cafb68e26688820f4)
I20240430 20:11:38.864127 19805 leader_election.cc:288] T e2d90ff17e574f88b2061b1c117d4b7d P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:38.866240 20061 raft_consensus.cc:504] T 7dfdecba959543a795dfcd3eeda29e58 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:38.866627 20085 raft_consensus.cc:482] T 724e51ff05ec4baebc2bb55e05c8eddc P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2efb542aa55e439b99ccb98d1c5c884a)
I20240430 20:11:38.866794 20085 raft_consensus.cc:504] T 724e51ff05ec4baebc2bb55e05c8eddc P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:38.867288 20061 leader_election.cc:288] T 7dfdecba959543a795dfcd3eeda29e58 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:38.867765 20019 leader_election.cc:288] T 77285afcafe04a829e485998bbbe00f0 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:38.868418 18988 tablet_bootstrap.cc:492] T 26ed12f498644d15bf45e884dcad172e P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:38.868593 18988 ts_tablet_manager.cc:1367] T 26ed12f498644d15bf45e884dcad172e P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.145s user 0.005s sys 0.005s
I20240430 20:11:38.868636 20061 raft_consensus.cc:482] T 81787eccca6c42c1b421377a3bf9ebe6 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3c51c33d5616482cafb68e26688820f4)
I20240430 20:11:38.868820 20061 raft_consensus.cc:504] T 81787eccca6c42c1b421377a3bf9ebe6 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:38.869882 20061 leader_election.cc:288] T 81787eccca6c42c1b421377a3bf9ebe6 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
W20240430 20:11:38.871230 18696 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.177.2:46027, user_credentials={real_user=slave}} blocked reactor thread for 320429us
I20240430 20:11:38.873567 19489 ts_tablet_manager.cc:1888] T cace1398492f4d968dcb6899385d527c P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:38.873924 18617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aec9c09cc0454acab6006bb3e10982b7" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:38.874018 18734 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59830 (request call id 183) took 4639 ms. Trace:
I20240430 20:11:38.877238 18744 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59830 (request call id 187) took 4599 ms. Trace:
I20240430 20:11:38.877633 20019 raft_consensus.cc:482] T 5bde72e2791a497db7dd08686ca36d80 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5e1c6397f09747f091534faf93542710)
I20240430 20:11:38.877791 18617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77285afcafe04a829e485998bbbe00f0" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:38.878230 18612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2d90ff17e574f88b2061b1c117d4b7d" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:38.878268 19048 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:51664 (request call id 175) took 3212 ms. Trace:
I20240430 20:11:38.878654 20107 raft_consensus.cc:482] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5e1c6397f09747f091534faf93542710)
I20240430 20:11:38.878368 19048 rpcz_store.cc:276] 0430 20:11:35.666131 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:38.480326 (+2814195us) service_pool.cc:225] Handling call
0430 20:11:38.481406 (+ 1080us) catalog_manager.cc:2072] Acquired catalog manager lock
0430 20:11:38.481904 (+ 498us) catalog_manager.cc:2118] Created new table and tablet info
0430 20:11:38.877968 (+396064us) catalog_manager.cc:2178] Wrote table and tablets to system table
0430 20:11:38.878104 (+ 136us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0430 20:11:38.878210 (+ 106us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:38.571543 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:38.573017 (+ 1474us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:38.573795 (+ 778us) write_op.cc:427] Acquired schema lock
0430 20:11:38.573798 (+ 3us) tablet.cc:660] Decoding operations
0430 20:11:38.578445 (+ 4647us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:38.578461 (+ 16us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:38.578468 (+ 7us) tablet.cc:683] Acquiring locks for 2 operations
0430 20:11:38.580326 (+ 1858us) tablet.cc:699] Row locks acquired
0430 20:11:38.580329 (+ 3us) write_op.cc:260] PREPARE: finished
0430 20:11:38.580658 (+ 329us) write_op.cc:270] Start()
0430 20:11:38.581413 (+ 755us) write_op.cc:276] Timestamp: P: 1714507898580600 usec, L: 0
0430 20:11:38.581423 (+ 10us) op_driver.cc:331] REPLICATION: starting
0430 20:11:38.585072 (+ 3649us) log.cc:842] Serialized 433 byte log entry
0430 20:11:38.852602 (+267530us) op_driver.cc:444] REPLICATION: finished
0430 20:11:38.862141 (+ 9539us) write_op.cc:301] APPLY: starting
0430 20:11:38.862162 (+ 21us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:38.862535 (+ 373us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:38.862572 (+ 37us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:38.864028 (+ 1456us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:38.864067 (+ 39us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0430 20:11:38.864074 (+ 7us) write_op.cc:312] APPLY: finished
0430 20:11:38.864161 (+ 87us) log.cc:842] Serialized 29 byte log entry
0430 20:11:38.877169 (+ 13008us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:38.877181 (+ 12us) write_op.cc:446] Released schema lock
0430 20:11:38.877186 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":9453,"num_ops":2,"prepare.queue_time_us":88461,"prepare.run_cpu_time_us":1503,"prepare.run_wall_time_us":15087,"replication_time_us":271154}]]}
W20240430 20:11:38.880612 18415 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.177.2:46027, user_credentials={real_user=slave}} blocked reactor thread for 44342.1us
I20240430 20:11:38.883242 19024 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:51664 (request call id 176) took 3216 ms. Trace:
I20240430 20:11:38.883369 19024 rpcz_store.cc:276] 0430 20:11:35.666230 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:38.483721 (+2817491us) service_pool.cc:225] Handling call
0430 20:11:38.484589 (+ 868us) catalog_manager.cc:2072] Acquired catalog manager lock
0430 20:11:38.485054 (+ 465us) catalog_manager.cc:2118] Created new table and tablet info
0430 20:11:38.882879 (+397825us) catalog_manager.cc:2178] Wrote table and tablets to system table
0430 20:11:38.883031 (+ 152us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0430 20:11:38.883174 (+ 143us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:38.588421 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:38.591440 (+ 3019us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:38.591804 (+ 364us) write_op.cc:427] Acquired schema lock
0430 20:11:38.591809 (+ 5us) tablet.cc:660] Decoding operations
0430 20:11:38.594744 (+ 2935us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:38.595271 (+ 527us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:38.595281 (+ 10us) tablet.cc:683] Acquiring locks for 2 operations
0430 20:11:38.597942 (+ 2661us) tablet.cc:699] Row locks acquired
0430 20:11:38.597947 (+ 5us) write_op.cc:260] PREPARE: finished
0430 20:11:38.598922 (+ 975us) write_op.cc:270] Start()
0430 20:11:38.625004 (+ 26082us) write_op.cc:276] Timestamp: P: 1714507898598489 usec, L: 0
0430 20:11:38.625015 (+ 11us) op_driver.cc:331] REPLICATION: starting
0430 20:11:38.687571 (+ 62556us) log.cc:842] Serialized 433 byte log entry
0430 20:11:38.852685 (+165114us) op_driver.cc:444] REPLICATION: finished
0430 20:11:38.876509 (+ 23824us) write_op.cc:301] APPLY: starting
0430 20:11:38.876531 (+ 22us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:38.876580 (+ 49us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:38.876619 (+ 39us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:38.876729 (+ 110us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:38.876766 (+ 37us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0430 20:11:38.876774 (+ 8us) write_op.cc:312] APPLY: finished
0430 20:11:38.876858 (+ 84us) log.cc:842] Serialized 29 byte log entry
0430 20:11:38.876927 (+ 69us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:38.876952 (+ 25us) write_op.cc:446] Released schema lock
0430 20:11:38.876962 (+ 10us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":23757,"num_ops":2,"prepare.queue_time_us":101109,"prepare.run_cpu_time_us":1619,"prepare.run_wall_time_us":100642,"replication_time_us":227651}]]}
I20240430 20:11:38.882241 18988 raft_consensus.cc:348] T 26ed12f498644d15bf45e884dcad172e P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:38.884665 18755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53a899fa5a424a0d99f1b47925e8c7ad" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:38.884765 18988 raft_consensus.cc:374] T 26ed12f498644d15bf45e884dcad172e P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:38.887231 20061 raft_consensus.cc:482] T 0b4d9e860d7c45fd91d7a566605c9283 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3c51c33d5616482cafb68e26688820f4)
I20240430 20:11:38.887497 20061 raft_consensus.cc:504] T 0b4d9e860d7c45fd91d7a566605c9283 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:38.887900 18730 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59830 (request call id 199) took 3934 ms. Trace:
I20240430 20:11:38.888000 20038 raft_consensus.cc:482] T a05e32a33aa7488eb287cc6314c09c44 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3c51c33d5616482cafb68e26688820f4)
I20240430 20:11:38.888036 18730 rpcz_store.cc:276] 0430 20:11:34.938958 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:37.487307 (+2548349us) service_pool.cc:225] Handling call
0430 20:11:37.493843 (+ 6536us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0430 20:11:37.493882 (+ 39us) ts_tablet_manager.cc:655] Creating new metadata...
0430 20:11:38.819906 (+1326024us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:38.873572 (+ 53666us) tablet_replica.cc:205] Creating consensus instance
0430 20:11:38.873732 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240430 20:11:38.888147 20038 raft_consensus.cc:504] T a05e32a33aa7488eb287cc6314c09c44 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:38.888300 20061 leader_election.cc:288] T 0b4d9e860d7c45fd91d7a566605c9283 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:38.888854 20038 leader_election.cc:288] T a05e32a33aa7488eb287cc6314c09c44 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:38.889106 19489 log.cc:1196] T cace1398492f4d968dcb6899385d527c P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/cace1398492f4d968dcb6899385d527c
I20240430 20:11:38.889750 19489 ts_tablet_manager.cc:1909] T cace1398492f4d968dcb6899385d527c P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:38.895457 20038 raft_consensus.cc:482] T 7bde1d6934784c9aa5911181e8df9eeb P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3c51c33d5616482cafb68e26688820f4)
I20240430 20:11:38.895685 20038 raft_consensus.cc:504] T 7bde1d6934784c9aa5911181e8df9eeb P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:38.896373 20038 leader_election.cc:288] T 7bde1d6934784c9aa5911181e8df9eeb P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:38.896940 18668 heartbeater.cc:347] Connected to a master server at 127.17.177.62:36441
I20240430 20:11:38.900010 19033 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: Sending DeleteTablet for 3 replicas of tablet 24dcc20832804229a2664873bf6ddaa7
I20240430 20:11:38.900604 19033 catalog_manager.cc:6554] ReplaceTablet: tablet 24dcc20832804229a2664873bf6ddaa7 deleted and replaced by tablet 62732af95c6a4c749c5b726f0460dc72
I20240430 20:11:38.900784 19033 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:51664 (request call id 177) took 2997 ms. Trace:
I20240430 20:11:38.900903 19033 rpcz_store.cc:276] 0430 20:11:35.903753 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:38.486508 (+2582755us) service_pool.cc:225] Handling call
0430 20:11:38.900715 (+414207us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:38.687754 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:38.687912 (+ 158us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:38.687923 (+ 11us) write_op.cc:427] Acquired schema lock
0430 20:11:38.687927 (+ 4us) tablet.cc:660] Decoding operations
0430 20:11:38.692740 (+ 4813us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:38.692758 (+ 18us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:38.692767 (+ 9us) tablet.cc:683] Acquiring locks for 2 operations
0430 20:11:38.692893 (+ 126us) tablet.cc:699] Row locks acquired
0430 20:11:38.692897 (+ 4us) write_op.cc:260] PREPARE: finished
0430 20:11:38.692948 (+ 51us) write_op.cc:270] Start()
0430 20:11:38.692996 (+ 48us) write_op.cc:276] Timestamp: P: 1714507898692884 usec, L: 0
0430 20:11:38.693003 (+ 7us) op_driver.cc:331] REPLICATION: starting
0430 20:11:38.716038 (+ 23035us) spinlock_profiling.cc:243] Waited 14 ms on lock 0x616000042ab8. stack: 00007feb16c74756 00007feb16c74510 00007feb15ca748d 0000000000955804 00000000009557cd 000000000095519a 00007feb22415b35 00007feb224ddbdb 00007feb224b9758 00007feb224b9005 00007feb23821b02 00007feb23820629 00007feb2382853b 00007feb238283b8 0000000000a60bcc 00007feb16cbab74
0430 20:11:38.716198 (+ 160us) log.cc:842] Serialized 625 byte log entry
0430 20:11:38.877336 (+161138us) op_driver.cc:444] REPLICATION: finished
0430 20:11:38.879695 (+ 2359us) write_op.cc:301] APPLY: starting
0430 20:11:38.879709 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:38.879752 (+ 43us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:38.879754 (+ 2us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:38.879846 (+ 92us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:38.879902 (+ 56us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0430 20:11:38.879910 (+ 8us) write_op.cc:312] APPLY: finished
0430 20:11:38.879979 (+ 69us) log.cc:842] Serialized 29 byte log entry
0430 20:11:38.880035 (+ 56us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:38.880053 (+ 18us) write_op.cc:446] Released schema lock
0430 20:11:38.880058 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":2318,"num_ops":2,"prepare.queue_time_us":200100,"prepare.run_cpu_time_us":1543,"prepare.run_wall_time_us":28536,"replication_time_us":184310,"spinlock_wait_cycles":32294656,"threads_started":1}]]}
I20240430 20:11:38.903661 20038 raft_consensus.cc:482] T 1555924b5f1444398f12ee82c3cb5b13 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3c51c33d5616482cafb68e26688820f4)
I20240430 20:11:38.903798 20038 raft_consensus.cc:504] T 1555924b5f1444398f12ee82c3cb5b13 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:38.904426 20038 leader_election.cc:288] T 1555924b5f1444398f12ee82c3cb5b13 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:38.907323 18588 tablet_service.cc:1514] Processing DeleteTablet for tablet 24dcc20832804229a2664873bf6ddaa7 with delete_type TABLET_DATA_DELETED (replaced by tablet 62732af95c6a4c749c5b726f0460dc72) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:38.907389 19784 raft_consensus.cc:482] T 294644d7a16b4ca2bbb6d106bcb8c7ff P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3c51c33d5616482cafb68e26688820f4)
I20240430 20:11:38.908552 18744 rpcz_store.cc:276] 0430 20:11:34.278198 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:35.855738 (+1577540us) service_pool.cc:225] Handling call
0430 20:11:35.868612 (+ 12874us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0430 20:11:35.868667 (+ 55us) ts_tablet_manager.cc:655] Creating new metadata...
0430 20:11:38.830982 (+2962315us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:38.877058 (+ 46076us) tablet_replica.cc:205] Creating consensus instance
0430 20:11:38.877197 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240430 20:11:38.910665 18734 rpcz_store.cc:276] 0430 20:11:34.234128 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:34.455024 (+220896us) service_pool.cc:225] Handling call
0430 20:11:34.455533 (+ 509us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0430 20:11:34.455558 (+ 25us) ts_tablet_manager.cc:655] Creating new metadata...
0430 20:11:38.792406 (+4336848us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:38.873815 (+ 81409us) tablet_replica.cc:205] Creating consensus instance
0430 20:11:38.873940 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240430 20:11:38.910496 20019 raft_consensus.cc:504] T 5bde72e2791a497db7dd08686ca36d80 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:38.912067 20019 leader_election.cc:288] T 5bde72e2791a497db7dd08686ca36d80 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:38.912175 19805 raft_consensus.cc:482] T 5809e7c9c2204ba293426c24129fb16e P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5e1c6397f09747f091534faf93542710)
I20240430 20:11:38.911417 19028 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet acd23f0a63d5448aaa7ed78422e67bad (table default.Table_972f13be4dca4cd19b9a04534dc59f51 [id=69f37db4de634a3185613e40013e0ed7]). Expected version 1 got 0
I20240430 20:11:38.912611 20019 raft_consensus.cc:482] T fae9a35a2ac844b4bc2ebdf214aa7171 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5e1c6397f09747f091534faf93542710)
I20240430 20:11:38.912756 20019 raft_consensus.cc:504] T fae9a35a2ac844b4bc2ebdf214aa7171 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:38.912895 20107 raft_consensus.cc:504] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:38.912724 19805 raft_consensus.cc:504] T 5809e7c9c2204ba293426c24129fb16e P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:38.913341 19028 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet fd40967ab4a04001819e16c06a6c4300 (table default.Table_4130caf534004edfb413056d7f6cfd52 [id=da9ad991953a40c4b77b31317597ad1d]). Expected version 4 got 0
I20240430 20:11:38.914326 18754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "724e51ff05ec4baebc2bb55e05c8eddc" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:38.914326 18762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aec9c09cc0454acab6006bb3e10982b7" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:38.915359 18762 raft_consensus.cc:2454] T aec9c09cc0454acab6006bb3e10982b7 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
I20240430 20:11:38.915377 19028 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 53a899fa5a424a0d99f1b47925e8c7ad (table default.Table_ffb3192f443d437ab63f56f628e596ad [id=608890834da843fa9af0cfc0d548029e]). Expected version 2 got 0
I20240430 20:11:38.915596 18760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b72fe1f0713b4076a81a3fc1c74f7fa7" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:38.915766 18760 raft_consensus.cc:2454] T b72fe1f0713b4076a81a3fc1c74f7fa7 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
I20240430 20:11:38.915985 20107 leader_election.cc:288] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:38.920579 18754 raft_consensus.cc:2454] T 724e51ff05ec4baebc2bb55e05c8eddc P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
I20240430 20:11:38.920796 20100 raft_consensus.cc:482] T a745334c12c545c1ba825c7552c60359 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5e1c6397f09747f091534faf93542710)
I20240430 20:11:38.921504 20107 raft_consensus.cc:482] T 680553d4d0bc4fcb839235c1832757e1 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5e1c6397f09747f091534faf93542710)
I20240430 20:11:38.921651 20025 raft_consensus.cc:482] T 1555924b5f1444398f12ee82c3cb5b13 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3c51c33d5616482cafb68e26688820f4)
I20240430 20:11:38.921800 18763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c37876694c4a4abb1091c631dab7a5" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:38.921833 20107 raft_consensus.cc:504] T 680553d4d0bc4fcb839235c1832757e1 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:38.922158 19784 raft_consensus.cc:504] T 294644d7a16b4ca2bbb6d106bcb8c7ff P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:38.922189 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 63) took 7627 ms. Trace:
I20240430 20:11:38.922679 18724 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59830 (request call id 163) took 5034 ms. Trace:
W20240430 20:11:38.922837 18211 leader_election.cc:334] T 00000000000000000000000000000000 P 89df84ccf5394c9bb60d4e4ed872fc2e [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 88b3f98793354c7aaa5c106e8ec31ed8 (127.17.177.62:36441): Timed out: RequestConsensusVote RPC to 127.17.177.62:36441 timed out after 0.531s (SENT)
I20240430 20:11:38.922780 18724 rpcz_store.cc:276] 0430 20:11:33.875608 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:33.875780 (+ 172us) service_pool.cc:225] Handling call
0430 20:11:34.463492 (+587712us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0430 20:11:34.463523 (+ 31us) ts_tablet_manager.cc:655] Creating new metadata...
0430 20:11:38.739217 (+4275694us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:38.910143 (+170926us) tablet_replica.cc:205] Creating consensus instance
0430 20:11:38.910314 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240430 20:11:38.923002 18211 leader_election.cc:302] T 00000000000000000000000000000000 P 89df84ccf5394c9bb60d4e4ed872fc2e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89df84ccf5394c9bb60d4e4ed872fc2e; no voters: 88b3f98793354c7aaa5c106e8ec31ed8, b632d6ef4c104a058a6db208024cab33
I20240430 20:11:38.923489 18983 ts_tablet_manager.cc:1398] T 2a1e50f99c5d4b44bdbb51b58a92bc75 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.131s user 0.003s sys 0.000s
I20240430 20:11:38.923899 18983 tablet_bootstrap.cc:492] T a9e0659564894b0e8cb7f600bb07a085 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:38.925791 19033 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_cb067ab866334a64a0186737511a9855"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:38.926535 19033 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cb067ab866334a64a0186737511a9855 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:38.928035 20085 leader_election.cc:288] T 724e51ff05ec4baebc2bb55e05c8eddc P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:38.928071 19024 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_d06ed4569aa2456d85107f8a351d2ea1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:38.928587 19024 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d06ed4569aa2456d85107f8a351d2ea1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:38.929960 19048 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_088b70cec77744aaacf26aa6d4adebbd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:38.930419 19048 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_088b70cec77744aaacf26aa6d4adebbd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:38.931979 20038 raft_consensus.cc:482] T 01436e5b6232494babbb414342f03090 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3c51c33d5616482cafb68e26688820f4)
I20240430 20:11:38.932188 20038 raft_consensus.cc:504] T 01436e5b6232494babbb414342f03090 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:38.933007 20038 leader_election.cc:288] T 01436e5b6232494babbb414342f03090 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:38.933254 20038 raft_consensus.cc:482] T 44fa6917a85c46f3aaa363df46ea586b P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3c51c33d5616482cafb68e26688820f4)
I20240430 20:11:38.933396 20038 raft_consensus.cc:504] T 44fa6917a85c46f3aaa363df46ea586b P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:38.922812 19489 rpcz_store.cc:276] 0430 20:11:31.264170 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:31.264234 (+ 64us) service_pool.cc:225] Handling call
0430 20:11:38.630385 (+7366151us) ts_tablet_manager.cc:1120] Deleting tablet cace1398492f4d968dcb6899385d527c
0430 20:11:38.630403 (+ 18us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:38.747426 (+117023us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:38.873418 (+125992us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:38.891470 (+ 18052us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7363960}
I20240430 20:11:38.934065 20038 leader_election.cc:288] T 44fa6917a85c46f3aaa363df46ea586b P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:38.934424 20103 raft_consensus.cc:482] T 5bde72e2791a497db7dd08686ca36d80 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5e1c6397f09747f091534faf93542710)
I20240430 20:11:38.934661 20103 raft_consensus.cc:504] T 5bde72e2791a497db7dd08686ca36d80 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:38.935254 18454 tablet_service.cc:1514] Processing DeleteTablet for tablet 24dcc20832804229a2664873bf6ddaa7 with delete_type TABLET_DATA_DELETED (replaced by tablet 62732af95c6a4c749c5b726f0460dc72) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:38.935451 18482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81787eccca6c42c1b421377a3bf9ebe6" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:38.935411 20100 raft_consensus.cc:504] T a745334c12c545c1ba825c7552c60359 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:38.936162 20100 leader_election.cc:288] T a745334c12c545c1ba825c7552c60359 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:38.935222 18474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9e7660eccc448dbb99d135f82bfd261" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:38.936579 18415 leader_election.cc:302] T 802336602d8f4ac4bd7e0872d6f3dae9 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:38.931152 20105 raft_consensus.cc:482] T b76a4105a2974fe280e759349037d027 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3c51c33d5616482cafb68e26688820f4)
I20240430 20:11:38.937112 20061 raft_consensus.cc:482] T 294644d7a16b4ca2bbb6d106bcb8c7ff P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3c51c33d5616482cafb68e26688820f4)
I20240430 20:11:38.937193 19784 leader_election.cc:288] T 294644d7a16b4ca2bbb6d106bcb8c7ff P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:38.937256 20061 raft_consensus.cc:504] T 294644d7a16b4ca2bbb6d106bcb8c7ff P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:38.937812 20061 leader_election.cc:288] T 294644d7a16b4ca2bbb6d106bcb8c7ff P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:38.938604 19495 ts_tablet_manager.cc:1909] T c5fd0730133c472299c93f4b0830e527 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:38.939767 20025 raft_consensus.cc:504] T 1555924b5f1444398f12ee82c3cb5b13 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:38.942742 20103 leader_election.cc:288] T 5bde72e2791a497db7dd08686ca36d80 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:38.943995 18754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcc9698abda4491b98e1e879589913a8" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:38.937105 20105 raft_consensus.cc:504] T b76a4105a2974fe280e759349037d027 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:38.944425 18752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77285afcafe04a829e485998bbbe00f0" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:38.944598 18752 raft_consensus.cc:2454] T 77285afcafe04a829e485998bbbe00f0 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
I20240430 20:11:38.944890 19805 leader_election.cc:288] T 5809e7c9c2204ba293426c24129fb16e P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:38.945173 18481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dfdecba959543a795dfcd3eeda29e58" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:38.947063 18474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b4d9e860d7c45fd91d7a566605c9283" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:38.948954 18482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a05e32a33aa7488eb287cc6314c09c44" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:38.944762 18763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2d90ff17e574f88b2061b1c117d4b7d" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:38.949750 18754 raft_consensus.cc:2454] T fcc9698abda4491b98e1e879589913a8 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 0.
I20240430 20:11:38.949777 18988 raft_consensus.cc:729] T 26ed12f498644d15bf45e884dcad172e P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:38.950150 18753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e826459c3f4437da81c9e6e09f0c22f" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:38.950686 20085 raft_consensus.cc:482] T 6e826459c3f4437da81c9e6e09f0c22f P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2efb542aa55e439b99ccb98d1c5c884a)
I20240430 20:11:38.950913 19471 ts_tablet_manager.cc:1888] T 13c1107ea4d04eb78a74d3f658c72fb3 P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:38.951555 20100 raft_consensus.cc:2781] T 802336602d8f4ac4bd7e0872d6f3dae9 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:38.952283 18481 raft_consensus.cc:2454] T 7dfdecba959543a795dfcd3eeda29e58 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
I20240430 20:11:38.952245 18475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01436e5b6232494babbb414342f03090" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:38.952327 20134 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 89df84ccf5394c9bb60d4e4ed872fc2e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240430 20:11:38.953044 19471 log.cc:1196] T 13c1107ea4d04eb78a74d3f658c72fb3 P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/13c1107ea4d04eb78a74d3f658c72fb3
I20240430 20:11:38.954969 18481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44fa6917a85c46f3aaa363df46ea586b" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
W20240430 20:11:38.956176 18558 leader_election.cc:341] T c9e7660eccc448dbb99d135f82bfd261 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
W20240430 20:11:38.956660 18558 leader_election.cc:341] T 01436e5b6232494babbb414342f03090 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:38.956926 18558 leader_election.cc:302] T 7dfdecba959543a795dfcd3eeda29e58 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:38.957250 20096 raft_consensus.cc:2781] T 7dfdecba959543a795dfcd3eeda29e58 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240430 20:11:38.957355 20096 raft_consensus.cc:482] T 7dfdecba959543a795dfcd3eeda29e58 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3c51c33d5616482cafb68e26688820f4)
I20240430 20:11:38.957509 20096 raft_consensus.cc:3037] T 7dfdecba959543a795dfcd3eeda29e58 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Advancing to term 2
I20240430 20:11:38.960685 20096 raft_consensus.cc:504] T 7dfdecba959543a795dfcd3eeda29e58 P 5e1c6397f09747f091534faf93542710 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:38.961367 18475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "294644d7a16b4ca2bbb6d106bcb8c7ff" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:38.961498 20096 leader_election.cc:288] T 7dfdecba959543a795dfcd3eeda29e58 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 2 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:38.961612 18475 raft_consensus.cc:2454] T 294644d7a16b4ca2bbb6d106bcb8c7ff P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
I20240430 20:11:38.950903 20105 leader_election.cc:288] T b76a4105a2974fe280e759349037d027 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:38.961935 20100 raft_consensus.cc:686] T 802336602d8f4ac4bd7e0872d6f3dae9 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Becoming Leader. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Running, Role: LEADER
I20240430 20:11:38.962648 18558 leader_election.cc:302] T 294644d7a16b4ca2bbb6d106bcb8c7ff P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:38.962957 20061 raft_consensus.cc:2781] T 294644d7a16b4ca2bbb6d106bcb8c7ff P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240430 20:11:38.963232 20061 raft_consensus.cc:482] T 294644d7a16b4ca2bbb6d106bcb8c7ff P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3c51c33d5616482cafb68e26688820f4)
I20240430 20:11:38.963518 20061 raft_consensus.cc:3037] T 294644d7a16b4ca2bbb6d106bcb8c7ff P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Advancing to term 2
I20240430 20:11:38.964504 20100 consensus_queue.cc:237] T 802336602d8f4ac4bd7e0872d6f3dae9 P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:38.967058 18475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dfdecba959543a795dfcd3eeda29e58" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:38.967330 18475 raft_consensus.cc:3037] T 7dfdecba959543a795dfcd3eeda29e58 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Advancing to term 2
I20240430 20:11:38.969961 20061 raft_consensus.cc:504] T 294644d7a16b4ca2bbb6d106bcb8c7ff P 5e1c6397f09747f091534faf93542710 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:38.970768 18474 raft_consensus.cc:2454] T 0b4d9e860d7c45fd91d7a566605c9283 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
I20240430 20:11:38.970831 18482 raft_consensus.cc:2454] T a05e32a33aa7488eb287cc6314c09c44 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
I20240430 20:11:38.971158 20025 leader_election.cc:288] T 1555924b5f1444398f12ee82c3cb5b13 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:38.972965 20061 leader_election.cc:288] T 294644d7a16b4ca2bbb6d106bcb8c7ff P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 2 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:38.975415 19471 ts_tablet_manager.cc:1909] T 13c1107ea4d04eb78a74d3f658c72fb3 P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:38.978320 18988 consensus_queue.cc:260] T 26ed12f498644d15bf45e884dcad172e P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:38.980144 19001 catalog_manager.cc:4830] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): tablet cace1398492f4d968dcb6899385d527c (table default.Table_a8d5ac73c6d047698d87b3d1ba34e66f [id=9c80193cde2346c982cf84ae9612e2f7]) successfully deleted
I20240430 20:11:38.980273 18985 tablet_bootstrap.cc:492] T 01436e5b6232494babbb414342f03090 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:38.981318 18764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a05e32a33aa7488eb287cc6314c09c44" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:38.981344 20107 leader_election.cc:288] T 680553d4d0bc4fcb839235c1832757e1 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:38.980788 18696 leader_election.cc:302] T b72fe1f0713b4076a81a3fc1c74f7fa7 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710
I20240430 20:11:38.982017 18755 raft_consensus.cc:3037] T 53a899fa5a424a0d99f1b47925e8c7ad P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:38.982056 20019 leader_election.cc:288] T fae9a35a2ac844b4bc2ebdf214aa7171 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:38.983291 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 71) took 7942 ms. Trace:
I20240430 20:11:38.985399 18985 ts_tablet_manager.cc:1367] T 01436e5b6232494babbb414342f03090 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 1.466s user 0.026s sys 0.033s
I20240430 20:11:38.985435 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 54) took 7887 ms. Trace:
I20240430 20:11:38.985649 19495 rpcz_store.cc:276] 0430 20:11:31.059612 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:31.061041 (+ 1429us) service_pool.cc:225] Handling call
0430 20:11:38.605869 (+7544828us) ts_tablet_manager.cc:1120] Deleting tablet c5fd0730133c472299c93f4b0830e527
0430 20:11:38.605881 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:38.610251 (+ 4370us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:38.846316 (+236065us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:38.947037 (+100721us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4905,"tablet-delete.queue_time_us":7492410}
I20240430 20:11:38.986233 19004 catalog_manager.cc:4830] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): tablet c5fd0730133c472299c93f4b0830e527 (table default.Table_ea83912975774b6089fd85c252f45ea5 [id=99cbf7b92c7943838fce9f3ba42babde]) successfully deleted
I20240430 20:11:38.986621 18476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1555924b5f1444398f12ee82c3cb5b13" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:38.980507 18473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bde1d6934784c9aa5911181e8df9eeb" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:38.987623 19495 tablet_replica.cc:330] T d9e578c99058424a9b86390641f9c34f P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:38.988092 18558 leader_election.cc:302] T 0b4d9e860d7c45fd91d7a566605c9283 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:38.988232 18476 raft_consensus.cc:2454] T 1555924b5f1444398f12ee82c3cb5b13 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
I20240430 20:11:38.988306 18473 raft_consensus.cc:2454] T 7bde1d6934784c9aa5911181e8df9eeb P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
I20240430 20:11:38.988418 18558 leader_election.cc:302] T a05e32a33aa7488eb287cc6314c09c44 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:38.988786 18558 leader_election.cc:302] T 1555924b5f1444398f12ee82c3cb5b13 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:38.989066 18558 leader_election.cc:302] T 7bde1d6934784c9aa5911181e8df9eeb P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:38.989437 20060 raft_consensus.cc:2781] T a05e32a33aa7488eb287cc6314c09c44 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240430 20:11:38.989563 20060 raft_consensus.cc:482] T a05e32a33aa7488eb287cc6314c09c44 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3c51c33d5616482cafb68e26688820f4)
I20240430 20:11:38.989666 20060 raft_consensus.cc:3037] T a05e32a33aa7488eb287cc6314c09c44 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Advancing to term 2
I20240430 20:11:38.990233 20096 raft_consensus.cc:2781] T 0b4d9e860d7c45fd91d7a566605c9283 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240430 20:11:38.990414 20096 raft_consensus.cc:482] T 0b4d9e860d7c45fd91d7a566605c9283 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3c51c33d5616482cafb68e26688820f4)
I20240430 20:11:38.990564 20096 raft_consensus.cc:3037] T 0b4d9e860d7c45fd91d7a566605c9283 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Advancing to term 2
I20240430 20:11:38.992421 20040 raft_consensus.cc:2781] T 1555924b5f1444398f12ee82c3cb5b13 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240430 20:11:38.992554 20040 raft_consensus.cc:482] T 1555924b5f1444398f12ee82c3cb5b13 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3c51c33d5616482cafb68e26688820f4)
I20240430 20:11:38.992641 20040 raft_consensus.cc:3037] T 1555924b5f1444398f12ee82c3cb5b13 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Advancing to term 2
I20240430 20:11:38.992974 20060 raft_consensus.cc:504] T a05e32a33aa7488eb287cc6314c09c44 P 5e1c6397f09747f091534faf93542710 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:38.993983 20060 leader_election.cc:288] T a05e32a33aa7488eb287cc6314c09c44 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 2 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:38.994184 20060 raft_consensus.cc:2781] T 7bde1d6934784c9aa5911181e8df9eeb P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240430 20:11:38.994267 20060 raft_consensus.cc:482] T 7bde1d6934784c9aa5911181e8df9eeb P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3c51c33d5616482cafb68e26688820f4)
I20240430 20:11:38.994376 20060 raft_consensus.cc:3037] T 7bde1d6934784c9aa5911181e8df9eeb P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Advancing to term 2
I20240430 20:11:38.994642 20096 raft_consensus.cc:504] T 0b4d9e860d7c45fd91d7a566605c9283 P 5e1c6397f09747f091534faf93542710 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:38.995541 20096 leader_election.cc:288] T 0b4d9e860d7c45fd91d7a566605c9283 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 2 election: Requested vote from peers 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:38.997535 18764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75ebf051383e42529dcb5f7dd0770a4b" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:38.997790 18764 raft_consensus.cc:2454] T 75ebf051383e42529dcb5f7dd0770a4b P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 1.
I20240430 20:11:38.998512 20107 raft_consensus.cc:2726] T b72fe1f0713b4076a81a3fc1c74f7fa7 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:38.984643 19471 rpcz_store.cc:276] 0430 20:11:31.038563 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:31.038632 (+ 69us) service_pool.cc:225] Handling call
0430 20:11:38.738020 (+7699388us) ts_tablet_manager.cc:1120] Deleting tablet 13c1107ea4d04eb78a74d3f658c72fb3
0430 20:11:38.738193 (+ 173us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:38.830137 (+ 91944us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:38.942468 (+112331us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:38.980972 (+ 38504us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7699138}
I20240430 20:11:38.998723 18724 tablet_service.cc:1514] Processing DeleteTablet for tablet 24dcc20832804229a2664873bf6ddaa7 with delete_type TABLET_DATA_DELETED (replaced by tablet 62732af95c6a4c749c5b726f0460dc72) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:38.998778 18763 raft_consensus.cc:2454] T e2d90ff17e574f88b2061b1c117d4b7d P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
W20240430 20:11:38.998919 18696 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.177.2:46027, user_credentials={real_user=slave}} blocked reactor thread for 46347.6us
I20240430 20:11:38.997596 18482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "294644d7a16b4ca2bbb6d106bcb8c7ff" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:38.999300 18725 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59830 (request call id 184) took 4649 ms. Trace:
I20240430 20:11:38.999403 18725 rpcz_store.cc:276] 0430 20:11:34.235814 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:34.792506 (+556692us) service_pool.cc:225] Handling call
0430 20:11:34.793036 (+ 530us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0430 20:11:34.793067 (+ 31us) ts_tablet_manager.cc:655] Creating new metadata...
0430 20:11:38.711260 (+3918193us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:38.882579 (+171319us) tablet_replica.cc:205] Creating consensus instance
0430 20:11:38.885075 (+ 2496us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240430 20:11:39.000907 18758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3e74e76ab2b4fe492ffcc1838656fd1" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:39.001498 18754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5546f9edeea44d7a6c5cb954f5157ba" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:39.001812 18482 raft_consensus.cc:3037] T 294644d7a16b4ca2bbb6d106bcb8c7ff P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Advancing to term 2
I20240430 20:11:38.998637 20040 raft_consensus.cc:504] T 1555924b5f1444398f12ee82c3cb5b13 P 5e1c6397f09747f091534faf93542710 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:39.001857 20085 raft_consensus.cc:504] T 6e826459c3f4437da81c9e6e09f0c22f P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:39.002513 20040 leader_election.cc:288] T 1555924b5f1444398f12ee82c3cb5b13 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 2 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:39.002749 18753 raft_consensus.cc:2454] T 6e826459c3f4437da81c9e6e09f0c22f P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
I20240430 20:11:39.003450 18754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd40967ab4a04001819e16c06a6c4300" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:39.003420 18761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6eac92dfa5f4456bc49571878f0f8dd" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:39.003412 18749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24dcc20832804229a2664873bf6ddaa7" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:39.004693 18999 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet 13c1107ea4d04eb78a74d3f658c72fb3 (table default.Table_52dcbd45fadf4a4d85c4533f1d69f976 [id=77aff20b7b63407c91f98a01a6f46e4c]) successfully deleted
I20240430 20:11:39.004845 18988 ts_tablet_manager.cc:1398] T 26ed12f498644d15bf45e884dcad172e P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.112s user 0.004s sys 0.000s
I20240430 20:11:39.005021 20085 leader_election.cc:288] T 6e826459c3f4437da81c9e6e09f0c22f P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:39.004909 18756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "802336602d8f4ac4bd7e0872d6f3dae9" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:39.005717 18749 raft_consensus.cc:2379] T 24dcc20832804229a2664873bf6ddaa7 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 2efb542aa55e439b99ccb98d1c5c884a in current term 1: Already voted for candidate 3c51c33d5616482cafb68e26688820f4 in this term.
I20240430 20:11:39.006728 18757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66c142419ca4e1399ecd1c8d1e62a59" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:39.007864 18761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6eac92dfa5f4456bc49571878f0f8dd" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:39.007531 19471 tablet_replica.cc:330] T b688e03ea71940c69bda5520b62c1fc3 P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:39.009742 18988 tablet_bootstrap.cc:492] T 802336602d8f4ac4bd7e0872d6f3dae9 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:39.009414 18764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b76a4105a2974fe280e759349037d027" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:39.010466 18757 raft_consensus.cc:2454] T c66c142419ca4e1399ecd1c8d1e62a59 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 1.
I20240430 20:11:39.010867 18756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a745334c12c545c1ba825c7552c60359" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:39.018128 19471 raft_consensus.cc:2229] T b688e03ea71940c69bda5520b62c1fc3 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:39.019927 19495 raft_consensus.cc:2229] T d9e578c99058424a9b86390641f9c34f P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus shutting down.
W20240430 20:11:39.019989 18417 consensus_peers.cc:485] T cace1398492f4d968dcb6899385d527c P 2efb542aa55e439b99ccb98d1c5c884a -> Peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Couldn't send request to peer 3c51c33d5616482cafb68e26688820f4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cace1398492f4d968dcb6899385d527c. This is attempt 1: this message will repeat every 5th retry.
I20240430 20:11:39.020359 19495 raft_consensus.cc:2258] T d9e578c99058424a9b86390641f9c34f P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:38.999876 18985 raft_consensus.cc:348] T 01436e5b6232494babbb414342f03090 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:39.020525 18985 raft_consensus.cc:374] T 01436e5b6232494babbb414342f03090 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:39.020960 19471 raft_consensus.cc:2258] T b688e03ea71940c69bda5520b62c1fc3 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:39.021667 19495 ts_tablet_manager.cc:1875] T d9e578c99058424a9b86390641f9c34f P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:39.019973 18475 raft_consensus.cc:2454] T 7dfdecba959543a795dfcd3eeda29e58 P 2efb542aa55e439b99ccb98d1c5c884a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 2.
I20240430 20:11:39.022097 18985 raft_consensus.cc:729] T 01436e5b6232494babbb414342f03090 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:39.022131 18417 leader_election.cc:302] T a05e32a33aa7488eb287cc6314c09c44 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a; no voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710
I20240430 20:11:39.022347 18985 consensus_queue.cc:260] T 01436e5b6232494babbb414342f03090 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:39.022722 18417 leader_election.cc:302] T 75ebf051383e42529dcb5f7dd0770a4b P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters:
I20240430 20:11:39.022718 19784 raft_consensus.cc:2726] T a05e32a33aa7488eb287cc6314c09c44 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240430 20:11:39.023094 18558 leader_election.cc:302] T 7dfdecba959543a795dfcd3eeda29e58 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:39.023801 20040 raft_consensus.cc:2781] T 7dfdecba959543a795dfcd3eeda29e58 P 5e1c6397f09747f091534faf93542710 [term 2 FOLLOWER]: Leader election won for term 2
I20240430 20:11:39.023957 20040 raft_consensus.cc:686] T 7dfdecba959543a795dfcd3eeda29e58 P 5e1c6397f09747f091534faf93542710 [term 2 LEADER]: Becoming Leader. State: Replica: 5e1c6397f09747f091534faf93542710, State: Running, Role: LEADER
I20240430 20:11:39.023953 18473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b4d9e860d7c45fd91d7a566605c9283" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:39.024276 20040 consensus_queue.cc:237] T 7dfdecba959543a795dfcd3eeda29e58 P 5e1c6397f09747f091534faf93542710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:39.024456 18476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a05e32a33aa7488eb287cc6314c09c44" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:39.024616 18476 raft_consensus.cc:3037] T a05e32a33aa7488eb287cc6314c09c44 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Advancing to term 2
I20240430 20:11:39.025617 20025 raft_consensus.cc:2781] T 75ebf051383e42529dcb5f7dd0770a4b P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240430 20:11:39.025885 18985 ts_tablet_manager.cc:1398] T 01436e5b6232494babbb414342f03090 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.038s user 0.001s sys 0.001s
I20240430 20:11:39.026186 18985 tablet_bootstrap.cc:492] T a9e0659564894b0e8cb7f600bb07a085 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:39.025509 20103 consensus_queue.cc:1035] T b72fe1f0713b4076a81a3fc1c74f7fa7 P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20240430 20:11:39.027269 18473 raft_consensus.cc:3037] T 0b4d9e860d7c45fd91d7a566605c9283 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Advancing to term 2
I20240430 20:11:39.026572 18475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1555924b5f1444398f12ee82c3cb5b13" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:39.027773 20025 raft_consensus.cc:482] T 75ebf051383e42529dcb5f7dd0770a4b P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:39.027917 20025 raft_consensus.cc:3037] T 75ebf051383e42529dcb5f7dd0770a4b P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Advancing to term 2
I20240430 20:11:39.029333 19029 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet acd23f0a63d5448aaa7ed78422e67bad (table default.Table_972f13be4dca4cd19b9a04534dc59f51 [id=69f37db4de634a3185613e40013e0ed7]). Expected version 1 got 0
I20240430 20:11:39.029582 19029 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet a05e32a33aa7488eb287cc6314c09c44 (table default.Table_2d1cfc78363a47aa8dba7ef7d80f8c9a [id=955d833ff8f14bf6957abadf9ef669e2]). Expected version 2 got 0
I20240430 20:11:39.030113 19029 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 12c6e131fa904d23865b8673b58a7e33 (table default.Table_a9dc8fc6a4aa46dc914d0e327a4de554 [id=19f4e6decedf4ee59fccc0a24a05cedd]). Expected version 2 got 0
I20240430 20:11:39.032238 18475 raft_consensus.cc:3037] T 1555924b5f1444398f12ee82c3cb5b13 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Advancing to term 2
I20240430 20:11:39.035578 19471 ts_tablet_manager.cc:1875] T b688e03ea71940c69bda5520b62c1fc3 P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:39.036296 18483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2d90ff17e574f88b2061b1c117d4b7d" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:39.037199 18472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77285afcafe04a829e485998bbbe00f0" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:39.038710 18477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "724e51ff05ec4baebc2bb55e05c8eddc" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:39.039638 18480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bde72e2791a497db7dd08686ca36d80" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:39.042829 18481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aec9c09cc0454acab6006bb3e10982b7" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:39.043949 18479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fae9a35a2ac844b4bc2ebdf214aa7171" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:39.045527 18472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5809e7c9c2204ba293426c24129fb16e" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:39.045715 18472 raft_consensus.cc:2454] T 5809e7c9c2204ba293426c24129fb16e P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:39.048149 18480 raft_consensus.cc:2454] T 5bde72e2791a497db7dd08686ca36d80 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:39.049589 18477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd6083f12c7e4d1e8a2006b3ddf9aa3a" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:39.049810 18479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680553d4d0bc4fcb839235c1832757e1" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:39.050642 18477 raft_consensus.cc:2454] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:39.051775 18415 leader_election.cc:302] T fcc9698abda4491b98e1e879589913a8 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters: 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:39.051951 18476 raft_consensus.cc:2454] T a05e32a33aa7488eb287cc6314c09c44 P 2efb542aa55e439b99ccb98d1c5c884a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 2.
I20240430 20:11:39.053022 18474 raft_consensus.cc:1226] T a05e32a33aa7488eb287cc6314c09c44 P 2efb542aa55e439b99ccb98d1c5c884a [term 2 FOLLOWER]: Rejecting Update request from peer 3c51c33d5616482cafb68e26688820f4 for earlier term 1. Current term is 2. Ops: [1.2-1.2]
I20240430 20:11:39.053309 18482 raft_consensus.cc:2454] T 294644d7a16b4ca2bbb6d106bcb8c7ff P 2efb542aa55e439b99ccb98d1c5c884a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 2.
I20240430 20:11:39.054150 18558 leader_election.cc:302] T a05e32a33aa7488eb287cc6314c09c44 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:39.054534 18558 leader_election.cc:302] T 294644d7a16b4ca2bbb6d106bcb8c7ff P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:39.055094 20024 raft_consensus.cc:2781] T a05e32a33aa7488eb287cc6314c09c44 P 5e1c6397f09747f091534faf93542710 [term 2 FOLLOWER]: Leader election won for term 2
I20240430 20:11:39.055243 20024 raft_consensus.cc:686] T a05e32a33aa7488eb287cc6314c09c44 P 5e1c6397f09747f091534faf93542710 [term 2 LEADER]: Becoming Leader. State: Replica: 5e1c6397f09747f091534faf93542710, State: Running, Role: LEADER
I20240430 20:11:39.055568 20024 consensus_queue.cc:237] T a05e32a33aa7488eb287cc6314c09c44 P 5e1c6397f09747f091534faf93542710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:39.055722 20025 raft_consensus.cc:504] T 75ebf051383e42529dcb5f7dd0770a4b P 2efb542aa55e439b99ccb98d1c5c884a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:39.056334 20025 leader_election.cc:288] T 75ebf051383e42529dcb5f7dd0770a4b P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 election: Requested vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:39.056655 18473 raft_consensus.cc:2454] T 0b4d9e860d7c45fd91d7a566605c9283 P 2efb542aa55e439b99ccb98d1c5c884a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 2.
I20240430 20:11:39.056753 18983 tablet_bootstrap.cc:654] T a9e0659564894b0e8cb7f600bb07a085 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:39.056875 18985 tablet_bootstrap.cc:654] T a9e0659564894b0e8cb7f600bb07a085 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:39.057232 20047 raft_consensus.cc:2781] T 294644d7a16b4ca2bbb6d106bcb8c7ff P 5e1c6397f09747f091534faf93542710 [term 2 FOLLOWER]: Leader election won for term 2
I20240430 20:11:39.057410 20047 raft_consensus.cc:686] T 294644d7a16b4ca2bbb6d106bcb8c7ff P 5e1c6397f09747f091534faf93542710 [term 2 LEADER]: Becoming Leader. State: Replica: 5e1c6397f09747f091534faf93542710, State: Running, Role: LEADER
I20240430 20:11:39.057729 20047 consensus_queue.cc:237] T 294644d7a16b4ca2bbb6d106bcb8c7ff P 5e1c6397f09747f091534faf93542710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:39.058858 18558 leader_election.cc:302] T 0b4d9e860d7c45fd91d7a566605c9283 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:39.059413 19861 raft_consensus.cc:2781] T 0b4d9e860d7c45fd91d7a566605c9283 P 5e1c6397f09747f091534faf93542710 [term 2 FOLLOWER]: Leader election won for term 2
I20240430 20:11:39.059552 19861 raft_consensus.cc:686] T 0b4d9e860d7c45fd91d7a566605c9283 P 5e1c6397f09747f091534faf93542710 [term 2 LEADER]: Becoming Leader. State: Replica: 5e1c6397f09747f091534faf93542710, State: Running, Role: LEADER
I20240430 20:11:39.059725 20025 raft_consensus.cc:2781] T fcc9698abda4491b98e1e879589913a8 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:39.059772 19861 consensus_queue.cc:237] T 0b4d9e860d7c45fd91d7a566605c9283 P 5e1c6397f09747f091534faf93542710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:39.059943 20025 raft_consensus.cc:686] T fcc9698abda4491b98e1e879589913a8 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Becoming Leader. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Running, Role: LEADER
I20240430 20:11:39.005932 18751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12c6e131fa904d23865b8673b58a7e33" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:39.063623 18475 raft_consensus.cc:2454] T 1555924b5f1444398f12ee82c3cb5b13 P 2efb542aa55e439b99ccb98d1c5c884a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 2.
W20240430 20:11:39.064105 18417 leader_election.cc:341] T e5111d7607dd42188a0176cf5be2ec05 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Not found: Tablet not found: e5111d7607dd42188a0176cf5be2ec05
I20240430 20:11:39.064332 18481 raft_consensus.cc:1226] T 1555924b5f1444398f12ee82c3cb5b13 P 2efb542aa55e439b99ccb98d1c5c884a [term 2 FOLLOWER]: Rejecting Update request from peer 3c51c33d5616482cafb68e26688820f4 for earlier term 1. Current term is 2. Ops: []
I20240430 20:11:39.064370 18558 leader_election.cc:302] T 1555924b5f1444398f12ee82c3cb5b13 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
W20240430 20:11:39.064327 18417 leader_election.cc:341] T fd40967ab4a04001819e16c06a6c4300 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:39.064678 20038 raft_consensus.cc:2781] T 1555924b5f1444398f12ee82c3cb5b13 P 5e1c6397f09747f091534faf93542710 [term 2 FOLLOWER]: Leader election won for term 2
I20240430 20:11:39.064843 20038 raft_consensus.cc:686] T 1555924b5f1444398f12ee82c3cb5b13 P 5e1c6397f09747f091534faf93542710 [term 2 LEADER]: Becoming Leader. State: Replica: 5e1c6397f09747f091534faf93542710, State: Running, Role: LEADER
W20240430 20:11:39.065042 18415 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.177.2:46027, user_credentials={real_user=slave}} blocked reactor thread for 113829us
I20240430 20:11:39.065117 20038 consensus_queue.cc:237] T 1555924b5f1444398f12ee82c3cb5b13 P 5e1c6397f09747f091534faf93542710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
W20240430 20:11:39.066478 18417 leader_election.cc:341] T e6eac92dfa5f4456bc49571878f0f8dd P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
W20240430 20:11:39.067463 18417 leader_election.cc:341] T 802336602d8f4ac4bd7e0872d6f3dae9 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:39.063643 20025 consensus_queue.cc:237] T fcc9698abda4491b98e1e879589913a8 P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:39.070050 18475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e826459c3f4437da81c9e6e09f0c22f" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
W20240430 20:11:39.077163 18415 leader_election.cc:341] T 76d6e9d547ca4a848eae852944b1d117 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:39.078461 18760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acd23f0a63d5448aaa7ed78422e67bad" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:39.080503 18755 raft_consensus.cc:2454] T 53a899fa5a424a0d99f1b47925e8c7ad P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
I20240430 20:11:39.078015 18750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76d6e9d547ca4a848eae852944b1d117" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:39.085366 18752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75ebf051383e42529dcb5f7dd0770a4b" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:39.085618 18752 raft_consensus.cc:2379] T 75ebf051383e42529dcb5f7dd0770a4b P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 5e1c6397f09747f091534faf93542710 in current term 1: Already voted for candidate 3c51c33d5616482cafb68e26688820f4 in this term.
W20240430 20:11:39.085709 18415 leader_election.cc:341] T e5111d7607dd42188a0176cf5be2ec05 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Not found: Tablet not found: e5111d7607dd42188a0176cf5be2ec05
I20240430 20:11:39.085832 18753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "372b135dd01f44f98cd713ca8857ba61" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:39.085994 19495 ts_tablet_manager.cc:1888] T d9e578c99058424a9b86390641f9c34f P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:39.086179 19495 log.cc:1196] T d9e578c99058424a9b86390641f9c34f P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/d9e578c99058424a9b86390641f9c34f
I20240430 20:11:39.086205 18749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "294644d7a16b4ca2bbb6d106bcb8c7ff" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:39.086249 18754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bde72e2791a497db7dd08686ca36d80" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:39.086532 18758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1555924b5f1444398f12ee82c3cb5b13" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:39.086835 18478 raft_consensus.cc:1226] T 0b4d9e860d7c45fd91d7a566605c9283 P 2efb542aa55e439b99ccb98d1c5c884a [term 2 FOLLOWER]: Rejecting Update request from peer 3c51c33d5616482cafb68e26688820f4 for earlier term 1. Current term is 2. Ops: []
I20240430 20:11:39.087112 19495 ts_tablet_manager.cc:1909] T d9e578c99058424a9b86390641f9c34f P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:39.087069 18696 leader_election.cc:302] T 75ebf051383e42529dcb5f7dd0770a4b P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710
I20240430 20:11:39.087239 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 64) took 7677 ms. Trace:
I20240430 20:11:39.087375 19489 rpcz_store.cc:276] 0430 20:11:31.266648 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:31.266698 (+ 50us) service_pool.cc:225] Handling call
0430 20:11:38.934246 (+7667548us) ts_tablet_manager.cc:1120] Deleting tablet f77086a5675f44e99e8a867b694df4b8
0430 20:11:38.934262 (+ 16us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:38.943834 (+ 9572us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7661627}
W20240430 20:11:39.087519 18696 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.177.2:46027, user_credentials={real_user=slave}} blocked reactor thread for 85917.4us
W20240430 20:11:39.088027 19001 catalog_manager.cc:4797] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): delete failed for tablet f77086a5675f44e99e8a867b694df4b8 because the tablet was not found. No further retry: Not found: Tablet not found: f77086a5675f44e99e8a867b694df4b8
I20240430 20:11:39.088546 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 52) took 8036 ms. Trace:
I20240430 20:11:39.088526 20019 consensus_queue.cc:1035] T e3e74e76ab2b4fe492ffcc1838656fd1 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240430 20:11:39.088702 19495 rpcz_store.cc:276] 0430 20:11:31.052385 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:31.052591 (+ 206us) service_pool.cc:225] Handling call
0430 20:11:38.985930 (+7933339us) ts_tablet_manager.cc:1120] Deleting tablet d9e578c99058424a9b86390641f9c34f
0430 20:11:38.985940 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:39.051500 (+ 65560us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:39.085255 (+ 33755us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:39.088510 (+ 3255us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10521,"tablet-delete.queue_time_us":7871838}
I20240430 20:11:39.089257 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 64) took 7978 ms. Trace:
I20240430 20:11:39.089308 19004 catalog_manager.cc:4830] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): tablet d9e578c99058424a9b86390641f9c34f (table default.Table_40857db02a904104a4850e51d5996da2 [id=64a1f78e3f3f425d9c350ced9e169ac5]) successfully deleted
I20240430 20:11:39.092136 19495 rpcz_store.cc:276] 0430 20:11:31.111112 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:31.113405 (+ 2293us) service_pool.cc:225] Handling call
0430 20:11:39.089118 (+7975713us) ts_tablet_manager.cc:1120] Deleting tablet 0a9066d895744c5bb7d9532b2940603c
0430 20:11:39.089128 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:39.089226 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7974650}
I20240430 20:11:39.092567 18988 tablet_bootstrap.cc:654] T 802336602d8f4ac4bd7e0872d6f3dae9 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:39.092603 20130 raft_consensus.cc:2726] T 75ebf051383e42529dcb5f7dd0770a4b P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240430 20:11:39.092527 19495 tablet_replica.cc:330] T 255c13ad8fae4ea4aa8e7b7aabb7602e P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
W20240430 20:11:39.092753 19004 catalog_manager.cc:4810] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): delete failed for tablet 0a9066d895744c5bb7d9532b2940603c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0a9066d895744c5bb7d9532b2940603c already in progress: creating tablet
I20240430 20:11:39.095028 20060 raft_consensus.cc:504] T 7bde1d6934784c9aa5911181e8df9eeb P 5e1c6397f09747f091534faf93542710 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:39.097874 19033 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:51664 (request call id 180) took 2671 ms. Trace:
I20240430 20:11:39.098469 19495 raft_consensus.cc:2229] T 255c13ad8fae4ea4aa8e7b7aabb7602e P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:39.098681 19495 raft_consensus.cc:2258] T 255c13ad8fae4ea4aa8e7b7aabb7602e P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:39.098022 19033 rpcz_store.cc:276] 0430 20:11:36.426838 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:38.901627 (+2474789us) service_pool.cc:225] Handling call
0430 20:11:38.925643 (+ 24016us) spinlock_profiling.cc:243] Waited 20.1 ms on lock 0x616000040654. stack: 00007feb16c74756 00007feb16c74510 00007feb15ca748d 0000000000955804 00000000009557cd 00007feb257b315d 00007feb257b3253 00007feb224e5d41 00007feb2a0d134d 00007feb2a25dcb3 00007feb210462a6 00007feb21045fd5 00007feb1d97c04e 00007feb1d97af2e 00007feb1d981405 00007feb1d981d2f
0430 20:11:38.926686 (+ 1043us) catalog_manager.cc:2072] Acquired catalog manager lock
0430 20:11:38.927194 (+ 508us) catalog_manager.cc:2118] Created new table and tablet info
0430 20:11:39.097506 (+170312us) catalog_manager.cc:2178] Wrote table and tablets to system table
0430 20:11:39.097655 (+ 149us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0430 20:11:39.097790 (+ 135us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:38.948743 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:38.948878 (+ 135us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:38.948886 (+ 8us) write_op.cc:427] Acquired schema lock
0430 20:11:38.948888 (+ 2us) tablet.cc:660] Decoding operations
0430 20:11:39.042350 (+ 93462us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:39.042366 (+ 16us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:39.042372 (+ 6us) tablet.cc:683] Acquiring locks for 2 operations
0430 20:11:39.048499 (+ 6127us) tablet.cc:699] Row locks acquired
0430 20:11:39.048502 (+ 3us) write_op.cc:260] PREPARE: finished
0430 20:11:39.057765 (+ 9263us) write_op.cc:270] Start()
0430 20:11:39.057822 (+ 57us) write_op.cc:276] Timestamp: P: 1714507899048495 usec, L: 0
0430 20:11:39.057829 (+ 7us) op_driver.cc:331] REPLICATION: starting
0430 20:11:39.090033 (+ 32204us) log.cc:842] Serialized 433 byte log entry
0430 20:11:39.095475 (+ 5442us) op_driver.cc:444] REPLICATION: finished
0430 20:11:39.095640 (+ 165us) write_op.cc:301] APPLY: starting
0430 20:11:39.095660 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:39.095712 (+ 52us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:39.095742 (+ 30us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:39.095852 (+ 110us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:39.095892 (+ 40us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0430 20:11:39.097205 (+ 1313us) write_op.cc:312] APPLY: finished
0430 20:11:39.097329 (+ 124us) log.cc:842] Serialized 29 byte log entry
0430 20:11:39.097388 (+ 59us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:39.097408 (+ 20us) write_op.cc:446] Released schema lock
0430 20:11:39.097415 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":46305664,"child_traces":[["op",{"apply.queue_time_us":112,"num_ops":2,"prepare.queue_time_us":20903,"prepare.run_cpu_time_us":1061,"prepare.run_wall_time_us":141405,"raft.queue_time_us":657,"raft.run_cpu_time_us":283,"raft.run_wall_time_us":283,"replication_time_us":37619}]]}
I20240430 20:11:39.099216 19033 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_a39fa485d1e6488b951a9681aa5f3f00"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:39.099870 19033 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a39fa485d1e6488b951a9681aa5f3f00 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:39.102202 18415 leader_election.cc:302] T e5111d7607dd42188a0176cf5be2ec05 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a; no voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710
I20240430 20:11:39.103066 18450 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e69ca12bd10494fb013614741bc166c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:27 UTC) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:39.103268 18415 leader_election.cc:302] T 24dcc20832804229a2664873bf6ddaa7 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a; no voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710
W20240430 20:11:39.105646 18699 leader_election.cc:341] T 15674aeb56b74210a4a2809608fd4457 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Not found: Tablet not found: 15674aeb56b74210a4a2809608fd4457
I20240430 20:11:39.105723 19489 tablet_replica.cc:330] T 15674aeb56b74210a4a2809608fd4457 P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:39.105654 19784 raft_consensus.cc:2726] T 24dcc20832804229a2664873bf6ddaa7 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240430 20:11:39.105655 20100 raft_consensus.cc:2726] T e5111d7607dd42188a0176cf5be2ec05 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:39.106302 19495 ts_tablet_manager.cc:1875] T 255c13ad8fae4ea4aa8e7b7aabb7602e P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240430 20:11:39.106353 18696 leader_election.cc:341] T 0fe1247349344863a6273a769ce8ddf6 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Not found: Tablet not found: 0fe1247349344863a6273a769ce8ddf6
W20240430 20:11:39.106527 18415 leader_election.cc:341] T fd40967ab4a04001819e16c06a6c4300 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Illegal state: must be running to vote when last-logged opid is not known
W20240430 20:11:39.106642 18696 leader_election.cc:341] T 15674aeb56b74210a4a2809608fd4457 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Not found: Tablet not found: 15674aeb56b74210a4a2809608fd4457
I20240430 20:11:39.106700 18415 leader_election.cc:302] T fd40967ab4a04001819e16c06a6c4300 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a; no voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710
I20240430 20:11:39.107417 18696 leader_election.cc:302] T 15674aeb56b74210a4a2809608fd4457 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710
I20240430 20:11:39.107439 18415 leader_election.cc:302] T e3e74e76ab2b4fe492ffcc1838656fd1 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters: 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:39.107762 19784 raft_consensus.cc:2781] T e3e74e76ab2b4fe492ffcc1838656fd1 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader pre-election won for term 2
W20240430 20:11:39.107860 18696 leader_election.cc:341] T cace1398492f4d968dcb6899385d527c P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Not found: Tablet not found: cace1398492f4d968dcb6899385d527c
I20240430 20:11:39.108528 19784 raft_consensus.cc:482] T e3e74e76ab2b4fe492ffcc1838656fd1 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3c51c33d5616482cafb68e26688820f4)
I20240430 20:11:39.108667 19784 raft_consensus.cc:3037] T e3e74e76ab2b4fe492ffcc1838656fd1 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Advancing to term 2
I20240430 20:11:39.109853 18696 leader_election.cc:302] T 24dcc20832804229a2664873bf6ddaa7 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:39.111254 19489 raft_consensus.cc:2229] T 15674aeb56b74210a4a2809608fd4457 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:39.111411 20130 raft_consensus.cc:2684] T 15674aeb56b74210a4a2809608fd4457 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Received pre-election callback for term 2 while not running: Illegal state: RaftConsensus is not running: State = Stopping
W20240430 20:11:39.111779 18699 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.177.1:41217, user_credentials={real_user=slave}} blocked reactor thread for 44545.5us
I20240430 20:11:39.114202 20130 raft_consensus.cc:2781] T 24dcc20832804229a2664873bf6ddaa7 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:39.114241 18699 leader_election.cc:302] T 12c6e131fa904d23865b8673b58a7e33 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710
I20240430 20:11:39.110556 18614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a745334c12c545c1ba825c7552c60359" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:39.110888 18622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "294644d7a16b4ca2bbb6d106bcb8c7ff" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:39.114845 18621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680553d4d0bc4fcb839235c1832757e1" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:39.115322 18605 raft_consensus.cc:1226] T 7dfdecba959543a795dfcd3eeda29e58 P 5e1c6397f09747f091534faf93542710 [term 2 LEADER]: Rejecting Update request from peer 3c51c33d5616482cafb68e26688820f4 for earlier term 1. Current term is 2. Ops: []
I20240430 20:11:39.110983 18612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "372b135dd01f44f98cd713ca8857ba61" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:39.111094 18618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75ebf051383e42529dcb5f7dd0770a4b" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:39.115679 18612 raft_consensus.cc:2454] T 372b135dd01f44f98cd713ca8857ba61 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 1.
I20240430 20:11:39.111220 18617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6eac92dfa5f4456bc49571878f0f8dd" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:39.115927 18617 raft_consensus.cc:3037] T e6eac92dfa5f4456bc49571878f0f8dd P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:39.111747 18610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bde72e2791a497db7dd08686ca36d80" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:39.111995 18608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b76a4105a2974fe280e759349037d027" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:39.112277 18607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1555924b5f1444398f12ee82c3cb5b13" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:39.113013 18613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd6083f12c7e4d1e8a2006b3ddf9aa3a" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:39.113301 18623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "724e51ff05ec4baebc2bb55e05c8eddc" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:39.113612 18609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bde72e2791a497db7dd08686ca36d80" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:39.113703 18619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9e7660eccc448dbb99d135f82bfd261" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:39.113898 18611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5809e7c9c2204ba293426c24129fb16e" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:39.113979 18620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fae9a35a2ac844b4bc2ebdf214aa7171" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:39.119612 20130 raft_consensus.cc:686] T 24dcc20832804229a2664873bf6ddaa7 P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Becoming Leader. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Running, Role: LEADER
I20240430 20:11:39.120088 19489 raft_consensus.cc:2258] T 15674aeb56b74210a4a2809608fd4457 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:39.120872 19489 ts_tablet_manager.cc:1875] T 15674aeb56b74210a4a2809608fd4457 P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:39.123265 20131 raft_consensus.cc:2726] T 12c6e131fa904d23865b8673b58a7e33 P 3c51c33d5616482cafb68e26688820f4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240430 20:11:39.124941 19024 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:51664 (request call id 179) took 2756 ms. Trace:
I20240430 20:11:39.125085 19024 rpcz_store.cc:276] 0430 20:11:36.368753 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:38.884040 (+2515287us) service_pool.cc:225] Handling call
0430 20:11:38.927957 (+ 43917us) spinlock_profiling.cc:243] Waited 37.9 ms on lock 0x616000040654. stack: 00007feb16c74756 00007feb16c74510 00007feb15ca748d 0000000000955804 00000000009557cd 00007feb257b315d 00007feb257b3253 00007feb224e5d41 00007feb2a0d134d 00007feb2a25dcb3 00007feb210462a6 00007feb21045fd5 00007feb1d97c04e 00007feb1d97af2e 00007feb1d981405 00007feb1d981d2f
0430 20:11:38.928688 (+ 731us) catalog_manager.cc:2072] Acquired catalog manager lock
0430 20:11:38.929095 (+ 407us) catalog_manager.cc:2118] Created new table and tablet info
0430 20:11:39.124570 (+195475us) catalog_manager.cc:2178] Wrote table and tablets to system table
0430 20:11:39.124696 (+ 126us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0430 20:11:39.124847 (+ 151us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:39.090182 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:39.090306 (+ 124us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:39.090313 (+ 7us) write_op.cc:427] Acquired schema lock
0430 20:11:39.090315 (+ 2us) tablet.cc:660] Decoding operations
0430 20:11:39.090434 (+ 119us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:39.090450 (+ 16us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:39.090459 (+ 9us) tablet.cc:683] Acquiring locks for 2 operations
0430 20:11:39.113193 (+ 22734us) tablet.cc:699] Row locks acquired
0430 20:11:39.113199 (+ 6us) write_op.cc:260] PREPARE: finished
0430 20:11:39.113252 (+ 53us) write_op.cc:270] Start()
0430 20:11:39.113299 (+ 47us) write_op.cc:276] Timestamp: P: 1714507899113193 usec, L: 0
0430 20:11:39.113303 (+ 4us) op_driver.cc:331] REPLICATION: starting
0430 20:11:39.116129 (+ 2826us) log.cc:842] Serialized 433 byte log entry
0430 20:11:39.123795 (+ 7666us) op_driver.cc:444] REPLICATION: finished
0430 20:11:39.124031 (+ 236us) write_op.cc:301] APPLY: starting
0430 20:11:39.124051 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:39.124097 (+ 46us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:39.124134 (+ 37us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:39.124273 (+ 139us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:39.124312 (+ 39us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0430 20:11:39.124320 (+ 8us) write_op.cc:312] APPLY: finished
0430 20:11:39.124395 (+ 75us) log.cc:842] Serialized 29 byte log entry
0430 20:11:39.124458 (+ 63us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:39.124483 (+ 25us) write_op.cc:446] Released schema lock
0430 20:11:39.124490 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":87115136,"child_traces":[["op",{"apply.queue_time_us":176,"num_ops":2,"prepare.queue_time_us":160546,"prepare.run_cpu_time_us":930,"prepare.run_wall_time_us":26063,"raft.queue_time_us":627,"raft.run_cpu_time_us":377,"raft.run_wall_time_us":410,"replication_time_us":10467}]]}
I20240430 20:11:39.126233 19024 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_d4469d8c485e427c8e245855b9d5328b" } new_table_name: "default.Table_1844f6b1bda84812825384e14a2fd94e" modify_external_catalogs: true
I20240430 20:11:39.129375 19048 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:51664 (request call id 178) took 2760 ms. Trace:
I20240430 20:11:39.129480 19048 rpcz_store.cc:276] 0430 20:11:36.368639 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:38.879078 (+2510439us) service_pool.cc:225] Handling call
0430 20:11:38.929859 (+ 50781us) spinlock_profiling.cc:243] Waited 42.9 ms on lock 0x616000040654. stack: 00007feb16c74756 00007feb16c74510 00007feb15ca748d 0000000000955804 00000000009557cd 00007feb257b315d 00007feb257b3253 00007feb224e5d41 00007feb2a0d134d 00007feb2a25dcb3 00007feb210462a6 00007feb21045fd5 00007feb1d97c04e 00007feb1d97af2e 00007feb1d981405 00007feb1d981d2f
0430 20:11:38.930516 (+ 657us) catalog_manager.cc:2072] Acquired catalog manager lock
0430 20:11:38.931003 (+ 487us) catalog_manager.cc:2118] Created new table and tablet info
0430 20:11:39.129070 (+198067us) catalog_manager.cc:2178] Wrote table and tablets to system table
0430 20:11:39.129208 (+ 138us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0430 20:11:39.129318 (+ 110us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:39.116286 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:39.116438 (+ 152us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:39.116445 (+ 7us) write_op.cc:427] Acquired schema lock
0430 20:11:39.116448 (+ 3us) tablet.cc:660] Decoding operations
0430 20:11:39.116553 (+ 105us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:39.116565 (+ 12us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:39.116569 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
0430 20:11:39.116651 (+ 82us) tablet.cc:699] Row locks acquired
0430 20:11:39.116653 (+ 2us) write_op.cc:260] PREPARE: finished
0430 20:11:39.116685 (+ 32us) write_op.cc:270] Start()
0430 20:11:39.116718 (+ 33us) write_op.cc:276] Timestamp: P: 1714507899116629 usec, L: 0
0430 20:11:39.116721 (+ 3us) op_driver.cc:331] REPLICATION: starting
0430 20:11:39.123006 (+ 6285us) log.cc:842] Serialized 433 byte log entry
0430 20:11:39.128419 (+ 5413us) op_driver.cc:444] REPLICATION: finished
0430 20:11:39.128597 (+ 178us) write_op.cc:301] APPLY: starting
0430 20:11:39.128610 (+ 13us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:39.128653 (+ 43us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:39.128687 (+ 34us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:39.128792 (+ 105us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:39.128822 (+ 30us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0430 20:11:39.128829 (+ 7us) write_op.cc:312] APPLY: finished
0430 20:11:39.128910 (+ 81us) log.cc:842] Serialized 29 byte log entry
0430 20:11:39.128971 (+ 61us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:39.128991 (+ 20us) write_op.cc:446] Released schema lock
0430 20:11:39.128997 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":98721024,"child_traces":[["op",{"apply.queue_time_us":134,"num_ops":2,"prepare.queue_time_us":184466,"prepare.run_cpu_time_us":761,"prepare.run_wall_time_us":6811,"replication_time_us":11678}]]}
I20240430 20:11:39.130268 19048 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_b3741e0be6594cfab0de42dec8e1f2fe" } new_table_name: "default.Table_e686cdfcbcc046749fa5b51578a64794" modify_external_catalogs: true
I20240430 20:11:39.132187 19028 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_343ee22ed6554f328aa58881079e1905" } new_table_name: "default.Table_69892a8264a4477db48ee6248758e046" modify_external_catalogs: true
I20240430 20:11:39.132516 18809 heartbeater.cc:347] Connected to a master server at 127.17.177.62:36441
I20240430 20:11:39.132653 18699 leader_election.cc:302] T cace1398492f4d968dcb6899385d527c P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710
I20240430 20:11:39.134188 18619 raft_consensus.cc:2379] T c9e7660eccc448dbb99d135f82bfd261 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c51c33d5616482cafb68e26688820f4 in current term 1: Already voted for candidate 5e1c6397f09747f091534faf93542710 in this term.
I20240430 20:11:39.135725 18415 leader_election.cc:302] T 372b135dd01f44f98cd713ca8857ba61 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:39.135775 20060 leader_election.cc:288] T 7bde1d6934784c9aa5911181e8df9eeb P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 2 election: Requested vote from peers 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
W20240430 20:11:39.136504 18557 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.177.3:44867, user_credentials={real_user=slave}} blocked reactor thread for 55975.9us
W20240430 20:11:39.136894 18699 raft_consensus.cc:2670] T cace1398492f4d968dcb6899385d527c P 3c51c33d5616482cafb68e26688820f4: unable to run election callback: Service unavailable: Thread pool token was shut down
I20240430 20:11:39.135432 20130 consensus_queue.cc:237] T 24dcc20832804229a2664873bf6ddaa7 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:39.142724 20103 raft_consensus.cc:2781] T 372b135dd01f44f98cd713ca8857ba61 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240430 20:11:39.142858 20103 raft_consensus.cc:482] T 372b135dd01f44f98cd713ca8857ba61 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3c51c33d5616482cafb68e26688820f4)
I20240430 20:11:39.137631 18449 tablet_service.cc:1514] Processing DeleteTablet for tablet c5fd0730133c472299c93f4b0830e527 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:25 UTC) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:39.143335 20103 raft_consensus.cc:3037] T 372b135dd01f44f98cd713ca8857ba61 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Advancing to term 2
W20240430 20:11:39.143818 18699 leader_election.cc:341] T 0fe1247349344863a6273a769ce8ddf6 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Not found: Tablet not found: 0fe1247349344863a6273a769ce8ddf6
I20240430 20:11:39.144207 18699 leader_election.cc:302] T 0fe1247349344863a6273a769ce8ddf6 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710
I20240430 20:11:39.144985 20085 raft_consensus.cc:2726] T 0fe1247349344863a6273a769ce8ddf6 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240430 20:11:39.145058 18699 leader_election.cc:302] T e2d90ff17e574f88b2061b1c117d4b7d P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710
I20240430 20:11:39.145457 18699 leader_election.cc:302] T 77285afcafe04a829e485998bbbe00f0 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710
I20240430 20:11:39.145748 20085 raft_consensus.cc:2726] T e2d90ff17e574f88b2061b1c117d4b7d P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240430 20:11:39.146284 19033 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:51664 (request call id 181) took 2677 ms. Trace:
I20240430 20:11:39.146358 20019 raft_consensus.cc:2726] T 77285afcafe04a829e485998bbbe00f0 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240430 20:11:39.146723 20100 raft_consensus.cc:2726] T fd40967ab4a04001819e16c06a6c4300 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:39.146395 19033 rpcz_store.cc:276] 0430 20:11:36.468464 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:39.098944 (+2630480us) service_pool.cc:225] Handling call
0430 20:11:39.099993 (+ 1049us) catalog_manager.cc:2072] Acquired catalog manager lock
0430 20:11:39.100558 (+ 565us) catalog_manager.cc:2118] Created new table and tablet info
0430 20:11:39.145993 (+ 45435us) catalog_manager.cc:2178] Wrote table and tablets to system table
0430 20:11:39.146122 (+ 129us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0430 20:11:39.146221 (+ 99us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:39.123136 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:39.138690 (+ 15554us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:39.138699 (+ 9us) write_op.cc:427] Acquired schema lock
0430 20:11:39.138703 (+ 4us) tablet.cc:660] Decoding operations
0430 20:11:39.138838 (+ 135us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:39.138857 (+ 19us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:39.138864 (+ 7us) tablet.cc:683] Acquiring locks for 2 operations
0430 20:11:39.138971 (+ 107us) tablet.cc:699] Row locks acquired
0430 20:11:39.138973 (+ 2us) write_op.cc:260] PREPARE: finished
0430 20:11:39.139008 (+ 35us) write_op.cc:270] Start()
0430 20:11:39.139058 (+ 50us) write_op.cc:276] Timestamp: P: 1714507899138947 usec, L: 0
0430 20:11:39.139077 (+ 19us) op_driver.cc:331] REPLICATION: starting
0430 20:11:39.139271 (+ 194us) log.cc:842] Serialized 433 byte log entry
0430 20:11:39.145417 (+ 6146us) op_driver.cc:444] REPLICATION: finished
0430 20:11:39.145539 (+ 122us) write_op.cc:301] APPLY: starting
0430 20:11:39.145555 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:39.145599 (+ 44us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:39.145628 (+ 29us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:39.145725 (+ 97us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:39.145744 (+ 19us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0430 20:11:39.145749 (+ 5us) write_op.cc:312] APPLY: finished
0430 20:11:39.145807 (+ 58us) log.cc:842] Serialized 29 byte log entry
0430 20:11:39.145873 (+ 66us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:39.145902 (+ 29us) write_op.cc:446] Released schema lock
0430 20:11:39.145910 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":73,"num_ops":2,"prepare.queue_time_us":21967,"prepare.run_cpu_time_us":822,"prepare.run_wall_time_us":16236,"raft.queue_time_us":895,"raft.run_cpu_time_us":241,"raft.run_wall_time_us":241,"replication_time_us":6316}]]}
I20240430 20:11:39.147246 19033 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_3ec50a6d9cd84592becff23d2cfa0c03" } modify_external_catalogs: true
I20240430 20:11:39.147568 19033 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_3ec50a6d9cd84592becff23d2cfa0c03" } modify_external_catalogs: true
I20240430 20:11:39.150015 19784 raft_consensus.cc:504] T e3e74e76ab2b4fe492ffcc1838656fd1 P 2efb542aa55e439b99ccb98d1c5c884a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:39.150864 19784 leader_election.cc:288] T e3e74e76ab2b4fe492ffcc1838656fd1 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 election: Requested vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:39.153002 18528 heartbeater.cc:347] Connected to a master server at 127.17.177.62:36441
I20240430 20:11:39.114636 18606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e826459c3f4437da81c9e6e09f0c22f" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:39.161388 20103 raft_consensus.cc:504] T 372b135dd01f44f98cd713ca8857ba61 P 2efb542aa55e439b99ccb98d1c5c884a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:39.162503 20103 leader_election.cc:288] T 372b135dd01f44f98cd713ca8857ba61 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:39.163216 18985 tablet_bootstrap.cc:492] T a9e0659564894b0e8cb7f600bb07a085 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:39.164556 18985 ts_tablet_manager.cc:1367] T a9e0659564894b0e8cb7f600bb07a085 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.138s user 0.008s sys 0.009s
I20240430 20:11:39.167393 18985 raft_consensus.cc:348] T a9e0659564894b0e8cb7f600bb07a085 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:39.169600 18985 raft_consensus.cc:374] T a9e0659564894b0e8cb7f600bb07a085 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:39.169867 18985 raft_consensus.cc:729] T a9e0659564894b0e8cb7f600bb07a085 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:39.170441 18985 consensus_queue.cc:260] T a9e0659564894b0e8cb7f600bb07a085 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:39.170943 18699 leader_election.cc:302] T aec9c09cc0454acab6006bb3e10982b7 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710
I20240430 20:11:39.171566 18699 leader_election.cc:302] T 5809e7c9c2204ba293426c24129fb16e P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters:
I20240430 20:11:39.172309 20019 raft_consensus.cc:2726] T aec9c09cc0454acab6006bb3e10982b7 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240430 20:11:39.172524 20019 raft_consensus.cc:2781] T 5809e7c9c2204ba293426c24129fb16e P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240430 20:11:39.172605 20019 raft_consensus.cc:482] T 5809e7c9c2204ba293426c24129fb16e P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 5e1c6397f09747f091534faf93542710)
I20240430 20:11:39.172730 20019 raft_consensus.cc:3037] T 5809e7c9c2204ba293426c24129fb16e P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Advancing to term 2
I20240430 20:11:39.172008 18985 ts_tablet_manager.cc:1398] T a9e0659564894b0e8cb7f600bb07a085 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240430 20:11:39.176091 20019 raft_consensus.cc:504] T 5809e7c9c2204ba293426c24129fb16e P 3c51c33d5616482cafb68e26688820f4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:39.176869 18985 tablet_bootstrap.cc:492] T c9e7660eccc448dbb99d135f82bfd261 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:39.177423 20019 leader_election.cc:288] T 5809e7c9c2204ba293426c24129fb16e P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 2 election: Requested vote from peers 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:39.176537 18699 leader_election.cc:302] T 5bde72e2791a497db7dd08686ca36d80 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters:
I20240430 20:11:39.178718 18699 leader_election.cc:302] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters:
I20240430 20:11:39.179106 20107 raft_consensus.cc:2781] T 5bde72e2791a497db7dd08686ca36d80 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240430 20:11:39.179204 20107 raft_consensus.cc:482] T 5bde72e2791a497db7dd08686ca36d80 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 5e1c6397f09747f091534faf93542710)
I20240430 20:11:39.179314 20107 raft_consensus.cc:3037] T 5bde72e2791a497db7dd08686ca36d80 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Advancing to term 2
I20240430 20:11:39.179883 18985 tablet_bootstrap.cc:654] T c9e7660eccc448dbb99d135f82bfd261 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:39.182438 20107 raft_consensus.cc:504] T 5bde72e2791a497db7dd08686ca36d80 P 3c51c33d5616482cafb68e26688820f4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:39.183476 20107 leader_election.cc:288] T 5bde72e2791a497db7dd08686ca36d80 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 2 election: Requested vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:39.183872 20107 raft_consensus.cc:2781] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240430 20:11:39.184032 20107 raft_consensus.cc:482] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 5e1c6397f09747f091534faf93542710)
I20240430 20:11:39.184178 20107 raft_consensus.cc:3037] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Advancing to term 2
I20240430 20:11:39.187044 20085 consensus_queue.cc:1046] T a05e32a33aa7488eb287cc6314c09c44 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240430 20:11:39.187372 20085 consensus_queue.cc:1046] T 1555924b5f1444398f12ee82c3cb5b13 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240430 20:11:39.187428 18417 leader_election.cc:341] T e6eac92dfa5f4456bc49571878f0f8dd P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:39.187688 20085 consensus_queue.cc:1046] T 0b4d9e860d7c45fd91d7a566605c9283 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: INVALID_TERM, Last received: 1.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240430 20:11:39.187731 20019 raft_consensus.cc:3032] T a05e32a33aa7488eb287cc6314c09c44 P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Stepping down as leader of term 1
I20240430 20:11:39.187991 20085 raft_consensus.cc:3032] T 1555924b5f1444398f12ee82c3cb5b13 P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Stepping down as leader of term 1
I20240430 20:11:39.188019 20019 raft_consensus.cc:729] T a05e32a33aa7488eb287cc6314c09c44 P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Running, Role: LEADER
I20240430 20:11:39.188093 20085 raft_consensus.cc:729] T 1555924b5f1444398f12ee82c3cb5b13 P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Running, Role: LEADER
I20240430 20:11:39.188328 20085 consensus_queue.cc:260] T 1555924b5f1444398f12ee82c3cb5b13 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:39.188791 20085 raft_consensus.cc:3037] T 1555924b5f1444398f12ee82c3cb5b13 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Advancing to term 2
I20240430 20:11:39.189023 18417 leader_election.cc:302] T b76a4105a2974fe280e759349037d027 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a; no voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710
I20240430 20:11:39.190429 20025 raft_consensus.cc:2726] T b76a4105a2974fe280e759349037d027 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240430 20:11:39.191939 18985 tablet_bootstrap.cc:492] T c9e7660eccc448dbb99d135f82bfd261 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:39.192528 18417 leader_election.cc:302] T c66c142419ca4e1399ecd1c8d1e62a59 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters: 5e1c6397f09747f091534faf93542710
I20240430 20:11:39.200062 18417 leader_election.cc:302] T a745334c12c545c1ba825c7552c60359 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a; no voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710
I20240430 20:11:39.200443 18417 leader_election.cc:302] T 12c6e131fa904d23865b8673b58a7e33 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a; no voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710
I20240430 20:11:39.200929 18417 leader_election.cc:302] T 294644d7a16b4ca2bbb6d106bcb8c7ff P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a; no voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710
I20240430 20:11:39.201277 18417 leader_election.cc:302] T 5bde72e2791a497db7dd08686ca36d80 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a; no voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710
I20240430 20:11:39.201640 18417 leader_election.cc:302] T 1555924b5f1444398f12ee82c3cb5b13 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a; no voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710
I20240430 20:11:39.201938 20071 raft_consensus.cc:2781] T c66c142419ca4e1399ecd1c8d1e62a59 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240430 20:11:39.194773 18481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bde72e2791a497db7dd08686ca36d80" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:39.202072 20071 raft_consensus.cc:482] T c66c142419ca4e1399ecd1c8d1e62a59 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 5e1c6397f09747f091534faf93542710)
I20240430 20:11:39.202205 20071 raft_consensus.cc:3037] T c66c142419ca4e1399ecd1c8d1e62a59 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Advancing to term 2
I20240430 20:11:39.202466 19784 raft_consensus.cc:2726] T a745334c12c545c1ba825c7552c60359 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240430 20:11:39.203204 19784 raft_consensus.cc:2726] T 12c6e131fa904d23865b8673b58a7e33 P 2efb542aa55e439b99ccb98d1c5c884a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240430 20:11:39.203608 19784 raft_consensus.cc:2726] T 294644d7a16b4ca2bbb6d106bcb8c7ff P 2efb542aa55e439b99ccb98d1c5c884a [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240430 20:11:39.203927 19784 raft_consensus.cc:2726] T 5bde72e2791a497db7dd08686ca36d80 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240430 20:11:39.204331 19784 raft_consensus.cc:2726] T 1555924b5f1444398f12ee82c3cb5b13 P 2efb542aa55e439b99ccb98d1c5c884a [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240430 20:11:39.195084 18478 raft_consensus.cc:1226] T e3e74e76ab2b4fe492ffcc1838656fd1 P 2efb542aa55e439b99ccb98d1c5c884a [term 2 FOLLOWER]: Rejecting Update request from peer 3c51c33d5616482cafb68e26688820f4 for earlier term 1. Current term is 2. Ops: []
W20240430 20:11:39.202085 18417 leader_election.cc:341] T 76d6e9d547ca4a848eae852944b1d117 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:39.207530 18417 leader_election.cc:302] T 76d6e9d547ca4a848eae852944b1d117 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a; no voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710
I20240430 20:11:39.208114 19784 raft_consensus.cc:2726] T 76d6e9d547ca4a848eae852944b1d117 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:39.195344 18475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5809e7c9c2204ba293426c24129fb16e" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:39.208597 20071 raft_consensus.cc:504] T c66c142419ca4e1399ecd1c8d1e62a59 P 2efb542aa55e439b99ccb98d1c5c884a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:39.208863 18475 raft_consensus.cc:3037] T 5809e7c9c2204ba293426c24129fb16e P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Advancing to term 2
I20240430 20:11:39.209491 19805 raft_consensus.cc:3032] T 0b4d9e860d7c45fd91d7a566605c9283 P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Stepping down as leader of term 1
W20240430 20:11:39.209637 20140 log.cc:924] Time spent T 7dfdecba959543a795dfcd3eeda29e58 P 5e1c6397f09747f091534faf93542710: Append to log took a long time: real 0.083s user 0.000s sys 0.000s
I20240430 20:11:39.212639 18475 raft_consensus.cc:2454] T 5809e7c9c2204ba293426c24129fb16e P 2efb542aa55e439b99ccb98d1c5c884a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 2.
I20240430 20:11:39.213430 18985 ts_tablet_manager.cc:1367] T c9e7660eccc448dbb99d135f82bfd261 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.036s user 0.003s sys 0.001s
I20240430 20:11:39.209401 20071 leader_election.cc:288] T c66c142419ca4e1399ecd1c8d1e62a59 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 election: Requested vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:39.215986 18609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3e74e76ab2b4fe492ffcc1838656fd1" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:39.216573 19495 ts_tablet_manager.cc:1888] T 255c13ad8fae4ea4aa8e7b7aabb7602e P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:39.216725 19495 log.cc:1196] T 255c13ad8fae4ea4aa8e7b7aabb7602e P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/255c13ad8fae4ea4aa8e7b7aabb7602e
I20240430 20:11:39.217128 18985 raft_consensus.cc:348] T c9e7660eccc448dbb99d135f82bfd261 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:39.188325 20019 consensus_queue.cc:260] T a05e32a33aa7488eb287cc6314c09c44 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:39.222719 19805 raft_consensus.cc:729] T 0b4d9e860d7c45fd91d7a566605c9283 P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Running, Role: LEADER
I20240430 20:11:39.223186 19805 consensus_queue.cc:260] T 0b4d9e860d7c45fd91d7a566605c9283 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:39.228389 18607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66c142419ca4e1399ecd1c8d1e62a59" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:39.228549 18985 raft_consensus.cc:374] T c9e7660eccc448dbb99d135f82bfd261 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:39.228675 18985 raft_consensus.cc:729] T c9e7660eccc448dbb99d135f82bfd261 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:39.229656 18613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5809e7c9c2204ba293426c24129fb16e" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:39.229938 18611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "372b135dd01f44f98cd713ca8857ba61" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:39.231745 18611 raft_consensus.cc:3037] T 372b135dd01f44f98cd713ca8857ba61 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Advancing to term 2
I20240430 20:11:39.229254 18475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bde1d6934784c9aa5911181e8df9eeb" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:39.232538 18985 consensus_queue.cc:260] T c9e7660eccc448dbb99d135f82bfd261 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:39.233840 20105 raft_consensus.cc:482] T a9e0659564894b0e8cb7f600bb07a085 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:39.234012 20105 raft_consensus.cc:504] T a9e0659564894b0e8cb7f600bb07a085 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:39.235013 20105 leader_election.cc:288] T a9e0659564894b0e8cb7f600bb07a085 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:39.235564 18985 ts_tablet_manager.cc:1398] T c9e7660eccc448dbb99d135f82bfd261 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.022s user 0.002s sys 0.000s
I20240430 20:11:39.235853 18985 tablet_bootstrap.cc:492] T f5546f9edeea44d7a6c5cb954f5157ba P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:39.237293 18557 leader_election.cc:302] T aec9c09cc0454acab6006bb3e10982b7 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a
I20240430 20:11:39.239240 18985 tablet_bootstrap.cc:654] T f5546f9edeea44d7a6c5cb954f5157ba P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:39.217147 19495 ts_tablet_manager.cc:1909] T 255c13ad8fae4ea4aa8e7b7aabb7602e P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:39.231293 18608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bde72e2791a497db7dd08686ca36d80" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:39.247802 20060 raft_consensus.cc:2781] T aec9c09cc0454acab6006bb3e10982b7 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20240430 20:11:39.247972 18557 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.177.3:44867, user_credentials={real_user=slave}} blocked reactor thread for 98143.4us
I20240430 20:11:39.246770 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 55) took 8152 ms. Trace:
I20240430 20:11:39.251935 18611 raft_consensus.cc:2454] T 372b135dd01f44f98cd713ca8857ba61 P 5e1c6397f09747f091534faf93542710 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 2.
I20240430 20:11:39.252440 18415 leader_election.cc:302] T 372b135dd01f44f98cd713ca8857ba61 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:39.252557 18608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9e0659564894b0e8cb7f600bb07a085" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:39.253130 20105 raft_consensus.cc:2781] T 372b135dd01f44f98cd713ca8857ba61 P 2efb542aa55e439b99ccb98d1c5c884a [term 2 FOLLOWER]: Leader election won for term 2
I20240430 20:11:39.253268 20105 raft_consensus.cc:686] T 372b135dd01f44f98cd713ca8857ba61 P 2efb542aa55e439b99ccb98d1c5c884a [term 2 LEADER]: Becoming Leader. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Running, Role: LEADER
I20240430 20:11:39.253571 20105 consensus_queue.cc:237] T 372b135dd01f44f98cd713ca8857ba61 P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:39.250340 19495 rpcz_store.cc:276] 0430 20:11:31.094235 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:31.094405 (+ 170us) service_pool.cc:225] Handling call
0430 20:11:39.092459 (+7998054us) ts_tablet_manager.cc:1120] Deleting tablet 255c13ad8fae4ea4aa8e7b7aabb7602e
0430 20:11:39.092468 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:39.161525 (+ 69057us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:39.216443 (+ 54918us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:39.246314 (+ 29871us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7972289}
I20240430 20:11:39.256228 19004 catalog_manager.cc:4830] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): tablet 255c13ad8fae4ea4aa8e7b7aabb7602e (table default.Table_a5ab8fc47d4e4d128ba4370b3249d8e5 [id=1238beeef04b476b8f4dc5af2fb68012]) successfully deleted
I20240430 20:11:39.256398 20025 consensus_queue.cc:1035] T b72fe1f0713b4076a81a3fc1c74f7fa7 P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240430 20:11:39.257264 18415 leader_election.cc:341] T a9e0659564894b0e8cb7f600bb07a085 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:39.258203 19495 tablet_replica.cc:330] T cace1398492f4d968dcb6899385d527c P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:39.260095 19805 raft_consensus.cc:3037] T 0b4d9e860d7c45fd91d7a566605c9283 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Advancing to term 2
I20240430 20:11:39.260118 18696 leader_election.cc:302] T 680553d4d0bc4fcb839235c1832757e1 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710
I20240430 20:11:39.263833 19495 raft_consensus.cc:2229] T cace1398492f4d968dcb6899385d527c P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Raft consensus shutting down.
I20240430 20:11:39.268297 20019 raft_consensus.cc:3037] T a05e32a33aa7488eb287cc6314c09c44 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Advancing to term 2
I20240430 20:11:39.271973 20060 raft_consensus.cc:482] T aec9c09cc0454acab6006bb3e10982b7 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 2efb542aa55e439b99ccb98d1c5c884a)
I20240430 20:11:39.272176 20060 raft_consensus.cc:3037] T aec9c09cc0454acab6006bb3e10982b7 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Advancing to term 2
I20240430 20:11:39.276861 20060 raft_consensus.cc:504] T aec9c09cc0454acab6006bb3e10982b7 P 5e1c6397f09747f091534faf93542710 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:39.284679 18475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aec9c09cc0454acab6006bb3e10982b7" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:39.280261 18557 leader_election.cc:302] T b72fe1f0713b4076a81a3fc1c74f7fa7 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a
I20240430 20:11:39.283027 18617 raft_consensus.cc:2454] T e6eac92dfa5f4456bc49571878f0f8dd P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 1.
I20240430 20:11:39.287178 19471 ts_tablet_manager.cc:1888] T b688e03ea71940c69bda5520b62c1fc3 P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:39.287321 19471 log.cc:1196] T b688e03ea71940c69bda5520b62c1fc3 P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/b688e03ea71940c69bda5520b62c1fc3
I20240430 20:11:39.287706 19471 ts_tablet_manager.cc:1909] T b688e03ea71940c69bda5520b62c1fc3 P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:39.288046 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 72) took 8248 ms. Trace:
I20240430 20:11:39.292680 18988 tablet_bootstrap.cc:492] T 802336602d8f4ac4bd7e0872d6f3dae9 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:39.285902 18415 leader_election.cc:302] T e6eac92dfa5f4456bc49571878f0f8dd P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters: 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:39.288195 19471 rpcz_store.cc:276] 0430 20:11:31.039126 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:31.047569 (+ 8443us) service_pool.cc:225] Handling call
0430 20:11:39.006525 (+7958956us) ts_tablet_manager.cc:1120] Deleting tablet b688e03ea71940c69bda5520b62c1fc3
0430 20:11:39.006842 (+ 317us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:39.217300 (+210458us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:39.285043 (+ 67743us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:39.288011 (+ 2968us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7958358}
I20240430 20:11:39.285672 19495 raft_consensus.cc:2258] T cace1398492f4d968dcb6899385d527c P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:39.293954 18999 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet b688e03ea71940c69bda5520b62c1fc3 (table default.Table_2ec9639eca4a47439b7859929cd705bc [id=6ffe241384bd42b7a22d41154ac59e3a]) successfully deleted
I20240430 20:11:39.292479 18985 tablet_bootstrap.cc:492] T f5546f9edeea44d7a6c5cb954f5157ba P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:39.294571 18985 ts_tablet_manager.cc:1367] T f5546f9edeea44d7a6c5cb954f5157ba P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.059s user 0.004s sys 0.000s
I20240430 20:11:39.289256 20060 leader_election.cc:288] T aec9c09cc0454acab6006bb3e10982b7 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 2 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:39.295905 18985 raft_consensus.cc:348] T f5546f9edeea44d7a6c5cb954f5157ba P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:39.296231 18985 raft_consensus.cc:374] T f5546f9edeea44d7a6c5cb954f5157ba P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:39.296329 18985 raft_consensus.cc:729] T f5546f9edeea44d7a6c5cb954f5157ba P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:39.296663 20025 raft_consensus.cc:2781] T e6eac92dfa5f4456bc49571878f0f8dd P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:39.296653 18985 consensus_queue.cc:260] T f5546f9edeea44d7a6c5cb954f5157ba P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:39.296859 20025 raft_consensus.cc:686] T e6eac92dfa5f4456bc49571878f0f8dd P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Becoming Leader. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Running, Role: LEADER
I20240430 20:11:39.297052 18762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9e7660eccc448dbb99d135f82bfd261" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:39.297245 18985 ts_tablet_manager.cc:1398] T f5546f9edeea44d7a6c5cb954f5157ba P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240430 20:11:39.297148 20025 consensus_queue.cc:237] T e6eac92dfa5f4456bc49571878f0f8dd P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:39.297292 18762 raft_consensus.cc:3037] T c9e7660eccc448dbb99d135f82bfd261 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:39.291225 20107 raft_consensus.cc:504] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 3c51c33d5616482cafb68e26688820f4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:39.297520 18985 tablet_bootstrap.cc:492] T 0eb4f3e7ece6496293e01b63620d644b P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:39.297853 20107 leader_election.cc:288] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 2 election: Requested vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:39.297976 20130 raft_consensus.cc:2726] T 680553d4d0bc4fcb839235c1832757e1 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240430 20:11:39.298785 18749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bde1d6934784c9aa5911181e8df9eeb" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:39.299865 18754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a05e32a33aa7488eb287cc6314c09c44" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:39.300062 18763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dfdecba959543a795dfcd3eeda29e58" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:39.300390 18750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81787eccca6c42c1b421377a3bf9ebe6" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:39.298230 18615 raft_consensus.cc:1226] T a05e32a33aa7488eb287cc6314c09c44 P 5e1c6397f09747f091534faf93542710 [term 2 LEADER]: Rejecting Update request from peer 3c51c33d5616482cafb68e26688820f4 for earlier term 1. Current term is 2. Ops: [1.2-1.2]
I20240430 20:11:39.300797 18758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b4d9e860d7c45fd91d7a566605c9283" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:39.300915 18760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dfdecba959543a795dfcd3eeda29e58" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:39.300997 20085 consensus_queue.cc:1046] T 7dfdecba959543a795dfcd3eeda29e58 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.007s
I20240430 20:11:39.301213 18751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44fa6917a85c46f3aaa363df46ea586b" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:39.301366 18557 leader_election.cc:302] T 724e51ff05ec4baebc2bb55e05c8eddc P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a
I20240430 20:11:39.303172 18985 tablet_bootstrap.cc:654] T 0eb4f3e7ece6496293e01b63620d644b P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:39.306160 19495 ts_tablet_manager.cc:1875] T cace1398492f4d968dcb6899385d527c P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:39.307647 18753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1555924b5f1444398f12ee82c3cb5b13" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:39.307797 18983 tablet_bootstrap.cc:492] T a9e0659564894b0e8cb7f600bb07a085 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:39.308131 19861 raft_consensus.cc:2781] T 724e51ff05ec4baebc2bb55e05c8eddc P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240430 20:11:39.308241 19861 raft_consensus.cc:482] T 724e51ff05ec4baebc2bb55e05c8eddc P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 2efb542aa55e439b99ccb98d1c5c884a)
I20240430 20:11:39.309759 19861 raft_consensus.cc:3037] T 724e51ff05ec4baebc2bb55e05c8eddc P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Advancing to term 2
I20240430 20:11:39.310376 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 73) took 8254 ms. Trace:
I20240430 20:11:39.312646 19471 rpcz_store.cc:276] 0430 20:11:31.048020 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:31.048067 (+ 47us) service_pool.cc:225] Handling call
0430 20:11:39.293445 (+8245378us) ts_tablet_manager.cc:1120] Deleting tablet e5111d7607dd42188a0176cf5be2ec05
0430 20:11:39.293453 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:39.302853 (+ 9400us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":8245168}
W20240430 20:11:39.310964 18557 leader_election.cc:341] T 47c37876694c4a4abb1091c631dab7a5 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:39.312940 18557 leader_election.cc:302] T 47c37876694c4a4abb1091c631dab7a5 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4
W20240430 20:11:39.313546 18557 leader_election.cc:341] T 13c1107ea4d04eb78a74d3f658c72fb3 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Not found: Tablet not found: 13c1107ea4d04eb78a74d3f658c72fb3
I20240430 20:11:39.313661 18557 leader_election.cc:302] T 13c1107ea4d04eb78a74d3f658c72fb3 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:39.314152 20040 raft_consensus.cc:2726] T 47c37876694c4a4abb1091c631dab7a5 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:39.310865 18983 ts_tablet_manager.cc:1367] T a9e0659564894b0e8cb7f600bb07a085 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.384s user 0.008s sys 0.004s
W20240430 20:11:39.315659 18999 catalog_manager.cc:4797] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): delete failed for tablet e5111d7607dd42188a0176cf5be2ec05 because the tablet was not found. No further retry: Not found: Tablet not found: e5111d7607dd42188a0176cf5be2ec05
I20240430 20:11:39.316095 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 74) took 8267 ms. Trace:
I20240430 20:11:39.317451 18983 raft_consensus.cc:348] T a9e0659564894b0e8cb7f600bb07a085 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:39.318718 20047 raft_consensus.cc:2781] T b72fe1f0713b4076a81a3fc1c74f7fa7 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240430 20:11:39.307919 18764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b4d9e860d7c45fd91d7a566605c9283" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
W20240430 20:11:39.319514 18557 raft_consensus.cc:2670] T 13c1107ea4d04eb78a74d3f658c72fb3 P 5e1c6397f09747f091534faf93542710: unable to run election callback: Service unavailable: Thread pool token was shut down
I20240430 20:11:39.320075 18757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a05e32a33aa7488eb287cc6314c09c44" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:39.322906 18983 raft_consensus.cc:374] T a9e0659564894b0e8cb7f600bb07a085 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:39.323029 18983 raft_consensus.cc:729] T a9e0659564894b0e8cb7f600bb07a085 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:39.324005 18696 leader_election.cc:302] T 724e51ff05ec4baebc2bb55e05c8eddc P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710
I20240430 20:11:39.324282 18988 ts_tablet_manager.cc:1367] T 802336602d8f4ac4bd7e0872d6f3dae9 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.286s user 0.008s sys 0.009s
I20240430 20:11:39.324910 19859 raft_consensus.cc:2726] T 724e51ff05ec4baebc2bb55e05c8eddc P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240430 20:11:39.325235 18696 leader_election.cc:302] T fae9a35a2ac844b4bc2ebdf214aa7171 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710
I20240430 20:11:39.323791 19471 rpcz_store.cc:276] 0430 20:11:31.048376 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:31.048412 (+ 36us) service_pool.cc:225] Handling call
0430 20:11:39.315975 (+8267563us) ts_tablet_manager.cc:1120] Deleting tablet 0e503a898406412ab977d3e048ea4b5d
0430 20:11:39.315984 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:39.316053 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":8267402}
I20240430 20:11:39.327267 18756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "294644d7a16b4ca2bbb6d106bcb8c7ff" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:39.325810 20047 raft_consensus.cc:482] T b72fe1f0713b4076a81a3fc1c74f7fa7 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 2efb542aa55e439b99ccb98d1c5c884a)
I20240430 20:11:39.328212 20047 raft_consensus.cc:3037] T b72fe1f0713b4076a81a3fc1c74f7fa7 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Advancing to term 2
I20240430 20:11:39.329574 18988 raft_consensus.cc:348] T 802336602d8f4ac4bd7e0872d6f3dae9 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:39.329777 19859 raft_consensus.cc:2726] T fae9a35a2ac844b4bc2ebdf214aa7171 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240430 20:11:39.330814 18755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "294644d7a16b4ca2bbb6d106bcb8c7ff" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:39.325907 18557 leader_election.cc:302] T 77285afcafe04a829e485998bbbe00f0 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a
I20240430 20:11:39.331734 19049 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet acd23f0a63d5448aaa7ed78422e67bad (table default.Table_972f13be4dca4cd19b9a04534dc59f51 [id=69f37db4de634a3185613e40013e0ed7]). Expected version 1 got 0
I20240430 20:11:39.332123 18557 leader_election.cc:302] T fcc9698abda4491b98e1e879589913a8 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a
I20240430 20:11:39.332471 18757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "372b135dd01f44f98cd713ca8857ba61" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:39.332548 18557 leader_election.cc:302] T e2d90ff17e574f88b2061b1c117d4b7d P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a
I20240430 20:11:39.333319 18588 tablet_service.cc:1467] Processing CreateTablet for tablet 8e9711d0a382424f9f758bfa883866a0 (DEFAULT_TABLE table=default.Table_b7c7025fbf2e4039aadd143d021bfef5 [id=db696ce66005401fa73746e55c071f44]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:39.333822 18588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e9711d0a382424f9f758bfa883866a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:39.332046 19049 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 2e79df8744e74b5991a79ce978ff0202 (table default.Table_276bbdc7cfe542d1a2936485a2aad1ec [id=f146a15e6e8a4acbb4c734c1c863da57]). Expected version 1 got 0
I20240430 20:11:39.335428 18449 tablet_service.cc:1467] Processing CreateTablet for tablet 8e9711d0a382424f9f758bfa883866a0 (DEFAULT_TABLE table=default.Table_b7c7025fbf2e4039aadd143d021bfef5 [id=db696ce66005401fa73746e55c071f44]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:39.335760 19049 catalog_manager.cc:5484] T e3e74e76ab2b4fe492ffcc1838656fd1 P 3c51c33d5616482cafb68e26688820f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c51c33d5616482cafb68e26688820f4 (127.17.177.3). New cstate: current_term: 1 leader_uuid: "3c51c33d5616482cafb68e26688820f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } health_report { overall_health: UNKNOWN } } }
I20240430 20:11:39.336619 18449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e9711d0a382424f9f758bfa883866a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:39.337829 18587 tablet_service.cc:1467] Processing CreateTablet for tablet 898b2c91adf34545808df9c786e218c9 (DEFAULT_TABLE table=default.Table_8e605def35724c5c945b61107596d81a [id=36f6e11d7b1843c3be3e8b8160e06d53]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:39.338265 18587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 898b2c91adf34545808df9c786e218c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:39.343194 18600 tablet_service.cc:1467] Processing CreateTablet for tablet 417351cbc9d54f9d84bb7155f39da119 (DEFAULT_TABLE table=default.Table_5d0995eff94c404fa97b08aa190f4a7f [id=6a55f1d4a8094a88bd8dd5a076630b24]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:39.343192 18450 tablet_service.cc:1467] Processing CreateTablet for tablet 417351cbc9d54f9d84bb7155f39da119 (DEFAULT_TABLE table=default.Table_5d0995eff94c404fa97b08aa190f4a7f [id=6a55f1d4a8094a88bd8dd5a076630b24]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:39.343699 18600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 417351cbc9d54f9d84bb7155f39da119. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:39.343709 18450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 417351cbc9d54f9d84bb7155f39da119. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:39.346560 18592 tablet_service.cc:1467] Processing CreateTablet for tablet 828ca913e2aa4102a6277bdbc6686ef5 (DEFAULT_TABLE table=default.Table_5d51defcfdfd4d7f8e7d341c5daf0cdf [id=bdab45e393374761bea47d965b81f043]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:39.346995 18592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 828ca913e2aa4102a6277bdbc6686ef5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:39.347208 18454 tablet_service.cc:1467] Processing CreateTablet for tablet 898b2c91adf34545808df9c786e218c9 (DEFAULT_TABLE table=default.Table_8e605def35724c5c945b61107596d81a [id=36f6e11d7b1843c3be3e8b8160e06d53]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:39.347617 18454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 898b2c91adf34545808df9c786e218c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:39.352245 20040 raft_consensus.cc:2781] T 77285afcafe04a829e485998bbbe00f0 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240430 20:11:39.352363 20040 raft_consensus.cc:482] T 77285afcafe04a829e485998bbbe00f0 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 2efb542aa55e439b99ccb98d1c5c884a)
I20240430 20:11:39.352478 20040 raft_consensus.cc:3037] T 77285afcafe04a829e485998bbbe00f0 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Advancing to term 2
I20240430 20:11:39.354908 20038 raft_consensus.cc:2741] T fcc9698abda4491b98e1e879589913a8 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20240430 20:11:39.332262 18756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66c142419ca4e1399ecd1c8d1e62a59" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:39.355114 20038 raft_consensus.cc:2781] T e2d90ff17e574f88b2061b1c117d4b7d P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240430 20:11:39.355211 20038 raft_consensus.cc:482] T e2d90ff17e574f88b2061b1c117d4b7d P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 2efb542aa55e439b99ccb98d1c5c884a)
I20240430 20:11:39.355294 20038 raft_consensus.cc:3037] T e2d90ff17e574f88b2061b1c117d4b7d P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Advancing to term 2
I20240430 20:11:39.356201 18478 raft_consensus.cc:1226] T 372b135dd01f44f98cd713ca8857ba61 P 2efb542aa55e439b99ccb98d1c5c884a [term 2 LEADER]: Rejecting Update request from peer 3c51c33d5616482cafb68e26688820f4 for earlier term 1. Current term is 2. Ops: []
I20240430 20:11:39.356582 18481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd6083f12c7e4d1e8a2006b3ddf9aa3a" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:39.357409 18699 leader_election.cc:302] T 5809e7c9c2204ba293426c24129fb16e P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters:
I20240430 20:11:39.357551 18734 tablet_service.cc:1467] Processing CreateTablet for tablet 898b2c91adf34545808df9c786e218c9 (DEFAULT_TABLE table=default.Table_8e605def35724c5c945b61107596d81a [id=36f6e11d7b1843c3be3e8b8160e06d53]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:39.358180 20038 raft_consensus.cc:504] T e2d90ff17e574f88b2061b1c117d4b7d P 5e1c6397f09747f091534faf93542710 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:39.358609 20085 consensus_queue.cc:1225] T 7dfdecba959543a795dfcd3eeda29e58 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Peer 5e1c6397f09747f091534faf93542710 log is divergent from this leader: its last log entry 2.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20240430 20:11:39.358611 18751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1555924b5f1444398f12ee82c3cb5b13" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:39.359114 20038 leader_election.cc:288] T e2d90ff17e574f88b2061b1c117d4b7d P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 2 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:39.359181 18753 raft_consensus.cc:2454] T 1555924b5f1444398f12ee82c3cb5b13 P 3c51c33d5616482cafb68e26688820f4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 2.
I20240430 20:11:39.359380 19489 ts_tablet_manager.cc:1888] T 15674aeb56b74210a4a2809608fd4457 P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:39.359423 18481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2d90ff17e574f88b2061b1c117d4b7d" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:39.359772 20131 raft_consensus.cc:2781] T 5809e7c9c2204ba293426c24129fb16e P 3c51c33d5616482cafb68e26688820f4 [term 2 FOLLOWER]: Leader election won for term 2
I20240430 20:11:39.360435 18725 tablet_service.cc:1467] Processing CreateTablet for tablet 8e9711d0a382424f9f758bfa883866a0 (DEFAULT_TABLE table=default.Table_b7c7025fbf2e4039aadd143d021bfef5 [id=db696ce66005401fa73746e55c071f44]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:39.360404 19489 log.cc:1196] T 15674aeb56b74210a4a2809608fd4457 P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/15674aeb56b74210a4a2809608fd4457
I20240430 20:11:39.361049 20040 raft_consensus.cc:504] T 77285afcafe04a829e485998bbbe00f0 P 5e1c6397f09747f091534faf93542710 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:39.361078 20019 consensus_queue.cc:1046] T 372b135dd01f44f98cd713ca8857ba61 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: INVALID_TERM, Last received: 1.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.003s
I20240430 20:11:39.361600 20107 raft_consensus.cc:482] T 26ed12f498644d15bf45e884dcad172e P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:39.360414 18724 tablet_service.cc:1467] Processing CreateTablet for tablet 417351cbc9d54f9d84bb7155f39da119 (DEFAULT_TABLE table=default.Table_5d0995eff94c404fa97b08aa190f4a7f [id=6a55f1d4a8094a88bd8dd5a076630b24]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:39.361953 20107 raft_consensus.cc:504] T 26ed12f498644d15bf45e884dcad172e P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:39.362145 20040 leader_election.cc:288] T 77285afcafe04a829e485998bbbe00f0 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 2 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:39.362538 18744 tablet_service.cc:1467] Processing CreateTablet for tablet 828ca913e2aa4102a6277bdbc6686ef5 (DEFAULT_TABLE table=default.Table_5d51defcfdfd4d7f8e7d341c5daf0cdf [id=bdab45e393374761bea47d965b81f043]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:39.363307 18481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77285afcafe04a829e485998bbbe00f0" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:39.364430 20019 consensus_queue.cc:1225] T 372b135dd01f44f98cd713ca8857ba61 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Peer 2efb542aa55e439b99ccb98d1c5c884a log is divergent from this leader: its last log entry 2.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20240430 20:11:39.324002 18983 consensus_queue.cc:260] T a9e0659564894b0e8cb7f600bb07a085 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:39.365254 18983 ts_tablet_manager.cc:1398] T a9e0659564894b0e8cb7f600bb07a085 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.050s user 0.001s sys 0.001s
I20240430 20:11:39.365463 19805 raft_consensus.cc:3032] T 372b135dd01f44f98cd713ca8857ba61 P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Stepping down as leader of term 1
I20240430 20:11:39.365590 19805 raft_consensus.cc:729] T 372b135dd01f44f98cd713ca8857ba61 P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Running, Role: LEADER
I20240430 20:11:39.365626 18983 tablet_bootstrap.cc:492] T fd40967ab4a04001819e16c06a6c4300 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:39.365890 19805 consensus_queue.cc:260] T 372b135dd01f44f98cd713ca8857ba61 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:39.366273 19805 raft_consensus.cc:3037] T 372b135dd01f44f98cd713ca8857ba61 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Advancing to term 2
I20240430 20:11:39.367465 18983 tablet_bootstrap.cc:654] T fd40967ab4a04001819e16c06a6c4300 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:39.369743 18762 raft_consensus.cc:2454] T c9e7660eccc448dbb99d135f82bfd261 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
I20240430 20:11:39.370025 18764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75ebf051383e42529dcb5f7dd0770a4b" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:39.370218 18764 raft_consensus.cc:3037] T 75ebf051383e42529dcb5f7dd0770a4b P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Advancing to term 2
I20240430 20:11:39.373203 18764 raft_consensus.cc:2454] T 75ebf051383e42529dcb5f7dd0770a4b P 3c51c33d5616482cafb68e26688820f4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 2.
I20240430 20:11:39.373514 18750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3e74e76ab2b4fe492ffcc1838656fd1" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:39.373572 18988 raft_consensus.cc:374] T 802336602d8f4ac4bd7e0872d6f3dae9 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:39.373706 18988 raft_consensus.cc:729] T 802336602d8f4ac4bd7e0872d6f3dae9 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:39.373939 18988 consensus_queue.cc:260] T 802336602d8f4ac4bd7e0872d6f3dae9 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:39.374493 18988 ts_tablet_manager.cc:1398] T 802336602d8f4ac4bd7e0872d6f3dae9 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.050s user 0.000s sys 0.002s
I20240430 20:11:39.374657 18417 leader_election.cc:302] T 75ebf051383e42529dcb5f7dd0770a4b P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters: 5e1c6397f09747f091534faf93542710
I20240430 20:11:39.374835 18988 tablet_bootstrap.cc:492] T 20d166756819406ba85044f4fee9c4ce P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
W20240430 20:11:39.374985 18696 consensus_peers.cc:485] T eed6338ad3d64859af087ee15e1ec798 P 3c51c33d5616482cafb68e26688820f4 -> Peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Couldn't send request to peer 5e1c6397f09747f091534faf93542710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: eed6338ad3d64859af087ee15e1ec798. This is attempt 1: this message will repeat every 5th retry.
I20240430 20:11:39.374981 20071 raft_consensus.cc:2781] T 75ebf051383e42529dcb5f7dd0770a4b P 2efb542aa55e439b99ccb98d1c5c884a [term 2 FOLLOWER]: Leader election won for term 2
W20240430 20:11:39.375084 18696 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.177.2:46027, user_credentials={real_user=slave}} blocked reactor thread for 44849.8us
I20240430 20:11:39.375269 18754 raft_consensus.cc:2396] T a05e32a33aa7488eb287cc6314c09c44 P 3c51c33d5616482cafb68e26688820f4 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e1c6397f09747f091534faf93542710 for term 2 because replica has last-logged OpId of term: 1 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20240430 20:11:39.375254 18557 leader_election.cc:341] T f5546f9edeea44d7a6c5cb954f5157ba P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:39.375372 18557 leader_election.cc:302] T f5546f9edeea44d7a6c5cb954f5157ba P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:39.375396 18756 raft_consensus.cc:3037] T c66c142419ca4e1399ecd1c8d1e62a59 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Advancing to term 2
I20240430 20:11:39.375545 18696 leader_election.cc:302] T c9e7660eccc448dbb99d135f82bfd261 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710
I20240430 20:11:39.375838 18557 leader_election.cc:302] T 6e826459c3f4437da81c9e6e09f0c22f P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a
I20240430 20:11:39.375985 18696 leader_election.cc:302] T 6e826459c3f4437da81c9e6e09f0c22f P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710
I20240430 20:11:39.376083 20061 raft_consensus.cc:2726] T f5546f9edeea44d7a6c5cb954f5157ba P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:39.376243 20061 raft_consensus.cc:2781] T 6e826459c3f4437da81c9e6e09f0c22f P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240430 20:11:39.376328 20061 raft_consensus.cc:482] T 6e826459c3f4437da81c9e6e09f0c22f P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 2efb542aa55e439b99ccb98d1c5c884a)
I20240430 20:11:39.376407 20061 raft_consensus.cc:3037] T 6e826459c3f4437da81c9e6e09f0c22f P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Advancing to term 2
I20240430 20:11:39.376750 18696 leader_election.cc:302] T 5bde72e2791a497db7dd08686ca36d80 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710
I20240430 20:11:39.376746 20019 raft_consensus.cc:2726] T c9e7660eccc448dbb99d135f82bfd261 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:39.377002 20019 raft_consensus.cc:2726] T 6e826459c3f4437da81c9e6e09f0c22f P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240430 20:11:39.377372 20085 raft_consensus.cc:2726] T 5bde72e2791a497db7dd08686ca36d80 P 3c51c33d5616482cafb68e26688820f4 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240430 20:11:39.378414 18756 raft_consensus.cc:2454] T c66c142419ca4e1399ecd1c8d1e62a59 P 3c51c33d5616482cafb68e26688820f4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 2.
I20240430 20:11:39.378731 18763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9e0659564894b0e8cb7f600bb07a085" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:39.379102 19859 consensus_queue.cc:1046] T e3e74e76ab2b4fe492ffcc1838656fd1 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.021s
I20240430 20:11:39.379186 18615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26ed12f498644d15bf45e884dcad172e" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
W20240430 20:11:39.379271 18557 leader_election.cc:341] T acd23f0a63d5448aaa7ed78422e67bad P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:39.379381 19859 raft_consensus.cc:3032] T e3e74e76ab2b4fe492ffcc1838656fd1 P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Stepping down as leader of term 1
I20240430 20:11:39.379496 19859 raft_consensus.cc:729] T e3e74e76ab2b4fe492ffcc1838656fd1 P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Running, Role: LEADER
I20240430 20:11:39.379832 19859 consensus_queue.cc:260] T e3e74e76ab2b4fe492ffcc1838656fd1 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:39.380250 19859 raft_consensus.cc:3037] T e3e74e76ab2b4fe492ffcc1838656fd1 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Advancing to term 2
I20240430 20:11:39.380452 20096 raft_consensus.cc:482] T 2a1e50f99c5d4b44bdbb51b58a92bc75 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:39.380595 20096 raft_consensus.cc:504] T 2a1e50f99c5d4b44bdbb51b58a92bc75 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:39.381317 20096 leader_election.cc:288] T 2a1e50f99c5d4b44bdbb51b58a92bc75 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:39.382138 20061 raft_consensus.cc:504] T 6e826459c3f4437da81c9e6e09f0c22f P 5e1c6397f09747f091534faf93542710 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:39.382820 20061 leader_election.cc:288] T 6e826459c3f4437da81c9e6e09f0c22f P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 2 election: Requested vote from peers 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:39.382947 18734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 898b2c91adf34545808df9c786e218c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:39.382994 20131 raft_consensus.cc:686] T 5809e7c9c2204ba293426c24129fb16e P 3c51c33d5616482cafb68e26688820f4 [term 2 LEADER]: Becoming Leader. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Running, Role: LEADER
I20240430 20:11:39.383330 20131 consensus_queue.cc:237] T 5809e7c9c2204ba293426c24129fb16e P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:39.383515 18557 leader_election.cc:302] T 81787eccca6c42c1b421377a3bf9ebe6 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:39.384068 18557 leader_election.cc:302] T 44fa6917a85c46f3aaa363df46ea586b P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:39.384449 18619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd6083f12c7e4d1e8a2006b3ddf9aa3a" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:39.384614 18752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01436e5b6232494babbb414342f03090" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:39.384975 20130 raft_consensus.cc:3032] T 7dfdecba959543a795dfcd3eeda29e58 P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Stepping down as leader of term 1
I20240430 20:11:39.385340 19489 ts_tablet_manager.cc:1909] T 15674aeb56b74210a4a2809608fd4457 P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:39.385299 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 75) took 8336 ms. Trace:
I20240430 20:11:39.385459 19471 rpcz_store.cc:276] 0430 20:11:31.048682 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:31.048722 (+ 40us) service_pool.cc:225] Handling call
0430 20:11:39.385138 (+8336416us) ts_tablet_manager.cc:1120] Deleting tablet 5f56ac2afd444d63a6bfd96988464bbb
0430 20:11:39.385164 (+ 26us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:39.385246 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":8336188}
I20240430 20:11:39.385824 19471 tablet_replica.cc:330] T 2a1e50f99c5d4b44bdbb51b58a92bc75 P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:39.386075 19471 raft_consensus.cc:2229] T 2a1e50f99c5d4b44bdbb51b58a92bc75 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Raft consensus shutting down.
W20240430 20:11:39.386497 18999 catalog_manager.cc:4797] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): delete failed for tablet 5f56ac2afd444d63a6bfd96988464bbb because the tablet was not found. No further retry: Not found: Tablet not found: 5f56ac2afd444d63a6bfd96988464bbb
I20240430 20:11:39.388235 18758 raft_consensus.cc:2454] T 0b4d9e860d7c45fd91d7a566605c9283 P 3c51c33d5616482cafb68e26688820f4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 2.
I20240430 20:11:39.388185 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 66) took 8115 ms. Trace:
I20240430 20:11:39.388506 18696 leader_election.cc:302] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710
I20240430 20:11:39.388901 20130 raft_consensus.cc:729] T 7dfdecba959543a795dfcd3eeda29e58 P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Running, Role: LEADER
I20240430 20:11:39.388867 19861 raft_consensus.cc:504] T 724e51ff05ec4baebc2bb55e05c8eddc P 5e1c6397f09747f091534faf93542710 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:39.389564 19489 rpcz_store.cc:276] 0430 20:11:31.270236 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:31.270314 (+ 78us) service_pool.cc:225] Handling call
0430 20:11:39.087718 (+7817404us) ts_tablet_manager.cc:1120] Deleting tablet 15674aeb56b74210a4a2809608fd4457
0430 20:11:39.087728 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:39.168665 (+ 80937us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:39.267669 (+ 99004us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:39.385690 (+118021us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7814648}
I20240430 20:11:39.390280 20130 consensus_queue.cc:260] T 7dfdecba959543a795dfcd3eeda29e58 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:39.390681 20130 raft_consensus.cc:3037] T 7dfdecba959543a795dfcd3eeda29e58 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Advancing to term 2
I20240430 20:11:39.390765 20071 raft_consensus.cc:686] T 75ebf051383e42529dcb5f7dd0770a4b P 2efb542aa55e439b99ccb98d1c5c884a [term 2 LEADER]: Becoming Leader. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Running, Role: LEADER
I20240430 20:11:39.390947 18478 raft_consensus.cc:1226] T c66c142419ca4e1399ecd1c8d1e62a59 P 2efb542aa55e439b99ccb98d1c5c884a [term 2 FOLLOWER]: Rejecting Update request from peer 5e1c6397f09747f091534faf93542710 for earlier term 1. Current term is 2. Ops: []
I20240430 20:11:39.391465 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 65) took 8121 ms. Trace:
I20240430 20:11:39.391589 19489 rpcz_store.cc:276] 0430 20:11:31.269750 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:31.269841 (+ 91us) service_pool.cc:225] Handling call
0430 20:11:39.390548 (+8120707us) ts_tablet_manager.cc:1120] Deleting tablet f5546f9edeea44d7a6c5cb954f5157ba
0430 20:11:39.390561 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:39.391412 (+ 851us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":8116615}
I20240430 20:11:39.391968 19489 tablet_replica.cc:330] T 0fe1247349344863a6273a769ce8ddf6 P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:39.392218 19489 raft_consensus.cc:2229] T 0fe1247349344863a6273a769ce8ddf6 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:39.392426 19489 raft_consensus.cc:2258] T 0fe1247349344863a6273a769ce8ddf6 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:39.392419 19861 leader_election.cc:288] T 724e51ff05ec4baebc2bb55e05c8eddc P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 2 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:39.393059 19489 ts_tablet_manager.cc:1875] T 0fe1247349344863a6273a769ce8ddf6 P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:39.393880 19001 catalog_manager.cc:4830] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): tablet 15674aeb56b74210a4a2809608fd4457 (table default.Table_7974c4f995c047469ccc69cc0d0521f3 [id=efc0011648254789a4f3b41a1fffdf4f]) successfully deleted
I20240430 20:11:39.394002 19859 raft_consensus.cc:2726] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 3c51c33d5616482cafb68e26688820f4 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20240430 20:11:39.394220 19001 catalog_manager.cc:4810] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): delete failed for tablet f5546f9edeea44d7a6c5cb954f5157ba because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f5546f9edeea44d7a6c5cb954f5157ba already in progress: creating tablet
I20240430 20:11:39.390231 20107 leader_election.cc:288] T 26ed12f498644d15bf45e884dcad172e P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:39.389626 18724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 417351cbc9d54f9d84bb7155f39da119. 1 dirs total, 0 dirs full, 0 dirs failed
W20240430 20:11:39.390077 20019 consensus_queue.cc:1159] T e3e74e76ab2b4fe492ffcc1838656fd1 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "5e1c6397f09747f091534faf93542710" responder_term: 1 status { last_received { term: 1 index: 1 } last_committed_idx: 0 last_received_current_leader { term: 1 index: 1 } }
I20240430 20:11:39.394963 18725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e9711d0a382424f9f758bfa883866a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:39.395802 18744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 828ca913e2aa4102a6277bdbc6686ef5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:39.396834 18988 tablet_bootstrap.cc:654] T 20d166756819406ba85044f4fee9c4ce P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:39.402752 20071 consensus_queue.cc:237] T 75ebf051383e42529dcb5f7dd0770a4b P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:39.405969 19049 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 22fef8bb6b034db098c3372ae992f7a7 (table default.Table_4440df38610340758047115845b42b95 [id=7d186ca20c094cc19aef792fe1c094e9]). Expected version 1 got 0
I20240430 20:11:39.408212 18481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26ed12f498644d15bf45e884dcad172e" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:39.408205 18417 leader_election.cc:302] T e3e74e76ab2b4fe492ffcc1838656fd1 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a; no voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710
I20240430 20:11:39.409497 20105 raft_consensus.cc:2726] T e3e74e76ab2b4fe492ffcc1838656fd1 P 2efb542aa55e439b99ccb98d1c5c884a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240430 20:11:39.410473 18482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "724e51ff05ec4baebc2bb55e05c8eddc" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:39.412092 18475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a1e50f99c5d4b44bdbb51b58a92bc75" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:39.412292 18475 raft_consensus.cc:2454] T 2a1e50f99c5d4b44bdbb51b58a92bc75 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 0.
I20240430 20:11:39.409516 18474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e826459c3f4437da81c9e6e09f0c22f" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:39.413622 18478 raft_consensus.cc:1226] T 294644d7a16b4ca2bbb6d106bcb8c7ff P 2efb542aa55e439b99ccb98d1c5c884a [term 2 FOLLOWER]: Rejecting Update request from peer 3c51c33d5616482cafb68e26688820f4 for earlier term 1. Current term is 2. Ops: []
I20240430 20:11:39.414503 18417 leader_election.cc:302] T c66c142419ca4e1399ecd1c8d1e62a59 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters: 5e1c6397f09747f091534faf93542710
W20240430 20:11:39.414980 18417 leader_election.cc:341] T a9e0659564894b0e8cb7f600bb07a085 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:39.415117 18417 leader_election.cc:302] T a9e0659564894b0e8cb7f600bb07a085 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a; no voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710
I20240430 20:11:39.415763 20025 raft_consensus.cc:2781] T c66c142419ca4e1399ecd1c8d1e62a59 P 2efb542aa55e439b99ccb98d1c5c884a [term 2 FOLLOWER]: Leader election won for term 2
I20240430 20:11:39.415920 20025 raft_consensus.cc:686] T c66c142419ca4e1399ecd1c8d1e62a59 P 2efb542aa55e439b99ccb98d1c5c884a [term 2 LEADER]: Becoming Leader. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Running, Role: LEADER
I20240430 20:11:39.416177 20025 consensus_queue.cc:237] T c66c142419ca4e1399ecd1c8d1e62a59 P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:39.416946 20096 raft_consensus.cc:2726] T 81787eccca6c42c1b421377a3bf9ebe6 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240430 20:11:39.417217 20103 raft_consensus.cc:2726] T a9e0659564894b0e8cb7f600bb07a085 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:39.417307 19471 raft_consensus.cc:2258] T 2a1e50f99c5d4b44bdbb51b58a92bc75 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:39.419348 18985 tablet_bootstrap.cc:492] T 0eb4f3e7ece6496293e01b63620d644b P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:39.419492 18985 ts_tablet_manager.cc:1367] T 0eb4f3e7ece6496293e01b63620d644b P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.122s user 0.003s sys 0.002s
I20240430 20:11:39.420269 18985 raft_consensus.cc:348] T 0eb4f3e7ece6496293e01b63620d644b P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:39.420473 18985 raft_consensus.cc:374] T 0eb4f3e7ece6496293e01b63620d644b P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:39.420560 18985 raft_consensus.cc:729] T 0eb4f3e7ece6496293e01b63620d644b P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:39.420768 18985 consensus_queue.cc:260] T 0eb4f3e7ece6496293e01b63620d644b P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:39.421331 18985 ts_tablet_manager.cc:1398] T 0eb4f3e7ece6496293e01b63620d644b P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240430 20:11:39.421640 18985 tablet_bootstrap.cc:492] T 6e754af64cb64fd2a082adae7d5f24da P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
W20240430 20:11:39.422317 18415 consensus_peers.cc:485] T b688e03ea71940c69bda5520b62c1fc3 P 2efb542aa55e439b99ccb98d1c5c884a -> Peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Couldn't send request to peer 5e1c6397f09747f091534faf93542710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b688e03ea71940c69bda5520b62c1fc3. This is attempt 1: this message will repeat every 5th retry.
I20240430 20:11:39.423718 18985 tablet_bootstrap.cc:654] T 6e754af64cb64fd2a082adae7d5f24da P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:39.427273 20096 raft_consensus.cc:2726] T 44fa6917a85c46f3aaa363df46ea586b P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240430 20:11:39.427816 20096 consensus_queue.cc:1046] T c66c142419ca4e1399ecd1c8d1e62a59 P 5e1c6397f09747f091534faf93542710 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: INVALID_TERM, Last received: 1.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240430 20:11:39.428102 18558 leader_election.cc:302] T 2a1e50f99c5d4b44bdbb51b58a92bc75 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:39.428454 19471 ts_tablet_manager.cc:1875] T 2a1e50f99c5d4b44bdbb51b58a92bc75 P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:39.429536 20047 raft_consensus.cc:504] T b72fe1f0713b4076a81a3fc1c74f7fa7 P 5e1c6397f09747f091534faf93542710 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:39.428311 20096 raft_consensus.cc:3032] T c66c142419ca4e1399ecd1c8d1e62a59 P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Stepping down as leader of term 1
I20240430 20:11:39.430250 20096 raft_consensus.cc:729] T c66c142419ca4e1399ecd1c8d1e62a59 P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Running, Role: LEADER
W20240430 20:11:39.429652 18558 raft_consensus.cc:2670] T 2a1e50f99c5d4b44bdbb51b58a92bc75 P 5e1c6397f09747f091534faf93542710: unable to run election callback: Service unavailable: Thread pool token was shut down
I20240430 20:11:39.434370 18699 leader_election.cc:302] T 26ed12f498644d15bf45e884dcad172e P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710
I20240430 20:11:39.434581 18985 tablet_bootstrap.cc:492] T 6e754af64cb64fd2a082adae7d5f24da P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:39.435075 18608 raft_consensus.cc:1226] T b72fe1f0713b4076a81a3fc1c74f7fa7 P 5e1c6397f09747f091534faf93542710 [term 2 FOLLOWER]: Rejecting Update request from peer 2efb542aa55e439b99ccb98d1c5c884a for earlier term 1. Current term is 2. Ops: []
I20240430 20:11:39.440642 20053 log.cc:824] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: Log is configured to *not* fsync() on all Append() calls
I20240430 20:11:39.441515 20019 raft_consensus.cc:2726] T 26ed12f498644d15bf45e884dcad172e P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:39.441510 20107 consensus_queue.cc:1046] T 294644d7a16b4ca2bbb6d106bcb8c7ff P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: INVALID_TERM, Last received: 1.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.004s
I20240430 20:11:39.441861 20019 raft_consensus.cc:3032] T 294644d7a16b4ca2bbb6d106bcb8c7ff P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Stepping down as leader of term 1
I20240430 20:11:39.444093 20019 raft_consensus.cc:729] T 294644d7a16b4ca2bbb6d106bcb8c7ff P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Running, Role: LEADER
I20240430 20:11:39.444573 19049 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 12c6e131fa904d23865b8673b58a7e33 (table default.Table_a9dc8fc6a4aa46dc914d0e327a4de554 [id=19f4e6decedf4ee59fccc0a24a05cedd]). Expected version 2 got 0
I20240430 20:11:39.445300 19049 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 66dd767bd3f8494f94061fb89e5e5680 (table default.Table_9c1e6182953140bb9cd594f08458c7d9 [id=5227206f44d04fa89b4842b899f33afb]). Expected version 1 got 0
I20240430 20:11:39.445758 20019 consensus_queue.cc:260] T 294644d7a16b4ca2bbb6d106bcb8c7ff P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:39.445554 20096 consensus_queue.cc:260] T c66c142419ca4e1399ecd1c8d1e62a59 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:39.446383 20096 raft_consensus.cc:3037] T c66c142419ca4e1399ecd1c8d1e62a59 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Advancing to term 2
I20240430 20:11:39.449235 20019 raft_consensus.cc:3037] T 294644d7a16b4ca2bbb6d106bcb8c7ff P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Advancing to term 2
I20240430 20:11:39.455271 18724 tablet_service.cc:1467] Processing CreateTablet for tablet 6b51cbc731794083a88fe4bed1ddeebf (DEFAULT_TABLE table=default.Table_17cb787c6a4c4c9fa720ce8c4e326605 [id=8e0f5d8c941d423ea7b30a89513d5170]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:39.456317 18557 leader_election.cc:302] T c9e7660eccc448dbb99d135f82bfd261 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a
I20240430 20:11:39.457031 18557 leader_election.cc:302] T 01436e5b6232494babbb414342f03090 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:39.457321 20040 raft_consensus.cc:2781] T c9e7660eccc448dbb99d135f82bfd261 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:39.457396 18724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b51cbc731794083a88fe4bed1ddeebf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:39.457599 20040 raft_consensus.cc:686] T c9e7660eccc448dbb99d135f82bfd261 P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Becoming Leader. State: Replica: 5e1c6397f09747f091534faf93542710, State: Running, Role: LEADER
I20240430 20:11:39.457929 20040 consensus_queue.cc:237] T c9e7660eccc448dbb99d135f82bfd261 P 5e1c6397f09747f091534faf93542710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:39.458171 18734 tablet_service.cc:1467] Processing CreateTablet for tablet 62732af95c6a4c749c5b726f0460dc72 (DEFAULT_TABLE table=default.Table_be2ebe77f859489f9cd3e912cb1edc8b [id=2d44ec94131148048021b2ee6366ba30]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:39.458622 18734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62732af95c6a4c749c5b726f0460dc72. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:39.459427 19049 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 53a899fa5a424a0d99f1b47925e8c7ad (table default.Table_ffb3192f443d437ab63f56f628e596ad [id=608890834da843fa9af0cfc0d548029e]). Expected version 2 got 0
I20240430 20:11:39.459584 20060 raft_consensus.cc:2726] T 01436e5b6232494babbb414342f03090 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240430 20:11:39.459897 19049 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 6e754af64cb64fd2a082adae7d5f24da (table default.Table_208b12b8e9bd4f3e8bc9c6d422ada649 [id=1df4e95702cd46f59ab21b101049a9ed]). Expected version 1 got 0
I20240430 20:11:39.460556 19049 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet b22aaedb0a6c4fd9ad0ea8fa019001e1 (table default.Table_730a215d7af74a5bb753adca39dea676 [id=a5d580cffa2447d78fc9f2c8e31264f1]). Expected version 5 got 0
I20240430 20:11:39.461145 19049 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 2da900b6bbfa4fa99fe8494fd2e68da8 (table default.Table_19a91a7f17ff456ebb145e8dd89c0940 [id=67283dd3995a44c6a7a7bc8722e4f955]). Expected version 1 got 0
I20240430 20:11:39.465590 18725 tablet_service.cc:1467] Processing CreateTablet for tablet 8901327b3efb4201adce61a9266142ea (DEFAULT_TABLE table=default.Table_2582fc0f97c1470d8b6b21cd1240697e [id=fc059935f9c943389180fab316c33a05]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:39.466924 18725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8901327b3efb4201adce61a9266142ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:39.468351 18758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bde1d6934784c9aa5911181e8df9eeb" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:39.471125 18758 raft_consensus.cc:1226] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 3c51c33d5616482cafb68e26688820f4 [term 2 FOLLOWER]: Rejecting Update request from peer 5e1c6397f09747f091534faf93542710 for earlier term 1. Current term is 2. Ops: []
I20240430 20:11:39.475674 19489 ts_tablet_manager.cc:1888] T 0fe1247349344863a6273a769ce8ddf6 P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:39.478484 19489 log.cc:1196] T 0fe1247349344863a6273a769ce8ddf6 P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/0fe1247349344863a6273a769ce8ddf6
I20240430 20:11:39.479089 18985 ts_tablet_manager.cc:1367] T 6e754af64cb64fd2a082adae7d5f24da P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.000s
I20240430 20:11:39.482040 18454 tablet_service.cc:1467] Processing CreateTablet for tablet 6b51cbc731794083a88fe4bed1ddeebf (DEFAULT_TABLE table=default.Table_17cb787c6a4c4c9fa720ce8c4e326605 [id=8e0f5d8c941d423ea7b30a89513d5170]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:39.482221 18474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b72fe1f0713b4076a81a3fc1c74f7fa7" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:39.482286 18450 tablet_service.cc:1467] Processing CreateTablet for tablet 828ca913e2aa4102a6277bdbc6686ef5 (DEFAULT_TABLE table=default.Table_5d51defcfdfd4d7f8e7d341c5daf0cdf [id=bdab45e393374761bea47d965b81f043]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:39.482888 18478 raft_consensus.cc:1226] T 75ebf051383e42529dcb5f7dd0770a4b P 2efb542aa55e439b99ccb98d1c5c884a [term 2 LEADER]: Rejecting Update request from peer 5e1c6397f09747f091534faf93542710 for earlier term 1. Current term is 2. Ops: []
I20240430 20:11:39.487193 19489 ts_tablet_manager.cc:1909] T 0fe1247349344863a6273a769ce8ddf6 P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:39.492826 18754 raft_consensus.cc:1226] T 5bde72e2791a497db7dd08686ca36d80 P 3c51c33d5616482cafb68e26688820f4 [term 2 FOLLOWER]: Rejecting Update request from peer 5e1c6397f09747f091534faf93542710 for earlier term 1. Current term is 2. Ops: []
I20240430 20:11:39.493233 18750 raft_consensus.cc:1226] T 75ebf051383e42529dcb5f7dd0770a4b P 3c51c33d5616482cafb68e26688820f4 [term 2 FOLLOWER]: Rejecting Update request from peer 5e1c6397f09747f091534faf93542710 for earlier term 1. Current term is 2. Ops: []
I20240430 20:11:39.493407 18763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "724e51ff05ec4baebc2bb55e05c8eddc" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:39.493697 18764 raft_consensus.cc:1226] T 5809e7c9c2204ba293426c24129fb16e P 3c51c33d5616482cafb68e26688820f4 [term 2 LEADER]: Rejecting Update request from peer 5e1c6397f09747f091534faf93542710 for earlier term 1. Current term is 2. Ops: []
I20240430 20:11:39.493817 18752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a1e50f99c5d4b44bdbb51b58a92bc75" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:39.493999 18762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b72fe1f0713b4076a81a3fc1c74f7fa7" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:39.494189 18756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2d90ff17e574f88b2061b1c117d4b7d" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:39.494486 18754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e826459c3f4437da81c9e6e09f0c22f" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:39.494694 18754 raft_consensus.cc:3037] T 6e826459c3f4437da81c9e6e09f0c22f P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Advancing to term 2
I20240430 20:11:39.494832 18758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77285afcafe04a829e485998bbbe00f0" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:39.484114 20105 consensus_queue.cc:1046] T b72fe1f0713b4076a81a3fc1c74f7fa7 P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.059s
I20240430 20:11:39.495155 20105 raft_consensus.cc:3032] T b72fe1f0713b4076a81a3fc1c74f7fa7 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Stepping down as leader of term 1
I20240430 20:11:39.495661 18454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b51cbc731794083a88fe4bed1ddeebf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:39.496156 18446 tablet_service.cc:1467] Processing CreateTablet for tablet 8901327b3efb4201adce61a9266142ea (DEFAULT_TABLE table=default.Table_2582fc0f97c1470d8b6b21cd1240697e [id=fc059935f9c943389180fab316c33a05]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:39.496796 18592 tablet_service.cc:1467] Processing CreateTablet for tablet 62732af95c6a4c749c5b726f0460dc72 (DEFAULT_TABLE table=default.Table_be2ebe77f859489f9cd3e912cb1edc8b [id=2d44ec94131148048021b2ee6366ba30]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:39.497166 20047 leader_election.cc:288] T b72fe1f0713b4076a81a3fc1c74f7fa7 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 2 election: Requested vote from peers 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:39.497551 18449 tablet_service.cc:1467] Processing CreateTablet for tablet 62732af95c6a4c749c5b726f0460dc72 (DEFAULT_TABLE table=default.Table_be2ebe77f859489f9cd3e912cb1edc8b [id=2d44ec94131148048021b2ee6366ba30]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:39.497802 20105 raft_consensus.cc:729] T b72fe1f0713b4076a81a3fc1c74f7fa7 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Running, Role: LEADER
I20240430 20:11:39.497867 18754 raft_consensus.cc:2454] T 6e826459c3f4437da81c9e6e09f0c22f P 3c51c33d5616482cafb68e26688820f4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 2.
I20240430 20:11:39.498147 20105 consensus_queue.cc:260] T b72fe1f0713b4076a81a3fc1c74f7fa7 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:39.498441 18450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 828ca913e2aa4102a6277bdbc6686ef5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:39.498579 20105 raft_consensus.cc:3037] T b72fe1f0713b4076a81a3fc1c74f7fa7 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Advancing to term 2
I20240430 20:11:39.499034 19049 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 34130f2820064fa0898ef1f9d7de1c6d (table default.Table_343ee22ed6554f328aa58881079e1905 [id=5a1c4a7be2fc465fac89a50854eb3322]). Expected version 2 got 0
I20240430 20:11:39.499238 19049 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 99203cc522de424184c6f97453f166fd (table default.Table_cb2a4b9ceaf4431facfff2c9bcf94c5d [id=3f8690a437ae49a3beb009fa7899d5d0]). Expected version 1 got 0
I20240430 20:11:39.499632 19049 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 78ba48dadef548708153878810cc46f8 (table default.Table_b9983e101a6547649eb44ee04b5e1147 [id=29769be2b9ec4ecba9de5c0a65692f85]). Expected version 1 got 0
I20240430 20:11:39.495977 18760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aec9c09cc0454acab6006bb3e10982b7" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:39.500187 18592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62732af95c6a4c749c5b726f0460dc72. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:39.500063 20096 consensus_queue.cc:1046] T 75ebf051383e42529dcb5f7dd0770a4b P 5e1c6397f09747f091534faf93542710 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.009s
I20240430 20:11:39.500314 20096 consensus_queue.cc:1225] T 75ebf051383e42529dcb5f7dd0770a4b P 5e1c6397f09747f091534faf93542710 [LEADER]: Peer 2efb542aa55e439b99ccb98d1c5c884a log is divergent from this leader: its last log entry 2.1 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 0
I20240430 20:11:39.500876 20096 raft_consensus.cc:3032] T 75ebf051383e42529dcb5f7dd0770a4b P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Stepping down as leader of term 1
I20240430 20:11:39.500973 20096 raft_consensus.cc:729] T 75ebf051383e42529dcb5f7dd0770a4b P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Running, Role: LEADER
I20240430 20:11:39.500631 18600 tablet_service.cc:1467] Processing CreateTablet for tablet 6b51cbc731794083a88fe4bed1ddeebf (DEFAULT_TABLE table=default.Table_17cb787c6a4c4c9fa720ce8c4e326605 [id=8e0f5d8c941d423ea7b30a89513d5170]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:39.502563 20096 consensus_queue.cc:260] T 75ebf051383e42529dcb5f7dd0770a4b P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:39.502987 20096 raft_consensus.cc:3037] T 75ebf051383e42529dcb5f7dd0770a4b P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Advancing to term 2
I20240430 20:11:39.503134 18758 raft_consensus.cc:3037] T 26ed12f498644d15bf45e884dcad172e P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:39.503607 18600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b51cbc731794083a88fe4bed1ddeebf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:39.504213 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 67) took 8232 ms. Trace:
I20240430 20:11:39.505283 19489 rpcz_store.cc:276] 0430 20:11:31.271604 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:31.271691 (+ 87us) service_pool.cc:225] Handling call
0430 20:11:39.391879 (+8120188us) ts_tablet_manager.cc:1120] Deleting tablet 0fe1247349344863a6273a769ce8ddf6
0430 20:11:39.391888 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:39.432264 (+ 40376us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:39.475462 (+ 43198us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:39.503589 (+ 28127us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":8117598}
I20240430 20:11:39.508208 18608 raft_consensus.cc:1226] T e2d90ff17e574f88b2061b1c117d4b7d P 5e1c6397f09747f091534faf93542710 [term 2 FOLLOWER]: Rejecting Update request from peer 2efb542aa55e439b99ccb98d1c5c884a for earlier term 1. Current term is 2. Ops: []
I20240430 20:11:39.508869 18587 tablet_service.cc:1467] Processing CreateTablet for tablet 8901327b3efb4201adce61a9266142ea (DEFAULT_TABLE table=default.Table_2582fc0f97c1470d8b6b21cd1240697e [id=fc059935f9c943389180fab316c33a05]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:39.511446 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 68) took 8237 ms. Trace:
I20240430 20:11:39.513206 18988 tablet_bootstrap.cc:492] T 20d166756819406ba85044f4fee9c4ce P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:39.514256 18446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8901327b3efb4201adce61a9266142ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:39.515712 18449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62732af95c6a4c749c5b726f0460dc72. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:39.513366 19489 rpcz_store.cc:276] 0430 20:11:31.273639 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:31.273698 (+ 59us) service_pool.cc:225] Handling call
0430 20:11:39.508416 (+8234718us) ts_tablet_manager.cc:1120] Deleting tablet 2a1e50f99c5d4b44bdbb51b58a92bc75
0430 20:11:39.508620 (+ 204us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:39.510000 (+ 1380us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":8231225}
I20240430 20:11:39.515676 18988 ts_tablet_manager.cc:1367] T 20d166756819406ba85044f4fee9c4ce P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.140s user 0.004s sys 0.011s
I20240430 20:11:39.519951 20103 raft_consensus.cc:482] T 01436e5b6232494babbb414342f03090 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:39.507586 19001 catalog_manager.cc:4830] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): tablet 0fe1247349344863a6273a769ce8ddf6 (table default.Table_7bfb47ba65114665943b7edccabd5a8b [id=c487c255c03741199e57b71670bc84e0]) successfully deleted
W20240430 20:11:39.520377 19001 catalog_manager.cc:4810] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): delete failed for tablet 2a1e50f99c5d4b44bdbb51b58a92bc75 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2a1e50f99c5d4b44bdbb51b58a92bc75 already in progress: creating tablet
I20240430 20:11:39.521389 20103 raft_consensus.cc:504] T 01436e5b6232494babbb414342f03090 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:39.522182 19049 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet d61c279d20b0471a967942525ceb4fcb (table default.Table_1177828cad1f4a24bd397619dad96170 [id=a95fed1bc3fa4865bf9a35dfc51b659c]). Expected version 2 got 0
I20240430 20:11:39.522400 18557 leader_election.cc:302] T 7bde1d6934784c9aa5911181e8df9eeb P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:39.524796 19049 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 3ec3993042f5498b818bdf32d5087928 (table default.Table_ba82d0fed0264b818de8278e1d142e5d [id=27217d463b9c40ec926a1862a31f1f24]). Expected version 1 got 0
I20240430 20:11:39.525275 19049 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet c6e3d09402574f1e8fe9e4160c70a3ee (table default.Table_656db97355f64d7c8aa29b1352dfe9c8 [id=450f5d1c7e084a1bb88d035c0ccfcf82]). Expected version 1 got 0
I20240430 20:11:39.525664 18754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01436e5b6232494babbb414342f03090" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:39.526379 19049 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 46251c08277e498da04a41d009310c62 (table default.Table_4e56923d440944978f0aacc140a0dd8e [id=b1312a6cd1a54003b1db727a2a9d125d]). Expected version 1 got 0
I20240430 20:11:39.526718 19049 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 32c823d1700949ab8b22adcd83b7bfd1 (table default.Table_e8c73c89623d4b49b8d7f0a01d729bdd [id=5cd10682ce52414cb5a103e4433a6a0d]). Expected version 1 got 0
I20240430 20:11:39.527256 19049 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet acf3b6fead41483a82c43dc0286fab7c (table default.Table_01450a1e661d450a8cbe185e16e8250c [id=ce9911e0a91949b1997ee47321800255]). Expected version 1 got 0
I20240430 20:11:39.526773 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 70) took 8242 ms. Trace:
I20240430 20:11:39.527978 19049 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet fbdeb4714b8c4b93ab72822c43329aae (table default.Table_c12ca6938d424fbfa2e1df5cdb00fdbc [id=0fb93f36bc17408ea56bc4cafcfd8580]). Expected version 1 got 0
I20240430 20:11:39.529103 19049 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 76d6e9d547ca4a848eae852944b1d117 (table default.Table_104ae015924a48fd8d27220792f07470 [id=ff86c0357c164dd0991098c416087773]). Expected version 2 got 0
I20240430 20:11:39.527814 19489 rpcz_store.cc:276] 0430 20:11:31.276000 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:31.276061 (+ 61us) service_pool.cc:225] Handling call
0430 20:11:39.517132 (+8241071us) ts_tablet_manager.cc:1120] Deleting tablet e6eac92dfa5f4456bc49571878f0f8dd
0430 20:11:39.517143 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:39.518413 (+ 1270us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":8230737}
I20240430 20:11:39.530805 20103 leader_election.cc:288] T 01436e5b6232494babbb414342f03090 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
W20240430 20:11:39.530966 19001 catalog_manager.cc:4810] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): delete failed for tablet e6eac92dfa5f4456bc49571878f0f8dd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e6eac92dfa5f4456bc49571878f0f8dd already in progress: creating tablet
I20240430 20:11:39.532171 19049 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet fd40967ab4a04001819e16c06a6c4300 (table default.Table_4130caf534004edfb413056d7f6cfd52 [id=da9ad991953a40c4b77b31317597ad1d]). Expected version 4 got 0
I20240430 20:11:39.523733 18985 raft_consensus.cc:348] T 6e754af64cb64fd2a082adae7d5f24da P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:39.534943 18608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01436e5b6232494babbb414342f03090" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:39.535305 18985 raft_consensus.cc:374] T 6e754af64cb64fd2a082adae7d5f24da P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:39.535382 18587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8901327b3efb4201adce61a9266142ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:39.535421 18985 raft_consensus.cc:729] T 6e754af64cb64fd2a082adae7d5f24da P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:39.535738 19049 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet a59ef6c545dd41a28265a4c1b8ce10e0 (table default.Table_07e40af8f1c546048975c8edf88170d7 [id=86df5d161d8641c89070e7eed9a39360]). Expected version 2 got 0
I20240430 20:11:39.537838 19049 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet c9e7660eccc448dbb99d135f82bfd261 (table default.Table_685a26737bb046fb85ced15df7ae3423 [id=51d177e8498848b38c43e3f8d11f092c]). Expected version 1 got 0
I20240430 20:11:39.538156 18415 leader_election.cc:302] T 01436e5b6232494babbb414342f03090 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a; no voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710
I20240430 20:11:39.538292 19049 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet a05e32a33aa7488eb287cc6314c09c44 (table default.Table_2d1cfc78363a47aa8dba7ef7d80f8c9a [id=955d833ff8f14bf6957abadf9ef669e2]). Expected version 2 got 0
I20240430 20:11:39.538697 20025 raft_consensus.cc:2726] T 01436e5b6232494babbb414342f03090 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:39.520829 20105 consensus_queue.cc:1046] T e2d90ff17e574f88b2061b1c117d4b7d P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: INVALID_TERM, Last received: 1.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.014s
I20240430 20:11:39.540745 18985 consensus_queue.cc:260] T 6e754af64cb64fd2a082adae7d5f24da P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:39.541057 19033 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: Sending DeleteTablet for 3 replicas of tablet de1dfe6c82d34a4d8d02e466c55327ff
I20240430 20:11:39.541361 18985 ts_tablet_manager.cc:1398] T 6e754af64cb64fd2a082adae7d5f24da P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.062s user 0.002s sys 0.000s
I20240430 20:11:39.541664 18985 tablet_bootstrap.cc:492] T acf3b6fead41483a82c43dc0286fab7c P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:39.541563 19033 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:51664 (request call id 185) took 1128 ms. Trace:
I20240430 20:11:39.542804 18734 tablet_service.cc:1514] Processing DeleteTablet for tablet de1dfe6c82d34a4d8d02e466c55327ff with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:39 UTC) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:39.542773 19033 rpcz_store.cc:276] 0430 20:11:38.413264 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:39.147072 (+733808us) service_pool.cc:225] Handling call
0430 20:11:39.147700 (+ 628us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0430 20:11:39.147877 (+ 177us) catalog_manager.cc:2749] Modifying in-memory table state
0430 20:11:39.147957 (+ 80us) catalog_manager.cc:2758] Locking tablets
0430 20:11:39.308871 (+160914us) catalog_manager.cc:2772] Removing table and tablets from system table
0430 20:11:39.320580 (+ 11709us) catalog_manager.cc:2790] Removing table from by-name map
0430 20:11:39.320680 (+ 100us) catalog_manager.cc:2806] Committing in-memory state
0430 20:11:39.540987 (+220307us) catalog_manager.cc:2810] Aborting table tasks
0430 20:11:39.541532 (+ 545us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:39.310133 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:39.310280 (+ 147us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:39.310290 (+ 10us) write_op.cc:427] Acquired schema lock
0430 20:11:39.310293 (+ 3us) tablet.cc:660] Decoding operations
0430 20:11:39.310497 (+ 204us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:39.310512 (+ 15us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:39.310521 (+ 9us) tablet.cc:683] Acquiring locks for 2 operations
0430 20:11:39.310683 (+ 162us) tablet.cc:699] Row locks acquired
0430 20:11:39.310687 (+ 4us) write_op.cc:260] PREPARE: finished
0430 20:11:39.310736 (+ 49us) write_op.cc:270] Start()
0430 20:11:39.310780 (+ 44us) write_op.cc:276] Timestamp: P: 1714507899310703 usec, L: 0
0430 20:11:39.310786 (+ 6us) op_driver.cc:331] REPLICATION: starting
0430 20:11:39.311024 (+ 238us) log.cc:842] Serialized 759 byte log entry
0430 20:11:39.316691 (+ 5667us) op_driver.cc:444] REPLICATION: finished
0430 20:11:39.318450 (+ 1759us) write_op.cc:301] APPLY: starting
0430 20:11:39.318467 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:39.318511 (+ 44us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:39.318539 (+ 28us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:39.318630 (+ 91us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:39.318670 (+ 40us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0430 20:11:39.318677 (+ 7us) write_op.cc:312] APPLY: finished
0430 20:11:39.318751 (+ 74us) log.cc:842] Serialized 29 byte log entry
0430 20:11:39.318817 (+ 66us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:39.318843 (+ 26us) write_op.cc:446] Released schema lock
0430 20:11:39.318851 (+ 8us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:39.318925 (+ 74us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:39.318929 (+ 4us) write_op.cc:446] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1692,"apply.run_cpu_time_us":538,"apply.run_wall_time_us":538,"num_ops":2,"prepare.queue_time_us":201,"prepare.run_cpu_time_us":1016,"prepare.run_wall_time_us":1002,"raft.queue_time_us":345,"raft.run_cpu_time_us":257,"raft.run_wall_time_us":257,"replication_time_us":5876}]]}
I20240430 20:11:39.543859 19489 tablet_replica.cc:330] T 5bde72e2791a497db7dd08686ca36d80 P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:39.547329 20105 raft_consensus.cc:3032] T e2d90ff17e574f88b2061b1c117d4b7d P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Stepping down as leader of term 1
I20240430 20:11:39.547540 20105 raft_consensus.cc:729] T e2d90ff17e574f88b2061b1c117d4b7d P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Running, Role: LEADER
I20240430 20:11:39.545967 19489 raft_consensus.cc:2229] T 5bde72e2791a497db7dd08686ca36d80 P 3c51c33d5616482cafb68e26688820f4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:39.548964 18447 tablet_service.cc:1514] Processing DeleteTablet for tablet de1dfe6c82d34a4d8d02e466c55327ff with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:39 UTC) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:39.549002 20105 consensus_queue.cc:260] T e2d90ff17e574f88b2061b1c117d4b7d P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:39.549510 20105 raft_consensus.cc:3037] T e2d90ff17e574f88b2061b1c117d4b7d P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Advancing to term 2
I20240430 20:11:39.553682 18810 heartbeater.cc:503] Master 127.17.177.60:37225 was elected leader, sending a full tablet report...
I20240430 20:11:39.548445 18988 raft_consensus.cc:348] T 20d166756819406ba85044f4fee9c4ce P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:39.553902 19489 raft_consensus.cc:2258] T 5bde72e2791a497db7dd08686ca36d80 P 3c51c33d5616482cafb68e26688820f4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:39.556720 18985 tablet_bootstrap.cc:654] T acf3b6fead41483a82c43dc0286fab7c P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
W20240430 20:11:39.561383 19028 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 34130f2820064fa0898ef1f9d7de1c6d consensus state has no leader
I20240430 20:11:39.561610 19028 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:51664 (request call id 182) took 1148 ms. Trace:
I20240430 20:11:39.562041 20047 raft_consensus.cc:2726] T 7bde1d6934784c9aa5911181e8df9eeb P 5e1c6397f09747f091534faf93542710 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240430 20:11:39.561699 19028 rpcz_store.cc:276] 0430 20:11:38.412994 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:39.117690 (+704696us) service_pool.cc:225] Handling call
0430 20:11:39.132112 (+ 14422us) spinlock_profiling.cc:243] Waited 12.8 ms on lock 0x616000040654. stack: 00007feb16c74756 00007feb16c74510 00007feb15ca748d 0000000000955804 00000000009557cd 00007feb257b315d 00007feb257b3253 00007feb224e5d41 00007feb2a0d134d 00007feb2a25ff46 00007feb21049c26 00007feb21049955 00007feb1d97c04e 00007feb1d97af2e 00007feb1d981405 00007feb1d981d2f
0430 20:11:39.136301 (+ 4189us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0430 20:11:39.136558 (+ 257us) catalog_manager.cc:3613] Apply alter schema
0430 20:11:39.136728 (+ 170us) catalog_manager.cc:3636] Acquired catalog manager lock
0430 20:11:39.136793 (+ 65us) catalog_manager.cc:3783] Updating metadata on disk
0430 20:11:39.196846 (+ 60053us) catalog_manager.cc:3814] Committing alterations to in-memory state
0430 20:11:39.561550 (+364704us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:39.140451 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:39.140551 (+ 100us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:39.140555 (+ 4us) write_op.cc:427] Acquired schema lock
0430 20:11:39.140557 (+ 2us) tablet.cc:660] Decoding operations
0430 20:11:39.140614 (+ 57us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:39.140622 (+ 8us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:39.140629 (+ 7us) tablet.cc:683] Acquiring locks for 1 operations
0430 20:11:39.140668 (+ 39us) tablet.cc:699] Row locks acquired
0430 20:11:39.140670 (+ 2us) write_op.cc:260] PREPARE: finished
0430 20:11:39.140692 (+ 22us) write_op.cc:270] Start()
0430 20:11:39.140715 (+ 23us) write_op.cc:276] Timestamp: P: 1714507899140636 usec, L: 0
0430 20:11:39.140720 (+ 5us) op_driver.cc:331] REPLICATION: starting
0430 20:11:39.140869 (+ 149us) log.cc:842] Serialized 405 byte log entry
0430 20:11:39.153842 (+ 12973us) op_driver.cc:444] REPLICATION: finished
0430 20:11:39.154384 (+ 542us) write_op.cc:301] APPLY: starting
0430 20:11:39.154393 (+ 9us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:39.154420 (+ 27us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:39.154440 (+ 20us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:39.154483 (+ 43us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:39.154510 (+ 27us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0430 20:11:39.154517 (+ 7us) write_op.cc:312] APPLY: finished
0430 20:11:39.154574 (+ 57us) log.cc:842] Serialized 23 byte log entry
0430 20:11:39.154674 (+ 100us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:39.154688 (+ 14us) write_op.cc:446] Released schema lock
0430 20:11:39.154693 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:39.154734 (+ 41us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:39.154738 (+ 4us) write_op.cc:446] Released schema lock
Metrics: {"spinlock_wait_cycles":29410048,"child_traces":[["op",{"apply.queue_time_us":515,"apply.run_cpu_time_us":391,"apply.run_wall_time_us":390,"num_ops":1,"prepare.queue_time_us":3139,"prepare.run_cpu_time_us":470,"prepare.run_wall_time_us":469,"replication_time_us":13107}]]}
I20240430 20:11:39.562633 19028 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_1840b0e89a9840fe92bd1ca13c915806"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240430 20:11:39.562826 20060 consensus_queue.cc:1046] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 5e1c6397f09747f091534faf93542710 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.031s
W20240430 20:11:39.563237 19028 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1840b0e89a9840fe92bd1ca13c915806 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:39.563232 19861 consensus_queue.cc:1046] T 5bde72e2791a497db7dd08686ca36d80 P 5e1c6397f09747f091534faf93542710 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.031s
W20240430 20:11:39.567377 19024 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ae20f44df50343289b0a512a4ea53692 consensus state has no leader
I20240430 20:11:39.567595 19024 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:51664 (request call id 183) took 1154 ms. Trace:
I20240430 20:11:39.567683 19024 rpcz_store.cc:276] 0430 20:11:38.413107 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:39.125983 (+712876us) service_pool.cc:225] Handling call
0430 20:11:39.126467 (+ 484us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0430 20:11:39.126786 (+ 319us) catalog_manager.cc:3613] Apply alter schema
0430 20:11:39.127004 (+ 218us) catalog_manager.cc:3636] Acquired catalog manager lock
0430 20:11:39.127077 (+ 73us) catalog_manager.cc:3783] Updating metadata on disk
0430 20:11:39.148023 (+ 20946us) catalog_manager.cc:3814] Committing alterations to in-memory state
0430 20:11:39.567532 (+419509us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:39.139402 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:39.139494 (+ 92us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:39.139501 (+ 7us) write_op.cc:427] Acquired schema lock
0430 20:11:39.139503 (+ 2us) tablet.cc:660] Decoding operations
0430 20:11:39.139581 (+ 78us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:39.139588 (+ 7us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:39.139592 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0430 20:11:39.139641 (+ 49us) tablet.cc:699] Row locks acquired
0430 20:11:39.139645 (+ 4us) write_op.cc:260] PREPARE: finished
0430 20:11:39.139677 (+ 32us) write_op.cc:270] Start()
0430 20:11:39.139707 (+ 30us) write_op.cc:276] Timestamp: P: 1714507899139617 usec, L: 0
0430 20:11:39.139713 (+ 6us) op_driver.cc:331] REPLICATION: starting
0430 20:11:39.139898 (+ 185us) log.cc:842] Serialized 405 byte log entry
0430 20:11:39.145483 (+ 5585us) op_driver.cc:444] REPLICATION: finished
0430 20:11:39.145563 (+ 80us) write_op.cc:301] APPLY: starting
0430 20:11:39.145571 (+ 8us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:39.145599 (+ 28us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:39.145662 (+ 63us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:39.145699 (+ 37us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:39.145733 (+ 34us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0430 20:11:39.145741 (+ 8us) write_op.cc:312] APPLY: finished
0430 20:11:39.145807 (+ 66us) log.cc:842] Serialized 23 byte log entry
0430 20:11:39.146009 (+ 202us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:39.146032 (+ 23us) write_op.cc:446] Released schema lock
0430 20:11:39.146036 (+ 4us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:39.146081 (+ 45us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:39.146084 (+ 3us) write_op.cc:446] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":50,"apply.run_cpu_time_us":417,"apply.run_wall_time_us":560,"mutex_wait_us":153,"num_ops":1,"prepare.queue_time_us":11871,"prepare.run_cpu_time_us":578,"prepare.run_wall_time_us":578,"raft.queue_time_us":554,"raft.run_cpu_time_us":3,"raft.run_wall_time_us":3,"replication_time_us":5754}]]}
I20240430 20:11:39.568377 19024 master_service.cc:946] ReplaceTablet: received request to replace tablet f5236574672b477a9026e91adcaf70ec from {username='slave'} at 127.0.0.1:51664
I20240430 20:11:39.568759 19024 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:51664 (request call id 187) took 1105 ms. Trace:
I20240430 20:11:39.568866 19024 rpcz_store.cc:276] 0430 20:11:38.463337 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:39.568295 (+1104958us) service_pool.cc:225] Handling call
0430 20:11:39.568724 (+ 429us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240430 20:11:39.569196 19024 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_97a0991bf9284215b482c532e9dfc113" } modify_external_catalogs: true
I20240430 20:11:39.568338 18988 raft_consensus.cc:374] T 20d166756819406ba85044f4fee9c4ce P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:39.569394 19024 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_97a0991bf9284215b482c532e9dfc113" } modify_external_catalogs: true
I20240430 20:11:39.569985 18988 raft_consensus.cc:729] T 20d166756819406ba85044f4fee9c4ce P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:39.573635 20060 raft_consensus.cc:3032] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Stepping down as leader of term 1
I20240430 20:11:39.573758 20060 raft_consensus.cc:729] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Running, Role: LEADER
I20240430 20:11:39.574023 20060 consensus_queue.cc:260] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:39.574375 20060 raft_consensus.cc:3037] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Advancing to term 2
I20240430 20:11:39.573711 19028 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:51664 (request call id 186) took 1160 ms. Trace:
I20240430 20:11:39.575999 19028 rpcz_store.cc:276] 0430 20:11:38.413354 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:39.562368 (+1149014us) service_pool.cc:225] Handling call
0430 20:11:39.563340 (+ 972us) catalog_manager.cc:2072] Acquired catalog manager lock
0430 20:11:39.563829 (+ 489us) catalog_manager.cc:2118] Created new table and tablet info
0430 20:11:39.573399 (+ 9570us) catalog_manager.cc:2178] Wrote table and tablets to system table
0430 20:11:39.573565 (+ 166us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0430 20:11:39.573659 (+ 94us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:39.564961 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:39.565098 (+ 137us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:39.565107 (+ 9us) write_op.cc:427] Acquired schema lock
0430 20:11:39.565111 (+ 4us) tablet.cc:660] Decoding operations
0430 20:11:39.565214 (+ 103us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:39.565231 (+ 17us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:39.565235 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
0430 20:11:39.565322 (+ 87us) tablet.cc:699] Row locks acquired
0430 20:11:39.565324 (+ 2us) write_op.cc:260] PREPARE: finished
0430 20:11:39.565366 (+ 42us) write_op.cc:270] Start()
0430 20:11:39.565401 (+ 35us) write_op.cc:276] Timestamp: P: 1714507899565311 usec, L: 0
0430 20:11:39.565404 (+ 3us) op_driver.cc:331] REPLICATION: starting
0430 20:11:39.565625 (+ 221us) log.cc:842] Serialized 433 byte log entry
0430 20:11:39.571256 (+ 5631us) op_driver.cc:444] REPLICATION: finished
0430 20:11:39.571361 (+ 105us) write_op.cc:301] APPLY: starting
0430 20:11:39.571377 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:39.571433 (+ 56us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:39.571460 (+ 27us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:39.571576 (+ 116us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:39.571615 (+ 39us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0430 20:11:39.571623 (+ 8us) write_op.cc:312] APPLY: finished
0430 20:11:39.571703 (+ 80us) log.cc:842] Serialized 29 byte log entry
0430 20:11:39.571770 (+ 67us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:39.571799 (+ 29us) write_op.cc:446] Released schema lock
0430 20:11:39.571806 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:39.571861 (+ 55us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:39.571865 (+ 4us) write_op.cc:446] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":36,"apply.run_cpu_time_us":552,"apply.run_wall_time_us":567,"num_ops":2,"prepare.queue_time_us":612,"prepare.run_cpu_time_us":777,"prepare.run_wall_time_us":778,"raft.queue_time_us":43,"raft.run_cpu_time_us":279,"raft.run_wall_time_us":278,"replication_time_us":5826}]]}
I20240430 20:11:39.574865 18988 consensus_queue.cc:260] T 20d166756819406ba85044f4fee9c4ce P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:39.576668 19489 ts_tablet_manager.cc:1875] T 5bde72e2791a497db7dd08686ca36d80 P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:39.579372 18557 leader_election.cc:302] T 724e51ff05ec4baebc2bb55e05c8eddc P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:39.580310 19028 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_423319f67e734a64a731bf33ef868beb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:39.581179 19028 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_423319f67e734a64a731bf33ef868beb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:39.583985 18988 ts_tablet_manager.cc:1398] T 20d166756819406ba85044f4fee9c4ce P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.067s user 0.001s sys 0.003s
I20240430 20:11:39.584591 18593 tablet_service.cc:1514] Processing DeleteTablet for tablet de1dfe6c82d34a4d8d02e466c55327ff with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:39 UTC) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:39.586498 18608 raft_consensus.cc:1226] T 724e51ff05ec4baebc2bb55e05c8eddc P 5e1c6397f09747f091534faf93542710 [term 2 FOLLOWER]: Rejecting Update request from peer 2efb542aa55e439b99ccb98d1c5c884a for earlier term 1. Current term is 2. Ops: []
I20240430 20:11:39.587162 18988 tablet_bootstrap.cc:492] T b688e03ea71940c69bda5520b62c1fc3 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:39.588961 19024 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: Sending DeleteTablet for 3 replicas of tablet 90a694bd5db649afa50f3c4e752939c1
I20240430 20:11:39.589814 18725 tablet_service.cc:1514] Processing DeleteTablet for tablet 90a694bd5db649afa50f3c4e752939c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:39 UTC) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:39.590070 19471 ts_tablet_manager.cc:1888] T 2a1e50f99c5d4b44bdbb51b58a92bc75 P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:39.590278 19024 master_service.cc:946] ReplaceTablet: received request to replace tablet 04b3dff86d5c48a09d03da58bd48f360 from {username='slave'} at 127.0.0.1:51664
I20240430 20:11:39.592069 20105 consensus_queue.cc:1046] T 724e51ff05ec4baebc2bb55e05c8eddc P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: INVALID_TERM, Last received: 1.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.005s
I20240430 20:11:39.592443 20105 raft_consensus.cc:3032] T 724e51ff05ec4baebc2bb55e05c8eddc P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Stepping down as leader of term 1
I20240430 20:11:39.592563 20105 raft_consensus.cc:729] T 724e51ff05ec4baebc2bb55e05c8eddc P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Running, Role: LEADER
I20240430 20:11:39.592823 20105 consensus_queue.cc:260] T 724e51ff05ec4baebc2bb55e05c8eddc P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:39.593233 20105 raft_consensus.cc:3037] T 724e51ff05ec4baebc2bb55e05c8eddc P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Advancing to term 2
I20240430 20:11:39.596589 19861 raft_consensus.cc:3032] T 5bde72e2791a497db7dd08686ca36d80 P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Stepping down as leader of term 1
I20240430 20:11:39.596457 18450 tablet_service.cc:1514] Processing DeleteTablet for tablet 90a694bd5db649afa50f3c4e752939c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:39 UTC) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:39.596694 20047 raft_consensus.cc:2726] T 724e51ff05ec4baebc2bb55e05c8eddc P 5e1c6397f09747f091534faf93542710 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240430 20:11:39.604671 19861 raft_consensus.cc:729] T 5bde72e2791a497db7dd08686ca36d80 P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Running, Role: LEADER
I20240430 20:11:39.604487 19495 ts_tablet_manager.cc:1888] T cace1398492f4d968dcb6899385d527c P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20240430 20:11:39.606542 18557 leader_election.cc:341] T 2a1e50f99c5d4b44bdbb51b58a92bc75 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:39.606029 19495 log.cc:1196] T cace1398492f4d968dcb6899385d527c P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/cace1398492f4d968dcb6899385d527c
I20240430 20:11:39.607991 20047 consensus_queue.cc:1035] T 53a899fa5a424a0d99f1b47925e8c7ad P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240430 20:11:39.610687 18557 leader_election.cc:302] T b72fe1f0713b4076a81a3fc1c74f7fa7 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:39.613164 19861 consensus_queue.cc:260] T 5bde72e2791a497db7dd08686ca36d80 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:39.613914 20040 consensus_queue.cc:1046] T 5809e7c9c2204ba293426c24129fb16e P 5e1c6397f09747f091534faf93542710 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.022s
I20240430 20:11:39.614097 19495 ts_tablet_manager.cc:1909] T cace1398492f4d968dcb6899385d527c P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:39.615002 20040 consensus_queue.cc:1225] T 5809e7c9c2204ba293426c24129fb16e P 5e1c6397f09747f091534faf93542710 [LEADER]: Peer 3c51c33d5616482cafb68e26688820f4 log is divergent from this leader: its last log entry 2.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20240430 20:11:39.615044 18588 tablet_service.cc:1514] Processing DeleteTablet for tablet 90a694bd5db649afa50f3c4e752939c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:39 UTC) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:39.616207 19861 raft_consensus.cc:3037] T 5bde72e2791a497db7dd08686ca36d80 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Advancing to term 2
I20240430 20:11:39.617233 18557 leader_election.cc:302] T e2d90ff17e574f88b2061b1c117d4b7d P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:39.616814 18988 tablet_bootstrap.cc:654] T b688e03ea71940c69bda5520b62c1fc3 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:39.622509 20061 raft_consensus.cc:2726] T e2d90ff17e574f88b2061b1c117d4b7d P 5e1c6397f09747f091534faf93542710 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240430 20:11:39.622754 18475 raft_consensus.cc:3037] T 01436e5b6232494babbb414342f03090 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:39.623658 20047 raft_consensus.cc:2726] T b72fe1f0713b4076a81a3fc1c74f7fa7 P 5e1c6397f09747f091534faf93542710 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240430 20:11:39.627976 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 60) took 8518 ms. Trace:
I20240430 20:11:39.628422 19495 rpcz_store.cc:276] 0430 20:11:31.109483 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:31.109529 (+ 46us) service_pool.cc:225] Handling call
0430 20:11:39.257564 (+8148035us) ts_tablet_manager.cc:1120] Deleting tablet cace1398492f4d968dcb6899385d527c
0430 20:11:39.257576 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:39.475365 (+217789us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:39.600420 (+125055us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:39.627870 (+ 27450us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":8136137}
I20240430 20:11:39.629472 19004 catalog_manager.cc:4830] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): tablet cace1398492f4d968dcb6899385d527c (table default.Table_a8d5ac73c6d047698d87b3d1ba34e66f [id=9c80193cde2346c982cf84ae9612e2f7]) successfully deleted
I20240430 20:11:39.629894 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 62) took 8519 ms. Trace:
I20240430 20:11:39.630231 19495 rpcz_store.cc:276] 0430 20:11:31.110216 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:31.110253 (+ 37us) service_pool.cc:225] Handling call
0430 20:11:39.629555 (+8519302us) ts_tablet_manager.cc:1120] Deleting tablet 0fe1247349344863a6273a769ce8ddf6
0430 20:11:39.629569 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:39.629803 (+ 234us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":8508079}
W20240430 20:11:39.631080 19004 catalog_manager.cc:4797] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): delete failed for tablet 0fe1247349344863a6273a769ce8ddf6 because the tablet was not found. No further retry: Not found: Tablet not found: 0fe1247349344863a6273a769ce8ddf6
I20240430 20:11:39.632146 18619 raft_consensus.cc:1226] T 77285afcafe04a829e485998bbbe00f0 P 5e1c6397f09747f091534faf93542710 [term 2 FOLLOWER]: Rejecting Update request from peer 2efb542aa55e439b99ccb98d1c5c884a for earlier term 1. Current term is 2. Ops: []
I20240430 20:11:39.633283 18557 leader_election.cc:302] T 77285afcafe04a829e485998bbbe00f0 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:39.635180 20105 consensus_queue.cc:1046] T 77285afcafe04a829e485998bbbe00f0 P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: INVALID_TERM, Last received: 1.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240430 20:11:39.635515 20105 raft_consensus.cc:3032] T 77285afcafe04a829e485998bbbe00f0 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Stepping down as leader of term 1
I20240430 20:11:39.635603 20105 raft_consensus.cc:729] T 77285afcafe04a829e485998bbbe00f0 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Running, Role: LEADER
I20240430 20:11:39.635898 20105 consensus_queue.cc:260] T 77285afcafe04a829e485998bbbe00f0 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:39.636248 20105 raft_consensus.cc:3037] T 77285afcafe04a829e485998bbbe00f0 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Advancing to term 2
W20240430 20:11:39.636509 19048 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a482f4622f774d60b4efd6827b8be7d7 consensus state has no leader
I20240430 20:11:39.636708 19048 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:51664 (request call id 184) took 1223 ms. Trace:
I20240430 20:11:39.636803 19048 rpcz_store.cc:276] 0430 20:11:38.413192 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:39.130090 (+716898us) service_pool.cc:225] Handling call
0430 20:11:39.130499 (+ 409us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0430 20:11:39.130789 (+ 290us) catalog_manager.cc:3613] Apply alter schema
0430 20:11:39.131026 (+ 237us) catalog_manager.cc:3636] Acquired catalog manager lock
0430 20:11:39.131122 (+ 96us) catalog_manager.cc:3783] Updating metadata on disk
0430 20:11:39.154839 (+ 23717us) catalog_manager.cc:3814] Committing alterations to in-memory state
0430 20:11:39.636646 (+481807us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:39.140005 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:39.140087 (+ 82us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:39.140093 (+ 6us) write_op.cc:427] Acquired schema lock
0430 20:11:39.140096 (+ 3us) tablet.cc:660] Decoding operations
0430 20:11:39.140164 (+ 68us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:39.140171 (+ 7us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:39.140175 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0430 20:11:39.140221 (+ 46us) tablet.cc:699] Row locks acquired
0430 20:11:39.140223 (+ 2us) write_op.cc:260] PREPARE: finished
0430 20:11:39.140247 (+ 24us) write_op.cc:270] Start()
0430 20:11:39.140267 (+ 20us) write_op.cc:276] Timestamp: P: 1714507899140191 usec, L: 0
0430 20:11:39.140270 (+ 3us) op_driver.cc:331] REPLICATION: starting
0430 20:11:39.140373 (+ 103us) log.cc:842] Serialized 405 byte log entry
0430 20:11:39.153779 (+ 13406us) op_driver.cc:444] REPLICATION: finished
0430 20:11:39.153921 (+ 142us) write_op.cc:301] APPLY: starting
0430 20:11:39.153936 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:39.153972 (+ 36us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:39.154007 (+ 35us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:39.154051 (+ 44us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:39.154081 (+ 30us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0430 20:11:39.154088 (+ 7us) write_op.cc:312] APPLY: finished
0430 20:11:39.154161 (+ 73us) log.cc:842] Serialized 23 byte log entry
0430 20:11:39.154225 (+ 64us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:39.154242 (+ 17us) write_op.cc:446] Released schema lock
0430 20:11:39.154248 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:39.154299 (+ 51us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:39.154305 (+ 6us) write_op.cc:446] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":95,"apply.run_cpu_time_us":433,"apply.run_wall_time_us":433,"num_ops":1,"prepare.queue_time_us":8395,"prepare.run_cpu_time_us":418,"prepare.run_wall_time_us":416,"replication_time_us":13483}]]}
I20240430 20:11:39.637459 19048 master_service.cc:946] ReplaceTablet: received request to replace tablet 53a899fa5a424a0d99f1b47925e8c7ad from {username='slave'} at 127.0.0.1:51664
I20240430 20:11:39.643448 19471 log.cc:1196] T 2a1e50f99c5d4b44bdbb51b58a92bc75 P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/2a1e50f99c5d4b44bdbb51b58a92bc75
I20240430 20:11:39.644443 19471 ts_tablet_manager.cc:1909] T 2a1e50f99c5d4b44bdbb51b58a92bc75 P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:39.644742 20047 raft_consensus.cc:2726] T 77285afcafe04a829e485998bbbe00f0 P 5e1c6397f09747f091534faf93542710 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240430 20:11:39.645071 18557 leader_election.cc:302] T 6e826459c3f4437da81c9e6e09f0c22f P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a
I20240430 20:11:39.645395 20047 raft_consensus.cc:2781] T 6e826459c3f4437da81c9e6e09f0c22f P 5e1c6397f09747f091534faf93542710 [term 2 FOLLOWER]: Leader election won for term 2
I20240430 20:11:39.645538 20047 raft_consensus.cc:686] T 6e826459c3f4437da81c9e6e09f0c22f P 5e1c6397f09747f091534faf93542710 [term 2 LEADER]: Becoming Leader. State: Replica: 5e1c6397f09747f091534faf93542710, State: Running, Role: LEADER
I20240430 20:11:39.645854 20047 consensus_queue.cc:237] T 6e826459c3f4437da81c9e6e09f0c22f P 5e1c6397f09747f091534faf93542710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:39.647504 19495 tablet_replica.cc:330] T 93d158de64994f3c9fa45a1250a4f533 P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:39.647770 18557 leader_election.cc:302] T aec9c09cc0454acab6006bb3e10982b7 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:39.648383 20061 raft_consensus.cc:2726] T aec9c09cc0454acab6006bb3e10982b7 P 5e1c6397f09747f091534faf93542710 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240430 20:11:39.648712 19495 raft_consensus.cc:2229] T 93d158de64994f3c9fa45a1250a4f533 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:39.649389 18608 raft_consensus.cc:1226] T aec9c09cc0454acab6006bb3e10982b7 P 5e1c6397f09747f091534faf93542710 [term 2 FOLLOWER]: Rejecting Update request from peer 2efb542aa55e439b99ccb98d1c5c884a for earlier term 1. Current term is 2. Ops: []
I20240430 20:11:39.649588 19495 raft_consensus.cc:2258] T 93d158de64994f3c9fa45a1250a4f533 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:39.650113 20025 consensus_queue.cc:1046] T aec9c09cc0454acab6006bb3e10982b7 P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: INVALID_TERM, Last received: 1.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240430 20:11:39.650424 20025 raft_consensus.cc:3032] T aec9c09cc0454acab6006bb3e10982b7 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Stepping down as leader of term 1
I20240430 20:11:39.650533 20025 raft_consensus.cc:729] T aec9c09cc0454acab6006bb3e10982b7 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Running, Role: LEADER
I20240430 20:11:39.650803 20025 consensus_queue.cc:260] T aec9c09cc0454acab6006bb3e10982b7 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:39.651630 20025 raft_consensus.cc:3037] T aec9c09cc0454acab6006bb3e10982b7 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Advancing to term 2
I20240430 20:11:39.652917 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 76) took 8420 ms. Trace:
I20240430 20:11:39.653028 19471 rpcz_store.cc:276] 0430 20:11:31.231563 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:31.231625 (+ 62us) service_pool.cc:225] Handling call
0430 20:11:39.385750 (+8154125us) ts_tablet_manager.cc:1120] Deleting tablet 2a1e50f99c5d4b44bdbb51b58a92bc75
0430 20:11:39.385763 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:39.463870 (+ 78107us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:39.589906 (+126036us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:39.652020 (+ 62114us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":31027,"tablet-delete.queue_time_us":8153724}
I20240430 20:11:39.653435 19471 tablet_replica.cc:330] T 5bde72e2791a497db7dd08686ca36d80 P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:39.653625 19471 raft_consensus.cc:2229] T 5bde72e2791a497db7dd08686ca36d80 P 5e1c6397f09747f091534faf93542710 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:39.653800 19471 raft_consensus.cc:2258] T 5bde72e2791a497db7dd08686ca36d80 P 5e1c6397f09747f091534faf93542710 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:39.654335 19471 ts_tablet_manager.cc:1875] T 5bde72e2791a497db7dd08686ca36d80 P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:39.658479 18999 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet 2a1e50f99c5d4b44bdbb51b58a92bc75 (table default.Table_c12ca6938d424fbfa2e1df5cdb00fdbc [id=0fb93f36bc17408ea56bc4cafcfd8580]) successfully deleted
I20240430 20:11:39.659171 20019 raft_consensus.cc:482] T 20d166756819406ba85044f4fee9c4ce P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:39.659307 20060 raft_consensus.cc:3032] T 5809e7c9c2204ba293426c24129fb16e P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Stepping down as leader of term 1
I20240430 20:11:39.659400 20060 raft_consensus.cc:729] T 5809e7c9c2204ba293426c24129fb16e P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Running, Role: LEADER
I20240430 20:11:39.659361 20019 raft_consensus.cc:504] T 20d166756819406ba85044f4fee9c4ce P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:39.660080 20019 leader_election.cc:288] T 20d166756819406ba85044f4fee9c4ce P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:39.659942 20060 consensus_queue.cc:260] T 5809e7c9c2204ba293426c24129fb16e P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:39.660409 20060 raft_consensus.cc:3037] T 5809e7c9c2204ba293426c24129fb16e P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Advancing to term 2
I20240430 20:11:39.660895 18608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20d166756819406ba85044f4fee9c4ce" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:39.664695 20181 tablet.cc:1722] T 12c6e131fa904d23865b8673b58a7e33 P 3c51c33d5616482cafb68e26688820f4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240430 20:11:39.668612 20061 consensus_queue.cc:1035] T 53a899fa5a424a0d99f1b47925e8c7ad P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:39.676983 18988 tablet_bootstrap.cc:492] T b688e03ea71940c69bda5520b62c1fc3 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:39.677629 19495 ts_tablet_manager.cc:1875] T 93d158de64994f3c9fa45a1250a4f533 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:39.678427 19471 ts_tablet_manager.cc:1888] T 5bde72e2791a497db7dd08686ca36d80 P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:39.682284 19048 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: Sending DeleteTablet for 3 replicas of tablet 53a899fa5a424a0d99f1b47925e8c7ad
I20240430 20:11:39.683033 19048 catalog_manager.cc:6554] ReplaceTablet: tablet 53a899fa5a424a0d99f1b47925e8c7ad deleted and replaced by tablet aa9998f4e17147aa8f2b35c01980463a
I20240430 20:11:39.683458 19048 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_c264c250888a459694c33184912ab76d" } new_table_name: "default.Table_1280cfe32d5b41abb4d3eb1bd70641bf" modify_external_catalogs: true
I20240430 20:11:39.683874 19471 log.cc:1196] T 5bde72e2791a497db7dd08686ca36d80 P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/5bde72e2791a497db7dd08686ca36d80
I20240430 20:11:39.686729 18988 ts_tablet_manager.cc:1367] T b688e03ea71940c69bda5520b62c1fc3 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.094s user 0.007s sys 0.000s
I20240430 20:11:39.686264 18600 tablet_service.cc:1514] Processing DeleteTablet for tablet 53a899fa5a424a0d99f1b47925e8c7ad with delete_type TABLET_DATA_DELETED (replaced by tablet aa9998f4e17147aa8f2b35c01980463a) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:39.687657 18725 tablet_service.cc:1514] Processing DeleteTablet for tablet 53a899fa5a424a0d99f1b47925e8c7ad with delete_type TABLET_DATA_DELETED (replaced by tablet aa9998f4e17147aa8f2b35c01980463a) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:39.681277 18983 tablet_bootstrap.cc:492] T fd40967ab4a04001819e16c06a6c4300 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:39.688150 18983 ts_tablet_manager.cc:1367] T fd40967ab4a04001819e16c06a6c4300 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.323s user 0.010s sys 0.010s
I20240430 20:11:39.689545 19033 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet a05e32a33aa7488eb287cc6314c09c44 (table default.Table_2d1cfc78363a47aa8dba7ef7d80f8c9a [id=955d833ff8f14bf6957abadf9ef669e2]). Expected version 2 got 0
I20240430 20:11:39.689958 19033 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 76d6e9d547ca4a848eae852944b1d117 (table default.Table_104ae015924a48fd8d27220792f07470 [id=ff86c0357c164dd0991098c416087773]). Expected version 2 got 0
I20240430 20:11:39.690326 19033 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 12c6e131fa904d23865b8673b58a7e33 (table default.Table_a9dc8fc6a4aa46dc914d0e327a4de554 [id=19f4e6decedf4ee59fccc0a24a05cedd]). Expected version 2 got 0
I20240430 20:11:39.690774 19471 ts_tablet_manager.cc:1909] T 5bde72e2791a497db7dd08686ca36d80 P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:39.691514 19489 ts_tablet_manager.cc:1888] T 5bde72e2791a497db7dd08686ca36d80 P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:39.691668 19489 log.cc:1196] T 5bde72e2791a497db7dd08686ca36d80 P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/5bde72e2791a497db7dd08686ca36d80
I20240430 20:11:39.692132 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 77) took 8457 ms. Trace:
W20240430 20:11:39.692502 19048 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f5eaf2413e1d4e69b56fae61f768f0ce consensus state has no leader
I20240430 20:11:39.692971 19048 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_45d50fc81325443097375ae6e65a1633" } modify_external_catalogs: true
I20240430 20:11:39.693221 19048 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_45d50fc81325443097375ae6e65a1633" } modify_external_catalogs: true
I20240430 20:11:39.694857 19471 rpcz_store.cc:276] 0430 20:11:31.234329 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:31.234401 (+ 72us) service_pool.cc:225] Handling call
0430 20:11:39.653356 (+8418955us) ts_tablet_manager.cc:1120] Deleting tablet 5bde72e2791a497db7dd08686ca36d80
0430 20:11:39.653365 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:39.657358 (+ 3993us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:39.678256 (+ 20898us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:39.692076 (+ 13820us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":8418660}
I20240430 20:11:39.695544 19489 ts_tablet_manager.cc:1909] T 5bde72e2791a497db7dd08686ca36d80 P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:39.695700 18999 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet 5bde72e2791a497db7dd08686ca36d80 (table default.Table_9c1e6182953140bb9cd594f08458c7d9 [id=5227206f44d04fa89b4842b899f33afb]) successfully deleted
I20240430 20:11:39.694950 18988 raft_consensus.cc:348] T b688e03ea71940c69bda5520b62c1fc3 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:39.696316 19471 tablet_replica.cc:330] T dd9b893d48684cfea7c242b93ecba01f P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:39.696786 18809 heartbeater.cc:347] Connected to a master server at 127.17.177.62:36441
I20240430 20:11:39.696296 18988 raft_consensus.cc:374] T b688e03ea71940c69bda5520b62c1fc3 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:39.698056 19471 raft_consensus.cc:2229] T dd9b893d48684cfea7c242b93ecba01f P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:39.704917 18985 tablet_bootstrap.cc:492] T acf3b6fead41483a82c43dc0286fab7c P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
W20240430 20:11:39.710364 19000 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet a482f4622f774d60b4efd6827b8be7d7 (table default.Table_e686cdfcbcc046749fa5b51578a64794 [id=c3dc9f05316d46d6a3ef4d03f40877ec], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet a482f4622f774d60b4efd6827b8be7d7 consensus state has no leader
I20240430 20:11:39.694373 18983 raft_consensus.cc:348] T fd40967ab4a04001819e16c06a6c4300 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:39.712224 18983 raft_consensus.cc:374] T fd40967ab4a04001819e16c06a6c4300 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:39.716429 18983 raft_consensus.cc:729] T fd40967ab4a04001819e16c06a6c4300 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:39.716814 18983 consensus_queue.cc:260] T fd40967ab4a04001819e16c06a6c4300 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:39.717219 19471 raft_consensus.cc:2258] T dd9b893d48684cfea7c242b93ecba01f P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:39.697597 20177 tablet.cc:1722] T 12c6e131fa904d23865b8673b58a7e33 P 5e1c6397f09747f091534faf93542710: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240430 20:11:39.718492 18988 raft_consensus.cc:729] T b688e03ea71940c69bda5520b62c1fc3 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:39.723306 18988 consensus_queue.cc:260] T b688e03ea71940c69bda5520b62c1fc3 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:39.722170 18454 tablet_service.cc:1514] Processing DeleteTablet for tablet 53a899fa5a424a0d99f1b47925e8c7ad with delete_type TABLET_DATA_DELETED (replaced by tablet aa9998f4e17147aa8f2b35c01980463a) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:39.725782 20025 raft_consensus.cc:482] T f5546f9edeea44d7a6c5cb954f5157ba P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:39.726087 18985 ts_tablet_manager.cc:1367] T acf3b6fead41483a82c43dc0286fab7c P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.164s user 0.007s sys 0.010s
W20240430 20:11:39.726032 18417 consensus_peers.cc:485] T b688e03ea71940c69bda5520b62c1fc3 P 2efb542aa55e439b99ccb98d1c5c884a -> Peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Couldn't send request to peer 3c51c33d5616482cafb68e26688820f4. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240430 20:11:39.727093 18474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20d166756819406ba85044f4fee9c4ce" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:39.731127 18988 ts_tablet_manager.cc:1398] T b688e03ea71940c69bda5520b62c1fc3 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.042s user 0.003s sys 0.001s
I20240430 20:11:39.731263 20183 tablet.cc:1722] T 12c6e131fa904d23865b8673b58a7e33 P 2efb542aa55e439b99ccb98d1c5c884a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240430 20:11:39.734264 19471 ts_tablet_manager.cc:1875] T dd9b893d48684cfea7c242b93ecba01f P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:39.735730 18699 leader_election.cc:302] T 20d166756819406ba85044f4fee9c4ce P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710
I20240430 20:11:39.734751 18983 ts_tablet_manager.cc:1398] T fd40967ab4a04001819e16c06a6c4300 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.046s user 0.001s sys 0.001s
I20240430 20:11:39.739050 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 69) took 8450 ms. Trace:
I20240430 20:11:39.741019 19028 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 12c6e131fa904d23865b8673b58a7e33 (table default.Table_a9dc8fc6a4aa46dc914d0e327a4de554 [id=19f4e6decedf4ee59fccc0a24a05cedd]). Expected version 2 got 0
I20240430 20:11:39.741269 19028 catalog_manager.cc:5484] T a05e32a33aa7488eb287cc6314c09c44 P 5e1c6397f09747f091534faf93542710 reported cstate change: term changed from 1 to 2, leader changed from 3c51c33d5616482cafb68e26688820f4 (127.17.177.3) to 5e1c6397f09747f091534faf93542710 (127.17.177.2). New cstate: current_term: 2 leader_uuid: "5e1c6397f09747f091534faf93542710" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } health_report { overall_health: UNKNOWN } } }
I20240430 20:11:39.732626 20025 raft_consensus.cc:504] T f5546f9edeea44d7a6c5cb954f5157ba P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:39.741724 19028 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet a05e32a33aa7488eb287cc6314c09c44 (table default.Table_2d1cfc78363a47aa8dba7ef7d80f8c9a [id=955d833ff8f14bf6957abadf9ef669e2]). Expected version 2 got 0
I20240430 20:11:39.741989 19028 catalog_manager.cc:5484] T acd23f0a63d5448aaa7ed78422e67bad P 5e1c6397f09747f091534faf93542710 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e1c6397f09747f091534faf93542710 (127.17.177.2). New cstate: current_term: 1 leader_uuid: "5e1c6397f09747f091534faf93542710" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } health_report { overall_health: UNKNOWN } } }
I20240430 20:11:39.742779 19028 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet acd23f0a63d5448aaa7ed78422e67bad (table default.Table_972f13be4dca4cd19b9a04534dc59f51 [id=69f37db4de634a3185613e40013e0ed7]). Expected version 1 got 0
I20240430 20:11:39.743206 20025 leader_election.cc:288] T f5546f9edeea44d7a6c5cb954f5157ba P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:39.743566 20019 raft_consensus.cc:2726] T 20d166756819406ba85044f4fee9c4ce P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:39.743556 18988 tablet_bootstrap.cc:492] T 47c37876694c4a4abb1091c631dab7a5 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:39.744179 18608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5546f9edeea44d7a6c5cb954f5157ba" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:39.744298 19051 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: Sending DeleteTablet for 3 replicas of tablet aec9c09cc0454acab6006bb3e10982b7
I20240430 20:11:39.744417 18608 raft_consensus.cc:2454] T f5546f9edeea44d7a6c5cb954f5157ba P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 0.
I20240430 20:11:39.744891 19051 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:51664 (request call id 169) took 4283 ms. Trace:
I20240430 20:11:39.744984 19051 rpcz_store.cc:276] 0430 20:11:35.460886 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:36.266681 (+805795us) service_pool.cc:225] Handling call
0430 20:11:36.276273 (+ 9592us) spinlock_profiling.cc:243] Waited 9.3 ms on lock 0x616000040654. stack: 00007feb16c74756 00007feb16c74510 00007feb15ca748d 0000000000955804 00000000009557cd 00007feb257b315d 00007feb257b3253 00007feb224e5d41 00007feb2a0d134d 00007feb2a25eca6 00007feb21047f66 00007feb21047c95 00007feb1d97c04e 00007feb1d97af2e 00007feb1d981405 00007feb1d981d2f
0430 20:11:38.400442 (+2124169us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0430 20:11:38.400640 (+ 198us) catalog_manager.cc:2749] Modifying in-memory table state
0430 20:11:38.400716 (+ 76us) catalog_manager.cc:2758] Locking tablets
0430 20:11:38.828430 (+427714us) catalog_manager.cc:2772] Removing table and tablets from system table
0430 20:11:38.995246 (+166816us) catalog_manager.cc:2790] Removing table from by-name map
0430 20:11:38.995339 (+ 93us) catalog_manager.cc:2806] Committing in-memory state
0430 20:11:39.744250 (+748911us) catalog_manager.cc:2810] Aborting table tasks
0430 20:11:39.744805 (+ 555us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:38.881613 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:38.898825 (+ 17212us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:38.898833 (+ 8us) write_op.cc:427] Acquired schema lock
0430 20:11:38.898838 (+ 5us) tablet.cc:660] Decoding operations
0430 20:11:38.898999 (+ 161us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:38.899015 (+ 16us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:38.899023 (+ 8us) tablet.cc:683] Acquiring locks for 2 operations
0430 20:11:38.899170 (+ 147us) tablet.cc:699] Row locks acquired
0430 20:11:38.899174 (+ 4us) write_op.cc:260] PREPARE: finished
0430 20:11:38.899225 (+ 51us) write_op.cc:270] Start()
0430 20:11:38.899270 (+ 45us) write_op.cc:276] Timestamp: P: 1714507898899183 usec, L: 0
0430 20:11:38.899277 (+ 7us) op_driver.cc:331] REPLICATION: starting
0430 20:11:38.948448 (+ 49171us) log.cc:842] Serialized 793 byte log entry
0430 20:11:38.948639 (+ 191us) spinlock_profiling.cc:243] Waited 48.9 ms on lock 0x616000040654. stack: 00007feb16c74756 00007feb16c74510 00007feb15ca748d 0000000000955804 00000000009557cd 000000000095519a 00007feb224b9162 00007feb23821b02 00007feb23820629 00007feb2382853b 00007feb238283b8 0000000000a60bcc 00007feb16cbab74 00007feb16cbe19b 00007feb16cbe018 0000000000a60bcc
0430 20:11:38.991865 (+ 43226us) op_driver.cc:444] REPLICATION: finished
0430 20:11:38.994726 (+ 2861us) write_op.cc:301] APPLY: starting
0430 20:11:38.994748 (+ 22us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:38.994805 (+ 57us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:38.994808 (+ 3us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:38.994911 (+ 103us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:38.994951 (+ 40us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0430 20:11:38.994959 (+ 8us) write_op.cc:312] APPLY: finished
0430 20:11:38.995052 (+ 93us) log.cc:842] Serialized 29 byte log entry
0430 20:11:38.995128 (+ 76us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:38.995154 (+ 26us) write_op.cc:446] Released schema lock
0430 20:11:38.995161 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:38.995229 (+ 68us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:38.995233 (+ 4us) write_op.cc:446] Released schema lock
Metrics: {"spinlock_wait_cycles":21397760,"child_traces":[["op",{"apply.queue_time_us":2752,"apply.run_cpu_time_us":569,"apply.run_wall_time_us":569,"num_ops":2,"prepare.queue_time_us":52461,"prepare.run_cpu_time_us":1337,"prepare.run_wall_time_us":67097,"raft.queue_time_us":37821,"raft.run_cpu_time_us":362,"raft.run_wall_time_us":40574,"replication_time_us":92560,"spinlock_wait_cycles":112491904}]]}
I20240430 20:11:39.745566 18415 leader_election.cc:302] T f5546f9edeea44d7a6c5cb954f5157ba P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:39.746768 18758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5546f9edeea44d7a6c5cb954f5157ba" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:39.748219 18725 tablet_service.cc:1514] Processing DeleteTablet for tablet aec9c09cc0454acab6006bb3e10982b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:38 UTC) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:39.748178 18985 raft_consensus.cc:348] T acf3b6fead41483a82c43dc0286fab7c P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:39.748417 18985 raft_consensus.cc:374] T acf3b6fead41483a82c43dc0286fab7c P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:39.748538 18985 raft_consensus.cc:729] T acf3b6fead41483a82c43dc0286fab7c P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:39.748392 18454 tablet_service.cc:1514] Processing DeleteTablet for tablet aec9c09cc0454acab6006bb3e10982b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:38 UTC) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:39.748787 18985 consensus_queue.cc:260] T acf3b6fead41483a82c43dc0286fab7c P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
W20240430 20:11:39.749116 18417 leader_election.cc:341] T f5546f9edeea44d7a6c5cb954f5157ba P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:39.749060 19489 rpcz_store.cc:276] 0430 20:11:31.274623 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:31.274690 (+ 67us) service_pool.cc:225] Handling call
0430 20:11:39.530921 (+8256231us) ts_tablet_manager.cc:1120] Deleting tablet 5bde72e2791a497db7dd08686ca36d80
0430 20:11:39.532096 (+ 1175us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:39.641113 (+109017us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:39.688702 (+ 47589us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:39.725373 (+ 36671us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":8244484}
I20240430 20:11:39.749420 18985 ts_tablet_manager.cc:1398] T acf3b6fead41483a82c43dc0286fab7c P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.023s user 0.001s sys 0.001s
I20240430 20:11:39.749562 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 71) took 8462 ms. Trace:
I20240430 20:11:39.749667 19489 rpcz_store.cc:276] 0430 20:11:31.287302 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:31.288347 (+ 1045us) service_pool.cc:225] Handling call
0430 20:11:39.749389 (+8461042us) ts_tablet_manager.cc:1120] Deleting tablet a9e0659564894b0e8cb7f600bb07a085
0430 20:11:39.749402 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:39.749522 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":8460707}
I20240430 20:11:39.749742 19001 catalog_manager.cc:4830] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): tablet 5bde72e2791a497db7dd08686ca36d80 (table default.Table_9c1e6182953140bb9cd594f08458c7d9 [id=5227206f44d04fa89b4842b899f33afb]) successfully deleted
I20240430 20:11:39.749949 19489 tablet_replica.cc:330] T dd9b893d48684cfea7c242b93ecba01f P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:39.750033 20025 raft_consensus.cc:2781] T f5546f9edeea44d7a6c5cb954f5157ba P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:39.750133 20025 raft_consensus.cc:482] T f5546f9edeea44d7a6c5cb954f5157ba P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:39.750172 19489 raft_consensus.cc:2229] T dd9b893d48684cfea7c242b93ecba01f P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus shutting down.
W20240430 20:11:39.750212 19001 catalog_manager.cc:4810] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): delete failed for tablet a9e0659564894b0e8cb7f600bb07a085 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a9e0659564894b0e8cb7f600bb07a085 already in progress: creating tablet
I20240430 20:11:39.750253 20025 raft_consensus.cc:3037] T f5546f9edeea44d7a6c5cb954f5157ba P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:39.750348 19489 raft_consensus.cc:2258] T dd9b893d48684cfea7c242b93ecba01f P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:39.750730 20105 raft_consensus.cc:482] T c9e7660eccc448dbb99d135f82bfd261 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:39.750867 20105 raft_consensus.cc:504] T c9e7660eccc448dbb99d135f82bfd261 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:39.752405 18600 tablet_service.cc:1514] Processing DeleteTablet for tablet aec9c09cc0454acab6006bb3e10982b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:38 UTC) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:39.753669 18608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9e7660eccc448dbb99d135f82bfd261" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:39.754065 19489 ts_tablet_manager.cc:1875] T dd9b893d48684cfea7c242b93ecba01f P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:39.756567 19024 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: Sending DeleteTablet for 3 replicas of tablet 04b3dff86d5c48a09d03da58bd48f360
I20240430 20:11:39.757107 19024 catalog_manager.cc:6554] ReplaceTablet: tablet 04b3dff86d5c48a09d03da58bd48f360 deleted and replaced by tablet 89b774967624498fa2be499878b71124
I20240430 20:11:39.757218 19024 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:51664 (request call id 190) took 1040 ms. Trace:
I20240430 20:11:39.757587 18600 tablet_service.cc:1514] Processing DeleteTablet for tablet 04b3dff86d5c48a09d03da58bd48f360 with delete_type TABLET_DATA_DELETED (replaced by tablet 89b774967624498fa2be499878b71124) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:39.757318 19024 rpcz_store.cc:276] 0430 20:11:38.717075 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:39.590194 (+873119us) service_pool.cc:225] Handling call
0430 20:11:39.757189 (+166995us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:39.736342 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:39.736468 (+ 126us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:39.736474 (+ 6us) write_op.cc:427] Acquired schema lock
0430 20:11:39.736478 (+ 4us) tablet.cc:660] Decoding operations
0430 20:11:39.736585 (+ 107us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:39.736596 (+ 11us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:39.736600 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
0430 20:11:39.736722 (+ 122us) tablet.cc:699] Row locks acquired
0430 20:11:39.736725 (+ 3us) write_op.cc:260] PREPARE: finished
0430 20:11:39.736758 (+ 33us) write_op.cc:270] Start()
0430 20:11:39.736786 (+ 28us) write_op.cc:276] Timestamp: P: 1714507899736691 usec, L: 0
0430 20:11:39.736790 (+ 4us) op_driver.cc:331] REPLICATION: starting
0430 20:11:39.736971 (+ 181us) log.cc:842] Serialized 625 byte log entry
0430 20:11:39.755634 (+ 18663us) op_driver.cc:444] REPLICATION: finished
0430 20:11:39.755739 (+ 105us) write_op.cc:301] APPLY: starting
0430 20:11:39.755758 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:39.755807 (+ 49us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:39.755853 (+ 46us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:39.755963 (+ 110us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:39.756025 (+ 62us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0430 20:11:39.756033 (+ 8us) write_op.cc:312] APPLY: finished
0430 20:11:39.756129 (+ 96us) log.cc:842] Serialized 29 byte log entry
0430 20:11:39.756205 (+ 76us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:39.756229 (+ 24us) write_op.cc:446] Released schema lock
0430 20:11:39.756236 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:39.756293 (+ 57us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:39.756296 (+ 3us) write_op.cc:446] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":58,"apply.run_cpu_time_us":608,"apply.run_wall_time_us":608,"num_ops":2,"prepare.queue_time_us":49,"prepare.run_cpu_time_us":697,"prepare.run_wall_time_us":696,"replication_time_us":18818}]]}
I20240430 20:11:39.758111 19024 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_0604805f009c4d6486965426c82dff10" } new_table_name: "default.Table_6a30a0abb67d4a298f87216515a92153" modify_external_catalogs: true
I20240430 20:11:39.758843 18988 tablet_bootstrap.cc:654] T 47c37876694c4a4abb1091c631dab7a5 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:39.759315 18725 tablet_service.cc:1514] Processing DeleteTablet for tablet 04b3dff86d5c48a09d03da58bd48f360 with delete_type TABLET_DATA_DELETED (replaced by tablet 89b774967624498fa2be499878b71124) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:39.762604 18758 raft_consensus.cc:3037] T 20d166756819406ba85044f4fee9c4ce P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:39.762995 19023 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: Sending DeleteTablet for 3 replicas of tablet 294644d7a16b4ca2bbb6d106bcb8c7ff
I20240430 20:11:39.763597 19023 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:51664 (request call id 171) took 4097 ms. Trace:
I20240430 20:11:39.763695 19023 rpcz_store.cc:276] 0430 20:11:35.665749 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:37.451580 (+1785831us) service_pool.cc:225] Handling call
0430 20:11:37.452113 (+ 533us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0430 20:11:37.452398 (+ 285us) catalog_manager.cc:2749] Modifying in-memory table state
0430 20:11:37.452473 (+ 75us) catalog_manager.cc:2758] Locking tablets
0430 20:11:38.827295 (+1374822us) catalog_manager.cc:2772] Removing table and tablets from system table
0430 20:11:39.027398 (+200103us) catalog_manager.cc:2790] Removing table from by-name map
0430 20:11:39.027513 (+ 115us) catalog_manager.cc:2806] Committing in-memory state
0430 20:11:39.762940 (+735427us) catalog_manager.cc:2810] Aborting table tasks
0430 20:11:39.763535 (+ 595us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:38.829287 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:38.829447 (+ 160us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:38.829456 (+ 9us) write_op.cc:427] Acquired schema lock
0430 20:11:38.829460 (+ 4us) tablet.cc:660] Decoding operations
0430 20:11:38.829609 (+ 149us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:38.829645 (+ 36us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:38.829653 (+ 8us) tablet.cc:683] Acquiring locks for 2 operations
0430 20:11:38.829800 (+ 147us) tablet.cc:699] Row locks acquired
0430 20:11:38.829804 (+ 4us) write_op.cc:260] PREPARE: finished
0430 20:11:38.829847 (+ 43us) write_op.cc:270] Start()
0430 20:11:38.829886 (+ 39us) write_op.cc:276] Timestamp: P: 1714507898829812 usec, L: 0
0430 20:11:38.829890 (+ 4us) op_driver.cc:331] REPLICATION: starting
0430 20:11:38.853008 (+ 23118us) log.cc:842] Serialized 793 byte log entry
0430 20:11:38.853218 (+ 210us) spinlock_profiling.cc:243] Waited 22.9 ms on lock 0x616000040654. stack: 00007feb16c74756 00007feb16c74510 00007feb15ca748d 0000000000955804 00000000009557cd 000000000095519a 00007feb224b9162 00007feb23821b02 00007feb23820629 00007feb2382853b 00007feb238283b8 0000000000a60bcc 00007feb16cbab74 00007feb16cbe19b 00007feb16cbe018 0000000000a60bcc
0430 20:11:38.918757 (+ 65539us) op_driver.cc:444] REPLICATION: finished
0430 20:11:38.918905 (+ 148us) write_op.cc:301] APPLY: starting
0430 20:11:38.918924 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:38.918987 (+ 63us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:38.918992 (+ 5us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:38.919096 (+ 104us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:38.919145 (+ 49us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0430 20:11:38.919157 (+ 12us) write_op.cc:312] APPLY: finished
0430 20:11:38.919255 (+ 98us) log.cc:842] Serialized 29 byte log entry
0430 20:11:38.945262 (+ 26007us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:38.945301 (+ 39us) write_op.cc:446] Released schema lock
0430 20:11:38.945312 (+ 11us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:39.030759 (+ 85447us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:39.030773 (+ 14us) write_op.cc:446] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":99,"apply.run_cpu_time_us":715,"apply.run_wall_time_us":111947,"num_ops":2,"prepare.queue_time_us":1106,"prepare.run_cpu_time_us":9673,"prepare.run_wall_time_us":52286,"raft.queue_time_us":6063,"raft.run_cpu_time_us":5,"raft.run_wall_time_us":5,"replication_time_us":88827,"spinlock_wait_cycles":52595072,"thread_start_us":28272,"threads_started":1}]]}
I20240430 20:11:39.764372 19023 master_service.cc:946] ReplaceTablet: received request to replace tablet d61c279d20b0471a967942525ceb4fcb from {username='slave'} at 127.0.0.1:51664
W20240430 20:11:39.765772 19027 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.17.177.2:41749 (request call id 97) took 4210 ms (client timeout 1000 ms). Trace:
W20240430 20:11:39.765901 19027 rpcz_store.cc:269] 0430 20:11:35.555642 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:36.166816 (+611174us) service_pool.cc:225] Handling call
0430 20:11:36.274722 (+107906us) spinlock_profiling.cc:243] Waited 107 ms on lock 0x616000040654. stack: 00007feb16c74756 00007feb16c74510 00007feb15ca748d 0000000000955804 00000000009557cd 00007feb257b315d 00007feb257b3253 00007feb224e5d41 00007feb2a0d134d 00007feb2a258668 00007feb21043786 00007feb210434b5 00007feb1d97c04e 00007feb1d97af2e 00007feb1d981405 00007feb1d981d2f
0430 20:11:39.765691 (+3490969us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:37.682033 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:37.682570 (+ 537us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:37.682650 (+ 80us) write_op.cc:427] Acquired schema lock
0430 20:11:37.682653 (+ 3us) tablet.cc:660] Decoding operations
0430 20:11:37.683268 (+ 615us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:37.683348 (+ 80us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:37.683415 (+ 67us) tablet.cc:683] Acquiring locks for 1 operations
0430 20:11:37.683827 (+ 412us) tablet.cc:699] Row locks acquired
0430 20:11:37.683889 (+ 62us) write_op.cc:260] PREPARE: finished
0430 20:11:37.684134 (+ 245us) write_op.cc:270] Start()
0430 20:11:37.684358 (+ 224us) write_op.cc:276] Timestamp: P: 1714507897684022 usec, L: 0
0430 20:11:37.684365 (+ 7us) op_driver.cc:331] REPLICATION: starting
0430 20:11:37.687250 (+ 2885us) log.cc:842] Serialized 530 byte log entry
0430 20:11:37.688609 (+ 1359us) spinlock_profiling.cc:243] Waited 1.71 ms on lock 0x616000040654. stack: 00007feb16c74756 00007feb16c74510 00007feb15ca748d 0000000000955804 00000000009557cd 000000000095519a 00007feb224b9162 00007feb23821b02 00007feb23820629 00007feb2382853b 00007feb238283b8 0000000000a60bcc 00007feb16cbab74 00007feb16cbe19b 00007feb16cbe018 0000000000a60bcc
0430 20:11:37.695342 (+ 6733us) op_driver.cc:444] REPLICATION: finished
0430 20:11:37.695481 (+ 139us) write_op.cc:301] APPLY: starting
0430 20:11:37.695504 (+ 23us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:37.695567 (+ 63us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:37.695570 (+ 3us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:37.695636 (+ 66us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:37.695680 (+ 44us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0430 20:11:37.695691 (+ 11us) write_op.cc:312] APPLY: finished
0430 20:11:37.696031 (+ 340us) log.cc:842] Serialized 23 byte log entry
0430 20:11:37.696137 (+ 106us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:37.696166 (+ 29us) write_op.cc:446] Released schema lock
0430 20:11:37.696176 (+ 10us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:37.696246 (+ 70us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:37.696250 (+ 4us) write_op.cc:446] Released schema lock
Metrics: {"reported_tablets":7,"spinlock_wait_cycles":245525376,"child_traces":[["op",{"apply.queue_time_us":36,"apply.run_cpu_time_us":722,"apply.run_wall_time_us":822,"num_ops":1,"prepare.queue_time_us":219063,"prepare.run_cpu_time_us":2762,"prepare.run_wall_time_us":7049,"raft.queue_time_us":150,"raft.run_cpu_time_us":392,"raft.run_wall_time_us":862,"replication_time_us":10946,"spinlock_wait_cycles":4087296}]]}
I20240430 20:11:39.767045 19027 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_88b0eaf3d0a1461ab5e2130cb69f0dee"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240430 20:11:39.767218 18983 tablet_bootstrap.cc:492] T 76d6e9d547ca4a848eae852944b1d117 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
W20240430 20:11:39.767604 19027 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_88b0eaf3d0a1461ab5e2130cb69f0dee in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:39.768954 18983 tablet_bootstrap.cc:654] T 76d6e9d547ca4a848eae852944b1d117 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
W20240430 20:11:39.769148 19026 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.17.177.2:41749 (request call id 98) took 3282 ms (client timeout 1000 ms). Trace:
W20240430 20:11:39.769315 19026 rpcz_store.cc:269] 0430 20:11:36.487027 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:37.008373 (+521346us) service_pool.cc:225] Handling call
0430 20:11:37.044937 (+ 36564us) spinlock_profiling.cc:243] Waited 35.6 ms on lock 0x616000040654. stack: 00007feb16c74756 00007feb16c74510 00007feb15ca748d 0000000000955804 00000000009557cd 00007feb257b315d 00007feb257b3253 00007feb224e5d41 00007feb2a0d134d 00007feb2a258668 00007feb21043786 00007feb210434b5 00007feb1d97c04e 00007feb1d97af2e 00007feb1d981405 00007feb1d981d2f
0430 20:11:39.769096 (+2724159us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:38.534012 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:38.534252 (+ 240us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:38.534262 (+ 10us) write_op.cc:427] Acquired schema lock
0430 20:11:38.534265 (+ 3us) tablet.cc:660] Decoding operations
0430 20:11:38.534470 (+ 205us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:38.534481 (+ 11us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:38.534485 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0430 20:11:38.534654 (+ 169us) tablet.cc:699] Row locks acquired
0430 20:11:38.534656 (+ 2us) write_op.cc:260] PREPARE: finished
0430 20:11:38.534721 (+ 65us) write_op.cc:270] Start()
0430 20:11:38.534753 (+ 32us) write_op.cc:276] Timestamp: P: 1714507898534647 usec, L: 0
0430 20:11:38.534780 (+ 27us) op_driver.cc:331] REPLICATION: starting
0430 20:11:38.535039 (+ 259us) log.cc:842] Serialized 530 byte log entry
0430 20:11:38.813291 (+278252us) op_driver.cc:444] REPLICATION: finished
0430 20:11:38.817350 (+ 4059us) write_op.cc:301] APPLY: starting
0430 20:11:38.817373 (+ 23us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:38.817427 (+ 54us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:38.817455 (+ 28us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:38.817517 (+ 62us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:38.817553 (+ 36us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0430 20:11:38.823134 (+ 5581us) write_op.cc:312] APPLY: finished
0430 20:11:38.823268 (+ 134us) log.cc:842] Serialized 23 byte log entry
0430 20:11:38.825017 (+ 1749us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:38.825044 (+ 27us) write_op.cc:446] Released schema lock
0430 20:11:38.825055 (+ 11us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:38.852364 (+ 27309us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:38.852378 (+ 14us) write_op.cc:446] Released schema lock
Metrics: {"reported_tablets":10,"spinlock_wait_cycles":81821952,"child_traces":[["op",{"apply.queue_time_us":3965,"apply.run_cpu_time_us":738,"apply.run_wall_time_us":35132,"num_ops":1,"prepare.queue_time_us":118893,"prepare.run_cpu_time_us":876,"prepare.run_wall_time_us":1428,"replication_time_us":278475}]]}
I20240430 20:11:39.770308 19026 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_0366454b829949928c7b5e79bbf3a649"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240430 20:11:39.773622 18600 tablet_service.cc:1514] Processing DeleteTablet for tablet 294644d7a16b4ca2bbb6d106bcb8c7ff with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:37 UTC) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:39.773903 18587 tablet_service.cc:1514] Processing DeleteTablet for tablet eed6338ad3d64859af087ee15e1ec798 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:26 UTC) from {username='slave'} at 127.0.0.1:40218
W20240430 20:11:39.778998 19029 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.17.177.2:41749 (request call id 99) took 1374 ms (client timeout 1000 ms). Trace:
W20240430 20:11:39.779163 19029 rpcz_store.cc:269] 0430 20:11:38.404179 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:38.406002 (+ 1823us) service_pool.cc:225] Handling call
0430 20:11:39.778932 (+1372930us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":17}
I20240430 20:11:39.779659 19029 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_96aea8c4ff80428da8a0394f1bf7b493"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:39.780230 19029 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_96aea8c4ff80428da8a0394f1bf7b493 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:39.780861 19048 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: Sending DeleteTablet for 3 replicas of tablet 1f4e860484784aa9a2d28af2e9fb611c
I20240430 20:11:39.781486 19048 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:51664 (request call id 193) took 1064 ms. Trace:
I20240430 20:11:39.781596 19048 rpcz_store.cc:276] 0430 20:11:38.717321 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:39.692792 (+975471us) service_pool.cc:225] Handling call
0430 20:11:39.693337 (+ 545us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0430 20:11:39.693513 (+ 176us) catalog_manager.cc:2749] Modifying in-memory table state
0430 20:11:39.693609 (+ 96us) catalog_manager.cc:2758] Locking tablets
0430 20:11:39.737742 (+ 44133us) catalog_manager.cc:2772] Removing table and tablets from system table
0430 20:11:39.762137 (+ 24395us) catalog_manager.cc:2790] Removing table from by-name map
0430 20:11:39.780758 (+ 18621us) catalog_manager.cc:2806] Committing in-memory state
0430 20:11:39.780802 (+ 44us) catalog_manager.cc:2810] Aborting table tasks
0430 20:11:39.781425 (+ 623us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:39.738606 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:39.738725 (+ 119us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:39.738732 (+ 7us) write_op.cc:427] Acquired schema lock
0430 20:11:39.738735 (+ 3us) tablet.cc:660] Decoding operations
0430 20:11:39.738841 (+ 106us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:39.738852 (+ 11us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:39.738856 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
0430 20:11:39.738971 (+ 115us) tablet.cc:699] Row locks acquired
0430 20:11:39.738975 (+ 4us) write_op.cc:260] PREPARE: finished
0430 20:11:39.739009 (+ 34us) write_op.cc:270] Start()
0430 20:11:39.739040 (+ 31us) write_op.cc:276] Timestamp: P: 1714507899738943 usec, L: 0
0430 20:11:39.739045 (+ 5us) op_driver.cc:331] REPLICATION: starting
0430 20:11:39.739208 (+ 163us) log.cc:842] Serialized 728 byte log entry
0430 20:11:39.760982 (+ 21774us) op_driver.cc:444] REPLICATION: finished
0430 20:11:39.761097 (+ 115us) write_op.cc:301] APPLY: starting
0430 20:11:39.761111 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:39.761634 (+ 523us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:39.761660 (+ 26us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:39.761748 (+ 88us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:39.761785 (+ 37us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0430 20:11:39.761793 (+ 8us) write_op.cc:312] APPLY: finished
0430 20:11:39.761866 (+ 73us) log.cc:842] Serialized 29 byte log entry
0430 20:11:39.761945 (+ 79us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:39.761967 (+ 22us) write_op.cc:446] Released schema lock
0430 20:11:39.761973 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:39.762026 (+ 53us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:39.762029 (+ 3us) write_op.cc:446] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":69,"apply.run_cpu_time_us":980,"apply.run_wall_time_us":979,"num_ops":2,"prepare.queue_time_us":40,"prepare.run_cpu_time_us":645,"prepare.run_wall_time_us":667,"replication_time_us":21915}]]}
I20240430 20:11:39.782398 19048 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_7b1809d7d84b4e7896d1f10321e3f20c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:39.782972 19048 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7b1809d7d84b4e7896d1f10321e3f20c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240430 20:11:39.791492 19026 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0366454b829949928c7b5e79bbf3a649 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:39.792788 19033 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_273caa9a42a9474e8787c37032368654"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:39.793344 19033 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_273caa9a42a9474e8787c37032368654 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:39.801137 19028 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_6b67c0c03ceb4b9491ad5828022b357f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:39.801752 19028 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6b67c0c03ceb4b9491ad5828022b357f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:39.802953 20105 leader_election.cc:288] T c9e7660eccc448dbb99d135f82bfd261 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:39.803133 18454 tablet_service.cc:1514] Processing DeleteTablet for tablet 04b3dff86d5c48a09d03da58bd48f360 with delete_type TABLET_DATA_DELETED (replaced by tablet 89b774967624498fa2be499878b71124) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:39.803982 19048 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_75336e7a0e5f4c2d909f5c5c385bc379"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:39.804518 19048 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_75336e7a0e5f4c2d909f5c5c385bc379 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:39.805711 18985 tablet_bootstrap.cc:492] T 5d09baed576e4136bf6d756ca39d08b2 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:39.808629 18454 tablet_service.cc:1514] Processing DeleteTablet for tablet 294644d7a16b4ca2bbb6d106bcb8c7ff with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:37 UTC) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:39.809293 18449 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f4e860484784aa9a2d28af2e9fb611c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:39 UTC) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:39.810202 19023 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: Sending DeleteTablet for 3 replicas of tablet d61c279d20b0471a967942525ceb4fcb
I20240430 20:11:39.811123 19023 catalog_manager.cc:6554] ReplaceTablet: tablet d61c279d20b0471a967942525ceb4fcb deleted and replaced by tablet 69f88532952a4591bf1577541c3430e6
I20240430 20:11:39.811327 18725 tablet_service.cc:1514] Processing DeleteTablet for tablet 294644d7a16b4ca2bbb6d106bcb8c7ff with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:37 UTC) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:39.811700 19023 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:51664 (request call id 195) took 1008 ms. Trace:
I20240430 20:11:39.812047 18446 tablet_service.cc:1467] Processing CreateTablet for tablet eb6c96047b114ebba73db7a371f2da8e (DEFAULT_TABLE table=default.Table_d06ed4569aa2456d85107f8a351d2ea1 [id=e67d619e36c0485f8f22b4a81cb4cb6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:39.812150 18734 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f4e860484784aa9a2d28af2e9fb611c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:39 UTC) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:39.812474 18446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb6c96047b114ebba73db7a371f2da8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:39.814272 18734 tablet_service.cc:1467] Processing CreateTablet for tablet eb6c96047b114ebba73db7a371f2da8e (DEFAULT_TABLE table=default.Table_d06ed4569aa2456d85107f8a351d2ea1 [id=e67d619e36c0485f8f22b4a81cb4cb6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:39.814474 18760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9e7660eccc448dbb99d135f82bfd261" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:39.818372 19495 ts_tablet_manager.cc:1888] T 93d158de64994f3c9fa45a1250a4f533 P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:39.818538 19495 log.cc:1196] T 93d158de64994f3c9fa45a1250a4f533 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/93d158de64994f3c9fa45a1250a4f533
I20240430 20:11:39.819149 19023 rpcz_store.cc:276] 0430 20:11:38.803023 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:39.764279 (+961256us) service_pool.cc:225] Handling call
0430 20:11:39.811657 (+ 47378us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:39.790780 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:39.790862 (+ 82us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:39.790866 (+ 4us) write_op.cc:427] Acquired schema lock
0430 20:11:39.790868 (+ 2us) tablet.cc:660] Decoding operations
0430 20:11:39.791018 (+ 150us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:39.791027 (+ 9us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:39.791032 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0430 20:11:39.791095 (+ 63us) tablet.cc:699] Row locks acquired
0430 20:11:39.791097 (+ 2us) write_op.cc:260] PREPARE: finished
0430 20:11:39.791129 (+ 32us) write_op.cc:270] Start()
0430 20:11:39.791155 (+ 26us) write_op.cc:276] Timestamp: P: 1714507899791113 usec, L: 0
0430 20:11:39.791159 (+ 4us) op_driver.cc:331] REPLICATION: starting
0430 20:11:39.791320 (+ 161us) log.cc:842] Serialized 625 byte log entry
0430 20:11:39.804908 (+ 13588us) op_driver.cc:444] REPLICATION: finished
0430 20:11:39.805816 (+ 908us) write_op.cc:301] APPLY: starting
0430 20:11:39.805831 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:39.805878 (+ 47us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:39.805902 (+ 24us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:39.805998 (+ 96us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:39.806049 (+ 51us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0430 20:11:39.806056 (+ 7us) write_op.cc:312] APPLY: finished
0430 20:11:39.806132 (+ 76us) log.cc:842] Serialized 29 byte log entry
0430 20:11:39.806197 (+ 65us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:39.806213 (+ 16us) write_op.cc:446] Released schema lock
0430 20:11:39.806219 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:39.806268 (+ 49us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:39.806273 (+ 5us) write_op.cc:446] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":851,"apply.run_cpu_time_us":516,"apply.run_wall_time_us":516,"num_ops":2,"prepare.queue_time_us":645,"prepare.run_cpu_time_us":607,"prepare.run_wall_time_us":607,"replication_time_us":13722}]]}
I20240430 20:11:39.820160 19023 master_service.cc:946] ReplaceTablet: received request to replace tablet e2d90ff17e574f88b2061b1c117d4b7d from {username='slave'} at 127.0.0.1:51664
I20240430 20:11:39.820566 18734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb6c96047b114ebba73db7a371f2da8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:39.820791 18725 tablet_service.cc:1467] Processing CreateTablet for tablet 2aa2a489116940dea3173cad16d94fa5 (DEFAULT_TABLE table=default.Table_a39fa485d1e6488b951a9681aa5f3f00 [id=f02e978b8b7b49a38dc7e9d0cb406ad8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:39.823968 18725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2aa2a489116940dea3173cad16d94fa5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:39.828096 19489 ts_tablet_manager.cc:1888] T dd9b893d48684cfea7c242b93ecba01f P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:39.829845 18449 tablet_service.cc:1467] Processing CreateTablet for tablet 2aa2a489116940dea3173cad16d94fa5 (DEFAULT_TABLE table=default.Table_a39fa485d1e6488b951a9681aa5f3f00 [id=f02e978b8b7b49a38dc7e9d0cb406ad8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:39.830376 18449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2aa2a489116940dea3173cad16d94fa5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:39.830374 18730 tablet_service.cc:1514] Processing DeleteTablet for tablet d61c279d20b0471a967942525ceb4fcb with delete_type TABLET_DATA_DELETED (replaced by tablet 69f88532952a4591bf1577541c3430e6) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:39.830552 18724 tablet_service.cc:1467] Processing CreateTablet for tablet 35fdb052135541cfb42a8aeb03b64dc8 (DEFAULT_TABLE table=default.Table_088b70cec77744aaacf26aa6d4adebbd [id=3e0705b442cc45109c5750e006e0ff04]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:39.831396 18724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35fdb052135541cfb42a8aeb03b64dc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:39.832099 18450 tablet_service.cc:1467] Processing CreateTablet for tablet 489dc838744a49eba542009c86688aa5 (DEFAULT_TABLE table=default.Table_cb067ab866334a64a0186737511a9855 [id=6f4ee22d8d924af08a838970664ecda0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:39.832479 18450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 489dc838744a49eba542009c86688aa5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:39.832556 18447 tablet_service.cc:1514] Processing DeleteTablet for tablet d61c279d20b0471a967942525ceb4fcb with delete_type TABLET_DATA_DELETED (replaced by tablet 69f88532952a4591bf1577541c3430e6) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:39.833320 18454 tablet_service.cc:1467] Processing CreateTablet for tablet 35fdb052135541cfb42a8aeb03b64dc8 (DEFAULT_TABLE table=default.Table_088b70cec77744aaacf26aa6d4adebbd [id=3e0705b442cc45109c5750e006e0ff04]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:39.833688 18454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35fdb052135541cfb42a8aeb03b64dc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:39.833671 18744 tablet_service.cc:1467] Processing CreateTablet for tablet 489dc838744a49eba542009c86688aa5 (DEFAULT_TABLE table=default.Table_cb067ab866334a64a0186737511a9855 [id=6f4ee22d8d924af08a838970664ecda0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:39.834266 18985 tablet_bootstrap.cc:654] T 5d09baed576e4136bf6d756ca39d08b2 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:39.835835 18760 raft_consensus.cc:2379] T c9e7660eccc448dbb99d135f82bfd261 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2efb542aa55e439b99ccb98d1c5c884a in current term 1: Already voted for candidate 5e1c6397f09747f091534faf93542710 in this term.
I20240430 20:11:39.836462 18417 leader_election.cc:302] T c9e7660eccc448dbb99d135f82bfd261 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a; no voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710
I20240430 20:11:39.836794 20105 raft_consensus.cc:3037] T c9e7660eccc448dbb99d135f82bfd261 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:39.837919 18744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 489dc838744a49eba542009c86688aa5. 1 dirs total, 0 dirs full, 0 dirs failed
W20240430 20:11:39.840004 18557 consensus_peers.cc:485] T acd23f0a63d5448aaa7ed78422e67bad P 5e1c6397f09747f091534faf93542710 -> Peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Couldn't send request to peer 3c51c33d5616482cafb68e26688820f4. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240430 20:11:39.840984 20105 raft_consensus.cc:2726] T c9e7660eccc448dbb99d135f82bfd261 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:39.847923 20025 raft_consensus.cc:504] T f5546f9edeea44d7a6c5cb954f5157ba P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:39.845283 19489 log.cc:1196] T dd9b893d48684cfea7c242b93ecba01f P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/dd9b893d48684cfea7c242b93ecba01f
I20240430 20:11:39.849298 19489 ts_tablet_manager.cc:1909] T dd9b893d48684cfea7c242b93ecba01f P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:39.849706 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 74) took 8561 ms. Trace:
I20240430 20:11:39.849812 19489 rpcz_store.cc:276] 0430 20:11:31.287731 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:31.288725 (+ 994us) service_pool.cc:225] Handling call
0430 20:11:39.749898 (+8461173us) ts_tablet_manager.cc:1120] Deleting tablet dd9b893d48684cfea7c242b93ecba01f
0430 20:11:39.749906 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:39.813238 (+ 63332us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:39.827975 (+ 14737us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:39.849629 (+ 21654us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":8460986}
I20240430 20:11:39.850230 19489 tablet_replica.cc:330] T eed6338ad3d64859af087ee15e1ec798 P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:39.850450 19489 raft_consensus.cc:2229] T eed6338ad3d64859af087ee15e1ec798 P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Raft consensus shutting down.
I20240430 20:11:39.850791 20103 raft_consensus.cc:482] T 6e754af64cb64fd2a082adae7d5f24da P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:39.850836 19489 raft_consensus.cc:2258] T eed6338ad3d64859af087ee15e1ec798 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:39.850962 20103 raft_consensus.cc:504] T 6e754af64cb64fd2a082adae7d5f24da P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:39.851692 20103 leader_election.cc:288] T 6e754af64cb64fd2a082adae7d5f24da P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:39.852985 18608 raft_consensus.cc:1226] T 6e826459c3f4437da81c9e6e09f0c22f P 5e1c6397f09747f091534faf93542710 [term 2 LEADER]: Rejecting Update request from peer 2efb542aa55e439b99ccb98d1c5c884a for earlier term 1. Current term is 2. Ops: []
I20240430 20:11:39.853269 20025 leader_election.cc:288] T f5546f9edeea44d7a6c5cb954f5157ba P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Requested vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:39.853999 19001 catalog_manager.cc:4830] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): tablet dd9b893d48684cfea7c242b93ecba01f (table default.Table_4bce1dae3391497ebea5f1b536700d62 [id=8f0cb0ef65d84fb8964209dd501c0b5c]) successfully deleted
I20240430 20:11:39.854496 18760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e754af64cb64fd2a082adae7d5f24da" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:39.854660 18754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5546f9edeea44d7a6c5cb954f5157ba" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
W20240430 20:11:39.855221 18417 leader_election.cc:341] T 6e754af64cb64fd2a082adae7d5f24da P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:39.855230 18587 tablet_service.cc:1514] Processing DeleteTablet for tablet 5bde72e2791a497db7dd08686ca36d80 with delete_type TABLET_DATA_DELETED (replaced by tablet 66dd767bd3f8494f94061fb89e5e5680) from {username='slave'} at 127.0.0.1:40218
W20240430 20:11:39.855440 18417 leader_election.cc:341] T f5546f9edeea44d7a6c5cb954f5157ba P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:39.855477 18600 tablet_service.cc:1514] Processing DeleteTablet for tablet e6eac92dfa5f4456bc49571878f0f8dd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:29 UTC) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:39.858765 18608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e754af64cb64fd2a082adae7d5f24da" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:39.858979 20105 consensus_queue.cc:1046] T 6e826459c3f4437da81c9e6e09f0c22f P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: INVALID_TERM, Last received: 1.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240430 20:11:39.859112 18615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5546f9edeea44d7a6c5cb954f5157ba" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:39.859412 20105 consensus_queue.cc:1225] T 6e826459c3f4437da81c9e6e09f0c22f P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Peer 5e1c6397f09747f091534faf93542710 log is divergent from this leader: its last log entry 2.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20240430 20:11:39.859573 20071 raft_consensus.cc:3032] T 6e826459c3f4437da81c9e6e09f0c22f P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Stepping down as leader of term 1
I20240430 20:11:39.859652 20071 raft_consensus.cc:729] T 6e826459c3f4437da81c9e6e09f0c22f P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Running, Role: LEADER
I20240430 20:11:39.859930 20071 consensus_queue.cc:260] T 6e826459c3f4437da81c9e6e09f0c22f P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:39.860240 20071 raft_consensus.cc:3037] T 6e826459c3f4437da81c9e6e09f0c22f P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Advancing to term 2
I20240430 20:11:39.861613 18615 raft_consensus.cc:3037] T f5546f9edeea44d7a6c5cb954f5157ba P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:39.863020 19489 ts_tablet_manager.cc:1875] T eed6338ad3d64859af087ee15e1ec798 P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240430 20:11:39.863373 18415 leader_election.cc:341] T 6e754af64cb64fd2a082adae7d5f24da P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:39.863507 18415 leader_election.cc:302] T 6e754af64cb64fd2a082adae7d5f24da P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a; no voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710
I20240430 20:11:39.864135 20103 raft_consensus.cc:2726] T 6e754af64cb64fd2a082adae7d5f24da P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:39.864758 18600 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f4e860484784aa9a2d28af2e9fb611c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:39 UTC) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:39.866829 18588 tablet_service.cc:1467] Processing CreateTablet for tablet 35fdb052135541cfb42a8aeb03b64dc8 (DEFAULT_TABLE table=default.Table_088b70cec77744aaacf26aa6d4adebbd [id=3e0705b442cc45109c5750e006e0ff04]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:39.867429 18588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35fdb052135541cfb42a8aeb03b64dc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:39.867480 18600 tablet_service.cc:1467] Processing CreateTablet for tablet 2aa2a489116940dea3173cad16d94fa5 (DEFAULT_TABLE table=default.Table_a39fa485d1e6488b951a9681aa5f3f00 [id=f02e978b8b7b49a38dc7e9d0cb406ad8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:39.867452 18587 tablet_service.cc:1467] Processing CreateTablet for tablet eb6c96047b114ebba73db7a371f2da8e (DEFAULT_TABLE table=default.Table_d06ed4569aa2456d85107f8a351d2ea1 [id=e67d619e36c0485f8f22b4a81cb4cb6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:39.867976 19495 ts_tablet_manager.cc:1909] T 93d158de64994f3c9fa45a1250a4f533 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:39.868842 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 53) took 8815 ms. Trace:
I20240430 20:11:39.868948 19495 rpcz_store.cc:276] 0430 20:11:31.053321 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:31.053698 (+ 377us) service_pool.cc:225] Handling call
0430 20:11:39.631139 (+8577441us) ts_tablet_manager.cc:1120] Deleting tablet 93d158de64994f3c9fa45a1250a4f533
0430 20:11:39.646962 (+ 15823us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:39.807826 (+160864us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:39.818256 (+ 10430us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:39.868774 (+ 50518us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":8509140}
I20240430 20:11:39.869254 18600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2aa2a489116940dea3173cad16d94fa5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:39.869342 19495 tablet_replica.cc:330] T a9e0659564894b0e8cb7f600bb07a085 P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:39.869544 19495 raft_consensus.cc:2229] T a9e0659564894b0e8cb7f600bb07a085 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:39.869724 19495 raft_consensus.cc:2258] T a9e0659564894b0e8cb7f600bb07a085 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:39.869935 19004 catalog_manager.cc:4830] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): tablet 93d158de64994f3c9fa45a1250a4f533 (table default.Table_9d39d94262a7481aad7df7868a2aff26 [id=06b16d46a7484d0dbc7971f070d0f37e]) successfully deleted
I20240430 20:11:39.871466 18587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb6c96047b114ebba73db7a371f2da8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:39.872603 18592 tablet_service.cc:1467] Processing CreateTablet for tablet 489dc838744a49eba542009c86688aa5 (DEFAULT_TABLE table=default.Table_cb067ab866334a64a0186737511a9855 [id=6f4ee22d8d924af08a838970664ecda0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:39.872982 18592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 489dc838744a49eba542009c86688aa5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:39.875216 19495 ts_tablet_manager.cc:1875] T a9e0659564894b0e8cb7f600bb07a085 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:39.875445 18754 raft_consensus.cc:1261] T a05e32a33aa7488eb287cc6314c09c44 P 3c51c33d5616482cafb68e26688820f4 [term 2 FOLLOWER]: Refusing update from remote peer 5e1c6397f09747f091534faf93542710: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 2. (term mismatch)
I20240430 20:11:39.875563 18754 pending_rounds.cc:76] T a05e32a33aa7488eb287cc6314c09c44 P 3c51c33d5616482cafb68e26688820f4: Aborting all ops after (but not including) 1
I20240430 20:11:39.875630 18754 pending_rounds.cc:97] T a05e32a33aa7488eb287cc6314c09c44 P 3c51c33d5616482cafb68e26688820f4: Aborting uncommitted ALTER_SCHEMA_OP operation due to leader change: 1.2
W20240430 20:11:39.875787 18754 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:59830: Aborted: Op aborted by new leader
I20240430 20:11:39.875980 18754 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:59830 (request call id 203) took 2396 ms. Trace:
I20240430 20:11:39.876087 18754 rpcz_store.cc:276] 0430 20:11:37.479905 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:38.600603 (+1120698us) service_pool.cc:225] Handling call
0430 20:11:39.875944 (+1275341us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:38.741587 (+ 0us) alter_schema_op.cc:96] PREPARE ALTER-SCHEMA: Starting
0430 20:11:38.741757 (+ 170us) alter_schema_op.cc:57] Acquiring schema lock in exclusive mode
0430 20:11:38.741769 (+ 12us) alter_schema_op.cc:59] Acquired schema lock
0430 20:11:38.741777 (+ 8us) alter_schema_op.cc:111] PREPARE ALTER-SCHEMA: finished
0430 20:11:38.741830 (+ 53us) alter_schema_op.cc:119] START. Timestamp: 1714507898741770
0430 20:11:38.741833 (+ 3us) op_driver.cc:331] REPLICATION: starting
0430 20:11:38.742110 (+ 277us) log.cc:842] Serialized 173 byte log entry
0430 20:11:39.875713 (+1133603us) op_driver.cc:444] REPLICATION: finished
0430 20:11:39.875729 (+ 16us) op_driver.cc:372] HandleFailure(Aborted: Op aborted by new leader)
0430 20:11:39.875734 (+ 5us) alter_schema_op.cc:152] AlterSchemaCommitCallback: op aborted
Metrics: {"child_traces":[["op",{"prepare.queue_time_us":140512,"prepare.run_cpu_time_us":3199,"prepare.run_wall_time_us":91668,"replication_time_us":1133840,"thread_start_us":232303,"threads_started":2,"wal-append.queue_time_us":116910,"wal-append.run_cpu_time_us":720,"wal-append.run_wall_time_us":1003608}]]}
W20240430 20:11:39.877496 20106 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:59830: Illegal state: Cannot assign timestamp to op. Tablet is not in leader mode. Last heard from a leader: 1.135s ago.
I20240430 20:11:39.877717 20106 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:59830 (request call id 209) took 2262 ms. Trace:
I20240430 20:11:39.877817 20106 rpcz_store.cc:276] 0430 20:11:37.614991 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:38.607919 (+992928us) service_pool.cc:225] Handling call
0430 20:11:39.877678 (+1269759us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:38.833311 (+ 0us) alter_schema_op.cc:96] PREPARE ALTER-SCHEMA: Starting
0430 20:11:38.833421 (+ 110us) alter_schema_op.cc:57] Acquiring schema lock in exclusive mode
0430 20:11:39.877354 (+1043933us) alter_schema_op.cc:59] Acquired schema lock
0430 20:11:39.877364 (+ 10us) alter_schema_op.cc:111] PREPARE ALTER-SCHEMA: finished
0430 20:11:39.877425 (+ 61us) op_driver.cc:372] HandleFailure(Illegal state: Cannot assign timestamp to op. Tablet is not in leader mode. Last heard from a leader: 1.135s ago.)
0430 20:11:39.877431 (+ 6us) alter_schema_op.cc:152] AlterSchemaCommitCallback: op aborted
Metrics: {"child_traces":[["op",{"prepare.queue_time_us":225030}]]}
I20240430 20:11:39.869942 20071 raft_consensus.cc:482] T acf3b6fead41483a82c43dc0286fab7c P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:39.878556 20071 raft_consensus.cc:504] T acf3b6fead41483a82c43dc0286fab7c P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
W20240430 20:11:39.878887 19001 catalog_manager.cc:4909] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): alter failed for tablet a05e32a33aa7488eb287cc6314c09c44 (table default.Table_2d1cfc78363a47aa8dba7ef7d80f8c9a [id=955d833ff8f14bf6957abadf9ef669e2]): Aborted: Op aborted by new leader
I20240430 20:11:39.879307 20071 leader_election.cc:288] T acf3b6fead41483a82c43dc0286fab7c P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:39.870077 18528 heartbeater.cc:347] Connected to a master server at 127.17.177.62:36441
I20240430 20:11:39.879904 18754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acf3b6fead41483a82c43dc0286fab7c" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
W20240430 20:11:39.880430 18417 leader_election.cc:341] T acf3b6fead41483a82c43dc0286fab7c P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:39.881063 18593 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a1e50f99c5d4b44bdbb51b58a92bc75 with delete_type TABLET_DATA_DELETED (replaced by tablet fbdeb4714b8c4b93ab72822c43329aae) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:39.881345 18586 tablet_service.cc:1514] Processing DeleteTablet for tablet a9e0659564894b0e8cb7f600bb07a085 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:29 UTC) from {username='slave'} at 127.0.0.1:40218
W20240430 20:11:39.881426 20053 log.cc:924] Time spent T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: Append to log took a long time: real 0.058s user 0.001s sys 0.000s
I20240430 20:11:39.881466 18615 raft_consensus.cc:2454] T f5546f9edeea44d7a6c5cb954f5157ba P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 1.
I20240430 20:11:39.882364 18415 leader_election.cc:302] T f5546f9edeea44d7a6c5cb954f5157ba P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters: 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:39.882994 20060 consensus_queue.cc:1035] T a05e32a33aa7488eb287cc6314c09c44 P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240430 20:11:39.884002 18586 tablet_service.cc:1514] Processing DeleteTablet for tablet d61c279d20b0471a967942525ceb4fcb with delete_type TABLET_DATA_DELETED (replaced by tablet 69f88532952a4591bf1577541c3430e6) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:39.884173 18608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acf3b6fead41483a82c43dc0286fab7c" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:39.886693 18760 raft_consensus.cc:1261] T 7dfdecba959543a795dfcd3eeda29e58 P 3c51c33d5616482cafb68e26688820f4 [term 2 FOLLOWER]: Refusing update from remote peer 5e1c6397f09747f091534faf93542710: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20240430 20:11:39.888144 18415 leader_election.cc:341] T acf3b6fead41483a82c43dc0286fab7c P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:39.888327 18415 leader_election.cc:302] T acf3b6fead41483a82c43dc0286fab7c P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a; no voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710
I20240430 20:11:39.888975 20025 raft_consensus.cc:2781] T f5546f9edeea44d7a6c5cb954f5157ba P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:39.889133 20025 raft_consensus.cc:686] T f5546f9edeea44d7a6c5cb954f5157ba P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Becoming Leader. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Running, Role: LEADER
I20240430 20:11:39.889380 20025 consensus_queue.cc:237] T f5546f9edeea44d7a6c5cb954f5157ba P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:39.890774 20105 raft_consensus.cc:2726] T acf3b6fead41483a82c43dc0286fab7c P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:39.892573 20060 consensus_queue.cc:1035] T 7dfdecba959543a795dfcd3eeda29e58 P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240430 20:11:39.899684 18756 raft_consensus.cc:1261] T 1555924b5f1444398f12ee82c3cb5b13 P 3c51c33d5616482cafb68e26688820f4 [term 2 FOLLOWER]: Refusing update from remote peer 5e1c6397f09747f091534faf93542710: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240430 20:11:39.901451 20060 consensus_queue.cc:1035] T 1555924b5f1444398f12ee82c3cb5b13 P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240430 20:11:39.903409 19489 ts_tablet_manager.cc:1888] T eed6338ad3d64859af087ee15e1ec798 P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:39.903558 19489 log.cc:1196] T eed6338ad3d64859af087ee15e1ec798 P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/eed6338ad3d64859af087ee15e1ec798
I20240430 20:11:39.903998 19489 ts_tablet_manager.cc:1909] T eed6338ad3d64859af087ee15e1ec798 P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:39.904328 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 73) took 8616 ms. Trace:
I20240430 20:11:39.904438 19489 rpcz_store.cc:276] 0430 20:11:31.287617 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:31.289450 (+ 1833us) service_pool.cc:225] Handling call
0430 20:11:39.850133 (+8560683us) ts_tablet_manager.cc:1120] Deleting tablet eed6338ad3d64859af087ee15e1ec798
0430 20:11:39.850143 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:39.872436 (+ 22293us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:39.903261 (+ 30825us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:39.904286 (+ 1025us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":8560476}
I20240430 20:11:39.904919 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 72) took 8617 ms. Trace:
I20240430 20:11:39.905043 19489 rpcz_store.cc:276] 0430 20:11:31.287499 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:31.289703 (+ 2204us) service_pool.cc:225] Handling call
0430 20:11:39.904728 (+8615025us) ts_tablet_manager.cc:1120] Deleting tablet 4e69ca12bd10494fb013614741bc166c
0430 20:11:39.904739 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:39.904881 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":8614847}
I20240430 20:11:39.905400 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 75) took 8605 ms. Trace:
I20240430 20:11:39.905400 19001 catalog_manager.cc:4830] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): tablet eed6338ad3d64859af087ee15e1ec798 (table default.Table_1dc3c3ef217d4f8b9e8fdb60180c5d0f [id=bc29c76ef4a448a2863c278309b548e8]) successfully deleted
I20240430 20:11:39.905495 19489 rpcz_store.cc:276] 0430 20:11:31.299501 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:31.299582 (+ 81us) service_pool.cc:225] Handling call
0430 20:11:39.905280 (+8605698us) ts_tablet_manager.cc:1120] Deleting tablet 0a9066d895744c5bb7d9532b2940603c
0430 20:11:39.905289 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:39.905373 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":8599447}
I20240430 20:11:39.905740 19489 tablet_replica.cc:330] T b688e03ea71940c69bda5520b62c1fc3 P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:39.905956 19489 raft_consensus.cc:2229] T b688e03ea71940c69bda5520b62c1fc3 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Raft consensus shutting down.
W20240430 20:11:39.905995 19001 catalog_manager.cc:4810] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): delete failed for tablet 4e69ca12bd10494fb013614741bc166c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4e69ca12bd10494fb013614741bc166c already in progress: creating tablet
I20240430 20:11:39.906140 19489 raft_consensus.cc:2258] T b688e03ea71940c69bda5520b62c1fc3 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Raft consensus is shut down!
W20240430 20:11:39.906283 19001 catalog_manager.cc:4810] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): delete failed for tablet 0a9066d895744c5bb7d9532b2940603c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0a9066d895744c5bb7d9532b2940603c already in progress: creating tablet
I20240430 20:11:39.915585 18474 raft_consensus.cc:1261] T 7dfdecba959543a795dfcd3eeda29e58 P 2efb542aa55e439b99ccb98d1c5c884a [term 2 FOLLOWER]: Refusing update from remote peer 5e1c6397f09747f091534faf93542710: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240430 20:11:39.917461 20024 consensus_queue.cc:1035] T 7dfdecba959543a795dfcd3eeda29e58 P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240430 20:11:39.919490 18474 raft_consensus.cc:1261] T 1555924b5f1444398f12ee82c3cb5b13 P 2efb542aa55e439b99ccb98d1c5c884a [term 2 FOLLOWER]: Refusing update from remote peer 5e1c6397f09747f091534faf93542710: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240430 20:11:39.927364 19495 ts_tablet_manager.cc:1888] T a9e0659564894b0e8cb7f600bb07a085 P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:39.927546 19495 log.cc:1196] T a9e0659564894b0e8cb7f600bb07a085 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/a9e0659564894b0e8cb7f600bb07a085
I20240430 20:11:39.928036 19495 ts_tablet_manager.cc:1909] T a9e0659564894b0e8cb7f600bb07a085 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:39.928988 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 65) took 8808 ms. Trace:
I20240430 20:11:39.929121 19495 rpcz_store.cc:276] 0430 20:11:31.120897 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:31.120987 (+ 90us) service_pool.cc:225] Handling call
0430 20:11:39.869255 (+8748268us) ts_tablet_manager.cc:1120] Deleting tablet a9e0659564894b0e8cb7f600bb07a085
0430 20:11:39.869264 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:39.878260 (+ 8996us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:39.927170 (+ 48910us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:39.928935 (+ 1765us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":8733637}
I20240430 20:11:39.929734 19033 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_ad4322a6bb564abd80dad138208e420d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240430 20:11:39.930155 19026 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_e34613952dcb46618c3bc7cecb31fdaf" } modify_external_catalogs: true
I20240430 20:11:39.930359 19026 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_e34613952dcb46618c3bc7cecb31fdaf" } modify_external_catalogs: true
I20240430 20:11:39.938673 19489 ts_tablet_manager.cc:1875] T b688e03ea71940c69bda5520b62c1fc3 P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:39.942994 20024 consensus_queue.cc:1035] T 1555924b5f1444398f12ee82c3cb5b13 P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.022s
I20240430 20:11:39.947016 18988 tablet_bootstrap.cc:492] T 47c37876694c4a4abb1091c631dab7a5 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
W20240430 20:11:39.947494 19033 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ad4322a6bb564abd80dad138208e420d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:39.947360 18988 ts_tablet_manager.cc:1367] T 47c37876694c4a4abb1091c631dab7a5 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.215s user 0.005s sys 0.009s
I20240430 20:11:39.951072 18988 raft_consensus.cc:348] T 47c37876694c4a4abb1091c631dab7a5 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:39.951305 18988 raft_consensus.cc:374] T 47c37876694c4a4abb1091c631dab7a5 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:39.951436 18988 raft_consensus.cc:729] T 47c37876694c4a4abb1091c631dab7a5 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:39.952683 18988 consensus_queue.cc:260] T 47c37876694c4a4abb1091c631dab7a5 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:39.953281 18988 ts_tablet_manager.cc:1398] T 47c37876694c4a4abb1091c631dab7a5 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240430 20:11:39.956276 18608 raft_consensus.cc:1226] T 7bde1d6934784c9aa5911181e8df9eeb P 5e1c6397f09747f091534faf93542710 [term 2 FOLLOWER]: Rejecting Update request from peer 3c51c33d5616482cafb68e26688820f4 for earlier term 1. Current term is 2. Ops: []
I20240430 20:11:39.957283 20019 consensus_queue.cc:1046] T 7bde1d6934784c9aa5911181e8df9eeb P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240430 20:11:39.959153 18985 tablet_bootstrap.cc:492] T 5d09baed576e4136bf6d756ca39d08b2 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:39.959391 18985 ts_tablet_manager.cc:1367] T 5d09baed576e4136bf6d756ca39d08b2 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.205s user 0.003s sys 0.008s
I20240430 20:11:39.960264 18985 raft_consensus.cc:348] T 5d09baed576e4136bf6d756ca39d08b2 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:39.960683 18985 raft_consensus.cc:374] T 5d09baed576e4136bf6d756ca39d08b2 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:39.960805 18985 raft_consensus.cc:729] T 5d09baed576e4136bf6d756ca39d08b2 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:39.961369 18985 consensus_queue.cc:260] T 5d09baed576e4136bf6d756ca39d08b2 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:39.972955 20071 consensus_queue.cc:1035] T 802336602d8f4ac4bd7e0872d6f3dae9 P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:39.977375 18758 raft_consensus.cc:3037] T 802336602d8f4ac4bd7e0872d6f3dae9 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:39.977612 18985 ts_tablet_manager.cc:1398] T 5d09baed576e4136bf6d756ca39d08b2 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.018s user 0.000s sys 0.002s
I20240430 20:11:39.978075 18985 tablet_bootstrap.cc:492] T 0a9066d895744c5bb7d9532b2940603c P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:39.980327 18985 tablet_bootstrap.cc:654] T 0a9066d895744c5bb7d9532b2940603c P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:39.981403 18988 tablet_bootstrap.cc:492] T acd23f0a63d5448aaa7ed78422e67bad P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:39.984381 19489 ts_tablet_manager.cc:1888] T b688e03ea71940c69bda5520b62c1fc3 P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:39.984565 19489 log.cc:1196] T b688e03ea71940c69bda5520b62c1fc3 P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/b688e03ea71940c69bda5520b62c1fc3
I20240430 20:11:39.984994 19489 ts_tablet_manager.cc:1909] T b688e03ea71940c69bda5520b62c1fc3 P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:39.985364 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 76) took 8678 ms. Trace:
I20240430 20:11:39.985468 19489 rpcz_store.cc:276] 0430 20:11:31.306688 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:31.306781 (+ 93us) service_pool.cc:225] Handling call
0430 20:11:39.905676 (+8598895us) ts_tablet_manager.cc:1120] Deleting tablet b688e03ea71940c69bda5520b62c1fc3
0430 20:11:39.905683 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:39.967543 (+ 61860us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:39.984245 (+ 16702us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:39.985304 (+ 1059us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":8598358}
I20240430 20:11:39.985911 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 77) took 8677 ms. Trace:
I20240430 20:11:39.986011 19489 rpcz_store.cc:276] 0430 20:11:31.308299 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:31.308396 (+ 97us) service_pool.cc:225] Handling call
0430 20:11:39.985770 (+8677374us) ts_tablet_manager.cc:1120] Deleting tablet e5111d7607dd42188a0176cf5be2ec05
0430 20:11:39.985778 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:39.985859 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":8675053}
I20240430 20:11:39.986402 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 78) took 8675 ms. Trace:
I20240430 20:11:39.987295 19001 catalog_manager.cc:4830] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): tablet b688e03ea71940c69bda5520b62c1fc3 (table default.Table_2ec9639eca4a47439b7859929cd705bc [id=6ffe241384bd42b7a22d41154ac59e3a]) successfully deleted
W20240430 20:11:39.987594 19001 catalog_manager.cc:4797] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): delete failed for tablet e5111d7607dd42188a0176cf5be2ec05 because the tablet was not found. No further retry: Not found: Tablet not found: e5111d7607dd42188a0176cf5be2ec05
I20240430 20:11:39.987828 20071 consensus_queue.cc:1035] T 802336602d8f4ac4bd7e0872d6f3dae9 P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:39.988348 18988 tablet_bootstrap.cc:654] T acd23f0a63d5448aaa7ed78422e67bad P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:39.989306 19004 catalog_manager.cc:4830] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): tablet a9e0659564894b0e8cb7f600bb07a085 (table default.Table_3c5df662747b48bf8beb6ebe4d2ea618 [id=663c8efbd7ac4a148d5f7626f1522033]) successfully deleted
I20240430 20:11:39.989491 19495 tablet_replica.cc:330] T f77086a5675f44e99e8a867b694df4b8 P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:39.989737 19495 raft_consensus.cc:2229] T f77086a5675f44e99e8a867b694df4b8 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:39.989921 19495 raft_consensus.cc:2258] T f77086a5675f44e99e8a867b694df4b8 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:39.990552 19495 ts_tablet_manager.cc:1875] T f77086a5675f44e99e8a867b694df4b8 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:39.991235 20024 consensus_queue.cc:1035] T acd23f0a63d5448aaa7ed78422e67bad P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:39.996085 18985 tablet_bootstrap.cc:492] T 0a9066d895744c5bb7d9532b2940603c P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:39.996343 18985 ts_tablet_manager.cc:1367] T 0a9066d895744c5bb7d9532b2940603c P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.000s
I20240430 20:11:39.997304 18985 raft_consensus.cc:348] T 0a9066d895744c5bb7d9532b2940603c P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:39.997579 18985 raft_consensus.cc:374] T 0a9066d895744c5bb7d9532b2940603c P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:39.997736 18985 raft_consensus.cc:729] T 0a9066d895744c5bb7d9532b2940603c P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:39.998199 18985 consensus_queue.cc:260] T 0a9066d895744c5bb7d9532b2940603c P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:39.998919 18985 ts_tablet_manager.cc:1398] T 0a9066d895744c5bb7d9532b2940603c P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240430 20:11:39.999653 18985 tablet_bootstrap.cc:492] T 32c823d1700949ab8b22adcd83b7bfd1 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:40.001731 18985 tablet_bootstrap.cc:654] T 32c823d1700949ab8b22adcd83b7bfd1 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
W20240430 20:11:40.005128 18417 consensus_peers.cc:485] T dd9b893d48684cfea7c242b93ecba01f P 2efb542aa55e439b99ccb98d1c5c884a -> Peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Couldn't send request to peer 3c51c33d5616482cafb68e26688820f4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dd9b893d48684cfea7c242b93ecba01f. This is attempt 1: this message will repeat every 5th retry.
I20240430 20:11:40.016705 20197 raft_consensus.cc:3032] T 7bde1d6934784c9aa5911181e8df9eeb P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Stepping down as leader of term 1
I20240430 20:11:40.029472 18482 raft_consensus.cc:1261] T a05e32a33aa7488eb287cc6314c09c44 P 2efb542aa55e439b99ccb98d1c5c884a [term 2 FOLLOWER]: Refusing update from remote peer 5e1c6397f09747f091534faf93542710: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240430 20:11:40.027264 20197 raft_consensus.cc:729] T 7bde1d6934784c9aa5911181e8df9eeb P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Running, Role: LEADER
I20240430 20:11:40.032174 20024 consensus_queue.cc:1035] T a05e32a33aa7488eb287cc6314c09c44 P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
W20240430 20:11:40.039085 18696 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.177.2:46027, user_credentials={real_user=slave}} blocked reactor thread for 82005.1us
I20240430 20:11:39.986495 19489 rpcz_store.cc:276] 0430 20:11:31.311106 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:31.311262 (+ 156us) service_pool.cc:225] Handling call
0430 20:11:39.986253 (+8674991us) ts_tablet_manager.cc:1120] Deleting tablet 0e503a898406412ab977d3e048ea4b5d
0430 20:11:39.986261 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:39.986343 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":8673853}
I20240430 20:11:40.050943 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 79) took 8738 ms. Trace:
I20240430 20:11:40.051082 19489 rpcz_store.cc:276] 0430 20:11:31.312563 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:31.312614 (+ 51us) service_pool.cc:225] Handling call
0430 20:11:40.050759 (+8738145us) ts_tablet_manager.cc:1120] Deleting tablet 5f56ac2afd444d63a6bfd96988464bbb
0430 20:11:40.050771 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:40.050866 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":8737912}
I20240430 20:11:40.054950 19495 ts_tablet_manager.cc:1888] T f77086a5675f44e99e8a867b694df4b8 P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:40.055176 19495 log.cc:1196] T f77086a5675f44e99e8a867b694df4b8 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/f77086a5675f44e99e8a867b694df4b8
I20240430 20:11:40.055603 19495 ts_tablet_manager.cc:1909] T f77086a5675f44e99e8a867b694df4b8 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:40.055936 18983 tablet_bootstrap.cc:492] T 76d6e9d547ca4a848eae852944b1d117 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:40.056053 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 66) took 8928 ms. Trace:
I20240430 20:11:40.056268 18983 ts_tablet_manager.cc:1367] T 76d6e9d547ca4a848eae852944b1d117 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.289s user 0.021s sys 0.033s
I20240430 20:11:40.056167 19495 rpcz_store.cc:276] 0430 20:11:31.127712 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:31.127825 (+ 113us) service_pool.cc:225] Handling call
0430 20:11:39.989418 (+8861593us) ts_tablet_manager.cc:1120] Deleting tablet f77086a5675f44e99e8a867b694df4b8
0430 20:11:39.989429 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:40.010729 (+ 21300us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:40.054773 (+ 44044us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:40.055993 (+ 1220us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":8824712}
I20240430 20:11:40.056738 19004 catalog_manager.cc:4830] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): tablet f77086a5675f44e99e8a867b694df4b8 (table default.Table_cd719a43207f4bf983e1b8e6d236bb16 [id=47e0b04b17834deb833a8f31c123edc5]) successfully deleted
I20240430 20:11:40.057389 18983 raft_consensus.cc:348] T 76d6e9d547ca4a848eae852944b1d117 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:40.057762 18983 raft_consensus.cc:374] T 76d6e9d547ca4a848eae852944b1d117 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:40.057900 18983 raft_consensus.cc:729] T 76d6e9d547ca4a848eae852944b1d117 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:40.059230 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 81) took 8593 ms. Trace:
I20240430 20:11:40.059374 19489 rpcz_store.cc:276] 0430 20:11:31.458211 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:31.458321 (+ 110us) service_pool.cc:225] Handling call
0430 20:11:40.051317 (+8592996us) ts_tablet_manager.cc:1120] Deleting tablet 5bd0362ff41b4faca0205d136d596737
0430 20:11:40.051325 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:40.051429 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":8592764}
I20240430 20:11:40.059732 19489 tablet_replica.cc:330] T 77285afcafe04a829e485998bbbe00f0 P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:40.059967 19489 raft_consensus.cc:2229] T 77285afcafe04a829e485998bbbe00f0 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:40.060195 19489 raft_consensus.cc:2258] T 77285afcafe04a829e485998bbbe00f0 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:40.060984 19028 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_3e3885add2344068a5da1614890073e0" } new_table_name: "default.Table_b2d07520b9944a89ad62fca70396174e" modify_external_catalogs: true
I20240430 20:11:40.064657 18988 tablet_bootstrap.cc:492] T acd23f0a63d5448aaa7ed78422e67bad P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:40.064885 18988 ts_tablet_manager.cc:1367] T acd23f0a63d5448aaa7ed78422e67bad P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.111s user 0.010s sys 0.001s
I20240430 20:11:40.065759 18988 raft_consensus.cc:348] T acd23f0a63d5448aaa7ed78422e67bad P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:40.066044 18988 raft_consensus.cc:374] T acd23f0a63d5448aaa7ed78422e67bad P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:40.066154 18988 raft_consensus.cc:729] T acd23f0a63d5448aaa7ed78422e67bad P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:40.066107 20196 consensus_queue.cc:1035] T 24dcc20832804229a2664873bf6ddaa7 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:40.070636 19023 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: Sending DeleteTablet for 3 replicas of tablet e2d90ff17e574f88b2061b1c117d4b7d
I20240430 20:11:40.071198 19023 catalog_manager.cc:6554] ReplaceTablet: tablet e2d90ff17e574f88b2061b1c117d4b7d deleted and replaced by tablet a663030d37614053a7c0369841403422
I20240430 20:11:40.032585 20197 consensus_queue.cc:260] T 7bde1d6934784c9aa5911181e8df9eeb P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:40.074203 19029 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet fd40967ab4a04001819e16c06a6c4300 (table default.Table_4130caf534004edfb413056d7f6cfd52 [id=da9ad991953a40c4b77b31317597ad1d]). Expected version 4 got 0
I20240430 20:11:40.074966 18449 tablet_service.cc:1514] Processing DeleteTablet for tablet e2d90ff17e574f88b2061b1c117d4b7d with delete_type TABLET_DATA_DELETED (replaced by tablet a663030d37614053a7c0369841403422) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:40.076660 18449 tablet_service.cc:1467] Processing CreateTablet for tablet d4d6c37b578c4fe1aa666b0a6b0bc7c9 (DEFAULT_TABLE table=default.Table_423319f67e734a64a731bf33ef868beb [id=44f7bf1df366476292532972544fe690]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:40.077174 18449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4d6c37b578c4fe1aa666b0a6b0bc7c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:40.078337 19029 catalog_manager.cc:5484] T c9e7660eccc448dbb99d135f82bfd261 P 5e1c6397f09747f091534faf93542710 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e1c6397f09747f091534faf93542710 (127.17.177.2). New cstate: current_term: 1 leader_uuid: "5e1c6397f09747f091534faf93542710" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } health_report { overall_health: UNKNOWN } } }
I20240430 20:11:40.078840 19029 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet c9e7660eccc448dbb99d135f82bfd261 (table default.Table_685a26737bb046fb85ced15df7ae3423 [id=51d177e8498848b38c43e3f8d11f092c]). Expected version 1 got 0
I20240430 20:11:40.078876 18983 consensus_queue.cc:260] T 76d6e9d547ca4a848eae852944b1d117 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:40.079591 18983 ts_tablet_manager.cc:1398] T 76d6e9d547ca4a848eae852944b1d117 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.023s user 0.001s sys 0.001s
I20240430 20:11:40.078313 20197 raft_consensus.cc:3037] T 7bde1d6934784c9aa5911181e8df9eeb P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Advancing to term 2
I20240430 20:11:40.081293 18760 raft_consensus.cc:1261] T 294644d7a16b4ca2bbb6d106bcb8c7ff P 3c51c33d5616482cafb68e26688820f4 [term 2 FOLLOWER]: Refusing update from remote peer 5e1c6397f09747f091534faf93542710: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240430 20:11:40.084954 18446 tablet_service.cc:1467] Processing CreateTablet for tablet 3a03b6997921408cb5a65f4cae223e9c (DEFAULT_TABLE table=default.Table_1840b0e89a9840fe92bd1ca13c915806 [id=83b1ac2e9c3d4a10b02f96ebb17faf44]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:40.085245 18450 tablet_service.cc:1467] Processing CreateTablet for tablet aa9998f4e17147aa8f2b35c01980463a (DEFAULT_TABLE table=default.Table_ffb3192f443d437ab63f56f628e596ad [id=608890834da843fa9af0cfc0d548029e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:40.085413 18446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a03b6997921408cb5a65f4cae223e9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:40.085624 18450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa9998f4e17147aa8f2b35c01980463a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:40.088304 18586 tablet_service.cc:1467] Processing CreateTablet for tablet 89b774967624498fa2be499878b71124 (DEFAULT_TABLE table=default.Table_640bb0fa95ca44d49921276d665b9ca3 [id=9fa22833979f4c2f978de651145acef9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:40.088766 18586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89b774967624498fa2be499878b71124. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:40.092862 18454 tablet_service.cc:1467] Processing CreateTablet for tablet 89b774967624498fa2be499878b71124 (DEFAULT_TABLE table=default.Table_640bb0fa95ca44d49921276d665b9ca3 [id=9fa22833979f4c2f978de651145acef9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:40.093334 18454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89b774967624498fa2be499878b71124. 1 dirs total, 0 dirs full, 0 dirs failed
W20240430 20:11:40.095861 19001 catalog_manager.cc:4797] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): delete failed for tablet 5f56ac2afd444d63a6bfd96988464bbb because the tablet was not found. No further retry: Not found: Tablet not found: 5f56ac2afd444d63a6bfd96988464bbb
W20240430 20:11:40.096336 19001 catalog_manager.cc:4810] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): delete failed for tablet 5bd0362ff41b4faca0205d136d596737 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5bd0362ff41b4faca0205d136d596737 already in progress: creating tablet
I20240430 20:11:40.097118 18725 tablet_service.cc:1514] Processing DeleteTablet for tablet e2d90ff17e574f88b2061b1c117d4b7d with delete_type TABLET_DATA_DELETED (replaced by tablet a663030d37614053a7c0369841403422) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:40.097703 18744 tablet_service.cc:1467] Processing CreateTablet for tablet d4d6c37b578c4fe1aa666b0a6b0bc7c9 (DEFAULT_TABLE table=default.Table_423319f67e734a64a731bf33ef868beb [id=44f7bf1df366476292532972544fe690]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:40.098198 18744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4d6c37b578c4fe1aa666b0a6b0bc7c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:40.099048 18588 tablet_service.cc:1467] Processing CreateTablet for tablet aa9998f4e17147aa8f2b35c01980463a (DEFAULT_TABLE table=default.Table_ffb3192f443d437ab63f56f628e596ad [id=608890834da843fa9af0cfc0d548029e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:40.099496 19489 ts_tablet_manager.cc:1875] T 77285afcafe04a829e485998bbbe00f0 P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:40.099546 18588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa9998f4e17147aa8f2b35c01980463a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:40.102909 18724 tablet_service.cc:1467] Processing CreateTablet for tablet 3a03b6997921408cb5a65f4cae223e9c (DEFAULT_TABLE table=default.Table_1840b0e89a9840fe92bd1ca13c915806 [id=83b1ac2e9c3d4a10b02f96ebb17faf44]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:40.103333 18724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a03b6997921408cb5a65f4cae223e9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:40.103453 18592 tablet_service.cc:1467] Processing CreateTablet for tablet 3a03b6997921408cb5a65f4cae223e9c (DEFAULT_TABLE table=default.Table_1840b0e89a9840fe92bd1ca13c915806 [id=83b1ac2e9c3d4a10b02f96ebb17faf44]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:40.103871 18592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a03b6997921408cb5a65f4cae223e9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:40.103838 18988 consensus_queue.cc:260] T acd23f0a63d5448aaa7ed78422e67bad P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:40.106403 18988 ts_tablet_manager.cc:1398] T acd23f0a63d5448aaa7ed78422e67bad P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.041s user 0.001s sys 0.001s
I20240430 20:11:40.106837 18988 tablet_bootstrap.cc:492] T 4e69ca12bd10494fb013614741bc166c P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:40.106832 20024 raft_consensus.cc:482] T fd40967ab4a04001819e16c06a6c4300 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:40.106910 18734 tablet_service.cc:1467] Processing CreateTablet for tablet aa9998f4e17147aa8f2b35c01980463a (DEFAULT_TABLE table=default.Table_ffb3192f443d437ab63f56f628e596ad [id=608890834da843fa9af0cfc0d548029e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:40.107360 18734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa9998f4e17147aa8f2b35c01980463a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:40.111065 19033 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_8e605def35724c5c945b61107596d81a" } modify_external_catalogs: true
I20240430 20:11:40.111284 19033 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_8e605def35724c5c945b61107596d81a" } modify_external_catalogs: true
I20240430 20:11:40.111249 18600 tablet_service.cc:1514] Processing DeleteTablet for tablet e2d90ff17e574f88b2061b1c117d4b7d with delete_type TABLET_DATA_DELETED (replaced by tablet a663030d37614053a7c0369841403422) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:40.111282 20060 consensus_queue.cc:1035] T 294644d7a16b4ca2bbb6d106bcb8c7ff P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.027s
I20240430 20:11:40.112092 18725 tablet_service.cc:1467] Processing CreateTablet for tablet 89b774967624498fa2be499878b71124 (DEFAULT_TABLE table=default.Table_640bb0fa95ca44d49921276d665b9ca3 [id=9fa22833979f4c2f978de651145acef9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:40.112102 19023 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_1ebe0eb1a1984ee9bbb16d634ff2cedb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240430 20:11:40.112553 18725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89b774967624498fa2be499878b71124. 1 dirs total, 0 dirs full, 0 dirs failed
W20240430 20:11:40.112682 19023 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1ebe0eb1a1984ee9bbb16d634ff2cedb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:40.115114 19048 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_3be3b20c86ea424fbc173dd32e01f90c" } new_table_name: "default.Table_404b186c544e4b30bb36cb77d1a14120" modify_external_catalogs: true
I20240430 20:11:40.115648 20203 raft_consensus.cc:482] T 802336602d8f4ac4bd7e0872d6f3dae9 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2efb542aa55e439b99ccb98d1c5c884a)
I20240430 20:11:40.115800 20203 raft_consensus.cc:504] T 802336602d8f4ac4bd7e0872d6f3dae9 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:40.117143 20203 leader_election.cc:288] T 802336602d8f4ac4bd7e0872d6f3dae9 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:40.122378 18983 tablet_bootstrap.cc:492] T 0a9066d895744c5bb7d9532b2940603c P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:40.130252 19489 ts_tablet_manager.cc:1888] T 77285afcafe04a829e485998bbbe00f0 P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240430 20:11:40.130921 19495 tablet_replica.cc:330] T e6eac92dfa5f4456bc49571878f0f8dd P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:40.131158 19495 raft_consensus.cc:2229] T e6eac92dfa5f4456bc49571878f0f8dd P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Raft consensus shutting down.
I20240430 20:11:40.132743 19489 log.cc:1196] T 77285afcafe04a829e485998bbbe00f0 P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/77285afcafe04a829e485998bbbe00f0
I20240430 20:11:40.135329 19489 ts_tablet_manager.cc:1909] T 77285afcafe04a829e485998bbbe00f0 P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:40.139672 20203 raft_consensus.cc:482] T acd23f0a63d5448aaa7ed78422e67bad P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:40.139969 20203 raft_consensus.cc:504] T acd23f0a63d5448aaa7ed78422e67bad P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:40.141360 20203 leader_election.cc:288] T acd23f0a63d5448aaa7ed78422e67bad P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:40.141985 18474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acd23f0a63d5448aaa7ed78422e67bad" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:40.140173 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 82) took 8553 ms. Trace:
I20240430 20:11:40.142791 19489 rpcz_store.cc:276] 0430 20:11:31.586503 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:31.586710 (+ 207us) service_pool.cc:225] Handling call
0430 20:11:40.059643 (+8472933us) ts_tablet_manager.cc:1120] Deleting tablet 77285afcafe04a829e485998bbbe00f0
0430 20:11:40.059654 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:40.120500 (+ 60846us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:40.130055 (+ 9555us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:40.140126 (+ 10071us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":8472713}
I20240430 20:11:40.143059 20060 raft_consensus.cc:482] T a9e0659564894b0e8cb7f600bb07a085 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:40.143283 20060 raft_consensus.cc:504] T a9e0659564894b0e8cb7f600bb07a085 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:40.143522 19489 tablet_replica.cc:330] T 802336602d8f4ac4bd7e0872d6f3dae9 P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:40.143779 19489 raft_consensus.cc:2229] T 802336602d8f4ac4bd7e0872d6f3dae9 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:40.143985 19489 raft_consensus.cc:2258] T 802336602d8f4ac4bd7e0872d6f3dae9 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:40.144397 19001 catalog_manager.cc:4830] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): tablet 77285afcafe04a829e485998bbbe00f0 (table default.Table_e63f64a411094f8db2ef641f2257ca79 [id=040dec3a3d804cdfbdada359bbe5123b]) successfully deleted
I20240430 20:11:40.146283 20024 raft_consensus.cc:504] T fd40967ab4a04001819e16c06a6c4300 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:40.146827 19489 ts_tablet_manager.cc:1875] T 802336602d8f4ac4bd7e0872d6f3dae9 P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:40.147006 18617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "802336602d8f4ac4bd7e0872d6f3dae9" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:40.148708 18754 raft_consensus.cc:1261] T 0b4d9e860d7c45fd91d7a566605c9283 P 3c51c33d5616482cafb68e26688820f4 [term 2 FOLLOWER]: Refusing update from remote peer 5e1c6397f09747f091534faf93542710: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240430 20:11:40.149037 18760 raft_consensus.cc:1261] T 372b135dd01f44f98cd713ca8857ba61 P 3c51c33d5616482cafb68e26688820f4 [term 2 FOLLOWER]: Refusing update from remote peer 2efb542aa55e439b99ccb98d1c5c884a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240430 20:11:40.146800 18587 tablet_service.cc:1467] Processing CreateTablet for tablet d4d6c37b578c4fe1aa666b0a6b0bc7c9 (DEFAULT_TABLE table=default.Table_423319f67e734a64a731bf33ef868beb [id=44f7bf1df366476292532972544fe690]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:40.150274 18475 raft_consensus.cc:1261] T 0b4d9e860d7c45fd91d7a566605c9283 P 2efb542aa55e439b99ccb98d1c5c884a [term 2 FOLLOWER]: Refusing update from remote peer 5e1c6397f09747f091534faf93542710: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240430 20:11:40.150334 18619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acd23f0a63d5448aaa7ed78422e67bad" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:40.151542 18696 leader_election.cc:302] T acd23f0a63d5448aaa7ed78422e67bad P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710
I20240430 20:11:40.151940 20019 raft_consensus.cc:2726] T acd23f0a63d5448aaa7ed78422e67bad P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:40.152818 18754 raft_consensus.cc:3037] T fcc9698abda4491b98e1e879589913a8 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:40.154351 18985 tablet_bootstrap.cc:492] T 32c823d1700949ab8b22adcd83b7bfd1 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:40.159979 18988 tablet_bootstrap.cc:654] T 4e69ca12bd10494fb013614741bc166c P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:40.163878 18988 tablet_bootstrap.cc:492] T 4e69ca12bd10494fb013614741bc166c P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:40.164072 18988 ts_tablet_manager.cc:1367] T 4e69ca12bd10494fb013614741bc166c P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.057s user 0.000s sys 0.004s
I20240430 20:11:40.164310 19489 ts_tablet_manager.cc:1888] T 802336602d8f4ac4bd7e0872d6f3dae9 P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:40.164484 19489 log.cc:1196] T 802336602d8f4ac4bd7e0872d6f3dae9 P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/802336602d8f4ac4bd7e0872d6f3dae9
I20240430 20:11:40.164988 19489 ts_tablet_manager.cc:1909] T 802336602d8f4ac4bd7e0872d6f3dae9 P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:40.165163 18988 raft_consensus.cc:348] T 4e69ca12bd10494fb013614741bc166c P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:40.165395 18988 raft_consensus.cc:374] T 4e69ca12bd10494fb013614741bc166c P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:40.165527 18988 raft_consensus.cc:729] T 4e69ca12bd10494fb013614741bc166c P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:40.165807 18988 consensus_queue.cc:260] T 4e69ca12bd10494fb013614741bc166c P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:40.166612 18988 ts_tablet_manager.cc:1398] T 4e69ca12bd10494fb013614741bc166c P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20240430 20:11:40.166960 18988 tablet_bootstrap.cc:492] T 255c13ad8fae4ea4aa8e7b7aabb7602e P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:40.165395 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 83) took 8578 ms. Trace:
I20240430 20:11:40.169485 19489 rpcz_store.cc:276] 0430 20:11:31.586655 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:31.586959 (+ 304us) service_pool.cc:225] Handling call
0430 20:11:40.143437 (+8556478us) ts_tablet_manager.cc:1120] Deleting tablet 802336602d8f4ac4bd7e0872d6f3dae9
0430 20:11:40.143455 (+ 18us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:40.155344 (+ 11889us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:40.164163 (+ 8819us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:40.165310 (+ 1147us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":8556017}
I20240430 20:11:40.169875 19489 tablet_replica.cc:330] T 26ed12f498644d15bf45e884dcad172e P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:40.170061 19489 raft_consensus.cc:2229] T 26ed12f498644d15bf45e884dcad172e P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:40.170282 19489 raft_consensus.cc:2258] T 26ed12f498644d15bf45e884dcad172e P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:40.169404 18988 tablet_bootstrap.cc:654] T 255c13ad8fae4ea4aa8e7b7aabb7602e P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:40.171566 20071 consensus_queue.cc:1035] T fcc9698abda4491b98e1e879589913a8 P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.015s
I20240430 20:11:40.171766 19001 catalog_manager.cc:4830] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): tablet 802336602d8f4ac4bd7e0872d6f3dae9 (table default.Table_d997b723a0dd48f6bc2d42c9bfd6bc14 [id=7737ec4807974640aa8556cb1c3ca3ef]) successfully deleted
I20240430 20:11:40.172132 20105 consensus_queue.cc:1035] T 372b135dd01f44f98cd713ca8857ba61 P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.022s
I20240430 20:11:40.173070 18474 raft_consensus.cc:1261] T 294644d7a16b4ca2bbb6d106bcb8c7ff P 2efb542aa55e439b99ccb98d1c5c884a [term 2 FOLLOWER]: Refusing update from remote peer 5e1c6397f09747f091534faf93542710: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240430 20:11:40.173398 18478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "802336602d8f4ac4bd7e0872d6f3dae9" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:40.173842 18985 ts_tablet_manager.cc:1367] T 32c823d1700949ab8b22adcd83b7bfd1 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.155s user 0.035s sys 0.031s
I20240430 20:11:40.174814 19489 ts_tablet_manager.cc:1875] T 26ed12f498644d15bf45e884dcad172e P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:40.175309 18985 raft_consensus.cc:348] T 32c823d1700949ab8b22adcd83b7bfd1 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:40.175611 18985 raft_consensus.cc:374] T 32c823d1700949ab8b22adcd83b7bfd1 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:40.175720 18985 raft_consensus.cc:729] T 32c823d1700949ab8b22adcd83b7bfd1 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:40.175997 18985 consensus_queue.cc:260] T 32c823d1700949ab8b22adcd83b7bfd1 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:40.176694 18985 ts_tablet_manager.cc:1398] T 32c823d1700949ab8b22adcd83b7bfd1 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240430 20:11:40.177323 18699 leader_election.cc:302] T 802336602d8f4ac4bd7e0872d6f3dae9 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710
W20240430 20:11:40.177544 18699 raft_consensus.cc:2670] T 802336602d8f4ac4bd7e0872d6f3dae9 P 3c51c33d5616482cafb68e26688820f4: unable to run election callback: Service unavailable: Thread pool token was shut down
I20240430 20:11:40.179044 18985 tablet_bootstrap.cc:492] T 3fd0e8c360734dcbb75dc2958b70c2b8 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:40.182273 19489 ts_tablet_manager.cc:1888] T 26ed12f498644d15bf45e884dcad172e P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:40.182459 19489 log.cc:1196] T 26ed12f498644d15bf45e884dcad172e P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/26ed12f498644d15bf45e884dcad172e
I20240430 20:11:40.185308 19495 pending_rounds.cc:62] T e6eac92dfa5f4456bc49571878f0f8dd P 2efb542aa55e439b99ccb98d1c5c884a: Trying to abort 1 pending ops.
I20240430 20:11:40.189062 19861 consensus_queue.cc:1035] T 0b4d9e860d7c45fd91d7a566605c9283 P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.039s
I20240430 20:11:40.189479 20047 consensus_queue.cc:1035] T 0b4d9e860d7c45fd91d7a566605c9283 P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240430 20:11:40.192013 18762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9e0659564894b0e8cb7f600bb07a085" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:40.196075 19489 ts_tablet_manager.cc:1909] T 26ed12f498644d15bf45e884dcad172e P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:40.196512 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 84) took 8553 ms. Trace:
I20240430 20:11:40.196648 19489 rpcz_store.cc:276] 0430 20:11:31.643029 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:31.643088 (+ 59us) service_pool.cc:225] Handling call
0430 20:11:40.169784 (+8526696us) ts_tablet_manager.cc:1120] Deleting tablet 26ed12f498644d15bf45e884dcad172e
0430 20:11:40.169794 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:40.178851 (+ 9057us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:40.182102 (+ 3251us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:40.196463 (+ 14361us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":8526488}
I20240430 20:11:40.197974 19001 catalog_manager.cc:4830] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): tablet 26ed12f498644d15bf45e884dcad172e (table default.Table_3f42877693b04aaa837c89b8983d5a72 [id=6d499a625cee4df2a4f40ee189cc6220]) successfully deleted
I20240430 20:11:40.198259 19023 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_c5c549bbf87d46c7886dad5f93086cd5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:40.198944 19023 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c5c549bbf87d46c7886dad5f93086cd5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:40.199326 18988 tablet_bootstrap.cc:492] T 255c13ad8fae4ea4aa8e7b7aabb7602e P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:40.199522 18988 ts_tablet_manager.cc:1367] T 255c13ad8fae4ea4aa8e7b7aabb7602e P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.033s user 0.007s sys 0.002s
I20240430 20:11:40.204135 19471 ts_tablet_manager.cc:1888] T dd9b893d48684cfea7c242b93ecba01f P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:40.206681 18983 tablet_bootstrap.cc:654] T 0a9066d895744c5bb7d9532b2940603c P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:40.209882 20060 leader_election.cc:288] T a9e0659564894b0e8cb7f600bb07a085 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
W20240430 20:11:40.210253 18557 leader_election.cc:341] T a9e0659564894b0e8cb7f600bb07a085 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:40.212824 18988 raft_consensus.cc:348] T 255c13ad8fae4ea4aa8e7b7aabb7602e P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:40.213099 18988 raft_consensus.cc:374] T 255c13ad8fae4ea4aa8e7b7aabb7602e P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:40.213173 18988 raft_consensus.cc:729] T 255c13ad8fae4ea4aa8e7b7aabb7602e P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:40.213431 18988 consensus_queue.cc:260] T 255c13ad8fae4ea4aa8e7b7aabb7602e P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:40.214146 18988 ts_tablet_manager.cc:1398] T 255c13ad8fae4ea4aa8e7b7aabb7602e P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.015s user 0.001s sys 0.001s
I20240430 20:11:40.215947 19489 tablet_replica.cc:330] T 7dfdecba959543a795dfcd3eeda29e58 P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:40.216240 19489 raft_consensus.cc:2229] T 7dfdecba959543a795dfcd3eeda29e58 P 3c51c33d5616482cafb68e26688820f4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:40.216298 18758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd40967ab4a04001819e16c06a6c4300" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:40.216454 19489 raft_consensus.cc:2258] T 7dfdecba959543a795dfcd3eeda29e58 P 3c51c33d5616482cafb68e26688820f4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:40.216862 18988 tablet_bootstrap.cc:492] T 0a9066d895744c5bb7d9532b2940603c P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:40.221176 20024 leader_election.cc:288] T fd40967ab4a04001819e16c06a6c4300 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:40.223117 18988 tablet_bootstrap.cc:654] T 0a9066d895744c5bb7d9532b2940603c P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
W20240430 20:11:40.223594 18415 consensus_peers.cc:485] T dd9b893d48684cfea7c242b93ecba01f P 2efb542aa55e439b99ccb98d1c5c884a -> Peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Couldn't send request to peer 5e1c6397f09747f091534faf93542710. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240430 20:11:40.223594 20071 consensus_queue.cc:1035] T fcc9698abda4491b98e1e879589913a8 P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.013s
I20240430 20:11:40.227806 19489 ts_tablet_manager.cc:1875] T 7dfdecba959543a795dfcd3eeda29e58 P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:40.229144 19051 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet acd23f0a63d5448aaa7ed78422e67bad (table default.Table_972f13be4dca4cd19b9a04534dc59f51 [id=69f37db4de634a3185613e40013e0ed7]). Expected version 1 got 0
W20240430 20:11:40.231253 18557 leader_election.cc:341] T fd40967ab4a04001819e16c06a6c4300 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:40.231472 18983 tablet_bootstrap.cc:492] T 0a9066d895744c5bb7d9532b2940603c P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:40.231868 18587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4d6c37b578c4fe1aa666b0a6b0bc7c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:40.237442 18988 tablet_bootstrap.cc:492] T 0a9066d895744c5bb7d9532b2940603c P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:40.237617 18988 ts_tablet_manager.cc:1367] T 0a9066d895744c5bb7d9532b2940603c P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.002s
I20240430 20:11:40.228596 20040 consensus_queue.cc:1035] T 294644d7a16b4ca2bbb6d106bcb8c7ff P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.038s
I20240430 20:11:40.238468 18988 raft_consensus.cc:348] T 0a9066d895744c5bb7d9532b2940603c P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:40.238704 18988 raft_consensus.cc:374] T 0a9066d895744c5bb7d9532b2940603c P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:40.238799 18988 raft_consensus.cc:729] T 0a9066d895744c5bb7d9532b2940603c P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:40.239056 18988 consensus_queue.cc:260] T 0a9066d895744c5bb7d9532b2940603c P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:40.239645 18988 ts_tablet_manager.cc:1398] T 0a9066d895744c5bb7d9532b2940603c P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240430 20:11:40.239955 18988 tablet_bootstrap.cc:492] T fd40967ab4a04001819e16c06a6c4300 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:40.241736 18988 tablet_bootstrap.cc:654] T fd40967ab4a04001819e16c06a6c4300 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:40.243037 19051 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 2e79df8744e74b5991a79ce978ff0202 (table default.Table_276bbdc7cfe542d1a2936485a2aad1ec [id=f146a15e6e8a4acbb4c734c1c863da57]). Expected version 1 got 0
I20240430 20:11:40.243867 19051 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 22fef8bb6b034db098c3372ae992f7a7 (table default.Table_4440df38610340758047115845b42b95 [id=7d186ca20c094cc19aef792fe1c094e9]). Expected version 1 got 0
I20240430 20:11:40.244299 19051 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 12c6e131fa904d23865b8673b58a7e33 (table default.Table_a9dc8fc6a4aa46dc914d0e327a4de554 [id=19f4e6decedf4ee59fccc0a24a05cedd]). Expected version 2 got 0
I20240430 20:11:40.245229 19051 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 66dd767bd3f8494f94061fb89e5e5680 (table default.Table_9c1e6182953140bb9cd594f08458c7d9 [id=5227206f44d04fa89b4842b899f33afb]). Expected version 1 got 0
I20240430 20:11:40.245889 19051 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet f5eaf2413e1d4e69b56fae61f768f0ce (table default.Table_1280cfe32d5b41abb4d3eb1bd70641bf [id=8489b872074a42b6abb713e420968141]). Expected version 1 got 0
I20240430 20:11:40.247068 19051 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet c9e7660eccc448dbb99d135f82bfd261 (table default.Table_685a26737bb046fb85ced15df7ae3423 [id=51d177e8498848b38c43e3f8d11f092c]). Expected version 1 got 0
I20240430 20:11:40.249049 19471 log.cc:1196] T dd9b893d48684cfea7c242b93ecba01f P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/dd9b893d48684cfea7c242b93ecba01f
I20240430 20:11:40.236871 19495 pending_rounds.cc:68] T e6eac92dfa5f4456bc49571878f0f8dd P 2efb542aa55e439b99ccb98d1c5c884a: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7022624355523739648 op_type: NO_OP noop_request { }
I20240430 20:11:40.249495 19495 raft_consensus.cc:2866] T e6eac92dfa5f4456bc49571878f0f8dd P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240430 20:11:40.249842 19495 raft_consensus.cc:2258] T e6eac92dfa5f4456bc49571878f0f8dd P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:40.250931 19051 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet a05e32a33aa7488eb287cc6314c09c44 (table default.Table_2d1cfc78363a47aa8dba7ef7d80f8c9a [id=955d833ff8f14bf6957abadf9ef669e2]). Expected version 2 got 0
I20240430 20:11:40.251334 19051 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 488d0e609b224674932f890939664df2 (table default.Table_58f35a2120874472978ca0e5f816e86c [id=ff18001c6d814afc8c1768d0c84a6bac]). Expected version 1 got 0
I20240430 20:11:40.251646 19051 catalog_manager.cc:5484] T 5809e7c9c2204ba293426c24129fb16e P 3c51c33d5616482cafb68e26688820f4 reported cstate change: term changed from 1 to 2, leader changed from 5e1c6397f09747f091534faf93542710 (127.17.177.2) to 3c51c33d5616482cafb68e26688820f4 (127.17.177.3). New cstate: current_term: 2 leader_uuid: "3c51c33d5616482cafb68e26688820f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } health_report { overall_health: HEALTHY } } }
I20240430 20:11:40.252516 19051 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet a59ef6c545dd41a28265a4c1b8ce10e0 (table default.Table_07e40af8f1c546048975c8edf88170d7 [id=86df5d161d8641c89070e7eed9a39360]). Expected version 2 got 0
I20240430 20:11:40.253383 18983 ts_tablet_manager.cc:1367] T 0a9066d895744c5bb7d9532b2940603c P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.152s user 0.004s sys 0.004s
I20240430 20:11:40.252946 19023 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_999c55e7bef74339b7b3ab1142599447" } new_table_name: "default.Table_763f0800e8bb4fa69976886416250f42" modify_external_catalogs: true
I20240430 20:11:40.253998 18985 tablet_bootstrap.cc:654] T 3fd0e8c360734dcbb75dc2958b70c2b8 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:40.255949 19051 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet fd40967ab4a04001819e16c06a6c4300 (table default.Table_4130caf534004edfb413056d7f6cfd52 [id=da9ad991953a40c4b77b31317597ad1d]). Expected version 4 got 0
I20240430 20:11:40.256928 19051 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet a482f4622f774d60b4efd6827b8be7d7 (table default.Table_e686cdfcbcc046749fa5b51578a64794 [id=c3dc9f05316d46d6a3ef4d03f40877ec]). Expected version 1 got 0
I20240430 20:11:40.260383 19051 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet ed0c37a71ede4bfc9eb5ad1a4365f25e (table default.Table_a2e85efc6b454524b0b122102ec07002 [id=86d795a82e674d859104741ebd04f4e8]). Expected version 1 got 0
I20240430 20:11:40.267071 19051 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet b22aaedb0a6c4fd9ad0ea8fa019001e1 (table default.Table_730a215d7af74a5bb753adca39dea676 [id=a5d580cffa2447d78fc9f2c8e31264f1]). Expected version 5 got 0
I20240430 20:11:40.267987 18988 tablet_bootstrap.cc:492] T fd40967ab4a04001819e16c06a6c4300 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:40.268240 18988 ts_tablet_manager.cc:1367] T fd40967ab4a04001819e16c06a6c4300 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.028s user 0.005s sys 0.000s
I20240430 20:11:40.269129 19471 ts_tablet_manager.cc:1909] T dd9b893d48684cfea7c242b93ecba01f P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:40.269165 18988 raft_consensus.cc:348] T fd40967ab4a04001819e16c06a6c4300 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:40.269560 18988 raft_consensus.cc:374] T fd40967ab4a04001819e16c06a6c4300 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:40.269681 18988 raft_consensus.cc:729] T fd40967ab4a04001819e16c06a6c4300 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:40.270032 18988 consensus_queue.cc:260] T fd40967ab4a04001819e16c06a6c4300 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:40.270114 18983 raft_consensus.cc:348] T 0a9066d895744c5bb7d9532b2940603c P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:40.270319 18983 raft_consensus.cc:374] T 0a9066d895744c5bb7d9532b2940603c P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:40.270416 18983 raft_consensus.cc:729] T 0a9066d895744c5bb7d9532b2940603c P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:40.270685 18983 consensus_queue.cc:260] T 0a9066d895744c5bb7d9532b2940603c P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:40.271008 18988 ts_tablet_manager.cc:1398] T fd40967ab4a04001819e16c06a6c4300 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240430 20:11:40.271267 18983 ts_tablet_manager.cc:1398] T 0a9066d895744c5bb7d9532b2940603c P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.018s user 0.002s sys 0.000s
I20240430 20:11:40.271517 18988 tablet_bootstrap.cc:492] T f5546f9edeea44d7a6c5cb954f5157ba P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:40.272595 19051 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 12795670f21447a8964fe1efc6e92398 (table default.Table_421069f84cb844368066c93102a8faf8 [id=d5cea90cbc2c48df9d8b80da1eccc949]). Expected version 2 got 0
I20240430 20:11:40.272816 19495 ts_tablet_manager.cc:1875] T e6eac92dfa5f4456bc49571878f0f8dd P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:40.273995 18988 tablet_bootstrap.cc:654] T f5546f9edeea44d7a6c5cb954f5157ba P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:40.274289 19051 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 62732af95c6a4c749c5b726f0460dc72 (table default.Table_be2ebe77f859489f9cd3e912cb1edc8b [id=2d44ec94131148048021b2ee6366ba30]). Expected version 2 got 0
I20240430 20:11:40.275012 19051 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 9c73d0a6bc414d3b9900570848d19a51 (table default.Table_49a012a7c4c8457fadb0a8f0fc9654c6 [id=eecb2f8eeef547c5ab1abe6a2430e592]). Expected version 1 got 0
I20240430 20:11:40.275494 19051 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 99203cc522de424184c6f97453f166fd (table default.Table_cb2a4b9ceaf4431facfff2c9bcf94c5d [id=3f8690a437ae49a3beb009fa7899d5d0]). Expected version 1 got 0
I20240430 20:11:40.276701 19051 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 8e91920e7e36418d8c27d1f0bfd9b981 (table default.Table_e860191f401c434e8cf0085ef41952b9 [id=f393bcc9669c4ea985a2d40deb161f0c]). Expected version 1 got 0
I20240430 20:11:40.286319 20103 raft_consensus.cc:482] T 5d09baed576e4136bf6d756ca39d08b2 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:40.286517 20103 raft_consensus.cc:504] T 5d09baed576e4136bf6d756ca39d08b2 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:40.287117 18474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd40967ab4a04001819e16c06a6c4300" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:40.287309 20103 leader_election.cc:288] T 5d09baed576e4136bf6d756ca39d08b2 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
W20240430 20:11:40.287590 18558 leader_election.cc:341] T a9e0659564894b0e8cb7f600bb07a085 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Not found: Tablet not found: a9e0659564894b0e8cb7f600bb07a085
I20240430 20:11:40.287617 20197 consensus_queue.cc:1035] T 24dcc20832804229a2664873bf6ddaa7 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:40.287708 18558 leader_election.cc:302] T a9e0659564894b0e8cb7f600bb07a085 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:40.287322 18474 raft_consensus.cc:2454] T fd40967ab4a04001819e16c06a6c4300 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 0.
I20240430 20:11:40.288664 18758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d09baed576e4136bf6d756ca39d08b2" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:40.289350 18619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d09baed576e4136bf6d756ca39d08b2" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:40.290520 18558 leader_election.cc:302] T fd40967ab4a04001819e16c06a6c4300 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters: 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:40.290890 20047 raft_consensus.cc:2726] T a9e0659564894b0e8cb7f600bb07a085 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:40.291085 20047 raft_consensus.cc:2781] T fd40967ab4a04001819e16c06a6c4300 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:40.291146 20047 raft_consensus.cc:482] T fd40967ab4a04001819e16c06a6c4300 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:40.291204 20047 raft_consensus.cc:3037] T fd40967ab4a04001819e16c06a6c4300 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:40.293802 20047 raft_consensus.cc:504] T fd40967ab4a04001819e16c06a6c4300 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
W20240430 20:11:40.294544 18417 leader_election.cc:341] T 5d09baed576e4136bf6d756ca39d08b2 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:40.294775 20047 leader_election.cc:288] T fd40967ab4a04001819e16c06a6c4300 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Requested vote from peers 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:40.295814 19051 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 2da900b6bbfa4fa99fe8494fd2e68da8 (table default.Table_19a91a7f17ff456ebb145e8dd89c0940 [id=67283dd3995a44c6a7a7bc8722e4f955]). Expected version 1 got 0
I20240430 20:11:40.296135 18758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd40967ab4a04001819e16c06a6c4300" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:40.296343 19051 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 78ba48dadef548708153878810cc46f8 (table default.Table_b9983e101a6547649eb44ee04b5e1147 [id=29769be2b9ec4ecba9de5c0a65692f85]). Expected version 1 got 0
I20240430 20:11:40.296417 18758 raft_consensus.cc:3037] T fd40967ab4a04001819e16c06a6c4300 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:40.296613 19051 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 6e754af64cb64fd2a082adae7d5f24da (table default.Table_208b12b8e9bd4f3e8bc9c6d422ada649 [id=1df4e95702cd46f59ab21b101049a9ed]). Expected version 1 got 0
I20240430 20:11:40.296753 19051 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 34130f2820064fa0898ef1f9d7de1c6d (table default.Table_69892a8264a4477db48ee6248758e046 [id=5a1c4a7be2fc465fac89a50854eb3322]). Expected version 3 got 0
I20240430 20:11:40.297005 19051 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet ae20f44df50343289b0a512a4ea53692 (table default.Table_1844f6b1bda84812825384e14a2fd94e [id=71da9acc3cff4d95bbac4a686b708fd5]). Expected version 1 got 0
I20240430 20:11:40.297163 19051 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 3ec3993042f5498b818bdf32d5087928 (table default.Table_ba82d0fed0264b818de8278e1d142e5d [id=27217d463b9c40ec926a1862a31f1f24]). Expected version 1 got 0
I20240430 20:11:40.297614 19051 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet c6e3d09402574f1e8fe9e4160c70a3ee (table default.Table_656db97355f64d7c8aa29b1352dfe9c8 [id=450f5d1c7e084a1bb88d035c0ccfcf82]). Expected version 1 got 0
I20240430 20:11:40.297827 19051 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 46251c08277e498da04a41d009310c62 (table default.Table_4e56923d440944978f0aacc140a0dd8e [id=b1312a6cd1a54003b1db727a2a9d125d]). Expected version 1 got 0
I20240430 20:11:40.298013 19051 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 32c823d1700949ab8b22adcd83b7bfd1 (table default.Table_e8c73c89623d4b49b8d7f0a01d729bdd [id=5cd10682ce52414cb5a103e4433a6a0d]). Expected version 1 got 0
I20240430 20:11:40.298391 19051 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet acf3b6fead41483a82c43dc0286fab7c (table default.Table_01450a1e661d450a8cbe185e16e8250c [id=ce9911e0a91949b1997ee47321800255]). Expected version 1 got 0
I20240430 20:11:40.298738 19051 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet fbdeb4714b8c4b93ab72822c43329aae (table default.Table_c12ca6938d424fbfa2e1df5cdb00fdbc [id=0fb93f36bc17408ea56bc4cafcfd8580]). Expected version 1 got 0
I20240430 20:11:40.298998 19051 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 76d6e9d547ca4a848eae852944b1d117 (table default.Table_104ae015924a48fd8d27220792f07470 [id=ff86c0357c164dd0991098c416087773]). Expected version 2 got 0
W20240430 20:11:40.298488 18415 leader_election.cc:341] T 5d09baed576e4136bf6d756ca39d08b2 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:40.300454 18415 leader_election.cc:302] T 5d09baed576e4136bf6d756ca39d08b2 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a; no voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710
I20240430 20:11:40.309052 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 78) took 9041 ms. Trace:
I20240430 20:11:40.309204 19471 rpcz_store.cc:276] 0430 20:11:31.268097 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:31.268162 (+ 65us) service_pool.cc:225] Handling call
0430 20:11:39.696249 (+8428087us) ts_tablet_manager.cc:1120] Deleting tablet dd9b893d48684cfea7c242b93ecba01f
0430 20:11:39.696260 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:40.052271 (+356011us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:40.146725 (+ 94454us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:40.309013 (+162288us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":31029,"tablet-delete.queue_time_us":8427780}
I20240430 20:11:40.309860 18999 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet dd9b893d48684cfea7c242b93ecba01f (table default.Table_4bce1dae3391497ebea5f1b536700d62 [id=8f0cb0ef65d84fb8964209dd501c0b5c]) successfully deleted
I20240430 20:11:40.312798 19029 catalog_manager.cc:6029] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: default.Table_a9dc8fc6a4aa46dc914d0e327a4de554 [id=19f4e6decedf4ee59fccc0a24a05cedd] alter complete (version 2)
I20240430 20:11:40.314126 19029 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_c06225c5a599482dab4f418fd13c593f" } new_table_name: "default.Table_a4ea055447464ca6af32fe80bab34c42" modify_external_catalogs: true
W20240430 20:11:40.321113 19028 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ebf74809b52f47f3b0ed21b6ac0f541a consensus state has no leader
I20240430 20:11:40.324343 20071 raft_consensus.cc:2726] T 5d09baed576e4136bf6d756ca39d08b2 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:40.321098 18478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd40967ab4a04001819e16c06a6c4300" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:40.324703 19033 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: Sending DeleteTablet for 3 replicas of tablet 898b2c91adf34545808df9c786e218c9
I20240430 20:11:40.324775 18478 raft_consensus.cc:3037] T fd40967ab4a04001819e16c06a6c4300 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:40.325556 19033 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_64d14df4e12146ecb85239e7ca03cf6e" } modify_external_catalogs: true
I20240430 20:11:40.325819 19033 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_64d14df4e12146ecb85239e7ca03cf6e" } modify_external_catalogs: true
I20240430 20:11:40.329577 18587 tablet_service.cc:1514] Processing DeleteTablet for tablet 898b2c91adf34545808df9c786e218c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:40 UTC) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:40.339334 18478 raft_consensus.cc:2454] T fd40967ab4a04001819e16c06a6c4300 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
I20240430 20:11:40.339336 18450 tablet_service.cc:1514] Processing DeleteTablet for tablet 898b2c91adf34545808df9c786e218c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:40 UTC) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:40.342829 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 80) took 8880 ms. Trace:
I20240430 20:11:40.343060 19471 rpcz_store.cc:276] 0430 20:11:31.462171 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:31.462231 (+ 60us) service_pool.cc:225] Handling call
0430 20:11:40.310243 (+8848012us) ts_tablet_manager.cc:1120] Deleting tablet 5bd0362ff41b4faca0205d136d596737
0430 20:11:40.342623 (+ 32380us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:40.342797 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":8847770}
W20240430 20:11:40.343807 18999 catalog_manager.cc:4810] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): delete failed for tablet 5bd0362ff41b4faca0205d136d596737 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5bd0362ff41b4faca0205d136d596737 already in progress: creating tablet
I20240430 20:11:40.344324 19471 tablet_replica.cc:330] T 802336602d8f4ac4bd7e0872d6f3dae9 P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:40.344648 19471 raft_consensus.cc:2229] T 802336602d8f4ac4bd7e0872d6f3dae9 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:40.344938 19471 raft_consensus.cc:2258] T 802336602d8f4ac4bd7e0872d6f3dae9 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:40.345496 18983 tablet_bootstrap.cc:492] T acf3b6fead41483a82c43dc0286fab7c P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:40.347590 18983 tablet_bootstrap.cc:654] T acf3b6fead41483a82c43dc0286fab7c P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:40.349409 18617 raft_consensus.cc:1261] T 372b135dd01f44f98cd713ca8857ba61 P 5e1c6397f09747f091534faf93542710 [term 2 FOLLOWER]: Refusing update from remote peer 2efb542aa55e439b99ccb98d1c5c884a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
W20240430 20:11:40.350785 18696 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.177.2:46027, user_credentials={real_user=slave}} blocked reactor thread for 48157.9us
I20240430 20:11:40.350971 20197 raft_consensus.cc:482] T 0a9066d895744c5bb7d9532b2940603c P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:40.352105 20197 raft_consensus.cc:504] T 0a9066d895744c5bb7d9532b2940603c P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:40.352730 20197 leader_election.cc:288] T 0a9066d895744c5bb7d9532b2940603c P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:40.354955 18615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a9066d895744c5bb7d9532b2940603c" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:40.355253 18615 raft_consensus.cc:2454] T 0a9066d895744c5bb7d9532b2940603c P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:40.357632 18744 tablet_service.cc:1514] Processing DeleteTablet for tablet 898b2c91adf34545808df9c786e218c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:40 UTC) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:40.357867 18760 raft_consensus.cc:1261] T 75ebf051383e42529dcb5f7dd0770a4b P 3c51c33d5616482cafb68e26688820f4 [term 2 FOLLOWER]: Refusing update from remote peer 2efb542aa55e439b99ccb98d1c5c884a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20240430 20:11:40.358274 18696 leader_election.cc:302] T 0a9066d895744c5bb7d9532b2940603c P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:40.359488 20234 consensus_queue.cc:1035] T 75ebf051383e42529dcb5f7dd0770a4b P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:40.360545 20197 raft_consensus.cc:2781] T 0a9066d895744c5bb7d9532b2940603c P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:40.360661 20197 raft_consensus.cc:482] T 0a9066d895744c5bb7d9532b2940603c P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:40.360781 20197 raft_consensus.cc:3037] T 0a9066d895744c5bb7d9532b2940603c P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:40.364022 20197 raft_consensus.cc:504] T 0a9066d895744c5bb7d9532b2940603c P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:40.364715 18758 raft_consensus.cc:2454] T fd40967ab4a04001819e16c06a6c4300 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
I20240430 20:11:40.365614 18557 leader_election.cc:302] T fd40967ab4a04001819e16c06a6c4300 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:40.366071 20047 consensus_queue.cc:1035] T c9e7660eccc448dbb99d135f82bfd261 P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:40.367224 20047 raft_consensus.cc:2781] T fd40967ab4a04001819e16c06a6c4300 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:40.367518 20047 raft_consensus.cc:686] T fd40967ab4a04001819e16c06a6c4300 P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Becoming Leader. State: Replica: 5e1c6397f09747f091534faf93542710, State: Running, Role: LEADER
I20240430 20:11:40.368324 20047 consensus_queue.cc:237] T fd40967ab4a04001819e16c06a6c4300 P 5e1c6397f09747f091534faf93542710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:40.370896 18608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a9066d895744c5bb7d9532b2940603c" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:40.371181 18608 raft_consensus.cc:3037] T 0a9066d895744c5bb7d9532b2940603c P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:40.374895 20197 leader_election.cc:288] T 0a9066d895744c5bb7d9532b2940603c P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Requested vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:40.388373 18608 raft_consensus.cc:2454] T 0a9066d895744c5bb7d9532b2940603c P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:40.388967 18696 leader_election.cc:302] T 0a9066d895744c5bb7d9532b2940603c P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:40.389520 20197 raft_consensus.cc:2781] T 0a9066d895744c5bb7d9532b2940603c P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:40.389711 20197 raft_consensus.cc:686] T 0a9066d895744c5bb7d9532b2940603c P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Becoming Leader. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Running, Role: LEADER
I20240430 20:11:40.390019 20197 consensus_queue.cc:237] T 0a9066d895744c5bb7d9532b2940603c P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:40.395117 19026 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet a663030d37614053a7c0369841403422
W20240430 20:11:40.395237 19048 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0368d37dd9e2452982fb81f539c6805b consensus state has no leader
I20240430 20:11:40.395478 19026 master_service.cc:946] ReplaceTablet: received request to replace tablet 5809e7c9c2204ba293426c24129fb16e from {username='slave'} at 127.0.0.1:51664
I20240430 20:11:40.395679 19048 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_fa64149f685c4784bda92ae4f578dd92" } modify_external_catalogs: true
I20240430 20:11:40.395875 19048 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_fa64149f685c4784bda92ae4f578dd92" } modify_external_catalogs: true
W20240430 20:11:40.396153 19024 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a0baee91f5b44945b595d0e234af018e consensus state has no leader
I20240430 20:11:40.396307 19024 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:51664 (request call id 194) took 1678 ms. Trace:
I20240430 20:11:40.396420 19024 rpcz_store.cc:276] 0430 20:11:38.717404 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:39.757954 (+1040550us) service_pool.cc:225] Handling call
0430 20:11:39.758301 (+ 347us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0430 20:11:39.758550 (+ 249us) catalog_manager.cc:3613] Apply alter schema
0430 20:11:39.780935 (+ 22385us) catalog_manager.cc:3636] Acquired catalog manager lock
0430 20:11:39.781042 (+ 107us) catalog_manager.cc:3783] Updating metadata on disk
0430 20:11:39.796982 (+ 15940us) catalog_manager.cc:3814] Committing alterations to in-memory state
0430 20:11:40.396270 (+599288us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:39.781600 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:39.781704 (+ 104us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:39.781710 (+ 6us) write_op.cc:427] Acquired schema lock
0430 20:11:39.781713 (+ 3us) tablet.cc:660] Decoding operations
0430 20:11:39.781810 (+ 97us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:39.781820 (+ 10us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:39.781825 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0430 20:11:39.781897 (+ 72us) tablet.cc:699] Row locks acquired
0430 20:11:39.781901 (+ 4us) write_op.cc:260] PREPARE: finished
0430 20:11:39.781939 (+ 38us) write_op.cc:270] Start()
0430 20:11:39.781971 (+ 32us) write_op.cc:276] Timestamp: P: 1714507899781921 usec, L: 0
0430 20:11:39.781975 (+ 4us) op_driver.cc:331] REPLICATION: starting
0430 20:11:39.782159 (+ 184us) log.cc:842] Serialized 405 byte log entry
0430 20:11:39.796195 (+ 14036us) op_driver.cc:444] REPLICATION: finished
0430 20:11:39.796485 (+ 290us) write_op.cc:301] APPLY: starting
0430 20:11:39.796501 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:39.796546 (+ 45us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:39.796577 (+ 31us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:39.796627 (+ 50us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:39.796665 (+ 38us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0430 20:11:39.796673 (+ 8us) write_op.cc:312] APPLY: finished
0430 20:11:39.796743 (+ 70us) log.cc:842] Serialized 23 byte log entry
0430 20:11:39.796800 (+ 57us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:39.796814 (+ 14us) write_op.cc:446] Released schema lock
0430 20:11:39.796820 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:39.796865 (+ 45us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:39.796869 (+ 4us) write_op.cc:446] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":230,"apply.run_cpu_time_us":434,"apply.run_wall_time_us":434,"num_ops":1,"prepare.queue_time_us":49,"prepare.run_cpu_time_us":649,"prepare.run_wall_time_us":649,"replication_time_us":14200}]]}
I20240430 20:11:40.397145 19024 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_423319f67e734a64a731bf33ef868beb" } modify_external_catalogs: true
I20240430 20:11:40.397331 19024 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_423319f67e734a64a731bf33ef868beb" } modify_external_catalogs: true
I20240430 20:11:40.401130 19489 ts_tablet_manager.cc:1888] T 7dfdecba959543a795dfcd3eeda29e58 P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.2
I20240430 20:11:40.401309 19489 log.cc:1196] T 7dfdecba959543a795dfcd3eeda29e58 P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/7dfdecba959543a795dfcd3eeda29e58
I20240430 20:11:40.401851 19489 ts_tablet_manager.cc:1909] T 7dfdecba959543a795dfcd3eeda29e58 P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:40.402549 19471 ts_tablet_manager.cc:1875] T 802336602d8f4ac4bd7e0872d6f3dae9 P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240430 20:11:40.412286 19023 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 331c031d44854d679a27d10d9151d0ea consensus state has no leader
I20240430 20:11:40.416087 19495 ts_tablet_manager.cc:1888] T e6eac92dfa5f4456bc49571878f0f8dd P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:40.416236 19495 log.cc:1196] T e6eac92dfa5f4456bc49571878f0f8dd P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/e6eac92dfa5f4456bc49571878f0f8dd
I20240430 20:11:40.421093 20103 raft_consensus.cc:482] T 0eb4f3e7ece6496293e01b63620d644b P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:40.421248 20103 raft_consensus.cc:504] T 0eb4f3e7ece6496293e01b63620d644b P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:40.422294 20103 leader_election.cc:288] T 0eb4f3e7ece6496293e01b63620d644b P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:40.422441 19023 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_dd1d113759284183905a803b2733436d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240430 20:11:40.422669 18756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0eb4f3e7ece6496293e01b63620d644b" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
W20240430 20:11:40.423187 19023 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dd1d113759284183905a803b2733436d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240430 20:11:40.423178 18417 leader_election.cc:341] T 0eb4f3e7ece6496293e01b63620d644b P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
W20240430 20:11:40.425509 19029 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f7d3dea34cf54fe2b949d1b7ca68577a consensus state has no leader
I20240430 20:11:40.425979 19029 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_5f50cb33cf774b86b2380ee4a569dac9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:40.426812 19029 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5f50cb33cf774b86b2380ee4a569dac9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:40.428596 19024 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: Sending DeleteTablet for 3 replicas of tablet d4d6c37b578c4fe1aa666b0a6b0bc7c9
I20240430 20:11:40.429217 18726 tablet_service.cc:1514] Processing DeleteTablet for tablet 26ed12f498644d15bf45e884dcad172e with delete_type TABLET_DATA_DELETED (replaced by tablet c6378f1495144c458af49b3c19a2b67c) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:40.429283 18744 tablet_service.cc:1514] Processing DeleteTablet for tablet 802336602d8f4ac4bd7e0872d6f3dae9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:31 UTC) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:40.429452 18733 tablet_service.cc:1514] Processing DeleteTablet for tablet cace1398492f4d968dcb6899385d527c with delete_type TABLET_DATA_DELETED (replaced by tablet 01436e5b6232494babbb414342f03090) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:40.429530 18736 tablet_service.cc:1514] Processing DeleteTablet for tablet 5bd0362ff41b4faca0205d136d596737 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:31 UTC) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:40.429486 19024 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_d441c3bc84ed42358d1bc6e717108efb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240430 20:11:40.425638 20105 consensus_queue.cc:1035] T 372b135dd01f44f98cd713ca8857ba61 P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.004s
W20240430 20:11:40.429987 19024 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d441c3bc84ed42358d1bc6e717108efb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:40.430078 18587 tablet_service.cc:1514] Processing DeleteTablet for tablet d4d6c37b578c4fe1aa666b0a6b0bc7c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:40 UTC) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:40.430758 18725 tablet_service.cc:1514] Processing DeleteTablet for tablet dd9b893d48684cfea7c242b93ecba01f with delete_type TABLET_DATA_DELETED (replaced by tablet f6cfd089f08b47409fb2a6cf4f6b0385) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:40.430946 18617 raft_consensus.cc:1261] T 75ebf051383e42529dcb5f7dd0770a4b P 5e1c6397f09747f091534faf93542710 [term 2 FOLLOWER]: Refusing update from remote peer 2efb542aa55e439b99ccb98d1c5c884a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 1. (term mismatch)
I20240430 20:11:40.431015 18724 tablet_service.cc:1514] Processing DeleteTablet for tablet 34ab40d8d91e401db299ddee7175ba28 with delete_type TABLET_DATA_DELETED (replaced by tablet 5f02d8862e1f4ccfa9dbda73dbe18615) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:40.431166 18617 pending_rounds.cc:76] T 75ebf051383e42529dcb5f7dd0770a4b P 5e1c6397f09747f091534faf93542710: Aborting all ops after (but not including) 0
I20240430 20:11:40.431252 18734 tablet_service.cc:1514] Processing DeleteTablet for tablet 5bde72e2791a497db7dd08686ca36d80 with delete_type TABLET_DATA_DELETED (replaced by tablet 66dd767bd3f8494f94061fb89e5e5680) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:40.431349 18988 tablet_bootstrap.cc:492] T f5546f9edeea44d7a6c5cb954f5157ba P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:40.431279 18617 pending_rounds.cc:97] T 75ebf051383e42529dcb5f7dd0770a4b P 5e1c6397f09747f091534faf93542710: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20240430 20:11:40.431448 18730 tablet_service.cc:1514] Processing DeleteTablet for tablet 77285afcafe04a829e485998bbbe00f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:31 UTC) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:40.431519 18988 ts_tablet_manager.cc:1367] T f5546f9edeea44d7a6c5cb954f5157ba P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.160s user 0.005s sys 0.001s
W20240430 20:11:40.431653 19049 catalog_manager.cc:5676] Failed to send AlterTable RPC for tablet b22aaedb0a6c4fd9ad0ea8fa019001e1 (table default.Table_730a215d7af74a5bb753adca39dea676 [id=a5d580cffa2447d78fc9f2c8e31264f1], current schema version=5): Not found: failed to reset TS proxy: No leader found: Tablet b22aaedb0a6c4fd9ad0ea8fa019001e1 consensus state has no leader
I20240430 20:11:40.432128 18450 tablet_service.cc:1514] Processing DeleteTablet for tablet d4d6c37b578c4fe1aa666b0a6b0bc7c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:40 UTC) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:40.432245 18730 tablet_service.cc:1514] Processing DeleteTablet for tablet d4d6c37b578c4fe1aa666b0a6b0bc7c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:40 UTC) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:40.431640 18617 raft_consensus.cc:2866] T 75ebf051383e42529dcb5f7dd0770a4b P 5e1c6397f09747f091534faf93542710 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240430 20:11:40.432451 18724 tablet_service.cc:1514] Processing DeleteTablet for tablet a9e0659564894b0e8cb7f600bb07a085 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:29 UTC) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:40.432662 18734 tablet_service.cc:1514] Processing DeleteTablet for tablet 15674aeb56b74210a4a2809608fd4457 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:25 UTC) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:40.433007 18988 raft_consensus.cc:348] T f5546f9edeea44d7a6c5cb954f5157ba P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:40.433162 18734 tablet_service.cc:1514] Processing DeleteTablet for tablet eed6338ad3d64859af087ee15e1ec798 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:26 UTC) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:40.433450 18988 raft_consensus.cc:374] T f5546f9edeea44d7a6c5cb954f5157ba P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:40.433573 18988 raft_consensus.cc:729] T f5546f9edeea44d7a6c5cb954f5157ba P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:40.433774 18734 tablet_service.cc:1514] Processing DeleteTablet for tablet e6eac92dfa5f4456bc49571878f0f8dd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:29 UTC) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:40.434027 18734 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a1e50f99c5d4b44bdbb51b58a92bc75 with delete_type TABLET_DATA_DELETED (replaced by tablet fbdeb4714b8c4b93ab72822c43329aae) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:40.434252 18988 consensus_queue.cc:260] T f5546f9edeea44d7a6c5cb954f5157ba P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:40.434902 18988 ts_tablet_manager.cc:1398] T f5546f9edeea44d7a6c5cb954f5157ba P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20240430 20:11:40.435233 18734 tablet_service.cc:1514] Processing DeleteTablet for tablet f5546f9edeea44d7a6c5cb954f5157ba with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:27 UTC) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:40.435329 18724 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e69ca12bd10494fb013614741bc166c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:27 UTC) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:40.435537 18730 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fe1247349344863a6273a769ce8ddf6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:24 UTC) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:40.435570 18988 tablet_bootstrap.cc:492] T 76d6e9d547ca4a848eae852944b1d117 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:40.435745 18724 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a9066d895744c5bb7d9532b2940603c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:29 UTC) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:40.435750 18734 tablet_service.cc:1514] Processing DeleteTablet for tablet b688e03ea71940c69bda5520b62c1fc3 with delete_type TABLET_DATA_DELETED (replaced by tablet 7683caba80594627900a0185f772c131) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:40.437873 18988 tablet_bootstrap.cc:654] T 76d6e9d547ca4a848eae852944b1d117 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:40.438558 20105 consensus_queue.cc:1035] T 75ebf051383e42529dcb5f7dd0770a4b P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240430 20:11:40.440569 19049 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.17.177.3:42157 (request call id 82) took 1889 ms (client timeout 1000 ms). Trace:
W20240430 20:11:40.441201 19049 rpcz_store.cc:269] 0430 20:11:38.550373 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:38.800224 (+249851us) service_pool.cc:225] Handling call
0430 20:11:40.440235 (+1640011us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:39.637092 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:39.637207 (+ 115us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:39.637215 (+ 8us) write_op.cc:427] Acquired schema lock
0430 20:11:39.637217 (+ 2us) tablet.cc:660] Decoding operations
0430 20:11:39.637335 (+ 118us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:39.637349 (+ 14us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:39.637353 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0430 20:11:39.637446 (+ 93us) tablet.cc:699] Row locks acquired
0430 20:11:39.637450 (+ 4us) write_op.cc:260] PREPARE: finished
0430 20:11:39.637488 (+ 38us) write_op.cc:270] Start()
0430 20:11:39.637522 (+ 34us) write_op.cc:276] Timestamp: P: 1714507899637428 usec, L: 0
0430 20:11:39.637526 (+ 4us) op_driver.cc:331] REPLICATION: starting
0430 20:11:39.637697 (+ 171us) log.cc:842] Serialized 530 byte log entry
0430 20:11:39.646209 (+ 8512us) op_driver.cc:444] REPLICATION: finished
0430 20:11:39.646312 (+ 103us) write_op.cc:301] APPLY: starting
0430 20:11:39.646329 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:39.646373 (+ 44us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:39.646415 (+ 42us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:39.646472 (+ 57us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:39.646509 (+ 37us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0430 20:11:39.646516 (+ 7us) write_op.cc:312] APPLY: finished
0430 20:11:39.646619 (+ 103us) log.cc:842] Serialized 23 byte log entry
0430 20:11:39.646684 (+ 65us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:39.646706 (+ 22us) write_op.cc:446] Released schema lock
0430 20:11:39.646712 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:39.646768 (+ 56us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:39.646772 (+ 4us) write_op.cc:446] Released schema lock
Metrics: {"reported_tablets":127,"spinlock_wait_cycles":1610624,"child_traces":[["op",{"apply.queue_time_us":53,"apply.run_cpu_time_us":512,"apply.run_wall_time_us":511,"num_ops":1,"prepare.queue_time_us":50,"prepare.run_cpu_time_us":703,"prepare.run_wall_time_us":704,"raft.queue_time_us":56,"raft.run_cpu_time_us":234,"raft.run_wall_time_us":234,"replication_time_us":8658}]]}
I20240430 20:11:40.445292 19049 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_c354e1a404914b959beb48d1b88c76a0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240430 20:11:40.446390 19495 ts_tablet_manager.cc:1909] T e6eac92dfa5f4456bc49571878f0f8dd P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:40.446825 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 68) took 9282 ms. Trace:
I20240430 20:11:40.446916 19495 rpcz_store.cc:276] 0430 20:11:31.164686 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:31.164791 (+ 105us) service_pool.cc:225] Handling call
0430 20:11:40.130843 (+8966052us) ts_tablet_manager.cc:1120] Deleting tablet e6eac92dfa5f4456bc49571878f0f8dd
0430 20:11:40.130869 (+ 26us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:40.360584 (+229715us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:40.415937 (+ 55353us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:40.446799 (+ 30862us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2360,"tablet-delete.queue_time_us":8945579}
I20240430 20:11:40.448046 20203 raft_consensus.cc:482] T 4e69ca12bd10494fb013614741bc166c P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:40.447407 19004 catalog_manager.cc:4830] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): tablet e6eac92dfa5f4456bc49571878f0f8dd (table default.Table_ad1a80db9c774df6add2b60e93577217 [id=19ffa77f540b499482d5dc7b64b12b79]) successfully deleted
W20240430 20:11:40.449581 19049 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c354e1a404914b959beb48d1b88c76a0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:40.467646 18985 tablet_bootstrap.cc:492] T 3fd0e8c360734dcbb75dc2958b70c2b8 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:40.467815 18985 ts_tablet_manager.cc:1367] T 3fd0e8c360734dcbb75dc2958b70c2b8 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.291s user 0.009s sys 0.006s
I20240430 20:11:40.452865 19027 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet c9e7660eccc448dbb99d135f82bfd261 (table default.Table_685a26737bb046fb85ced15df7ae3423 [id=51d177e8498848b38c43e3f8d11f092c]). Expected version 1 got 0
I20240430 20:11:40.469714 19027 catalog_manager.cc:5484] T 372b135dd01f44f98cd713ca8857ba61 P 2efb542aa55e439b99ccb98d1c5c884a reported cstate change: term changed from 1 to 2, leader changed from 3c51c33d5616482cafb68e26688820f4 (127.17.177.3) to 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1). New cstate: current_term: 2 leader_uuid: "2efb542aa55e439b99ccb98d1c5c884a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } health_report { overall_health: UNKNOWN } } }
I20240430 20:11:40.470366 19027 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet acf3b6fead41483a82c43dc0286fab7c (table default.Table_01450a1e661d450a8cbe185e16e8250c [id=ce9911e0a91949b1997ee47321800255]). Expected version 1 got 0
I20240430 20:11:40.470346 20203 raft_consensus.cc:504] T 4e69ca12bd10494fb013614741bc166c P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:40.470630 19027 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 6e754af64cb64fd2a082adae7d5f24da (table default.Table_208b12b8e9bd4f3e8bc9c6d422ada649 [id=1df4e95702cd46f59ab21b101049a9ed]). Expected version 1 got 0
I20240430 20:11:40.455384 19028 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 76d6e9d547ca4a848eae852944b1d117 (table default.Table_104ae015924a48fd8d27220792f07470 [id=ff86c0357c164dd0991098c416087773]). Expected version 2 got 0
I20240430 20:11:40.461046 19026 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: Sending DeleteTablet for 3 replicas of tablet 5809e7c9c2204ba293426c24129fb16e
I20240430 20:11:40.471786 19028 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet acd23f0a63d5448aaa7ed78422e67bad (table default.Table_972f13be4dca4cd19b9a04534dc59f51 [id=69f37db4de634a3185613e40013e0ed7]). Expected version 1 got 0
I20240430 20:11:40.471860 19026 catalog_manager.cc:6554] ReplaceTablet: tablet 5809e7c9c2204ba293426c24129fb16e deleted and replaced by tablet 4391b5227fb64ef481542a962a5e21a5
I20240430 20:11:40.472004 19028 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet a05e32a33aa7488eb287cc6314c09c44 (table default.Table_2d1cfc78363a47aa8dba7ef7d80f8c9a [id=955d833ff8f14bf6957abadf9ef669e2]). Expected version 2 got 0
I20240430 20:11:40.472071 19026 master_service.cc:946] ReplaceTablet: received request to replace tablet fd40967ab4a04001819e16c06a6c4300 from {username='slave'} at 127.0.0.1:51664
W20240430 20:11:40.472111 18696 leader_election.cc:341] T 4e69ca12bd10494fb013614741bc166c P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Not found: Tablet not found: 4e69ca12bd10494fb013614741bc166c
I20240430 20:11:40.461342 19033 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: Sending DeleteTablet for 3 replicas of tablet f3711610610542128c4a84e17bce4a3e
I20240430 20:11:40.461902 19029 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_5572f0acadaa432e8570b159fd9b76bc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:40.473991 19029 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5572f0acadaa432e8570b159fd9b76bc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:40.474233 18449 tablet_service.cc:1514] Processing DeleteTablet for tablet f3711610610542128c4a84e17bce4a3e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:40 UTC) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:40.475198 18450 tablet_service.cc:1514] Processing DeleteTablet for tablet 5809e7c9c2204ba293426c24129fb16e with delete_type TABLET_DATA_DELETED (replaced by tablet 4391b5227fb64ef481542a962a5e21a5) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:40.475450 18587 tablet_service.cc:1514] Processing DeleteTablet for tablet 5809e7c9c2204ba293426c24129fb16e with delete_type TABLET_DATA_DELETED (replaced by tablet 4391b5227fb64ef481542a962a5e21a5) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:40.477660 19495 tablet_replica.cc:330] T f5546f9edeea44d7a6c5cb954f5157ba P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:40.477866 19495 raft_consensus.cc:2229] T f5546f9edeea44d7a6c5cb954f5157ba P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Raft consensus shutting down.
I20240430 20:11:40.478085 19495 pending_rounds.cc:62] T f5546f9edeea44d7a6c5cb954f5157ba P 2efb542aa55e439b99ccb98d1c5c884a: Trying to abort 1 pending ops.
I20240430 20:11:40.478127 19495 pending_rounds.cc:68] T f5546f9edeea44d7a6c5cb954f5157ba P 2efb542aa55e439b99ccb98d1c5c884a: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7022624357949984768 op_type: NO_OP noop_request { }
I20240430 20:11:40.478226 19495 raft_consensus.cc:2866] T f5546f9edeea44d7a6c5cb954f5157ba P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240430 20:11:40.478214 18734 tablet_service.cc:1514] Processing DeleteTablet for tablet 5809e7c9c2204ba293426c24129fb16e with delete_type TABLET_DATA_DELETED (replaced by tablet 4391b5227fb64ef481542a962a5e21a5) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:40.478300 19495 raft_consensus.cc:2258] T f5546f9edeea44d7a6c5cb954f5157ba P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:40.478734 18734 tablet_service.cc:1514] Processing DeleteTablet for tablet f3711610610542128c4a84e17bce4a3e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:40 UTC) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:40.478746 19033 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_d12caaff392a45ccb6c95a4a731b714a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:40.479400 19033 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d12caaff392a45ccb6c95a4a731b714a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:40.481587 18587 tablet_service.cc:1514] Processing DeleteTablet for tablet f3711610610542128c4a84e17bce4a3e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:40 UTC) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:40.462693 19023 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_8c841af5208742f9b56a4591c02c573e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240430 20:11:40.463461 19024 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_6d9a8c8bd13b42eca6834e707653a66b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240430 20:11:40.469014 19048 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: Sending DeleteTablet for 3 replicas of tablet d7c1fc7e81364fb890bbc13d7fb6848b
I20240430 20:11:40.483353 20203 leader_election.cc:288] T 4e69ca12bd10494fb013614741bc166c P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:40.487509 19028 master_service.cc:946] ReplaceTablet: received request to replace tablet e4cffa295d8c401aa8415ce7e8f56526 from {username='slave'} at 127.0.0.1:51664
I20240430 20:11:40.489742 18983 tablet_bootstrap.cc:492] T acf3b6fead41483a82c43dc0286fab7c P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:40.489892 18983 ts_tablet_manager.cc:1367] T acf3b6fead41483a82c43dc0286fab7c P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.144s user 0.005s sys 0.000s
I20240430 20:11:40.489956 19048 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_5dd3db8b666a491e8b33cbb5a0b54ff5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240430 20:11:40.490978 18587 tablet_service.cc:1514] Processing DeleteTablet for tablet d7c1fc7e81364fb890bbc13d7fb6848b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:40 UTC) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:40.491075 18734 tablet_service.cc:1514] Processing DeleteTablet for tablet d7c1fc7e81364fb890bbc13d7fb6848b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:40 UTC) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:40.491195 18450 tablet_service.cc:1514] Processing DeleteTablet for tablet d7c1fc7e81364fb890bbc13d7fb6848b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:40 UTC) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:40.491359 18983 raft_consensus.cc:348] T acf3b6fead41483a82c43dc0286fab7c P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:40.491557 18983 raft_consensus.cc:374] T acf3b6fead41483a82c43dc0286fab7c P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:40.491624 18983 raft_consensus.cc:729] T acf3b6fead41483a82c43dc0286fab7c P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:40.491832 18983 consensus_queue.cc:260] T acf3b6fead41483a82c43dc0286fab7c P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
W20240430 20:11:40.492480 19048 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5dd3db8b666a491e8b33cbb5a0b54ff5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:40.492532 18983 ts_tablet_manager.cc:1398] T acf3b6fead41483a82c43dc0286fab7c P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240430 20:11:40.492789 18983 tablet_bootstrap.cc:492] T 0eb4f3e7ece6496293e01b63620d644b P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:40.494138 18983 tablet_bootstrap.cc:654] T 0eb4f3e7ece6496293e01b63620d644b P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:40.496167 18988 tablet_bootstrap.cc:492] T 76d6e9d547ca4a848eae852944b1d117 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:40.496361 19026 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: Sending DeleteTablet for 3 replicas of tablet fd40967ab4a04001819e16c06a6c4300
I20240430 20:11:40.496415 18988 ts_tablet_manager.cc:1367] T 76d6e9d547ca4a848eae852944b1d117 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.061s user 0.005s sys 0.000s
I20240430 20:11:40.496930 19026 catalog_manager.cc:6554] ReplaceTablet: tablet fd40967ab4a04001819e16c06a6c4300 deleted and replaced by tablet dbb427aefa6b47b0805d1751fea4efcb
I20240430 20:11:40.497128 18450 tablet_service.cc:1514] Processing DeleteTablet for tablet fd40967ab4a04001819e16c06a6c4300 with delete_type TABLET_DATA_DELETED (replaced by tablet dbb427aefa6b47b0805d1751fea4efcb) from {username='slave'} at 127.0.0.1:41680
W20240430 20:11:40.497706 19023 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8c841af5208742f9b56a4591c02c573e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:40.497891 18449 tablet_service.cc:1514] Processing DeleteTablet for tablet e6eac92dfa5f4456bc49571878f0f8dd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:29 UTC) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:40.498122 18450 tablet_service.cc:1514] Processing DeleteTablet for tablet a9e0659564894b0e8cb7f600bb07a085 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:29 UTC) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:40.498584 18734 tablet_service.cc:1514] Processing DeleteTablet for tablet fd40967ab4a04001819e16c06a6c4300 with delete_type TABLET_DATA_DELETED (replaced by tablet dbb427aefa6b47b0805d1751fea4efcb) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:40.498962 18988 raft_consensus.cc:348] T 76d6e9d547ca4a848eae852944b1d117 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:40.499245 18988 raft_consensus.cc:374] T 76d6e9d547ca4a848eae852944b1d117 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:40.499748 18988 raft_consensus.cc:729] T 76d6e9d547ca4a848eae852944b1d117 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:40.500082 18988 consensus_queue.cc:260] T 76d6e9d547ca4a848eae852944b1d117 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:40.502473 18988 ts_tablet_manager.cc:1398] T 76d6e9d547ca4a848eae852944b1d117 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240430 20:11:40.502851 18988 tablet_bootstrap.cc:492] T a9e0659564894b0e8cb7f600bb07a085 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:40.504690 18988 tablet_bootstrap.cc:654] T a9e0659564894b0e8cb7f600bb07a085 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:40.505625 19029 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_ff3ffe1bdd1e497697316f4ac1731e3d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:40.518813 19024 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6d9a8c8bd13b42eca6834e707653a66b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:40.468710 18985 raft_consensus.cc:348] T 3fd0e8c360734dcbb75dc2958b70c2b8 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:40.535609 18985 raft_consensus.cc:374] T 3fd0e8c360734dcbb75dc2958b70c2b8 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:40.535753 18985 raft_consensus.cc:729] T 3fd0e8c360734dcbb75dc2958b70c2b8 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:40.536031 18985 consensus_queue.cc:260] T 3fd0e8c360734dcbb75dc2958b70c2b8 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:40.537098 18985 ts_tablet_manager.cc:1398] T 3fd0e8c360734dcbb75dc2958b70c2b8 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.069s user 0.001s sys 0.001s
I20240430 20:11:40.544163 19028 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: Sending DeleteTablet for 3 replicas of tablet e4cffa295d8c401aa8415ce7e8f56526
I20240430 20:11:40.504518 18608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0eb4f3e7ece6496293e01b63620d644b" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:40.517935 18587 tablet_service.cc:1514] Processing DeleteTablet for tablet fd40967ab4a04001819e16c06a6c4300 with delete_type TABLET_DATA_DELETED (replaced by tablet dbb427aefa6b47b0805d1751fea4efcb) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:40.542330 20047 raft_consensus.cc:482] T 76d6e9d547ca4a848eae852944b1d117 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240430 20:11:40.576085 18415 leader_election.cc:341] T 0eb4f3e7ece6496293e01b63620d644b P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:40.575953 20047 raft_consensus.cc:504] T 76d6e9d547ca4a848eae852944b1d117 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:40.576203 18415 leader_election.cc:302] T 0eb4f3e7ece6496293e01b63620d644b P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a; no voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710
I20240430 20:11:40.576387 18985 tablet_bootstrap.cc:492] T 2f61e3f88a6448dfa323204bd8d14cc5 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:40.576426 19028 catalog_manager.cc:6554] ReplaceTablet: tablet e4cffa295d8c401aa8415ce7e8f56526 deleted and replaced by tablet e5b85a066cdc46d48ccf8201e5423c29
I20240430 20:11:40.555956 18615 raft_consensus.cc:1261] T c66c142419ca4e1399ecd1c8d1e62a59 P 5e1c6397f09747f091534faf93542710 [term 2 FOLLOWER]: Refusing update from remote peer 2efb542aa55e439b99ccb98d1c5c884a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240430 20:11:40.577775 18985 tablet_bootstrap.cc:654] T 2f61e3f88a6448dfa323204bd8d14cc5 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:40.540462 20236 raft_consensus.cc:482] T 0a9066d895744c5bb7d9532b2940603c P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:40.543140 19495 ts_tablet_manager.cc:1875] T f5546f9edeea44d7a6c5cb954f5157ba P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:40.560127 20096 consensus_queue.cc:1035] T c9e7660eccc448dbb99d135f82bfd261 P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:40.571491 18588 tablet_service.cc:1514] Processing DeleteTablet for tablet e4cffa295d8c401aa8415ce7e8f56526 with delete_type TABLET_DATA_DELETED (replaced by tablet e5b85a066cdc46d48ccf8201e5423c29) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:40.578423 20236 raft_consensus.cc:504] T 0a9066d895744c5bb7d9532b2940603c P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:40.556579 18474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a9066d895744c5bb7d9532b2940603c" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:40.557446 18478 raft_consensus.cc:1261] T 5809e7c9c2204ba293426c24129fb16e P 2efb542aa55e439b99ccb98d1c5c884a [term 2 FOLLOWER]: Refusing update from remote peer 3c51c33d5616482cafb68e26688820f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240430 20:11:40.579207 19027 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_49e612b137f343d1ac3ec06ded7d42f3" } new_table_name: "default.Table_b49a7942aa8c4f58acb94c35d1cdbb70" modify_external_catalogs: true
I20240430 20:11:40.579540 18474 raft_consensus.cc:3037] T 0a9066d895744c5bb7d9532b2940603c P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:40.581750 20047 leader_election.cc:288] T 76d6e9d547ca4a848eae852944b1d117 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
W20240430 20:11:40.582401 19029 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ff3ffe1bdd1e497697316f4ac1731e3d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:40.582551 18474 raft_consensus.cc:2454] T 0a9066d895744c5bb7d9532b2940603c P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:40.582671 20235 raft_consensus.cc:2726] T 0eb4f3e7ece6496293e01b63620d644b P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:40.561374 18475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a9066d895744c5bb7d9532b2940603c" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:40.571260 18450 tablet_service.cc:1514] Processing DeleteTablet for tablet e4cffa295d8c401aa8415ce7e8f56526 with delete_type TABLET_DATA_DELETED (replaced by tablet e5b85a066cdc46d48ccf8201e5423c29) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:40.582981 18475 raft_consensus.cc:2362] T 0a9066d895744c5bb7d9532b2940603c P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1. Re-sending same reply.
I20240430 20:11:40.583557 18478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76d6e9d547ca4a848eae852944b1d117" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:40.583738 18478 raft_consensus.cc:2454] T 76d6e9d547ca4a848eae852944b1d117 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 0.
I20240430 20:11:40.583796 20235 consensus_queue.cc:1035] T c66c142419ca4e1399ecd1c8d1e62a59 P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240430 20:11:40.584053 18615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a9066d895744c5bb7d9532b2940603c" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:40.584340 18615 raft_consensus.cc:2379] T 0a9066d895744c5bb7d9532b2940603c P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2efb542aa55e439b99ccb98d1c5c884a in current term 1: Already voted for candidate 3c51c33d5616482cafb68e26688820f4 in this term.
I20240430 20:11:40.585106 18615 raft_consensus.cc:1261] T 5809e7c9c2204ba293426c24129fb16e P 5e1c6397f09747f091534faf93542710 [term 2 FOLLOWER]: Refusing update from remote peer 3c51c33d5616482cafb68e26688820f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240430 20:11:40.585466 20236 leader_election.cc:288] T 0a9066d895744c5bb7d9532b2940603c P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:40.587097 19471 ts_tablet_manager.cc:1888] T 802336602d8f4ac4bd7e0872d6f3dae9 P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:40.526497 18756 raft_consensus.cc:1261] T c66c142419ca4e1399ecd1c8d1e62a59 P 3c51c33d5616482cafb68e26688820f4 [term 2 FOLLOWER]: Refusing update from remote peer 2efb542aa55e439b99ccb98d1c5c884a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240430 20:11:40.544648 20197 raft_consensus.cc:482] T 47c37876694c4a4abb1091c631dab7a5 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:40.588119 20197 raft_consensus.cc:504] T 47c37876694c4a4abb1091c631dab7a5 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:40.589319 20197 leader_election.cc:288] T 47c37876694c4a4abb1091c631dab7a5 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:40.590212 19028 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_43c679fe95d64db79020bcef1cbe61cb" } modify_external_catalogs: true
I20240430 20:11:40.545223 18734 tablet_service.cc:1514] Processing DeleteTablet for tablet e4cffa295d8c401aa8415ce7e8f56526 with delete_type TABLET_DATA_DELETED (replaced by tablet e5b85a066cdc46d48ccf8201e5423c29) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:40.554706 20205 raft_consensus.cc:482] T 255c13ad8fae4ea4aa8e7b7aabb7602e P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240430 20:11:40.562958 18699 leader_election.cc:341] T 4e69ca12bd10494fb013614741bc166c P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Not found: Tablet not found: 4e69ca12bd10494fb013614741bc166c
I20240430 20:11:40.585860 20198 consensus_queue.cc:1035] T 5809e7c9c2204ba293426c24129fb16e P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240430 20:11:40.590759 18699 leader_election.cc:302] T 4e69ca12bd10494fb013614741bc166c P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710
I20240430 20:11:40.591151 20198 raft_consensus.cc:2726] T 4e69ca12bd10494fb013614741bc166c P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:40.591377 20205 raft_consensus.cc:504] T 255c13ad8fae4ea4aa8e7b7aabb7602e P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:40.591975 20198 consensus_queue.cc:1035] T 5809e7c9c2204ba293426c24129fb16e P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240430 20:11:40.592702 18758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a9066d895744c5bb7d9532b2940603c" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:40.592964 20205 leader_election.cc:288] T 255c13ad8fae4ea4aa8e7b7aabb7602e P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:40.593195 18756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76d6e9d547ca4a848eae852944b1d117" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:40.593366 18756 raft_consensus.cc:2454] T 76d6e9d547ca4a848eae852944b1d117 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 0.
I20240430 20:11:40.593829 18417 leader_election.cc:302] T 0a9066d895744c5bb7d9532b2940603c P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a; no voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710
I20240430 20:11:40.594303 20235 consensus_queue.cc:1035] T c66c142419ca4e1399ecd1c8d1e62a59 P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240430 20:11:40.595033 20235 raft_consensus.cc:2726] T 0a9066d895744c5bb7d9532b2940603c P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:40.602221 18988 tablet_bootstrap.cc:492] T a9e0659564894b0e8cb7f600bb07a085 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:40.602424 18988 ts_tablet_manager.cc:1367] T a9e0659564894b0e8cb7f600bb07a085 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.100s user 0.003s sys 0.000s
I20240430 20:11:40.603134 19048 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_7b1809d7d84b4e7896d1f10321e3f20c" } modify_external_catalogs: true
I20240430 20:11:40.603413 18988 raft_consensus.cc:348] T a9e0659564894b0e8cb7f600bb07a085 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:40.603627 18988 raft_consensus.cc:374] T a9e0659564894b0e8cb7f600bb07a085 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:40.603713 18988 raft_consensus.cc:729] T a9e0659564894b0e8cb7f600bb07a085 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:40.603956 18988 consensus_queue.cc:260] T a9e0659564894b0e8cb7f600bb07a085 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:40.604255 19495 ts_tablet_manager.cc:1888] T f5546f9edeea44d7a6c5cb954f5157ba P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:40.604374 19495 log.cc:1196] T f5546f9edeea44d7a6c5cb954f5157ba P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/f5546f9edeea44d7a6c5cb954f5157ba
I20240430 20:11:40.604684 19495 ts_tablet_manager.cc:1909] T f5546f9edeea44d7a6c5cb954f5157ba P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:40.605078 18988 ts_tablet_manager.cc:1398] T a9e0659564894b0e8cb7f600bb07a085 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240430 20:11:40.605393 18988 tablet_bootstrap.cc:492] T 2a1e50f99c5d4b44bdbb51b58a92bc75 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:40.606029 19048 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_7b1809d7d84b4e7896d1f10321e3f20c" } modify_external_catalogs: true
I20240430 20:11:40.607322 19033 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_d06ed4569aa2456d85107f8a351d2ea1" } new_table_name: "default.Table_7d5f6f18cbd64c869739872a7dff9be2" modify_external_catalogs: true
I20240430 20:11:40.608557 19028 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_43c679fe95d64db79020bcef1cbe61cb" } modify_external_catalogs: true
W20240430 20:11:40.614010 18699 leader_election.cc:341] T 47c37876694c4a4abb1091c631dab7a5 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Not found: Tablet not found: 47c37876694c4a4abb1091c631dab7a5
I20240430 20:11:40.615237 18988 tablet_bootstrap.cc:654] T 2a1e50f99c5d4b44bdbb51b58a92bc75 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
W20240430 20:11:40.616343 18699 leader_election.cc:341] T 255c13ad8fae4ea4aa8e7b7aabb7602e P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Not found: Tablet not found: 255c13ad8fae4ea4aa8e7b7aabb7602e
W20240430 20:11:40.616825 19027 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1462ecf1176c4e66be42abddbe36b00c consensus state has no leader
I20240430 20:11:40.618181 19027 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_c354e1a404914b959beb48d1b88c76a0" } new_table_name: "default.Table_63d833beade34542a349c8f691fd3b98" modify_external_catalogs: true
I20240430 20:11:40.631597 18734 tablet_service.cc:1467] Processing CreateTablet for tablet 2e1cac64cefe4336bd3207b5ca58b631 (DEFAULT_TABLE table=default.Table_ad4322a6bb564abd80dad138208e420d [id=0f4d7a1eb8834b559e79685fbb9de43b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:40.631764 18450 tablet_service.cc:1467] Processing CreateTablet for tablet 2e1cac64cefe4336bd3207b5ca58b631 (DEFAULT_TABLE table=default.Table_ad4322a6bb564abd80dad138208e420d [id=0f4d7a1eb8834b559e79685fbb9de43b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:40.631850 18724 tablet_service.cc:1467] Processing CreateTablet for tablet 3191aa54a9644783a2403991c3be27c8 (DEFAULT_TABLE table=default.Table_75336e7a0e5f4c2d909f5c5c385bc379 [id=be4876d319a24a958d50e3d0c508e95f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:40.632119 18734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e1cac64cefe4336bd3207b5ca58b631. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:40.632169 18450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e1cac64cefe4336bd3207b5ca58b631. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:40.632912 18730 tablet_service.cc:1467] Processing CreateTablet for tablet 13c2e69ae91f4957887fd64b90f6e435 (DEFAULT_TABLE table=default.Table_6b67c0c03ceb4b9491ad5828022b357f [id=368fcf43e863494ca260862d12331c0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:40.633268 18730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13c2e69ae91f4957887fd64b90f6e435. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:40.634853 18725 tablet_service.cc:1467] Processing CreateTablet for tablet a692b253db914531b8fc6ffaf3e350b5 (DEFAULT_TABLE table=default.Table_96aea8c4ff80428da8a0394f1bf7b493 [id=03ccb721140c4e0193e2e48d509430f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:40.635190 18725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a692b253db914531b8fc6ffaf3e350b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:40.636682 18736 tablet_service.cc:1467] Processing CreateTablet for tablet 2797eed593fb4b06a3c4d6a28e8075a2 (DEFAULT_TABLE table=default.Table_88b0eaf3d0a1461ab5e2130cb69f0dee [id=6ddeb7090daf419db901940d7a86acee]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:40.637058 18736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2797eed593fb4b06a3c4d6a28e8075a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:40.638875 18449 tablet_service.cc:1467] Processing CreateTablet for tablet 13c2e69ae91f4957887fd64b90f6e435 (DEFAULT_TABLE table=default.Table_6b67c0c03ceb4b9491ad5828022b357f [id=368fcf43e863494ca260862d12331c0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:40.639222 18449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13c2e69ae91f4957887fd64b90f6e435. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:40.642633 19049 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_421069f84cb844368066c93102a8faf8" } new_table_name: "default.Table_896bbae1f6944005b0272ca8ebc97218" modify_external_catalogs: true
I20240430 20:11:40.642881 18446 tablet_service.cc:1467] Processing CreateTablet for tablet a692b253db914531b8fc6ffaf3e350b5 (DEFAULT_TABLE table=default.Table_96aea8c4ff80428da8a0394f1bf7b493 [id=03ccb721140c4e0193e2e48d509430f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:40.643229 18446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a692b253db914531b8fc6ffaf3e350b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:40.643973 18733 tablet_service.cc:1467] Processing CreateTablet for tablet 1fe7c6f1ab54499facd4d01573d3e7bc (DEFAULT_TABLE table=default.Table_0366454b829949928c7b5e79bbf3a649 [id=edc38e64373242f7bb394a526bb2a4f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:40.644315 18733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fe7c6f1ab54499facd4d01573d3e7bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:40.646733 18724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3191aa54a9644783a2403991c3be27c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:40.651108 19026 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 32c823d1700949ab8b22adcd83b7bfd1 (table default.Table_e8c73c89623d4b49b8d7f0a01d729bdd [id=5cd10682ce52414cb5a103e4433a6a0d]). Expected version 1 got 0
I20240430 20:11:40.651513 19026 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet a05e32a33aa7488eb287cc6314c09c44 (table default.Table_2d1cfc78363a47aa8dba7ef7d80f8c9a [id=955d833ff8f14bf6957abadf9ef669e2]). Expected version 2 got 0
I20240430 20:11:40.651819 19026 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet acd23f0a63d5448aaa7ed78422e67bad (table default.Table_972f13be4dca4cd19b9a04534dc59f51 [id=69f37db4de634a3185613e40013e0ed7]). Expected version 1 got 0
I20240430 20:11:40.653542 18450 tablet_service.cc:1467] Processing CreateTablet for tablet 2797eed593fb4b06a3c4d6a28e8075a2 (DEFAULT_TABLE table=default.Table_88b0eaf3d0a1461ab5e2130cb69f0dee [id=6ddeb7090daf419db901940d7a86acee]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:40.653930 18450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2797eed593fb4b06a3c4d6a28e8075a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:40.653571 18447 tablet_service.cc:1467] Processing CreateTablet for tablet 1fe7c6f1ab54499facd4d01573d3e7bc (DEFAULT_TABLE table=default.Table_0366454b829949928c7b5e79bbf3a649 [id=edc38e64373242f7bb394a526bb2a4f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:40.655906 18447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fe7c6f1ab54499facd4d01573d3e7bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:40.670504 19024 master_service.cc:946] ReplaceTablet: received request to replace tablet 34130f2820064fa0898ef1f9d7de1c6d from {username='slave'} at 127.0.0.1:51664
W20240430 20:11:40.672498 19033 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet eb6c96047b114ebba73db7a371f2da8e consensus state has no leader
I20240430 20:11:40.672947 19033 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_273caa9a42a9474e8787c37032368654" } modify_external_catalogs: true
I20240430 20:11:40.673162 19033 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_273caa9a42a9474e8787c37032368654" } modify_external_catalogs: true
I20240430 20:11:40.691133 18724 tablet_service.cc:1467] Processing CreateTablet for tablet 4dac8b3838954d839094021f1c0503da (DEFAULT_TABLE table=default.Table_7b1809d7d84b4e7896d1f10321e3f20c [id=45839a14bd914828b0fea7675f093233]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:40.691221 19023 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet acd23f0a63d5448aaa7ed78422e67bad (table default.Table_972f13be4dca4cd19b9a04534dc59f51 [id=69f37db4de634a3185613e40013e0ed7]). Expected version 1 got 0
I20240430 20:11:40.691504 19023 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet a05e32a33aa7488eb287cc6314c09c44 (table default.Table_2d1cfc78363a47aa8dba7ef7d80f8c9a [id=955d833ff8f14bf6957abadf9ef669e2]). Expected version 2 got 0
I20240430 20:11:40.691568 18724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4dac8b3838954d839094021f1c0503da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:40.692914 19023 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet c9e7660eccc448dbb99d135f82bfd261 (table default.Table_685a26737bb046fb85ced15df7ae3423 [id=51d177e8498848b38c43e3f8d11f092c]). Expected version 1 got 0
I20240430 20:11:40.693231 19023 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet acf3b6fead41483a82c43dc0286fab7c (table default.Table_01450a1e661d450a8cbe185e16e8250c [id=ce9911e0a91949b1997ee47321800255]). Expected version 1 got 0
I20240430 20:11:40.696663 18450 tablet_service.cc:1467] Processing CreateTablet for tablet 4dac8b3838954d839094021f1c0503da (DEFAULT_TABLE table=default.Table_7b1809d7d84b4e7896d1f10321e3f20c [id=45839a14bd914828b0fea7675f093233]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:40.696970 18449 tablet_service.cc:1467] Processing CreateTablet for tablet 436579f473844c6ca1038b496201a550 (DEFAULT_TABLE table=default.Table_273caa9a42a9474e8787c37032368654 [id=d89426aa652c42e49f956ea85efb0d43]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:40.697935 18733 tablet_service.cc:1467] Processing CreateTablet for tablet a663030d37614053a7c0369841403422 (DEFAULT_TABLE table=default.Table_e34613952dcb46618c3bc7cecb31fdaf [id=8f1ad1f8d0224b3daa5d84dc6916227b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:40.698393 18733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a663030d37614053a7c0369841403422. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:40.696627 18446 tablet_service.cc:1467] Processing CreateTablet for tablet a663030d37614053a7c0369841403422 (DEFAULT_TABLE table=default.Table_e34613952dcb46618c3bc7cecb31fdaf [id=8f1ad1f8d0224b3daa5d84dc6916227b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:40.699271 18447 tablet_service.cc:1467] Processing CreateTablet for tablet 69f88532952a4591bf1577541c3430e6 (DEFAULT_TABLE table=default.Table_1177828cad1f4a24bd397619dad96170 [id=a95fed1bc3fa4865bf9a35dfc51b659c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:40.699900 18730 tablet_service.cc:1467] Processing CreateTablet for tablet 436579f473844c6ca1038b496201a550 (DEFAULT_TABLE table=default.Table_273caa9a42a9474e8787c37032368654 [id=d89426aa652c42e49f956ea85efb0d43]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:40.700316 18730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 436579f473844c6ca1038b496201a550. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:40.705410 18725 tablet_service.cc:1467] Processing CreateTablet for tablet 69f88532952a4591bf1577541c3430e6 (DEFAULT_TABLE table=default.Table_1177828cad1f4a24bd397619dad96170 [id=a95fed1bc3fa4865bf9a35dfc51b659c]), partition=RANGE (key) PARTITION UNBOUNDED
W20240430 20:11:40.710451 19027 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1050ab528c7d41559831652a1437d358 has no consensus state
I20240430 20:11:40.697046 18450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4dac8b3838954d839094021f1c0503da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:40.711169 18447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69f88532952a4591bf1577541c3430e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:40.711552 19028 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: Sending DeleteTablet for 3 replicas of tablet a9f37f0f7ce8425f9b24699513d41d61
I20240430 20:11:40.712527 19048 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: Sending DeleteTablet for 3 replicas of tablet 4dac8b3838954d839094021f1c0503da
I20240430 20:11:40.713785 18459 tablet_service.cc:1514] Processing DeleteTablet for tablet a9f37f0f7ce8425f9b24699513d41d61 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:40 UTC) from {username='slave'} at 127.0.0.1:41680
W20240430 20:11:40.714102 19000 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet ae20f44df50343289b0a512a4ea53692 (table default.Table_1844f6b1bda84812825384e14a2fd94e [id=71da9acc3cff4d95bbac4a686b708fd5], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet ae20f44df50343289b0a512a4ea53692 consensus state has no leader
I20240430 20:11:40.714246 18459 tablet_service.cc:1514] Processing DeleteTablet for tablet 4dac8b3838954d839094021f1c0503da with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:40 UTC) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:40.714666 18446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a663030d37614053a7c0369841403422. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:40.722721 18449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 436579f473844c6ca1038b496201a550. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:40.732816 18454 tablet_service.cc:1467] Processing CreateTablet for tablet 3191aa54a9644783a2403991c3be27c8 (DEFAULT_TABLE table=default.Table_75336e7a0e5f4c2d909f5c5c385bc379 [id=be4876d319a24a958d50e3d0c508e95f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:40.590334 18615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c37876694c4a4abb1091c631dab7a5" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:40.592381 18558 leader_election.cc:302] T 76d6e9d547ca4a848eae852944b1d117 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:40.733618 18615 raft_consensus.cc:2454] T 47c37876694c4a4abb1091c631dab7a5 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
W20240430 20:11:40.733669 18558 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.177.1:41217, user_credentials={real_user=slave}} blocked reactor thread for 141319us
I20240430 20:11:40.734464 20061 raft_consensus.cc:2781] T 76d6e9d547ca4a848eae852944b1d117 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:40.734627 20061 raft_consensus.cc:482] T 76d6e9d547ca4a848eae852944b1d117 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:40.734745 20061 raft_consensus.cc:3037] T 76d6e9d547ca4a848eae852944b1d117 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:40.734975 18587 tablet_service.cc:1467] Processing CreateTablet for tablet 13c2e69ae91f4957887fd64b90f6e435 (DEFAULT_TABLE table=default.Table_6b67c0c03ceb4b9491ad5828022b357f [id=368fcf43e863494ca260862d12331c0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:40.735366 18587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13c2e69ae91f4957887fd64b90f6e435. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:40.618978 18983 tablet_bootstrap.cc:492] T 0eb4f3e7ece6496293e01b63620d644b P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:40.736289 18983 ts_tablet_manager.cc:1367] T 0eb4f3e7ece6496293e01b63620d644b P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.244s user 0.003s sys 0.000s
I20240430 20:11:40.737273 18983 raft_consensus.cc:348] T 0eb4f3e7ece6496293e01b63620d644b P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:40.737524 18983 raft_consensus.cc:374] T 0eb4f3e7ece6496293e01b63620d644b P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:40.737455 20061 raft_consensus.cc:504] T 76d6e9d547ca4a848eae852944b1d117 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:40.737587 18983 raft_consensus.cc:729] T 0eb4f3e7ece6496293e01b63620d644b P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:40.737814 18983 consensus_queue.cc:260] T 0eb4f3e7ece6496293e01b63620d644b P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:40.738039 20061 leader_election.cc:288] T 76d6e9d547ca4a848eae852944b1d117 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Requested vote from peers 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:40.738490 18983 ts_tablet_manager.cc:1398] T 0eb4f3e7ece6496293e01b63620d644b P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240430 20:11:40.738871 18593 tablet_service.cc:1467] Processing CreateTablet for tablet 1fe7c6f1ab54499facd4d01573d3e7bc (DEFAULT_TABLE table=default.Table_0366454b829949928c7b5e79bbf3a649 [id=edc38e64373242f7bb394a526bb2a4f8]), partition=RANGE (key) PARTITION UNBOUNDED
W20240430 20:11:40.738979 18557 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.177.3:44867, user_credentials={real_user=slave}} blocked reactor thread for 145335us
I20240430 20:11:40.739199 18593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fe7c6f1ab54499facd4d01573d3e7bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:40.741855 18592 tablet_service.cc:1467] Processing CreateTablet for tablet 2e1cac64cefe4336bd3207b5ca58b631 (DEFAULT_TABLE table=default.Table_ad4322a6bb564abd80dad138208e420d [id=0f4d7a1eb8834b559e79685fbb9de43b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:40.684512 18668 heartbeater.cc:347] Connected to a master server at 127.17.177.62:36441
I20240430 20:11:40.742338 18588 tablet_service.cc:1467] Processing CreateTablet for tablet 3191aa54a9644783a2403991c3be27c8 (DEFAULT_TABLE table=default.Table_75336e7a0e5f4c2d909f5c5c385bc379 [id=be4876d319a24a958d50e3d0c508e95f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:40.742678 18588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3191aa54a9644783a2403991c3be27c8. 1 dirs total, 0 dirs full, 0 dirs failed
W20240430 20:11:40.744913 19049 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 12795670f21447a8964fe1efc6e92398 consensus state has no leader
I20240430 20:11:40.746918 18586 tablet_service.cc:1467] Processing CreateTablet for tablet a692b253db914531b8fc6ffaf3e350b5 (DEFAULT_TABLE table=default.Table_96aea8c4ff80428da8a0394f1bf7b493 [id=03ccb721140c4e0193e2e48d509430f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:40.748759 18586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a692b253db914531b8fc6ffaf3e350b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:40.714515 18736 tablet_service.cc:1514] Processing DeleteTablet for tablet a9f37f0f7ce8425f9b24699513d41d61 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:40 UTC) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:40.710748 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 85) took 9056 ms. Trace:
I20240430 20:11:40.715714 18734 tablet_service.cc:1514] Processing DeleteTablet for tablet 4dac8b3838954d839094021f1c0503da with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:40 UTC) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:40.747200 18600 tablet_service.cc:1467] Processing CreateTablet for tablet 2797eed593fb4b06a3c4d6a28e8075a2 (DEFAULT_TABLE table=default.Table_88b0eaf3d0a1461ab5e2130cb69f0dee [id=6ddeb7090daf419db901940d7a86acee]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:40.751909 19471 log.cc:1196] T 802336602d8f4ac4bd7e0872d6f3dae9 P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/802336602d8f4ac4bd7e0872d6f3dae9
I20240430 20:11:40.752225 18600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2797eed593fb4b06a3c4d6a28e8075a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:40.755213 19027 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_9049c9eeb10540d3815bab49caa1e82e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:40.755874 19027 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9049c9eeb10540d3815bab49caa1e82e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:40.755925 18725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69f88532952a4591bf1577541c3430e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:40.741585 18696 leader_election.cc:302] T 47c37876694c4a4abb1091c631dab7a5 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a
W20240430 20:11:40.756795 18696 leader_election.cc:341] T 255c13ad8fae4ea4aa8e7b7aabb7602e P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Not found: Tablet not found: 255c13ad8fae4ea4aa8e7b7aabb7602e
I20240430 20:11:40.756902 18696 leader_election.cc:302] T 255c13ad8fae4ea4aa8e7b7aabb7602e P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710
I20240430 20:11:40.747421 18601 tablet_service.cc:1467] Processing CreateTablet for tablet 4dac8b3838954d839094021f1c0503da (DEFAULT_TABLE table=default.Table_7b1809d7d84b4e7896d1f10321e3f20c [id=45839a14bd914828b0fea7675f093233]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:40.757447 18601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4dac8b3838954d839094021f1c0503da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:40.757447 19028 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_7956b5f5fb284c749d82c1980caa8828"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:40.758045 19028 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7956b5f5fb284c749d82c1980caa8828 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:40.758754 19471 ts_tablet_manager.cc:1909] T 802336602d8f4ac4bd7e0872d6f3dae9 P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:40.759069 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 82) took 9168 ms. Trace:
I20240430 20:11:40.759136 19471 rpcz_store.cc:276] 0430 20:11:31.590875 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:31.590926 (+ 51us) service_pool.cc:225] Handling call
0430 20:11:40.344262 (+8753336us) ts_tablet_manager.cc:1120] Deleting tablet 802336602d8f4ac4bd7e0872d6f3dae9
0430 20:11:40.344276 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:40.406122 (+ 61846us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:40.586988 (+180866us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:40.759049 (+172061us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":8666450}
I20240430 20:11:40.759259 19029 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_f4b025d97b144efa9449e830a6f44126" } new_table_name: "default.Table_3c8f9f7b22f64fb1bdc3f1550987233a" modify_external_catalogs: true
I20240430 20:11:40.759481 19471 tablet_replica.cc:330] T 26ed12f498644d15bf45e884dcad172e P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:40.759657 19471 raft_consensus.cc:2229] T 26ed12f498644d15bf45e884dcad172e P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Raft consensus shutting down.
I20240430 20:11:40.759922 19471 raft_consensus.cc:2258] T 26ed12f498644d15bf45e884dcad172e P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:40.748410 18459 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a9066d895744c5bb7d9532b2940603c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:29 UTC) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:40.760421 18454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3191aa54a9644783a2403991c3be27c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:40.760529 19048 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_43f360b4669c4bfd8d6849258825fac5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:40.761014 19048 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_43f360b4669c4bfd8d6849258825fac5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:40.762109 19049 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_a4ea055447464ca6af32fe80bab34c42" } modify_external_catalogs: true
I20240430 20:11:40.762293 19049 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_a4ea055447464ca6af32fe80bab34c42" } modify_external_catalogs: true
I20240430 20:11:40.747613 18591 tablet_service.cc:1467] Processing CreateTablet for tablet 436579f473844c6ca1038b496201a550 (DEFAULT_TABLE table=default.Table_273caa9a42a9474e8787c37032368654 [id=d89426aa652c42e49f956ea85efb0d43]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:40.763077 18591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 436579f473844c6ca1038b496201a550. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:40.747881 18589 tablet_service.cc:1467] Processing CreateTablet for tablet a663030d37614053a7c0369841403422 (DEFAULT_TABLE table=default.Table_e34613952dcb46618c3bc7cecb31fdaf [id=8f1ad1f8d0224b3daa5d84dc6916227b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:40.767042 18589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a663030d37614053a7c0369841403422. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:40.770526 19026 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_849e6e38123a4d8cae466d841850d6f9" } modify_external_catalogs: true
I20240430 20:11:40.748081 18603 tablet_service.cc:1467] Processing CreateTablet for tablet 69f88532952a4591bf1577541c3430e6 (DEFAULT_TABLE table=default.Table_1177828cad1f4a24bd397619dad96170 [id=a95fed1bc3fa4865bf9a35dfc51b659c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:40.770778 19026 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_849e6e38123a4d8cae466d841850d6f9" } modify_external_catalogs: true
I20240430 20:11:40.771005 18603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69f88532952a4591bf1577541c3430e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:40.748157 18595 tablet_service.cc:1514] Processing DeleteTablet for tablet a9f37f0f7ce8425f9b24699513d41d61 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:40 UTC) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:40.748224 18585 tablet_service.cc:1514] Processing DeleteTablet for tablet 4dac8b3838954d839094021f1c0503da with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:40 UTC) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:40.771737 18592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e1cac64cefe4336bd3207b5ca58b631. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:40.775626 19023 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_fd33fc89e9d84f5c87a6572acdfe2c94"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:40.776228 19023 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fd33fc89e9d84f5c87a6572acdfe2c94 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:40.784056 19024 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: Sending DeleteTablet for 3 replicas of tablet 34130f2820064fa0898ef1f9d7de1c6d
I20240430 20:11:40.784667 19024 catalog_manager.cc:6554] ReplaceTablet: tablet 34130f2820064fa0898ef1f9d7de1c6d deleted and replaced by tablet 5b26319488644d6a8b6be60460a5e7af
I20240430 20:11:40.784750 20205 raft_consensus.cc:2781] T 47c37876694c4a4abb1091c631dab7a5 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:40.784885 20205 raft_consensus.cc:482] T 47c37876694c4a4abb1091c631dab7a5 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:40.784955 20205 raft_consensus.cc:3037] T 47c37876694c4a4abb1091c631dab7a5 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:40.785182 18459 tablet_service.cc:1514] Processing DeleteTablet for tablet 34130f2820064fa0898ef1f9d7de1c6d with delete_type TABLET_DATA_DELETED (replaced by tablet 5b26319488644d6a8b6be60460a5e7af) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:40.785472 20198 raft_consensus.cc:2726] T 255c13ad8fae4ea4aa8e7b7aabb7602e P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:40.786880 18734 tablet_service.cc:1514] Processing DeleteTablet for tablet 34130f2820064fa0898ef1f9d7de1c6d with delete_type TABLET_DATA_DELETED (replaced by tablet 5b26319488644d6a8b6be60460a5e7af) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:40.789372 19033 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: Sending DeleteTablet for 3 replicas of tablet 436579f473844c6ca1038b496201a550
I20240430 20:11:40.791009 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 67) took 9661 ms. Trace:
I20240430 20:11:40.791137 18734 tablet_service.cc:1514] Processing DeleteTablet for tablet 436579f473844c6ca1038b496201a550 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:40 UTC) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:40.791105 19495 rpcz_store.cc:276] 0430 20:11:31.137801 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:31.137947 (+ 146us) service_pool.cc:225] Handling call
0430 20:11:40.477612 (+9339665us) ts_tablet_manager.cc:1120] Deleting tablet f5546f9edeea44d7a6c5cb954f5157ba
0430 20:11:40.477622 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:40.581836 (+104214us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:40.604143 (+ 22307us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:40.790985 (+186842us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":9274692}
I20240430 20:11:40.791579 19495 tablet_replica.cc:330] T b688e03ea71940c69bda5520b62c1fc3 P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:40.791558 19004 catalog_manager.cc:4830] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): tablet f5546f9edeea44d7a6c5cb954f5157ba (table default.Table_aa42ff1beec14dd6baf999b8962907e0 [id=836a4da2ef3a455db26af8149ca5415b]) successfully deleted
I20240430 20:11:40.791766 19495 raft_consensus.cc:2229] T b688e03ea71940c69bda5520b62c1fc3 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Raft consensus shutting down.
I20240430 20:11:40.792083 19495 pending_rounds.cc:62] T b688e03ea71940c69bda5520b62c1fc3 P 2efb542aa55e439b99ccb98d1c5c884a: Trying to abort 1 pending ops.
I20240430 20:11:40.792131 19495 pending_rounds.cc:68] T b688e03ea71940c69bda5520b62c1fc3 P 2efb542aa55e439b99ccb98d1c5c884a: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7022624352689512448 op_type: NO_OP noop_request { }
I20240430 20:11:40.792246 19495 raft_consensus.cc:2866] T b688e03ea71940c69bda5520b62c1fc3 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240430 20:11:40.792328 19495 raft_consensus.cc:2258] T b688e03ea71940c69bda5520b62c1fc3 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:40.792865 19495 ts_tablet_manager.cc:1875] T b688e03ea71940c69bda5520b62c1fc3 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:40.796056 18999 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet 802336602d8f4ac4bd7e0872d6f3dae9 (table default.Table_d997b723a0dd48f6bc2d42c9bfd6bc14 [id=7737ec4807974640aa8556cb1c3ca3ef]) successfully deleted
I20240430 20:11:40.796348 18475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76d6e9d547ca4a848eae852944b1d117" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:40.796545 18475 raft_consensus.cc:3037] T 76d6e9d547ca4a848eae852944b1d117 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:40.799229 18475 raft_consensus.cc:2454] T 76d6e9d547ca4a848eae852944b1d117 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
I20240430 20:11:40.800765 18586 tablet_service.cc:1514] Processing DeleteTablet for tablet 34130f2820064fa0898ef1f9d7de1c6d with delete_type TABLET_DATA_DELETED (replaced by tablet 5b26319488644d6a8b6be60460a5e7af) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:40.802742 18756 raft_consensus.cc:3037] T acd23f0a63d5448aaa7ed78422e67bad P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:40.802734 18449 tablet_service.cc:1514] Processing DeleteTablet for tablet 436579f473844c6ca1038b496201a550 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:40 UTC) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:40.803647 18758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76d6e9d547ca4a848eae852944b1d117" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:40.803843 18758 raft_consensus.cc:3037] T 76d6e9d547ca4a848eae852944b1d117 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:40.806833 18585 tablet_service.cc:1514] Processing DeleteTablet for tablet 436579f473844c6ca1038b496201a550 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:40 UTC) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:40.807458 18558 leader_election.cc:302] T 76d6e9d547ca4a848eae852944b1d117 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:40.807973 20047 raft_consensus.cc:2781] T 76d6e9d547ca4a848eae852944b1d117 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:40.808084 20047 raft_consensus.cc:686] T 76d6e9d547ca4a848eae852944b1d117 P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Becoming Leader. State: Replica: 5e1c6397f09747f091534faf93542710, State: Running, Role: LEADER
I20240430 20:11:40.808305 20047 consensus_queue.cc:237] T 76d6e9d547ca4a848eae852944b1d117 P 5e1c6397f09747f091534faf93542710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:40.815829 19033 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_77b10083620349b0bcbb86e5de19f2ce"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:40.816480 19033 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_77b10083620349b0bcbb86e5de19f2ce in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:40.820132 20040 raft_consensus.cc:482] T acf3b6fead41483a82c43dc0286fab7c P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:40.820293 20040 raft_consensus.cc:504] T acf3b6fead41483a82c43dc0286fab7c P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:40.820994 20040 leader_election.cc:288] T acf3b6fead41483a82c43dc0286fab7c P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:40.829396 19028 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_68a87b92c37d40e9a4e4f6e7f314e8bd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:40.830020 19028 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_68a87b92c37d40e9a4e4f6e7f314e8bd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:40.830646 18475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acf3b6fead41483a82c43dc0286fab7c" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:40.830852 18475 raft_consensus.cc:2454] T acf3b6fead41483a82c43dc0286fab7c P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 0.
I20240430 20:11:40.831434 18478 raft_consensus.cc:1261] T 6e826459c3f4437da81c9e6e09f0c22f P 2efb542aa55e439b99ccb98d1c5c884a [term 2 FOLLOWER]: Refusing update from remote peer 5e1c6397f09747f091534faf93542710: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240430 20:11:40.831571 19027 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_401b6bfce5ed4b1bb5643b8a53269b60"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:40.832150 19027 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_401b6bfce5ed4b1bb5643b8a53269b60 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:40.832384 19495 ts_tablet_manager.cc:1888] T b688e03ea71940c69bda5520b62c1fc3 P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:40.832509 19495 log.cc:1196] T b688e03ea71940c69bda5520b62c1fc3 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/b688e03ea71940c69bda5520b62c1fc3
I20240430 20:11:40.832852 19495 ts_tablet_manager.cc:1909] T b688e03ea71940c69bda5520b62c1fc3 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:40.833004 18558 leader_election.cc:302] T acf3b6fead41483a82c43dc0286fab7c P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:40.833609 20061 raft_consensus.cc:2781] T acf3b6fead41483a82c43dc0286fab7c P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:40.833688 20061 raft_consensus.cc:482] T acf3b6fead41483a82c43dc0286fab7c P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:40.833750 20061 raft_consensus.cc:3037] T acf3b6fead41483a82c43dc0286fab7c P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:40.833725 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 69) took 9655 ms. Trace:
I20240430 20:11:40.833807 19495 rpcz_store.cc:276] 0430 20:11:31.178759 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:31.178937 (+ 178us) service_pool.cc:225] Handling call
0430 20:11:40.791530 (+9612593us) ts_tablet_manager.cc:1120] Deleting tablet b688e03ea71940c69bda5520b62c1fc3
0430 20:11:40.791539 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:40.817052 (+ 25513us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:40.832280 (+ 15228us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:40.833706 (+ 1426us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":9582819}
I20240430 20:11:40.834259 19004 catalog_manager.cc:4830] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): tablet b688e03ea71940c69bda5520b62c1fc3 (table default.Table_2ec9639eca4a47439b7859929cd705bc [id=6ffe241384bd42b7a22d41154ac59e3a]) successfully deleted
I20240430 20:11:40.836896 20061 raft_consensus.cc:504] T acf3b6fead41483a82c43dc0286fab7c P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:40.842170 20061 leader_election.cc:288] T acf3b6fead41483a82c43dc0286fab7c P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:40.842922 18478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acf3b6fead41483a82c43dc0286fab7c" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:40.843130 18478 raft_consensus.cc:3037] T acf3b6fead41483a82c43dc0286fab7c P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:40.847868 19048 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_63d833beade34542a349c8f691fd3b98" } new_table_name: "default.Table_acddc4cbf6334546b5fb3e4752fef2a5" modify_external_catalogs: true
I20240430 20:11:40.849736 19495 tablet_replica.cc:330] T 29974dc2c7dd45b99b6cc0712378fc30 P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:40.850011 19495 raft_consensus.cc:2229] T 29974dc2c7dd45b99b6cc0712378fc30 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Raft consensus shutting down.
I20240430 20:11:40.840431 18983 tablet_bootstrap.cc:492] T 32c823d1700949ab8b22adcd83b7bfd1 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:40.850386 19495 raft_consensus.cc:2258] T 29974dc2c7dd45b99b6cc0712378fc30 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:40.851004 19495 ts_tablet_manager.cc:1875] T 29974dc2c7dd45b99b6cc0712378fc30 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:40.851833 18983 tablet_bootstrap.cc:654] T 32c823d1700949ab8b22adcd83b7bfd1 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:40.837321 20040 consensus_queue.cc:1035] T 6e826459c3f4437da81c9e6e09f0c22f P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240430 20:11:40.852586 19029 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fdf53061efd74fa29a9d8208d1fd210e consensus state has no leader
I20240430 20:11:40.853799 18983 tablet_bootstrap.cc:492] T 32c823d1700949ab8b22adcd83b7bfd1 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:40.854063 18983 ts_tablet_manager.cc:1367] T 32c823d1700949ab8b22adcd83b7bfd1 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.000s
I20240430 20:11:40.854296 18478 raft_consensus.cc:2454] T acf3b6fead41483a82c43dc0286fab7c P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
I20240430 20:11:40.854779 18558 leader_election.cc:302] T acf3b6fead41483a82c43dc0286fab7c P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:40.855011 18983 raft_consensus.cc:348] T 32c823d1700949ab8b22adcd83b7bfd1 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:40.855320 18983 raft_consensus.cc:374] T 32c823d1700949ab8b22adcd83b7bfd1 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:40.855432 18983 raft_consensus.cc:729] T 32c823d1700949ab8b22adcd83b7bfd1 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:40.855779 18983 consensus_queue.cc:260] T 32c823d1700949ab8b22adcd83b7bfd1 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:40.855181 20061 raft_consensus.cc:2781] T acf3b6fead41483a82c43dc0286fab7c P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:40.856678 20061 raft_consensus.cc:686] T acf3b6fead41483a82c43dc0286fab7c P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Becoming Leader. State: Replica: 5e1c6397f09747f091534faf93542710, State: Running, Role: LEADER
I20240430 20:11:40.857077 20061 consensus_queue.cc:237] T acf3b6fead41483a82c43dc0286fab7c P 5e1c6397f09747f091534faf93542710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:40.858554 18983 ts_tablet_manager.cc:1398] T 32c823d1700949ab8b22adcd83b7bfd1 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240430 20:11:40.859082 18983 tablet_bootstrap.cc:492] T 5d09baed576e4136bf6d756ca39d08b2 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:40.859581 18985 tablet_bootstrap.cc:492] T 2f61e3f88a6448dfa323204bd8d14cc5 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:40.863078 18983 tablet_bootstrap.cc:654] T 5d09baed576e4136bf6d756ca39d08b2 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:40.863698 18985 ts_tablet_manager.cc:1367] T 2f61e3f88a6448dfa323204bd8d14cc5 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.287s user 0.001s sys 0.001s
I20240430 20:11:40.864789 19049 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: Sending DeleteTablet for 3 replicas of tablet f7d3dea34cf54fe2b949d1b7ca68577a
I20240430 20:11:40.866776 18734 tablet_service.cc:1514] Processing DeleteTablet for tablet f7d3dea34cf54fe2b949d1b7ca68577a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:40 UTC) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:40.866444 18454 tablet_service.cc:1514] Processing DeleteTablet for tablet f7d3dea34cf54fe2b949d1b7ca68577a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:40 UTC) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:40.751807 19489 rpcz_store.cc:276] 0430 20:11:31.653970 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:31.654184 (+ 214us) service_pool.cc:225] Handling call
0430 20:11:40.197788 (+8543604us) ts_tablet_manager.cc:1120] Deleting tablet 7dfdecba959543a795dfcd3eeda29e58
0430 20:11:40.197797 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:40.397687 (+199890us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:40.400967 (+ 3280us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:40.710400 (+309433us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":8543345}
I20240430 20:11:40.871825 19489 tablet_replica.cc:330] T fcc9698abda4491b98e1e879589913a8 P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:40.872040 19489 raft_consensus.cc:2229] T fcc9698abda4491b98e1e879589913a8 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:40.872094 19001 catalog_manager.cc:4830] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): tablet 7dfdecba959543a795dfcd3eeda29e58 (table default.Table_da5216d025844afda5a83ada555bc120 [id=dc55b25c3ca14863a21042e805e3bf9a]) successfully deleted
I20240430 20:11:40.872192 19489 raft_consensus.cc:2258] T fcc9698abda4491b98e1e879589913a8 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:40.872566 18601 tablet_service.cc:1514] Processing DeleteTablet for tablet f7d3dea34cf54fe2b949d1b7ca68577a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:40 UTC) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:40.873131 19489 ts_tablet_manager.cc:1875] T fcc9698abda4491b98e1e879589913a8 P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:40.877481 19029 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_4991bbcb43c540629cbc285f0484d79d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:40.878185 19029 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4991bbcb43c540629cbc285f0484d79d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:40.880363 19489 ts_tablet_manager.cc:1888] T fcc9698abda4491b98e1e879589913a8 P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:40.880487 19489 log.cc:1196] T fcc9698abda4491b98e1e879589913a8 P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/fcc9698abda4491b98e1e879589913a8
I20240430 20:11:40.880451 19049 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_d8df23edb8ce480bb59a2ab0f1fc7521"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240430 20:11:40.880893 19489 ts_tablet_manager.cc:1909] T fcc9698abda4491b98e1e879589913a8 P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
W20240430 20:11:40.881055 19049 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d8df23edb8ce480bb59a2ab0f1fc7521 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:40.881258 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 91) took 9110 ms. Trace:
I20240430 20:11:40.881330 19489 rpcz_store.cc:276] 0430 20:11:31.770964 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:31.806488 (+ 35524us) service_pool.cc:225] Handling call
0430 20:11:40.871766 (+9065278us) ts_tablet_manager.cc:1120] Deleting tablet fcc9698abda4491b98e1e879589913a8
0430 20:11:40.871781 (+ 15us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:40.876437 (+ 4656us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:40.880256 (+ 3819us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:40.881238 (+ 982us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":9060766}
I20240430 20:11:40.881769 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 100) took 8826 ms. Trace:
I20240430 20:11:40.881837 19489 rpcz_store.cc:276] 0430 20:11:32.055171 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:32.055222 (+ 51us) service_pool.cc:225] Handling call
0430 20:11:40.881610 (+8826388us) ts_tablet_manager.cc:1120] Deleting tablet 5d09baed576e4136bf6d756ca39d08b2
0430 20:11:40.881620 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:40.881752 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":8826094}
I20240430 20:11:40.882153 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 103) took 8781 ms. Trace:
I20240430 20:11:40.882216 19489 rpcz_store.cc:276] 0430 20:11:32.100389 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:32.100457 (+ 68us) service_pool.cc:225] Handling call
0430 20:11:40.882039 (+8781582us) ts_tablet_manager.cc:1120] Deleting tablet 2f61e3f88a6448dfa323204bd8d14cc5
0430 20:11:40.882049 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:40.882140 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":8781303}
W20240430 20:11:40.884353 19001 catalog_manager.cc:4810] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): delete failed for tablet 5d09baed576e4136bf6d756ca39d08b2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5d09baed576e4136bf6d756ca39d08b2 already in progress: creating tablet
W20240430 20:11:40.884685 19001 catalog_manager.cc:4810] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): delete failed for tablet 2f61e3f88a6448dfa323204bd8d14cc5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2f61e3f88a6448dfa323204bd8d14cc5 already in progress: creating tablet
I20240430 20:11:40.891846 18758 raft_consensus.cc:2454] T 76d6e9d547ca4a848eae852944b1d117 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
I20240430 20:11:40.900362 20205 raft_consensus.cc:504] T 47c37876694c4a4abb1091c631dab7a5 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:40.900996 20205 leader_election.cc:288] T 47c37876694c4a4abb1091c631dab7a5 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Requested vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:40.901059 19023 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_0858797306c54d81aab9d77e8591e502" } new_table_name: "default.Table_53b6a25ee2c242dcb83b4759648af9f3" modify_external_catalogs: true
I20240430 20:11:40.901481 19495 ts_tablet_manager.cc:1888] T 29974dc2c7dd45b99b6cc0712378fc30 P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:40.901607 19495 log.cc:1196] T 29974dc2c7dd45b99b6cc0712378fc30 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/29974dc2c7dd45b99b6cc0712378fc30
I20240430 20:11:40.901800 18615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c37876694c4a4abb1091c631dab7a5" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:40.901995 19495 ts_tablet_manager.cc:1909] T 29974dc2c7dd45b99b6cc0712378fc30 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:40.901990 18615 raft_consensus.cc:3037] T 47c37876694c4a4abb1091c631dab7a5 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:40.904844 18615 raft_consensus.cc:2454] T 47c37876694c4a4abb1091c631dab7a5 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:40.905356 18760 raft_consensus.cc:1261] T 6e826459c3f4437da81c9e6e09f0c22f P 3c51c33d5616482cafb68e26688820f4 [term 2 FOLLOWER]: Refusing update from remote peer 5e1c6397f09747f091534faf93542710: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240430 20:11:40.905827 18760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acf3b6fead41483a82c43dc0286fab7c" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:40.907392 18760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acf3b6fead41483a82c43dc0286fab7c" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:40.909972 18809 heartbeater.cc:347] Connected to a master server at 127.17.177.62:36441
I20240430 20:11:40.911139 18696 leader_election.cc:302] T 47c37876694c4a4abb1091c631dab7a5 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:40.912698 20096 consensus_queue.cc:1035] T 6e826459c3f4437da81c9e6e09f0c22f P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240430 20:11:40.912986 20198 raft_consensus.cc:2781] T 47c37876694c4a4abb1091c631dab7a5 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:40.913105 20198 raft_consensus.cc:686] T 47c37876694c4a4abb1091c631dab7a5 P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Becoming Leader. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Running, Role: LEADER
W20240430 20:11:40.913271 18557 leader_election.cc:341] T acf3b6fead41483a82c43dc0286fab7c P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:40.913329 20198 consensus_queue.cc:237] T 47c37876694c4a4abb1091c631dab7a5 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:40.919093 19471 ts_tablet_manager.cc:1875] T 26ed12f498644d15bf45e884dcad172e P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:40.919672 19033 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_4bb81ccd60a148d197b8a4ebd9c9cc49"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:40.920444 19033 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4bb81ccd60a148d197b8a4ebd9c9cc49 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240430 20:11:40.921981 18557 leader_election.cc:341] T acf3b6fead41483a82c43dc0286fab7c P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
W20240430 20:11:40.923491 18699 leader_election.cc:341] T 47c37876694c4a4abb1091c631dab7a5 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Not found: Tablet not found: 47c37876694c4a4abb1091c631dab7a5
I20240430 20:11:40.925895 18985 raft_consensus.cc:348] T 2f61e3f88a6448dfa323204bd8d14cc5 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:40.926239 18985 raft_consensus.cc:374] T 2f61e3f88a6448dfa323204bd8d14cc5 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:40.926359 18985 raft_consensus.cc:729] T 2f61e3f88a6448dfa323204bd8d14cc5 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:40.926702 18985 consensus_queue.cc:260] T 2f61e3f88a6448dfa323204bd8d14cc5 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:40.927430 18985 ts_tablet_manager.cc:1398] T 2f61e3f88a6448dfa323204bd8d14cc5 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.060s user 0.002s sys 0.000s
I20240430 20:11:40.935317 19028 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_d441c3bc84ed42358d1bc6e717108efb" } modify_external_catalogs: true
I20240430 20:11:40.935565 19028 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_d441c3bc84ed42358d1bc6e717108efb" } modify_external_catalogs: true
I20240430 20:11:40.939087 18983 tablet_bootstrap.cc:492] T 5d09baed576e4136bf6d756ca39d08b2 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:40.939357 18983 ts_tablet_manager.cc:1367] T 5d09baed576e4136bf6d756ca39d08b2 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.080s user 0.010s sys 0.000s
I20240430 20:11:40.940215 18983 raft_consensus.cc:348] T 5d09baed576e4136bf6d756ca39d08b2 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:40.940698 18983 raft_consensus.cc:374] T 5d09baed576e4136bf6d756ca39d08b2 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:40.940830 18983 raft_consensus.cc:729] T 5d09baed576e4136bf6d756ca39d08b2 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:40.941279 19489 tablet_replica.cc:330] T 47c37876694c4a4abb1091c631dab7a5 P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:40.941193 18983 consensus_queue.cc:260] T 5d09baed576e4136bf6d756ca39d08b2 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:40.941846 18983 ts_tablet_manager.cc:1398] T 5d09baed576e4136bf6d756ca39d08b2 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240430 20:11:40.942193 18983 tablet_bootstrap.cc:492] T 6e754af64cb64fd2a082adae7d5f24da P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:40.944447 18983 tablet_bootstrap.cc:654] T 6e754af64cb64fd2a082adae7d5f24da P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:40.946507 19489 raft_consensus.cc:2229] T 47c37876694c4a4abb1091c631dab7a5 P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Raft consensus shutting down.
I20240430 20:11:40.946956 19489 pending_rounds.cc:62] T 47c37876694c4a4abb1091c631dab7a5 P 3c51c33d5616482cafb68e26688820f4: Trying to abort 1 pending ops.
I20240430 20:11:40.947016 19489 pending_rounds.cc:68] T 47c37876694c4a4abb1091c631dab7a5 P 3c51c33d5616482cafb68e26688820f4: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7022624362143834112 op_type: NO_OP noop_request { }
I20240430 20:11:40.947127 19489 raft_consensus.cc:2866] T 47c37876694c4a4abb1091c631dab7a5 P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240430 20:11:40.947206 19489 raft_consensus.cc:2258] T 47c37876694c4a4abb1091c631dab7a5 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:40.948597 18988 tablet_bootstrap.cc:492] T 2a1e50f99c5d4b44bdbb51b58a92bc75 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:40.948729 18988 ts_tablet_manager.cc:1367] T 2a1e50f99c5d4b44bdbb51b58a92bc75 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.343s user 0.019s sys 0.034s
I20240430 20:11:40.949267 19027 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_a39fa485d1e6488b951a9681aa5f3f00" } modify_external_catalogs: true
I20240430 20:11:40.949478 19027 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_a39fa485d1e6488b951a9681aa5f3f00" } modify_external_catalogs: true
I20240430 20:11:40.949455 18988 raft_consensus.cc:348] T 2a1e50f99c5d4b44bdbb51b58a92bc75 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:40.937429 20236 raft_consensus.cc:482] T 32c823d1700949ab8b22adcd83b7bfd1 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:40.953675 20236 raft_consensus.cc:504] T 32c823d1700949ab8b22adcd83b7bfd1 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:40.954434 20236 leader_election.cc:288] T 32c823d1700949ab8b22adcd83b7bfd1 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:40.955410 18985 tablet_bootstrap.cc:492] T ca64de16d40f4a44b8b411f5526ba11a P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:40.956926 18985 tablet_bootstrap.cc:654] T ca64de16d40f4a44b8b411f5526ba11a P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:40.952291 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 70) took 9747 ms. Trace:
I20240430 20:11:40.949633 18988 raft_consensus.cc:374] T 2a1e50f99c5d4b44bdbb51b58a92bc75 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:40.957592 18988 raft_consensus.cc:729] T 2a1e50f99c5d4b44bdbb51b58a92bc75 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:40.957579 19495 rpcz_store.cc:276] 0430 20:11:31.204564 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:31.204628 (+ 64us) service_pool.cc:225] Handling call
0430 20:11:40.849674 (+9645046us) ts_tablet_manager.cc:1120] Deleting tablet 29974dc2c7dd45b99b6cc0712378fc30
0430 20:11:40.849689 (+ 15us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:40.895224 (+ 45535us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:40.901364 (+ 6140us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:40.952265 (+ 50901us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":9635018}
I20240430 20:11:40.957856 18615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32c823d1700949ab8b22adcd83b7bfd1" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:40.957834 18988 consensus_queue.cc:260] T 2a1e50f99c5d4b44bdbb51b58a92bc75 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:40.958070 19495 tablet_replica.cc:330] T eed6338ad3d64859af087ee15e1ec798 P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:40.958079 18615 raft_consensus.cc:2454] T 32c823d1700949ab8b22adcd83b7bfd1 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 0.
I20240430 20:11:40.955555 18758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32c823d1700949ab8b22adcd83b7bfd1" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:40.958253 19495 raft_consensus.cc:2229] T eed6338ad3d64859af087ee15e1ec798 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:40.958295 19004 catalog_manager.cc:4830] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): tablet 29974dc2c7dd45b99b6cc0712378fc30 (table default.Table_28b63364f56b484eb48763e662dda489 [id=cc553a8f7052423b9724a3b28c63592d]) successfully deleted
I20240430 20:11:40.958549 19495 raft_consensus.cc:2258] T eed6338ad3d64859af087ee15e1ec798 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:40.958519 18415 leader_election.cc:302] T 32c823d1700949ab8b22adcd83b7bfd1 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
W20240430 20:11:40.958683 18417 leader_election.cc:341] T 32c823d1700949ab8b22adcd83b7bfd1 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
W20240430 20:11:40.956925 20270 consensus_queue.cc:1159] T 47c37876694c4a4abb1091c631dab7a5 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "3c51c33d5616482cafb68e26688820f4" status { last_received { term: 1 index: 1 } last_committed_idx: 0 last_received_current_leader { term: 1 index: 1 } }
I20240430 20:11:40.959129 19495 ts_tablet_manager.cc:1875] T eed6338ad3d64859af087ee15e1ec798 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:40.953414 18810 heartbeater.cc:503] Master 127.17.177.60:37225 was elected leader, sending a full tablet report...
I20240430 20:11:40.959618 20236 raft_consensus.cc:2781] T 32c823d1700949ab8b22adcd83b7bfd1 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:40.959699 20236 raft_consensus.cc:482] T 32c823d1700949ab8b22adcd83b7bfd1 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:40.959764 20236 raft_consensus.cc:3037] T 32c823d1700949ab8b22adcd83b7bfd1 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:40.960114 19489 ts_tablet_manager.cc:1875] T 47c37876694c4a4abb1091c631dab7a5 P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:40.962229 20236 raft_consensus.cc:504] T 32c823d1700949ab8b22adcd83b7bfd1 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:40.963868 18988 ts_tablet_manager.cc:1398] T 2a1e50f99c5d4b44bdbb51b58a92bc75 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.015s user 0.001s sys 0.001s
I20240430 20:11:40.964154 18988 tablet_bootstrap.cc:492] T e6eac92dfa5f4456bc49571878f0f8dd P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:40.965662 18988 tablet_bootstrap.cc:654] T e6eac92dfa5f4456bc49571878f0f8dd P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
W20240430 20:11:40.966774 19048 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1050ab528c7d41559831652a1437d358 has no consensus state
I20240430 20:11:40.967223 19048 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_d5e6a23db079462bb59fa312d8c07b7f" } new_table_name: "default.Table_c7c87a8e019944cab3ec99e9563ac01a" modify_external_catalogs: true
I20240430 20:11:40.968559 18615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32c823d1700949ab8b22adcd83b7bfd1" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:40.968750 18615 raft_consensus.cc:3037] T 32c823d1700949ab8b22adcd83b7bfd1 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:40.971422 18615 raft_consensus.cc:2454] T 32c823d1700949ab8b22adcd83b7bfd1 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 1.
I20240430 20:11:40.971854 18415 leader_election.cc:302] T 32c823d1700949ab8b22adcd83b7bfd1 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:40.972157 20105 raft_consensus.cc:2781] T 32c823d1700949ab8b22adcd83b7bfd1 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:40.972271 20105 raft_consensus.cc:686] T 32c823d1700949ab8b22adcd83b7bfd1 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Becoming Leader. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Running, Role: LEADER
I20240430 20:11:40.972564 20105 consensus_queue.cc:237] T 32c823d1700949ab8b22adcd83b7bfd1 P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:40.974376 20236 leader_election.cc:288] T 32c823d1700949ab8b22adcd83b7bfd1 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Requested vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:40.974893 18758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32c823d1700949ab8b22adcd83b7bfd1" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
W20240430 20:11:40.975461 18417 leader_election.cc:341] T 32c823d1700949ab8b22adcd83b7bfd1 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:40.983247 18725 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d09baed576e4136bf6d756ca39d08b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:31 UTC) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:40.985278 19029 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_ff3ffe1bdd1e497697316f4ac1731e3d" } new_table_name: "default.Table_de88cafd92b149f9b545369bff71033c" modify_external_catalogs: true
I20240430 20:11:40.991101 19471 ts_tablet_manager.cc:1888] T 26ed12f498644d15bf45e884dcad172e P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:40.991328 19471 log.cc:1196] T 26ed12f498644d15bf45e884dcad172e P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/26ed12f498644d15bf45e884dcad172e
I20240430 20:11:40.994562 18725 tablet_service.cc:1514] Processing DeleteTablet for tablet fcc9698abda4491b98e1e879589913a8 with delete_type TABLET_DATA_DELETED (replaced by tablet de1dfe6c82d34a4d8d02e466c55327ff) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:40.997952 18725 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f61e3f88a6448dfa323204bd8d14cc5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:31 UTC) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:41.006095 18985 tablet_bootstrap.cc:492] T ca64de16d40f4a44b8b411f5526ba11a P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:41.006878 18985 ts_tablet_manager.cc:1367] T ca64de16d40f4a44b8b411f5526ba11a P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.052s user 0.002s sys 0.001s
I20240430 20:11:41.007704 18985 raft_consensus.cc:348] T ca64de16d40f4a44b8b411f5526ba11a P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:41.008011 18985 raft_consensus.cc:374] T ca64de16d40f4a44b8b411f5526ba11a P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:41.008111 18985 raft_consensus.cc:729] T ca64de16d40f4a44b8b411f5526ba11a P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:41.008379 18985 consensus_queue.cc:260] T ca64de16d40f4a44b8b411f5526ba11a P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:41.008947 18985 ts_tablet_manager.cc:1398] T ca64de16d40f4a44b8b411f5526ba11a P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240430 20:11:41.009270 18985 tablet_bootstrap.cc:492] T 7542c7fe6cad452ab718f9ef54dbbbbe P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
W20240430 20:11:41.009944 18417 consensus_peers.cc:485] T fcc9698abda4491b98e1e879589913a8 P 2efb542aa55e439b99ccb98d1c5c884a -> Peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Couldn't send request to peer 3c51c33d5616482cafb68e26688820f4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fcc9698abda4491b98e1e879589913a8. This is attempt 1: this message will repeat every 5th retry.
I20240430 20:11:41.010452 18725 tablet_service.cc:1514] Processing DeleteTablet for tablet 7dfdecba959543a795dfcd3eeda29e58 with delete_type TABLET_DATA_DELETED (replaced by tablet 95383d9d5a234a4d9d841a8b114428b0) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:41.011039 18985 tablet_bootstrap.cc:654] T 7542c7fe6cad452ab718f9ef54dbbbbe P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:41.014394 19495 ts_tablet_manager.cc:1888] T eed6338ad3d64859af087ee15e1ec798 P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:41.015043 19471 ts_tablet_manager.cc:1909] T 26ed12f498644d15bf45e884dcad172e P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
W20240430 20:11:41.016077 19051 catalog_manager.cc:5676] Failed to send AlterTable RPC for tablet ed0c37a71ede4bfc9eb5ad1a4365f25e (table default.Table_a2e85efc6b454524b0b122102ec07002 [id=86d795a82e674d859104741ebd04f4e8], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet ed0c37a71ede4bfc9eb5ad1a4365f25e consensus state has no leader
I20240430 20:11:41.017462 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 83) took 9330 ms. Trace:
I20240430 20:11:40.995687 18983 tablet_bootstrap.cc:492] T 6e754af64cb64fd2a082adae7d5f24da P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:41.017817 18983 ts_tablet_manager.cc:1367] T 6e754af64cb64fd2a082adae7d5f24da P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.076s user 0.004s sys 0.000s
I20240430 20:11:41.018622 18983 raft_consensus.cc:348] T 6e754af64cb64fd2a082adae7d5f24da P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:41.018887 18983 raft_consensus.cc:374] T 6e754af64cb64fd2a082adae7d5f24da P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:41.018981 18983 raft_consensus.cc:729] T 6e754af64cb64fd2a082adae7d5f24da P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:41.019274 18983 consensus_queue.cc:260] T 6e754af64cb64fd2a082adae7d5f24da P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:41.019917 18983 ts_tablet_manager.cc:1398] T 6e754af64cb64fd2a082adae7d5f24da P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
W20240430 20:11:41.018965 19023 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 47bf64ecc41b445da4ff697e141243dd consensus state has no leader
I20240430 20:11:41.020480 19023 master_service.cc:946] ReplaceTablet: received request to replace tablet 95383d9d5a234a4d9d841a8b114428b0 from {username='slave'} at 127.0.0.1:51664
I20240430 20:11:41.020745 18983 tablet_bootstrap.cc:492] T 3fd0e8c360734dcbb75dc2958b70c2b8 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
W20240430 20:11:41.024453 19051 catalog_manager.cc:5676] Failed to send AlterTable RPC for tablet 62732af95c6a4c749c5b726f0460dc72 (table default.Table_be2ebe77f859489f9cd3e912cb1edc8b [id=2d44ec94131148048021b2ee6366ba30], current schema version=2): Not found: failed to reset TS proxy: No leader found: Tablet 62732af95c6a4c749c5b726f0460dc72 consensus state has no leader
W20240430 20:11:41.030779 19051 catalog_manager.cc:5676] Failed to send AlterTable RPC for tablet 9c73d0a6bc414d3b9900570848d19a51 (table default.Table_49a012a7c4c8457fadb0a8f0fc9654c6 [id=eecb2f8eeef547c5ab1abe6a2430e592], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 9c73d0a6bc414d3b9900570848d19a51 consensus state has no leader
I20240430 20:11:41.035928 19033 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_68a87b92c37d40e9a4e4f6e7f314e8bd" } modify_external_catalogs: true
I20240430 20:11:41.036198 19033 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_68a87b92c37d40e9a4e4f6e7f314e8bd" } modify_external_catalogs: true
I20240430 20:11:41.044050 18983 tablet_bootstrap.cc:654] T 3fd0e8c360734dcbb75dc2958b70c2b8 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:41.046880 18988 tablet_bootstrap.cc:492] T e6eac92dfa5f4456bc49571878f0f8dd P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:41.047129 18988 ts_tablet_manager.cc:1367] T e6eac92dfa5f4456bc49571878f0f8dd P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.083s user 0.003s sys 0.002s
I20240430 20:11:41.047928 18988 raft_consensus.cc:348] T e6eac92dfa5f4456bc49571878f0f8dd P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:41.048197 18988 raft_consensus.cc:374] T e6eac92dfa5f4456bc49571878f0f8dd P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:41.048290 18988 raft_consensus.cc:729] T e6eac92dfa5f4456bc49571878f0f8dd P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:41.048571 18988 consensus_queue.cc:260] T e6eac92dfa5f4456bc49571878f0f8dd P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:41.049204 18988 ts_tablet_manager.cc:1398] T e6eac92dfa5f4456bc49571878f0f8dd P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240430 20:11:41.050770 18988 tablet_bootstrap.cc:492] T 0eb4f3e7ece6496293e01b63620d644b P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:41.052313 18988 tablet_bootstrap.cc:654] T 0eb4f3e7ece6496293e01b63620d644b P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:41.055375 19471 rpcz_store.cc:276] 0430 20:11:31.687026 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:31.687095 (+ 69us) service_pool.cc:225] Handling call
0430 20:11:40.759432 (+9072337us) ts_tablet_manager.cc:1120] Deleting tablet 26ed12f498644d15bf45e884dcad172e
0430 20:11:40.759442 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:40.922361 (+162919us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:40.990973 (+ 68612us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:41.017435 (+ 26462us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":9072103}
I20240430 20:11:41.056169 18999 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet 26ed12f498644d15bf45e884dcad172e (table default.Table_3f42877693b04aaa837c89b8983d5a72 [id=6d499a625cee4df2a4f40ee189cc6220]) successfully deleted
I20240430 20:11:41.056651 19471 tablet_replica.cc:330] T 7dfdecba959543a795dfcd3eeda29e58 P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:41.056965 19471 raft_consensus.cc:2229] T 7dfdecba959543a795dfcd3eeda29e58 P 5e1c6397f09747f091534faf93542710 [term 2 LEADER]: Raft consensus shutting down.
I20240430 20:11:41.057413 19471 raft_consensus.cc:2258] T 7dfdecba959543a795dfcd3eeda29e58 P 5e1c6397f09747f091534faf93542710 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:41.058069 19471 ts_tablet_manager.cc:1875] T 7dfdecba959543a795dfcd3eeda29e58 P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240430 20:11:41.058842 19048 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3e5225f1f5f544ecbf8dfa470af3daca consensus state has no leader
I20240430 20:11:41.058868 19495 log.cc:1196] T eed6338ad3d64859af087ee15e1ec798 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/eed6338ad3d64859af087ee15e1ec798
I20240430 20:11:41.059896 19495 ts_tablet_manager.cc:1909] T eed6338ad3d64859af087ee15e1ec798 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:41.062469 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 72) took 9853 ms. Trace:
I20240430 20:11:41.066560 18983 tablet_bootstrap.cc:492] T 3fd0e8c360734dcbb75dc2958b70c2b8 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:41.066812 18983 ts_tablet_manager.cc:1367] T 3fd0e8c360734dcbb75dc2958b70c2b8 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.046s user 0.005s sys 0.001s
I20240430 20:11:41.067669 18983 raft_consensus.cc:348] T 3fd0e8c360734dcbb75dc2958b70c2b8 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:41.067945 18983 raft_consensus.cc:374] T 3fd0e8c360734dcbb75dc2958b70c2b8 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:41.068043 18983 raft_consensus.cc:729] T 3fd0e8c360734dcbb75dc2958b70c2b8 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:41.068382 18983 consensus_queue.cc:260] T 3fd0e8c360734dcbb75dc2958b70c2b8 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:41.062582 19495 rpcz_store.cc:276] 0430 20:11:31.209464 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:31.209520 (+ 56us) service_pool.cc:225] Handling call
0430 20:11:40.958011 (+9748491us) ts_tablet_manager.cc:1120] Deleting tablet eed6338ad3d64859af087ee15e1ec798
0430 20:11:40.958021 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:40.962208 (+ 4187us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:41.014280 (+ 52072us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:41.062438 (+ 48158us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":9728977}
I20240430 20:11:41.071038 19004 catalog_manager.cc:4830] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): tablet eed6338ad3d64859af087ee15e1ec798 (table default.Table_1dc3c3ef217d4f8b9e8fdb60180c5d0f [id=bc29c76ef4a448a2863c278309b548e8]) successfully deleted
I20240430 20:11:41.071125 19495 tablet_replica.cc:330] T f5236574672b477a9026e91adcaf70ec P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:41.071316 19495 raft_consensus.cc:2229] T f5236574672b477a9026e91adcaf70ec P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Raft consensus shutting down.
I20240430 20:11:41.071553 18528 heartbeater.cc:347] Connected to a master server at 127.17.177.62:36441
I20240430 20:11:41.071667 19495 raft_consensus.cc:2258] T f5236574672b477a9026e91adcaf70ec P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:41.072281 19495 ts_tablet_manager.cc:1875] T f5236574672b477a9026e91adcaf70ec P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:41.079789 18985 tablet_bootstrap.cc:492] T 7542c7fe6cad452ab718f9ef54dbbbbe P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:41.080022 18985 ts_tablet_manager.cc:1367] T 7542c7fe6cad452ab718f9ef54dbbbbe P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.071s user 0.003s sys 0.000s
I20240430 20:11:41.080732 18985 raft_consensus.cc:348] T 7542c7fe6cad452ab718f9ef54dbbbbe P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:41.080973 18985 raft_consensus.cc:374] T 7542c7fe6cad452ab718f9ef54dbbbbe P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:41.081068 18985 raft_consensus.cc:729] T 7542c7fe6cad452ab718f9ef54dbbbbe P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:41.081334 18985 consensus_queue.cc:260] T 7542c7fe6cad452ab718f9ef54dbbbbe P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:41.083462 18983 ts_tablet_manager.cc:1398] T 3fd0e8c360734dcbb75dc2958b70c2b8 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.017s user 0.001s sys 0.001s
I20240430 20:11:41.083452 18985 ts_tablet_manager.cc:1398] T 7542c7fe6cad452ab718f9ef54dbbbbe P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240430 20:11:41.083765 18985 tablet_bootstrap.cc:492] T 73453a7fa30342e29db552b17b62f0c1 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:41.086236 20198 raft_consensus.cc:482] T f5546f9edeea44d7a6c5cb954f5157ba P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:41.086346 18985 tablet_bootstrap.cc:654] T 73453a7fa30342e29db552b17b62f0c1 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:41.086386 20198 raft_consensus.cc:504] T f5546f9edeea44d7a6c5cb954f5157ba P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:41.087064 20198 leader_election.cc:288] T f5546f9edeea44d7a6c5cb954f5157ba P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:41.087879 18615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5546f9edeea44d7a6c5cb954f5157ba" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:41.088163 18615 raft_consensus.cc:2379] T f5546f9edeea44d7a6c5cb954f5157ba P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c51c33d5616482cafb68e26688820f4 in current term 1: Already voted for candidate 2efb542aa55e439b99ccb98d1c5c884a in this term.
W20240430 20:11:41.089547 18699 leader_election.cc:341] T f5546f9edeea44d7a6c5cb954f5157ba P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Not found: Tablet not found: f5546f9edeea44d7a6c5cb954f5157ba
I20240430 20:11:41.089658 18699 leader_election.cc:302] T f5546f9edeea44d7a6c5cb954f5157ba P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710
I20240430 20:11:41.090154 20198 raft_consensus.cc:3037] T f5546f9edeea44d7a6c5cb954f5157ba P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:41.092312 18983 tablet_bootstrap.cc:492] T 7542c7fe6cad452ab718f9ef54dbbbbe P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:41.092769 20198 raft_consensus.cc:2726] T f5546f9edeea44d7a6c5cb954f5157ba P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240430 20:11:41.094718 18121 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.17.177.2:46027, user_credentials={real_user=slave}} blocked reactor thread for 45468.2us
W20240430 20:11:41.096830 19029 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e041756c962b4a9f97c2cb314e7d8c05 has no consensus state
W20240430 20:11:41.095579 19051 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.17.177.3:42157 (request call id 83) took 1397 ms (client timeout 1000 ms). Trace:
I20240430 20:11:41.099992 19029 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_91c01cce23fe4f4182952019c19875c0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:41.100665 19029 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_91c01cce23fe4f4182952019c19875c0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:41.103860 18983 tablet_bootstrap.cc:654] T 7542c7fe6cad452ab718f9ef54dbbbbe P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:41.108822 18988 tablet_bootstrap.cc:492] T 0eb4f3e7ece6496293e01b63620d644b P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:41.108877 20053 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: Bootstrap replayed 1/1 log segments. Stats: ops{read=309 overwritten=0 applied=308 ignored=0} inserts{seen=284 ignored=0} mutations{seen=337 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240430 20:11:41.108989 18988 ts_tablet_manager.cc:1367] T 0eb4f3e7ece6496293e01b63620d644b P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.058s user 0.005s sys 0.000s
I20240430 20:11:41.109778 20053 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: Bootstrap complete.
I20240430 20:11:41.110716 18988 raft_consensus.cc:348] T 0eb4f3e7ece6496293e01b63620d644b P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:41.110924 18988 raft_consensus.cc:374] T 0eb4f3e7ece6496293e01b63620d644b P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:41.110980 18988 raft_consensus.cc:729] T 0eb4f3e7ece6496293e01b63620d644b P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:41.111192 18988 consensus_queue.cc:260] T 0eb4f3e7ece6496293e01b63620d644b P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:41.111703 18988 ts_tablet_manager.cc:1398] T 0eb4f3e7ece6496293e01b63620d644b P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240430 20:11:41.112046 18988 tablet_bootstrap.cc:492] T acf3b6fead41483a82c43dc0286fab7c P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:41.113785 18988 tablet_bootstrap.cc:654] T acf3b6fead41483a82c43dc0286fab7c P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:41.113551 19489 ts_tablet_manager.cc:1888] T 47c37876694c4a4abb1091c631dab7a5 P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:41.115231 19489 log.cc:1196] T 47c37876694c4a4abb1091c631dab7a5 P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/47c37876694c4a4abb1091c631dab7a5
I20240430 20:11:41.116806 19489 ts_tablet_manager.cc:1909] T 47c37876694c4a4abb1091c631dab7a5 P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
W20240430 20:11:41.118433 18417 consensus_peers.cc:485] T 802336602d8f4ac4bd7e0872d6f3dae9 P 2efb542aa55e439b99ccb98d1c5c884a -> Peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Couldn't send request to peer 3c51c33d5616482cafb68e26688820f4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 802336602d8f4ac4bd7e0872d6f3dae9. This is attempt 1: this message will repeat every 5th retry.
I20240430 20:11:41.120553 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 106) took 8994 ms. Trace:
I20240430 20:11:41.120945 19489 rpcz_store.cc:276] 0430 20:11:32.126397 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:32.126476 (+ 79us) service_pool.cc:225] Handling call
0430 20:11:40.941209 (+8814733us) ts_tablet_manager.cc:1120] Deleting tablet 47c37876694c4a4abb1091c631dab7a5
0430 20:11:40.941227 (+ 18us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:40.946778 (+ 5551us) spinlock_profiling.cc:243] Waited 4.97 ms on lock 0x6160000f7054. stack: 00007f30e7d63756 00007f30e7d63510 00007f30e6d9648d 0000000000955804 00000000009557cd 000000000095519a 00007f30f35d1a00 00007f30f48ab853 00007f30f733e4bc 00007f30f7370814 00007f30f7351b00 00007f30f735188d 0000000000a60bcc 00007f30e7da9b74 00007f30e7dad19b 00007f30e7dad018
0430 20:11:41.017102 (+ 70324us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:41.112924 (+ 95822us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:41.120534 (+ 7610us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":11429632,"tablet-delete.queue_time_us":8749839}
I20240430 20:11:41.122568 19489 tablet_replica.cc:330] T 75ebf051383e42529dcb5f7dd0770a4b P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:41.123312 19489 raft_consensus.cc:2229] T 75ebf051383e42529dcb5f7dd0770a4b P 3c51c33d5616482cafb68e26688820f4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:41.124027 19489 raft_consensus.cc:2258] T 75ebf051383e42529dcb5f7dd0770a4b P 3c51c33d5616482cafb68e26688820f4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:41.126725 19001 catalog_manager.cc:4830] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): tablet 47c37876694c4a4abb1091c631dab7a5 (table default.Table_85072f20d90d4470bf3f65189b19fd51 [id=0fd2754282af4b2b9fcc7952be704cd3]) successfully deleted
I20240430 20:11:41.129876 19489 ts_tablet_manager.cc:1875] T 75ebf051383e42529dcb5f7dd0770a4b P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:41.130364 19471 ts_tablet_manager.cc:1888] T 7dfdecba959543a795dfcd3eeda29e58 P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.2
I20240430 20:11:41.130570 19471 log.cc:1196] T 7dfdecba959543a795dfcd3eeda29e58 P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/7dfdecba959543a795dfcd3eeda29e58
I20240430 20:11:41.131043 19471 ts_tablet_manager.cc:1909] T 7dfdecba959543a795dfcd3eeda29e58 P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:41.132028 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 84) took 9444 ms. Trace:
I20240430 20:11:41.132812 19033 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet b4a2d638a5bb4af9bc6055eed6e51bb3
I20240430 20:11:41.134254 19033 master_service.cc:946] ReplaceTablet: received request to replace tablet e6c0c808bfc94be3837ab835292b3c1b from {username='slave'} at 127.0.0.1:51664
I20240430 20:11:41.135479 19029 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_94da8b9b6d284bd28c2310417f57e095"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:41.137423 19029 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_94da8b9b6d284bd28c2310417f57e095 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:41.137897 19495 ts_tablet_manager.cc:1888] T f5236574672b477a9026e91adcaf70ec P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:41.138022 19495 log.cc:1196] T f5236574672b477a9026e91adcaf70ec P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/f5236574672b477a9026e91adcaf70ec
I20240430 20:11:41.138365 19495 ts_tablet_manager.cc:1909] T f5236574672b477a9026e91adcaf70ec P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:41.138901 19471 rpcz_store.cc:276] 0430 20:11:31.687158 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:31.688095 (+ 937us) service_pool.cc:225] Handling call
0430 20:11:41.056579 (+9368484us) ts_tablet_manager.cc:1120] Deleting tablet 7dfdecba959543a795dfcd3eeda29e58
0430 20:11:41.056595 (+ 16us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:41.088627 (+ 32032us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:41.122569 (+ 33942us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:41.132000 (+ 9431us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":9367669}
I20240430 20:11:41.139391 19471 tablet_replica.cc:330] T 77285afcafe04a829e485998bbbe00f0 P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:41.139590 19471 raft_consensus.cc:2229] T 77285afcafe04a829e485998bbbe00f0 P 5e1c6397f09747f091534faf93542710 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:41.139791 19471 raft_consensus.cc:2258] T 77285afcafe04a829e485998bbbe00f0 P 5e1c6397f09747f091534faf93542710 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:41.139899 18999 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet 7dfdecba959543a795dfcd3eeda29e58 (table default.Table_da5216d025844afda5a83ada555bc120 [id=dc55b25c3ca14863a21042e805e3bf9a]) successfully deleted
W20240430 20:11:41.098421 19051 rpcz_store.cc:269] 0430 20:11:39.697276 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:39.745706 (+ 48430us) service_pool.cc:225] Handling call
0430 20:11:41.095141 (+1349435us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:40.398931 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:40.399027 (+ 96us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:40.399033 (+ 6us) write_op.cc:427] Acquired schema lock
0430 20:11:40.399036 (+ 3us) tablet.cc:660] Decoding operations
0430 20:11:40.399133 (+ 97us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:40.399143 (+ 10us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:40.399147 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0430 20:11:40.399200 (+ 53us) tablet.cc:699] Row locks acquired
0430 20:11:40.399202 (+ 2us) write_op.cc:260] PREPARE: finished
0430 20:11:40.399237 (+ 35us) write_op.cc:270] Start()
0430 20:11:40.399266 (+ 29us) write_op.cc:276] Timestamp: P: 1714507900399143 usec, L: 0
0430 20:11:40.399269 (+ 3us) op_driver.cc:331] REPLICATION: starting
0430 20:11:40.399453 (+ 184us) log.cc:842] Serialized 530 byte log entry
0430 20:11:40.422737 (+ 23284us) op_driver.cc:444] REPLICATION: finished
0430 20:11:40.423481 (+ 744us) write_op.cc:301] APPLY: starting
0430 20:11:40.423493 (+ 12us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:40.423524 (+ 31us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:40.423553 (+ 29us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:40.423599 (+ 46us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:40.423630 (+ 31us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0430 20:11:40.423638 (+ 8us) write_op.cc:312] APPLY: finished
0430 20:11:40.423703 (+ 65us) log.cc:842] Serialized 23 byte log entry
0430 20:11:40.423745 (+ 42us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:40.423759 (+ 14us) write_op.cc:446] Released schema lock
0430 20:11:40.423766 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:40.423823 (+ 57us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:40.423829 (+ 6us) write_op.cc:446] Released schema lock
Metrics: {"mutex_wait_us":4053,"reported_tablets":162,"child_traces":[["op",{"apply.queue_time_us":710,"apply.run_cpu_time_us":408,"apply.run_wall_time_us":408,"num_ops":1,"prepare.queue_time_us":2270,"prepare.run_cpu_time_us":600,"prepare.run_wall_time_us":600,"replication_time_us":23453}]]}
I20240430 20:11:41.140769 19051 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_57208e9f76f14782a806f0a2971a3a58"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:41.141309 19051 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_57208e9f76f14782a806f0a2971a3a58 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:41.148438 19471 ts_tablet_manager.cc:1875] T 77285afcafe04a829e485998bbbe00f0 P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:41.150151 19033 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: Sending DeleteTablet for 3 replicas of tablet e6c0c808bfc94be3837ab835292b3c1b
I20240430 20:11:41.150676 19033 catalog_manager.cc:6554] ReplaceTablet: tablet e6c0c808bfc94be3837ab835292b3c1b deleted and replaced by tablet 772eba5f24c549c791cad75565858285
I20240430 20:11:41.149798 20053 raft_consensus.cc:348] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8 [term 4 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b3f98793354c7aaa5c106e8ec31ed8" member_type: VOTER last_known_addr { host: "127.17.177.62" port: 36441 } } peers { permanent_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e" member_type: VOTER last_known_addr { host: "127.17.177.61" port: 34777 } } peers { permanent_uuid: "b632d6ef4c104a058a6db208024cab33" member_type: VOTER last_known_addr { host: "127.17.177.60" port: 37225 } }
I20240430 20:11:41.150920 19033 master_service.cc:946] ReplaceTablet: received request to replace tablet 283e5e73cbc04daba0de19f4541f9fd0 from {username='slave'} at 127.0.0.1:51664
I20240430 20:11:41.151754 18985 tablet_bootstrap.cc:492] T 73453a7fa30342e29db552b17b62f0c1 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:41.151899 18985 ts_tablet_manager.cc:1367] T 73453a7fa30342e29db552b17b62f0c1 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.068s user 0.001s sys 0.002s
I20240430 20:11:41.152185 18725 tablet_service.cc:1514] Processing DeleteTablet for tablet e6c0c808bfc94be3837ab835292b3c1b with delete_type TABLET_DATA_DELETED (replaced by tablet 772eba5f24c549c791cad75565858285) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:41.152478 18450 tablet_service.cc:1514] Processing DeleteTablet for tablet e6c0c808bfc94be3837ab835292b3c1b with delete_type TABLET_DATA_DELETED (replaced by tablet 772eba5f24c549c791cad75565858285) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:41.152573 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 71) took 9944 ms. Trace:
I20240430 20:11:41.152638 19495 rpcz_store.cc:276] 0430 20:11:31.207868 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:31.208016 (+ 148us) service_pool.cc:225] Handling call
0430 20:11:41.071070 (+9863054us) ts_tablet_manager.cc:1120] Deleting tablet f5236574672b477a9026e91adcaf70ec
0430 20:11:41.071081 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:41.124789 (+ 53708us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:41.137772 (+ 12983us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:41.152558 (+ 14786us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":9839077}
I20240430 20:11:41.153220 19004 catalog_manager.cc:4830] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): tablet f5236574672b477a9026e91adcaf70ec (table default.Table_b8388e076be44f45ab6ed2ab3bce74a8 [id=00dd4c11a1cc4018bb900da95768bb88]) successfully deleted
I20240430 20:11:41.153256 18985 raft_consensus.cc:348] T 73453a7fa30342e29db552b17b62f0c1 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:41.153457 18985 raft_consensus.cc:374] T 73453a7fa30342e29db552b17b62f0c1 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:41.153507 18985 raft_consensus.cc:729] T 73453a7fa30342e29db552b17b62f0c1 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:41.153811 18988 tablet_bootstrap.cc:492] T acf3b6fead41483a82c43dc0286fab7c P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:41.153734 18985 consensus_queue.cc:260] T 73453a7fa30342e29db552b17b62f0c1 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:41.153954 18988 ts_tablet_manager.cc:1367] T acf3b6fead41483a82c43dc0286fab7c P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.042s user 0.003s sys 0.000s
I20240430 20:11:41.154522 18601 tablet_service.cc:1514] Processing DeleteTablet for tablet e6c0c808bfc94be3837ab835292b3c1b with delete_type TABLET_DATA_DELETED (replaced by tablet 772eba5f24c549c791cad75565858285) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:41.154734 18988 raft_consensus.cc:348] T acf3b6fead41483a82c43dc0286fab7c P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:41.155521 18988 raft_consensus.cc:374] T acf3b6fead41483a82c43dc0286fab7c P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:41.155596 18988 raft_consensus.cc:729] T acf3b6fead41483a82c43dc0286fab7c P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:41.155838 18988 consensus_queue.cc:260] T acf3b6fead41483a82c43dc0286fab7c P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
W20240430 20:11:41.159636 18415 consensus_peers.cc:485] T 802336602d8f4ac4bd7e0872d6f3dae9 P 2efb542aa55e439b99ccb98d1c5c884a -> Peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Couldn't send request to peer 5e1c6397f09747f091534faf93542710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 802336602d8f4ac4bd7e0872d6f3dae9. This is attempt 1: this message will repeat every 5th retry.
I20240430 20:11:41.159715 18985 ts_tablet_manager.cc:1398] T 73453a7fa30342e29db552b17b62f0c1 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20240430 20:11:41.160126 20096 raft_consensus.cc:482] T 5d09baed576e4136bf6d756ca39d08b2 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:41.160279 20096 raft_consensus.cc:504] T 5d09baed576e4136bf6d756ca39d08b2 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:41.160939 20096 leader_election.cc:288] T 5d09baed576e4136bf6d756ca39d08b2 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:41.161491 18758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d09baed576e4136bf6d756ca39d08b2" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
W20240430 20:11:41.162019 18557 leader_election.cc:341] T 5d09baed576e4136bf6d756ca39d08b2 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:41.162555 18478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d09baed576e4136bf6d756ca39d08b2" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:41.162770 18478 raft_consensus.cc:2454] T 5d09baed576e4136bf6d756ca39d08b2 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 0.
I20240430 20:11:41.164011 18558 leader_election.cc:302] T 5d09baed576e4136bf6d756ca39d08b2 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters: 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:41.164367 20096 raft_consensus.cc:2781] T 5d09baed576e4136bf6d756ca39d08b2 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:41.164450 20096 raft_consensus.cc:482] T 5d09baed576e4136bf6d756ca39d08b2 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:41.164507 20096 raft_consensus.cc:3037] T 5d09baed576e4136bf6d756ca39d08b2 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:41.165436 18988 ts_tablet_manager.cc:1398] T acf3b6fead41483a82c43dc0286fab7c P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240430 20:11:41.165728 18988 tablet_bootstrap.cc:492] T 6e754af64cb64fd2a082adae7d5f24da P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:41.167410 18988 tablet_bootstrap.cc:654] T 6e754af64cb64fd2a082adae7d5f24da P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:41.167763 18985 tablet_bootstrap.cc:492] T c2b019f529a54228b804437bb025e31d P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:41.171419 20096 raft_consensus.cc:504] T 5d09baed576e4136bf6d756ca39d08b2 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:41.171514 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 74) took 9939 ms. Trace:
I20240430 20:11:41.171623 19495 rpcz_store.cc:276] 0430 20:11:31.239183 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:31.239247 (+ 64us) service_pool.cc:225] Handling call
0430 20:11:41.171404 (+9932157us) ts_tablet_manager.cc:1120] Deleting tablet 0e503a898406412ab977d3e048ea4b5d
0430 20:11:41.171414 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:41.171493 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":9922308}
I20240430 20:11:41.172241 19029 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_d12caaff392a45ccb6c95a4a731b714a" } new_table_name: "default.Table_6f80b003810342a6b03c6dbb3c955772" modify_external_catalogs: true
I20240430 20:11:41.172667 18985 tablet_bootstrap.cc:654] T c2b019f529a54228b804437bb025e31d P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:41.176009 19051 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_49a012a7c4c8457fadb0a8f0fc9654c6" } modify_external_catalogs: true
I20240430 20:11:41.176507 19051 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_49a012a7c4c8457fadb0a8f0fc9654c6" } modify_external_catalogs: true
I20240430 20:11:41.178246 18478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d09baed576e4136bf6d756ca39d08b2" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:41.178463 18478 raft_consensus.cc:3037] T 5d09baed576e4136bf6d756ca39d08b2 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:41.174448 18983 tablet_bootstrap.cc:492] T 7542c7fe6cad452ab718f9ef54dbbbbe P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:41.178674 18983 ts_tablet_manager.cc:1367] T 7542c7fe6cad452ab718f9ef54dbbbbe P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.086s user 0.006s sys 0.005s
I20240430 20:11:41.179412 18983 raft_consensus.cc:348] T 7542c7fe6cad452ab718f9ef54dbbbbe P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:41.179590 18983 raft_consensus.cc:374] T 7542c7fe6cad452ab718f9ef54dbbbbe P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:41.179638 18983 raft_consensus.cc:729] T 7542c7fe6cad452ab718f9ef54dbbbbe P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:41.179881 18983 consensus_queue.cc:260] T 7542c7fe6cad452ab718f9ef54dbbbbe P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:41.181115 18478 raft_consensus.cc:2454] T 5d09baed576e4136bf6d756ca39d08b2 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
I20240430 20:11:41.182052 18558 leader_election.cc:302] T 5d09baed576e4136bf6d756ca39d08b2 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:41.182325 20047 raft_consensus.cc:2781] T 5d09baed576e4136bf6d756ca39d08b2 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:41.182449 20047 raft_consensus.cc:686] T 5d09baed576e4136bf6d756ca39d08b2 P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Becoming Leader. State: Replica: 5e1c6397f09747f091534faf93542710, State: Running, Role: LEADER
I20240430 20:11:41.182682 20047 consensus_queue.cc:237] T 5d09baed576e4136bf6d756ca39d08b2 P 5e1c6397f09747f091534faf93542710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
W20240430 20:11:41.184626 19029 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 086b3dfa84c045a18c8c4f177dce606a has no consensus state
I20240430 20:11:41.184156 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 76) took 9941 ms. Trace:
I20240430 20:11:41.185163 19495 rpcz_store.cc:276] 0430 20:11:31.242757 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:31.242837 (+ 80us) service_pool.cc:225] Handling call
0430 20:11:41.184027 (+9941190us) ts_tablet_manager.cc:1120] Deleting tablet 5f56ac2afd444d63a6bfd96988464bbb
0430 20:11:41.184038 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:41.184136 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":9933314}
I20240430 20:11:41.185142 19029 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_8039a17f6e4b4e9d90222f99d53d90a1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240430 20:11:41.185611 19495 tablet_replica.cc:330] T e5111d7607dd42188a0176cf5be2ec05 P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
W20240430 20:11:41.185599 19004 catalog_manager.cc:4797] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): delete failed for tablet 5f56ac2afd444d63a6bfd96988464bbb because the tablet was not found. No further retry: Not found: Tablet not found: 5f56ac2afd444d63a6bfd96988464bbb
I20240430 20:11:41.185794 19495 raft_consensus.cc:2229] T e5111d7607dd42188a0176cf5be2ec05 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:41.185945 19495 raft_consensus.cc:2258] T e5111d7607dd42188a0176cf5be2ec05 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Raft consensus is shut down!
W20240430 20:11:41.186142 19029 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8039a17f6e4b4e9d90222f99d53d90a1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:41.186462 19495 ts_tablet_manager.cc:1875] T e5111d7607dd42188a0176cf5be2ec05 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:41.188458 18983 ts_tablet_manager.cc:1398] T 7542c7fe6cad452ab718f9ef54dbbbbe P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.010s user 0.001s sys 0.000s
I20240430 20:11:41.188565 20096 leader_election.cc:288] T 5d09baed576e4136bf6d756ca39d08b2 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:41.189041 18983 tablet_bootstrap.cc:492] T 2f61e3f88a6448dfa323204bd8d14cc5 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:41.191277 18983 tablet_bootstrap.cc:654] T 2f61e3f88a6448dfa323204bd8d14cc5 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:41.176892 18758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d09baed576e4136bf6d756ca39d08b2" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
W20240430 20:11:41.195196 18557 leader_election.cc:341] T 5d09baed576e4136bf6d756ca39d08b2 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:41.207028 19049 catalog_manager.cc:5484] T 76d6e9d547ca4a848eae852944b1d117 P 5e1c6397f09747f091534faf93542710 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e1c6397f09747f091534faf93542710 (127.17.177.2). New cstate: current_term: 1 leader_uuid: "5e1c6397f09747f091534faf93542710" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } health_report { overall_health: UNKNOWN } } }
I20240430 20:11:41.207530 19049 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 76d6e9d547ca4a848eae852944b1d117 (table default.Table_104ae015924a48fd8d27220792f07470 [id=ff86c0357c164dd0991098c416087773]). Expected version 2 got 0
I20240430 20:11:41.212006 18983 tablet_bootstrap.cc:492] T 2f61e3f88a6448dfa323204bd8d14cc5 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:41.212349 18983 ts_tablet_manager.cc:1367] T 2f61e3f88a6448dfa323204bd8d14cc5 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.023s user 0.001s sys 0.004s
I20240430 20:11:41.212998 19471 ts_tablet_manager.cc:1888] T 77285afcafe04a829e485998bbbe00f0 P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240430 20:11:41.213306 18725 tablet_service.cc:1467] Processing CreateTablet for tablet 330453ddeea6456e8ef93fa92608b0d3 (DEFAULT_TABLE table=default.Table_8c841af5208742f9b56a4591c02c573e [id=a50abf28790d4284a76ac6ee22c02606]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:41.213670 19471 log.cc:1196] T 77285afcafe04a829e485998bbbe00f0 P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/77285afcafe04a829e485998bbbe00f0
I20240430 20:11:41.213766 18725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 330453ddeea6456e8ef93fa92608b0d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:41.213999 18601 tablet_service.cc:1467] Processing CreateTablet for tablet 330453ddeea6456e8ef93fa92608b0d3 (DEFAULT_TABLE table=default.Table_8c841af5208742f9b56a4591c02c573e [id=a50abf28790d4284a76ac6ee22c02606]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:41.214401 18601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 330453ddeea6456e8ef93fa92608b0d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:41.214929 18988 tablet_bootstrap.cc:492] T 6e754af64cb64fd2a082adae7d5f24da P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:41.215042 18988 ts_tablet_manager.cc:1367] T 6e754af64cb64fd2a082adae7d5f24da P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.049s user 0.003s sys 0.000s
I20240430 20:11:41.216857 18988 raft_consensus.cc:348] T 6e754af64cb64fd2a082adae7d5f24da P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:41.217041 18988 raft_consensus.cc:374] T 6e754af64cb64fd2a082adae7d5f24da P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:41.217089 18988 raft_consensus.cc:729] T 6e754af64cb64fd2a082adae7d5f24da P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:41.217288 18988 consensus_queue.cc:260] T 6e754af64cb64fd2a082adae7d5f24da P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:41.217779 18988 ts_tablet_manager.cc:1398] T 6e754af64cb64fd2a082adae7d5f24da P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240430 20:11:41.218041 18988 tablet_bootstrap.cc:492] T 32c823d1700949ab8b22adcd83b7bfd1 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:41.220299 18988 tablet_bootstrap.cc:654] T 32c823d1700949ab8b22adcd83b7bfd1 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:41.224434 18450 tablet_service.cc:1467] Processing CreateTablet for tablet 330453ddeea6456e8ef93fa92608b0d3 (DEFAULT_TABLE table=default.Table_8c841af5208742f9b56a4591c02c573e [id=a50abf28790d4284a76ac6ee22c02606]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:41.224857 18450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 330453ddeea6456e8ef93fa92608b0d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:41.227831 20019 raft_consensus.cc:482] T 6e754af64cb64fd2a082adae7d5f24da P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:41.227982 20019 raft_consensus.cc:504] T 6e754af64cb64fd2a082adae7d5f24da P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:41.228623 20019 leader_election.cc:288] T 6e754af64cb64fd2a082adae7d5f24da P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:41.229120 18615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e754af64cb64fd2a082adae7d5f24da" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:41.229303 18615 raft_consensus.cc:2454] T 6e754af64cb64fd2a082adae7d5f24da P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:41.229485 20019 consensus_queue.cc:1035] T 0a9066d895744c5bb7d9532b2940603c P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:41.229661 18696 leader_election.cc:302] T 6e754af64cb64fd2a082adae7d5f24da P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:41.229882 20019 raft_consensus.cc:2781] T 6e754af64cb64fd2a082adae7d5f24da P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:41.229962 20019 raft_consensus.cc:482] T 6e754af64cb64fd2a082adae7d5f24da P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:41.230020 20019 raft_consensus.cc:3037] T 6e754af64cb64fd2a082adae7d5f24da P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:41.232496 20019 raft_consensus.cc:504] T 6e754af64cb64fd2a082adae7d5f24da P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:41.233923 18608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e754af64cb64fd2a082adae7d5f24da" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:41.233911 18475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e754af64cb64fd2a082adae7d5f24da" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:41.234090 18608 raft_consensus.cc:3037] T 6e754af64cb64fd2a082adae7d5f24da P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:41.234100 18475 raft_consensus.cc:3037] T 6e754af64cb64fd2a082adae7d5f24da P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:41.235626 19029 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_f30868d759f64cceb682278f0dbb51ae"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:41.236258 19029 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f30868d759f64cceb682278f0dbb51ae in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:41.236689 18608 raft_consensus.cc:2454] T 6e754af64cb64fd2a082adae7d5f24da P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:41.237192 18696 leader_election.cc:302] T 6e754af64cb64fd2a082adae7d5f24da P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:41.237356 18475 raft_consensus.cc:2454] T 6e754af64cb64fd2a082adae7d5f24da P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:41.237543 20198 raft_consensus.cc:2781] T 6e754af64cb64fd2a082adae7d5f24da P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:41.237675 20198 raft_consensus.cc:686] T 6e754af64cb64fd2a082adae7d5f24da P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Becoming Leader. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Running, Role: LEADER
I20240430 20:11:41.238013 20198 consensus_queue.cc:237] T 6e754af64cb64fd2a082adae7d5f24da P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:41.238240 20019 leader_election.cc:288] T 6e754af64cb64fd2a082adae7d5f24da P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Requested vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:41.239285 18478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e754af64cb64fd2a082adae7d5f24da" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:41.239475 18478 raft_consensus.cc:2362] T 6e754af64cb64fd2a082adae7d5f24da P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1. Re-sending same reply.
I20240430 20:11:41.240337 18725 tablet_service.cc:1467] Processing CreateTablet for tablet 086b3dfa84c045a18c8c4f177dce606a (DEFAULT_TABLE table=default.Table_6f80b003810342a6b03c6dbb3c955772 [id=197923633fc54a4582207942b51530e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:41.240983 18725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 086b3dfa84c045a18c8c4f177dce606a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:41.242946 18809 heartbeater.cc:347] Connected to a master server at 127.17.177.62:36441
I20240430 20:11:41.242125 18601 tablet_service.cc:1467] Processing CreateTablet for tablet 086b3dfa84c045a18c8c4f177dce606a (DEFAULT_TABLE table=default.Table_6f80b003810342a6b03c6dbb3c955772 [id=197923633fc54a4582207942b51530e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:41.244612 18601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 086b3dfa84c045a18c8c4f177dce606a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:41.245910 19495 ts_tablet_manager.cc:1888] T e5111d7607dd42188a0176cf5be2ec05 P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:41.246038 19495 log.cc:1196] T e5111d7607dd42188a0176cf5be2ec05 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/e5111d7607dd42188a0176cf5be2ec05
I20240430 20:11:41.246316 19495 ts_tablet_manager.cc:1909] T e5111d7607dd42188a0176cf5be2ec05 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:41.246894 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 75) took 10005 ms. Trace:
I20240430 20:11:41.246982 19495 rpcz_store.cc:276] 0430 20:11:31.241636 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:31.241683 (+ 47us) service_pool.cc:225] Handling call
0430 20:11:41.185563 (+9943880us) ts_tablet_manager.cc:1120] Deleting tablet e5111d7607dd42188a0176cf5be2ec05
0430 20:11:41.185572 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:41.192010 (+ 6438us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:41.245808 (+ 53798us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:41.246864 (+ 1056us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":9923159}
I20240430 20:11:41.247428 19004 catalog_manager.cc:4830] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): tablet e5111d7607dd42188a0176cf5be2ec05 (table default.Table_f61a45518e134c4496e4ac81fbe40fce [id=02d9fe0426694ee8b5fd70589fb66ead]) successfully deleted
I20240430 20:11:41.250936 18985 tablet_bootstrap.cc:492] T c2b019f529a54228b804437bb025e31d P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:41.251129 18985 ts_tablet_manager.cc:1367] T c2b019f529a54228b804437bb025e31d P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.083s user 0.003s sys 0.002s
I20240430 20:11:41.251885 18985 raft_consensus.cc:348] T c2b019f529a54228b804437bb025e31d P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:41.252149 18985 raft_consensus.cc:374] T c2b019f529a54228b804437bb025e31d P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:41.252238 18985 raft_consensus.cc:729] T c2b019f529a54228b804437bb025e31d P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:41.252542 18985 consensus_queue.cc:260] T c2b019f529a54228b804437bb025e31d P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:41.252776 18983 raft_consensus.cc:348] T 2f61e3f88a6448dfa323204bd8d14cc5 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:41.253010 18983 raft_consensus.cc:374] T 2f61e3f88a6448dfa323204bd8d14cc5 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:41.253108 18983 raft_consensus.cc:729] T 2f61e3f88a6448dfa323204bd8d14cc5 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:41.253311 18985 ts_tablet_manager.cc:1398] T c2b019f529a54228b804437bb025e31d P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240430 20:11:41.253345 18983 consensus_queue.cc:260] T 2f61e3f88a6448dfa323204bd8d14cc5 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:41.253572 18985 tablet_bootstrap.cc:492] T 87ac32a0ddb84676adc83038ef664591 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:41.254902 19495 tablet_replica.cc:330] T dd9b893d48684cfea7c242b93ecba01f P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:41.255092 19495 raft_consensus.cc:2229] T dd9b893d48684cfea7c242b93ecba01f P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Raft consensus shutting down.
I20240430 20:11:41.255434 19495 raft_consensus.cc:2258] T dd9b893d48684cfea7c242b93ecba01f P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:41.256059 19495 ts_tablet_manager.cc:1875] T dd9b893d48684cfea7c242b93ecba01f P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:41.257692 18983 ts_tablet_manager.cc:1398] T 2f61e3f88a6448dfa323204bd8d14cc5 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.045s user 0.001s sys 0.001s
I20240430 20:11:41.258368 18983 tablet_bootstrap.cc:492] T ca64de16d40f4a44b8b411f5526ba11a P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:41.259323 20047 raft_consensus.cc:482] T 0eb4f3e7ece6496293e01b63620d644b P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:41.259488 20047 raft_consensus.cc:504] T 0eb4f3e7ece6496293e01b63620d644b P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:41.260262 20047 leader_election.cc:288] T 0eb4f3e7ece6496293e01b63620d644b P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:41.260819 18758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0eb4f3e7ece6496293e01b63620d644b" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:41.261015 18758 raft_consensus.cc:2454] T 0eb4f3e7ece6496293e01b63620d644b P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 0.
I20240430 20:11:41.261421 18557 leader_election.cc:302] T 0eb4f3e7ece6496293e01b63620d644b P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:41.261662 20047 raft_consensus.cc:2781] T 0eb4f3e7ece6496293e01b63620d644b P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:41.261726 20047 raft_consensus.cc:482] T 0eb4f3e7ece6496293e01b63620d644b P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:41.261786 20047 raft_consensus.cc:3037] T 0eb4f3e7ece6496293e01b63620d644b P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:41.261696 18478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0eb4f3e7ece6496293e01b63620d644b" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:41.261936 18478 raft_consensus.cc:2454] T 0eb4f3e7ece6496293e01b63620d644b P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 0.
I20240430 20:11:41.264945 20105 raft_consensus.cc:482] T c2b019f529a54228b804437bb025e31d P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:41.265080 20105 raft_consensus.cc:504] T c2b019f529a54228b804437bb025e31d P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:41.265733 20105 leader_election.cc:288] T c2b019f529a54228b804437bb025e31d P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:41.266041 18983 tablet_bootstrap.cc:654] T ca64de16d40f4a44b8b411f5526ba11a P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:41.266834 18608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2b019f529a54228b804437bb025e31d" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
W20240430 20:11:41.267314 18415 leader_election.cc:341] T c2b019f529a54228b804437bb025e31d P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:41.268231 18988 tablet_bootstrap.cc:492] T 32c823d1700949ab8b22adcd83b7bfd1 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:41.269258 18988 ts_tablet_manager.cc:1367] T 32c823d1700949ab8b22adcd83b7bfd1 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.051s user 0.004s sys 0.000s
I20240430 20:11:41.270787 18758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2b019f529a54228b804437bb025e31d" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:41.271296 18988 raft_consensus.cc:348] T 32c823d1700949ab8b22adcd83b7bfd1 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:41.271548 18988 raft_consensus.cc:374] T 32c823d1700949ab8b22adcd83b7bfd1 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:41.271603 18988 raft_consensus.cc:729] T 32c823d1700949ab8b22adcd83b7bfd1 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:41.271821 18988 consensus_queue.cc:260] T 32c823d1700949ab8b22adcd83b7bfd1 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
W20240430 20:11:41.272289 18417 leader_election.cc:341] T c2b019f529a54228b804437bb025e31d P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:41.272403 18988 ts_tablet_manager.cc:1398] T 32c823d1700949ab8b22adcd83b7bfd1 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240430 20:11:41.272418 18417 leader_election.cc:302] T c2b019f529a54228b804437bb025e31d P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a; no voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710
I20240430 20:11:41.272459 20047 raft_consensus.cc:504] T 0eb4f3e7ece6496293e01b63620d644b P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:41.272671 18988 tablet_bootstrap.cc:492] T 5d09baed576e4136bf6d756ca39d08b2 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:41.272785 20105 raft_consensus.cc:2726] T c2b019f529a54228b804437bb025e31d P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:41.273272 20019 consensus_queue.cc:1035] T 0a9066d895744c5bb7d9532b2940603c P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:41.273512 18758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0eb4f3e7ece6496293e01b63620d644b" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:41.273706 18758 raft_consensus.cc:3037] T 0eb4f3e7ece6496293e01b63620d644b P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:41.273730 18478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0eb4f3e7ece6496293e01b63620d644b" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:41.273900 18478 raft_consensus.cc:3037] T 0eb4f3e7ece6496293e01b63620d644b P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:41.274466 18988 tablet_bootstrap.cc:654] T 5d09baed576e4136bf6d756ca39d08b2 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:41.277169 18478 raft_consensus.cc:2454] T 0eb4f3e7ece6496293e01b63620d644b P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
I20240430 20:11:41.277570 18558 leader_election.cc:302] T 0eb4f3e7ece6496293e01b63620d644b P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:41.277913 20096 raft_consensus.cc:2781] T 0eb4f3e7ece6496293e01b63620d644b P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:41.278028 20096 raft_consensus.cc:686] T 0eb4f3e7ece6496293e01b63620d644b P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Becoming Leader. State: Replica: 5e1c6397f09747f091534faf93542710, State: Running, Role: LEADER
I20240430 20:11:41.278295 20096 consensus_queue.cc:237] T 0eb4f3e7ece6496293e01b63620d644b P 5e1c6397f09747f091534faf93542710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:41.276383 18758 raft_consensus.cc:2454] T 0eb4f3e7ece6496293e01b63620d644b P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
I20240430 20:11:41.281379 18983 tablet_bootstrap.cc:492] T ca64de16d40f4a44b8b411f5526ba11a P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:41.281626 18983 ts_tablet_manager.cc:1367] T ca64de16d40f4a44b8b411f5526ba11a P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.023s user 0.001s sys 0.005s
I20240430 20:11:41.281903 20047 leader_election.cc:288] T 0eb4f3e7ece6496293e01b63620d644b P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:41.282367 18983 raft_consensus.cc:348] T ca64de16d40f4a44b8b411f5526ba11a P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:41.282579 18985 tablet_bootstrap.cc:654] T 87ac32a0ddb84676adc83038ef664591 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:41.283600 18983 raft_consensus.cc:374] T ca64de16d40f4a44b8b411f5526ba11a P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:41.283674 18983 raft_consensus.cc:729] T ca64de16d40f4a44b8b411f5526ba11a P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:41.283916 18983 consensus_queue.cc:260] T ca64de16d40f4a44b8b411f5526ba11a P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:41.284912 18983 ts_tablet_manager.cc:1398] T ca64de16d40f4a44b8b411f5526ba11a P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240430 20:11:41.288079 18450 tablet_service.cc:1467] Processing CreateTablet for tablet 086b3dfa84c045a18c8c4f177dce606a (DEFAULT_TABLE table=default.Table_6f80b003810342a6b03c6dbb3c955772 [id=197923633fc54a4582207942b51530e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:41.288517 18450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 086b3dfa84c045a18c8c4f177dce606a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:41.288905 18983 tablet_bootstrap.cc:492] T 43b5e2c005a34df08cc4335c6e014cb5 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:41.291775 18725 tablet_service.cc:1467] Processing CreateTablet for tablet 1050ab528c7d41559831652a1437d358 (DEFAULT_TABLE table=default.Table_acddc4cbf6334546b5fb3e4752fef2a5 [id=274cb15fd751410e83327ac4e8dc4629]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:41.292224 18725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1050ab528c7d41559831652a1437d358. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:41.214103 19471 ts_tablet_manager.cc:1909] T 77285afcafe04a829e485998bbbe00f0 P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:41.294375 18588 tablet_service.cc:1467] Processing CreateTablet for tablet 1050ab528c7d41559831652a1437d358 (DEFAULT_TABLE table=default.Table_acddc4cbf6334546b5fb3e4752fef2a5 [id=274cb15fd751410e83327ac4e8dc4629]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:41.299139 18588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1050ab528c7d41559831652a1437d358. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:41.301204 18988 tablet_bootstrap.cc:492] T 5d09baed576e4136bf6d756ca39d08b2 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:41.304843 18450 tablet_service.cc:1467] Processing CreateTablet for tablet 1050ab528c7d41559831652a1437d358 (DEFAULT_TABLE table=default.Table_acddc4cbf6334546b5fb3e4752fef2a5 [id=274cb15fd751410e83327ac4e8dc4629]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:41.305256 18450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1050ab528c7d41559831652a1437d358. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:41.305953 20096 consensus_queue.cc:1035] T fd40967ab4a04001819e16c06a6c4300 P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:41.294939 20047 raft_consensus.cc:482] T 3fd0e8c360734dcbb75dc2958b70c2b8 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:41.306882 20047 raft_consensus.cc:504] T 3fd0e8c360734dcbb75dc2958b70c2b8 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:41.307727 20047 leader_election.cc:288] T 3fd0e8c360734dcbb75dc2958b70c2b8 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:41.307657 19033 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_74b89f328a354c9a937566cdaf2a3377"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240430 20:11:41.290503 18983 tablet_bootstrap.cc:654] T 43b5e2c005a34df08cc4335c6e014cb5 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:41.308048 18475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd0e8c360734dcbb75dc2958b70c2b8" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
W20240430 20:11:41.308234 19033 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_74b89f328a354c9a937566cdaf2a3377 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:41.308228 18475 raft_consensus.cc:2454] T 3fd0e8c360734dcbb75dc2958b70c2b8 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 0.
I20240430 20:11:41.308661 18558 leader_election.cc:302] T 3fd0e8c360734dcbb75dc2958b70c2b8 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:41.309444 20047 raft_consensus.cc:2781] T 3fd0e8c360734dcbb75dc2958b70c2b8 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:41.309522 20047 raft_consensus.cc:482] T 3fd0e8c360734dcbb75dc2958b70c2b8 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:41.309583 20047 raft_consensus.cc:3037] T 3fd0e8c360734dcbb75dc2958b70c2b8 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:41.310693 18756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd0e8c360734dcbb75dc2958b70c2b8" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
W20240430 20:11:41.311478 18557 leader_election.cc:341] T 3fd0e8c360734dcbb75dc2958b70c2b8 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:41.313355 20053 raft_consensus.cc:729] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8 [term 4 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b3f98793354c7aaa5c106e8ec31ed8, State: Initialized, Role: FOLLOWER
I20240430 20:11:41.314047 18985 tablet_bootstrap.cc:492] T 87ac32a0ddb84676adc83038ef664591 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:41.314311 18985 ts_tablet_manager.cc:1367] T 87ac32a0ddb84676adc83038ef664591 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.061s user 0.005s sys 0.000s
I20240430 20:11:41.314512 20053 consensus_queue.cc:260] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 308, Last appended: 4.309, Last appended by leader: 309, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b3f98793354c7aaa5c106e8ec31ed8" member_type: VOTER last_known_addr { host: "127.17.177.62" port: 36441 } } peers { permanent_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e" member_type: VOTER last_known_addr { host: "127.17.177.61" port: 34777 } } peers { permanent_uuid: "b632d6ef4c104a058a6db208024cab33" member_type: VOTER last_known_addr { host: "127.17.177.60" port: 37225 } }
I20240430 20:11:41.315546 18985 raft_consensus.cc:348] T 87ac32a0ddb84676adc83038ef664591 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:41.315793 18985 raft_consensus.cc:374] T 87ac32a0ddb84676adc83038ef664591 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:41.315883 18985 raft_consensus.cc:729] T 87ac32a0ddb84676adc83038ef664591 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:41.316179 18985 consensus_queue.cc:260] T 87ac32a0ddb84676adc83038ef664591 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:41.316854 18985 ts_tablet_manager.cc:1398] T 87ac32a0ddb84676adc83038ef664591 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240430 20:11:41.317159 18985 tablet_bootstrap.cc:492] T 5f02d8862e1f4ccfa9dbda73dbe18615 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:41.319142 18985 tablet_bootstrap.cc:654] T 5f02d8862e1f4ccfa9dbda73dbe18615 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:41.301329 18988 ts_tablet_manager.cc:1367] T 5d09baed576e4136bf6d756ca39d08b2 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.029s user 0.004s sys 0.000s
I20240430 20:11:41.320981 18988 raft_consensus.cc:348] T 5d09baed576e4136bf6d756ca39d08b2 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:41.321251 18988 raft_consensus.cc:374] T 5d09baed576e4136bf6d756ca39d08b2 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:41.321333 18988 raft_consensus.cc:729] T 5d09baed576e4136bf6d756ca39d08b2 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:41.321569 18988 consensus_queue.cc:260] T 5d09baed576e4136bf6d756ca39d08b2 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:41.322084 18988 ts_tablet_manager.cc:1398] T 5d09baed576e4136bf6d756ca39d08b2 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240430 20:11:41.322369 18988 tablet_bootstrap.cc:492] T 7542c7fe6cad452ab718f9ef54dbbbbe P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:41.322805 20297 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 4 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b3f98793354c7aaa5c106e8ec31ed8" member_type: VOTER last_known_addr { host: "127.17.177.62" port: 36441 } } peers { permanent_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e" member_type: VOTER last_known_addr { host: "127.17.177.61" port: 34777 } } peers { permanent_uuid: "b632d6ef4c104a058a6db208024cab33" member_type: VOTER last_known_addr { host: "127.17.177.60" port: 37225 } } }
I20240430 20:11:41.323170 20297 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8 [sys.catalog]: This master's current role is: FOLLOWER
I20240430 20:11:41.324638 18988 tablet_bootstrap.cc:654] T 7542c7fe6cad452ab718f9ef54dbbbbe P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:41.325142 20053 sys_catalog.cc:564] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8 [sys.catalog]: configured and running, proceeding with master startup.
I20240430 20:11:41.334642 19979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 486 } } ignore_live_leader: false dest_uuid: "88b3f98793354c7aaa5c106e8ec31ed8" is_pre_election: true
W20240430 20:11:41.335633 19979 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33774 (request call id 0) took 2655 ms (client timeout 531 ms). Trace:
W20240430 20:11:41.335882 19979 rpcz_store.cc:269] 0430 20:11:38.680229 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:38.680451 (+ 222us) service_pool.cc:225] Handling call
0430 20:11:41.335575 (+2655124us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240430 20:11:41.339069 19495 ts_tablet_manager.cc:1888] T dd9b893d48684cfea7c242b93ecba01f P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:41.339212 19495 log.cc:1196] T dd9b893d48684cfea7c242b93ecba01f P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/dd9b893d48684cfea7c242b93ecba01f
I20240430 20:11:41.339669 19495 ts_tablet_manager.cc:1909] T dd9b893d48684cfea7c242b93ecba01f P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:41.340562 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 79) took 10078 ms. Trace:
I20240430 20:11:41.340665 19495 rpcz_store.cc:276] 0430 20:11:31.262510 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:31.262553 (+ 43us) service_pool.cc:225] Handling call
0430 20:11:41.254146 (+9991593us) ts_tablet_manager.cc:1120] Deleting tablet dd9b893d48684cfea7c242b93ecba01f
0430 20:11:41.254849 (+ 703us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:41.277979 (+ 23130us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:41.338940 (+ 60961us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:41.340539 (+ 1599us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":9991373}
I20240430 20:11:41.341444 19495 tablet_replica.cc:330] T 2a1e50f99c5d4b44bdbb51b58a92bc75 P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:41.341475 19004 catalog_manager.cc:4830] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): tablet dd9b893d48684cfea7c242b93ecba01f (table default.Table_4bce1dae3391497ebea5f1b536700d62 [id=8f0cb0ef65d84fb8964209dd501c0b5c]) successfully deleted
I20240430 20:11:41.341627 19495 raft_consensus.cc:2229] T 2a1e50f99c5d4b44bdbb51b58a92bc75 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:41.341761 19495 raft_consensus.cc:2258] T 2a1e50f99c5d4b44bdbb51b58a92bc75 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:41.342247 19495 ts_tablet_manager.cc:1875] T 2a1e50f99c5d4b44bdbb51b58a92bc75 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:41.349771 18985 tablet_bootstrap.cc:492] T 5f02d8862e1f4ccfa9dbda73dbe18615 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:41.350049 18985 ts_tablet_manager.cc:1367] T 5f02d8862e1f4ccfa9dbda73dbe18615 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.033s user 0.003s sys 0.000s
I20240430 20:11:41.351835 18985 raft_consensus.cc:348] T 5f02d8862e1f4ccfa9dbda73dbe18615 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:41.352123 18985 raft_consensus.cc:374] T 5f02d8862e1f4ccfa9dbda73dbe18615 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:41.352214 18985 raft_consensus.cc:729] T 5f02d8862e1f4ccfa9dbda73dbe18615 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:41.352492 18985 consensus_queue.cc:260] T 5f02d8862e1f4ccfa9dbda73dbe18615 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:41.353127 18985 ts_tablet_manager.cc:1398] T 5f02d8862e1f4ccfa9dbda73dbe18615 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240430 20:11:41.353475 18985 tablet_bootstrap.cc:492] T ae87f5e92a934e2182164c231bc76df0 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:41.353390 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 81) took 9762 ms. Trace:
I20240430 20:11:41.353530 19471 rpcz_store.cc:276] 0430 20:11:31.590753 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:31.590832 (+ 79us) service_pool.cc:225] Handling call
0430 20:11:41.139337 (+9548505us) ts_tablet_manager.cc:1120] Deleting tablet 77285afcafe04a829e485998bbbe00f0
0430 20:11:41.139347 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:41.199873 (+ 60526us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:41.212896 (+ 13023us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:41.353353 (+140457us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":9441921}
I20240430 20:11:41.354534 18999 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet 77285afcafe04a829e485998bbbe00f0 (table default.Table_e63f64a411094f8db2ef641f2257ca79 [id=040dec3a3d804cdfbdada359bbe5123b]) successfully deleted
I20240430 20:11:41.355072 19471 tablet_replica.cc:330] T fcc9698abda4491b98e1e879589913a8 P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:41.355227 18985 tablet_bootstrap.cc:654] T ae87f5e92a934e2182164c231bc76df0 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:41.355373 19471 raft_consensus.cc:2229] T fcc9698abda4491b98e1e879589913a8 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:41.355597 19471 raft_consensus.cc:2258] T fcc9698abda4491b98e1e879589913a8 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:41.356340 19471 ts_tablet_manager.cc:1875] T fcc9698abda4491b98e1e879589913a8 P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:41.364324 18983 tablet_bootstrap.cc:492] T 43b5e2c005a34df08cc4335c6e014cb5 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:41.364557 18983 ts_tablet_manager.cc:1367] T 43b5e2c005a34df08cc4335c6e014cb5 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.076s user 0.005s sys 0.015s
I20240430 20:11:41.365571 18983 raft_consensus.cc:348] T 43b5e2c005a34df08cc4335c6e014cb5 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:41.365847 18983 raft_consensus.cc:374] T 43b5e2c005a34df08cc4335c6e014cb5 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:41.366009 18983 raft_consensus.cc:729] T 43b5e2c005a34df08cc4335c6e014cb5 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:41.366304 18983 consensus_queue.cc:260] T 43b5e2c005a34df08cc4335c6e014cb5 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:41.366938 19028 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: Sending DeleteTablet for 3 replicas of tablet 3b7fdc6ec7e046c5af894f9a6eda17b0
I20240430 20:11:41.367785 19028 master_service.cc:946] ReplaceTablet: received request to replace tablet 73453a7fa30342e29db552b17b62f0c1 from {username='slave'} at 127.0.0.1:51664
I20240430 20:11:41.368631 19028 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_f2a19db5098448f79bf95cccb1abd368"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:41.369303 19028 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f2a19db5098448f79bf95cccb1abd368 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:41.371834 18588 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b7fdc6ec7e046c5af894f9a6eda17b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:40 UTC) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:41.373239 18983 ts_tablet_manager.cc:1398] T 43b5e2c005a34df08cc4335c6e014cb5 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20240430 20:11:41.377799 19049 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_c6d5c8ecce28428790b07d228879a0e4" } new_table_name: "default.Table_78987875650940c293247bc43db345ab" modify_external_catalogs: true
I20240430 20:11:41.379779 18983 tablet_bootstrap.cc:492] T ae87f5e92a934e2182164c231bc76df0 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:41.379987 19024 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet c9e7660eccc448dbb99d135f82bfd261 (table default.Table_685a26737bb046fb85ced15df7ae3423 [id=51d177e8498848b38c43e3f8d11f092c]). Expected version 1 got 0
I20240430 20:11:41.382917 18983 tablet_bootstrap.cc:654] T ae87f5e92a934e2182164c231bc76df0 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:41.388564 18588 tablet_service.cc:1514] Processing DeleteTablet for tablet 26ed12f498644d15bf45e884dcad172e with delete_type TABLET_DATA_DELETED (replaced by tablet c6378f1495144c458af49b3c19a2b67c) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:41.388837 18988 tablet_bootstrap.cc:492] T 7542c7fe6cad452ab718f9ef54dbbbbe P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:41.389035 18988 ts_tablet_manager.cc:1367] T 7542c7fe6cad452ab718f9ef54dbbbbe P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.067s user 0.004s sys 0.000s
I20240430 20:11:41.390061 18988 raft_consensus.cc:348] T 7542c7fe6cad452ab718f9ef54dbbbbe P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:41.390283 18988 raft_consensus.cc:374] T 7542c7fe6cad452ab718f9ef54dbbbbe P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:41.390368 18988 raft_consensus.cc:729] T 7542c7fe6cad452ab718f9ef54dbbbbe P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:41.390749 18988 consensus_queue.cc:260] T 7542c7fe6cad452ab718f9ef54dbbbbe P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:41.364951 20235 raft_consensus.cc:482] T 2f61e3f88a6448dfa323204bd8d14cc5 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240430 20:11:41.371191 18417 consensus_peers.cc:485] T 75ebf051383e42529dcb5f7dd0770a4b P 2efb542aa55e439b99ccb98d1c5c884a -> Peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Couldn't send request to peer 3c51c33d5616482cafb68e26688820f4. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240430 20:11:41.393939 20235 raft_consensus.cc:504] T 2f61e3f88a6448dfa323204bd8d14cc5 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:41.394171 18450 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b7fdc6ec7e046c5af894f9a6eda17b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:40 UTC) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:41.394459 18988 ts_tablet_manager.cc:1398] T 7542c7fe6cad452ab718f9ef54dbbbbe P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240430 20:11:41.395190 20235 leader_election.cc:288] T 2f61e3f88a6448dfa323204bd8d14cc5 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:41.395673 18756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f61e3f88a6448dfa323204bd8d14cc5" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:41.395681 18615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f61e3f88a6448dfa323204bd8d14cc5" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:41.395897 18615 raft_consensus.cc:2454] T 2f61e3f88a6448dfa323204bd8d14cc5 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 0.
W20240430 20:11:41.396265 18417 leader_election.cc:341] T 2f61e3f88a6448dfa323204bd8d14cc5 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:41.396571 18415 leader_election.cc:302] T 2f61e3f88a6448dfa323204bd8d14cc5 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters: 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:41.397034 20235 raft_consensus.cc:2781] T 2f61e3f88a6448dfa323204bd8d14cc5 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:41.397125 20235 raft_consensus.cc:482] T 2f61e3f88a6448dfa323204bd8d14cc5 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:41.397187 20235 raft_consensus.cc:3037] T 2f61e3f88a6448dfa323204bd8d14cc5 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:41.397651 18983 tablet_bootstrap.cc:492] T ae87f5e92a934e2182164c231bc76df0 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:41.397820 18983 ts_tablet_manager.cc:1367] T ae87f5e92a934e2182164c231bc76df0 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.006s
I20240430 20:11:41.398561 18983 raft_consensus.cc:348] T ae87f5e92a934e2182164c231bc76df0 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:41.399035 18983 raft_consensus.cc:374] T ae87f5e92a934e2182164c231bc76df0 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:41.399109 18983 raft_consensus.cc:729] T ae87f5e92a934e2182164c231bc76df0 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:41.382369 19024 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_5721e7a6b1ef44a780ff57d59fa76f64"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240430 20:11:41.399380 18983 consensus_queue.cc:260] T ae87f5e92a934e2182164c231bc76df0 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
W20240430 20:11:41.399849 19024 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5721e7a6b1ef44a780ff57d59fa76f64 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:41.400007 20235 raft_consensus.cc:504] T 2f61e3f88a6448dfa323204bd8d14cc5 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:41.402141 18756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f61e3f88a6448dfa323204bd8d14cc5" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:41.402289 18615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f61e3f88a6448dfa323204bd8d14cc5" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:41.402482 18615 raft_consensus.cc:3037] T 2f61e3f88a6448dfa323204bd8d14cc5 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
W20240430 20:11:41.402724 18417 leader_election.cc:341] T 2f61e3f88a6448dfa323204bd8d14cc5 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:41.402426 20235 leader_election.cc:288] T 2f61e3f88a6448dfa323204bd8d14cc5 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:41.405701 18615 raft_consensus.cc:2454] T 2f61e3f88a6448dfa323204bd8d14cc5 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 1.
I20240430 20:11:41.406334 18415 leader_election.cc:302] T 2f61e3f88a6448dfa323204bd8d14cc5 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters: 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:41.395776 18988 tablet_bootstrap.cc:492] T 2f61e3f88a6448dfa323204bd8d14cc5 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:41.406765 18725 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b7fdc6ec7e046c5af894f9a6eda17b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:40 UTC) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:41.407033 20235 raft_consensus.cc:2781] T 2f61e3f88a6448dfa323204bd8d14cc5 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:41.407337 20235 raft_consensus.cc:686] T 2f61e3f88a6448dfa323204bd8d14cc5 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Becoming Leader. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Running, Role: LEADER
I20240430 20:11:41.408478 18988 tablet_bootstrap.cc:654] T 2f61e3f88a6448dfa323204bd8d14cc5 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:41.407743 20235 consensus_queue.cc:237] T 2f61e3f88a6448dfa323204bd8d14cc5 P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:41.412431 18983 ts_tablet_manager.cc:1398] T ae87f5e92a934e2182164c231bc76df0 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.015s user 0.003s sys 0.000s
I20240430 20:11:41.418380 18983 tablet_bootstrap.cc:492] T 34130f2820064fa0898ef1f9d7de1c6d P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:41.420341 18983 tablet_bootstrap.cc:654] T 34130f2820064fa0898ef1f9d7de1c6d P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:41.424671 19495 ts_tablet_manager.cc:1888] T 2a1e50f99c5d4b44bdbb51b58a92bc75 P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:41.424896 19495 log.cc:1196] T 2a1e50f99c5d4b44bdbb51b58a92bc75 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/2a1e50f99c5d4b44bdbb51b58a92bc75
I20240430 20:11:41.425266 19495 ts_tablet_manager.cc:1909] T 2a1e50f99c5d4b44bdbb51b58a92bc75 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:41.426139 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 77) took 10178 ms. Trace:
I20240430 20:11:41.426288 19495 rpcz_store.cc:276] 0430 20:11:31.247464 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:31.247520 (+ 56us) service_pool.cc:225] Handling call
0430 20:11:41.341393 (+10093873us) ts_tablet_manager.cc:1120] Deleting tablet 2a1e50f99c5d4b44bdbb51b58a92bc75
0430 20:11:41.341403 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:41.346289 (+ 4886us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:41.424533 (+ 78244us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:41.426068 (+ 1535us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":10076644}
I20240430 20:11:41.426857 19004 catalog_manager.cc:4830] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): tablet 2a1e50f99c5d4b44bdbb51b58a92bc75 (table default.Table_c12ca6938d424fbfa2e1df5cdb00fdbc [id=0fb93f36bc17408ea56bc4cafcfd8580]) successfully deleted
I20240430 20:11:41.435036 20235 raft_consensus.cc:482] T 5f02d8862e1f4ccfa9dbda73dbe18615 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:41.435252 20235 raft_consensus.cc:504] T 5f02d8862e1f4ccfa9dbda73dbe18615 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:41.435985 20235 leader_election.cc:288] T 5f02d8862e1f4ccfa9dbda73dbe18615 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:41.436821 18615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f02d8862e1f4ccfa9dbda73dbe18615" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:41.437587 18983 tablet_bootstrap.cc:492] T 34130f2820064fa0898ef1f9d7de1c6d P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:41.437738 18756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f02d8862e1f4ccfa9dbda73dbe18615" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
W20240430 20:11:41.437914 18415 leader_election.cc:341] T 5f02d8862e1f4ccfa9dbda73dbe18615 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:41.437831 18983 ts_tablet_manager.cc:1367] T 34130f2820064fa0898ef1f9d7de1c6d P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.020s user 0.000s sys 0.003s
W20240430 20:11:41.438310 18417 leader_election.cc:341] T 5f02d8862e1f4ccfa9dbda73dbe18615 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:41.438419 18417 leader_election.cc:302] T 5f02d8862e1f4ccfa9dbda73dbe18615 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a; no voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710
I20240430 20:11:41.438829 20235 raft_consensus.cc:2726] T 5f02d8862e1f4ccfa9dbda73dbe18615 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:41.438925 18983 raft_consensus.cc:348] T 34130f2820064fa0898ef1f9d7de1c6d P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:41.439133 18983 raft_consensus.cc:374] T 34130f2820064fa0898ef1f9d7de1c6d P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:41.439215 18983 raft_consensus.cc:729] T 34130f2820064fa0898ef1f9d7de1c6d P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:41.439471 18983 consensus_queue.cc:260] T 34130f2820064fa0898ef1f9d7de1c6d P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:41.440183 18983 ts_tablet_manager.cc:1398] T 34130f2820064fa0898ef1f9d7de1c6d P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240430 20:11:41.440488 18983 tablet_bootstrap.cc:492] T 84568f90b9c54cf292099808d4cbf5dc P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:41.442247 18983 tablet_bootstrap.cc:654] T 84568f90b9c54cf292099808d4cbf5dc P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:41.445451 20198 raft_consensus.cc:482] T a9e0659564894b0e8cb7f600bb07a085 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:41.445638 20198 raft_consensus.cc:504] T a9e0659564894b0e8cb7f600bb07a085 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:41.445856 20047 raft_consensus.cc:504] T 3fd0e8c360734dcbb75dc2958b70c2b8 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:41.446322 19495 tablet_replica.cc:330] T 5bde72e2791a497db7dd08686ca36d80 P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:41.446365 20198 leader_election.cc:288] T a9e0659564894b0e8cb7f600bb07a085 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:41.446686 20047 leader_election.cc:288] T 3fd0e8c360734dcbb75dc2958b70c2b8 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:41.448009 18615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9e0659564894b0e8cb7f600bb07a085" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:41.448200 18615 raft_consensus.cc:2454] T a9e0659564894b0e8cb7f600bb07a085 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:41.448647 18696 leader_election.cc:302] T a9e0659564894b0e8cb7f600bb07a085 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:41.448679 18756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd0e8c360734dcbb75dc2958b70c2b8" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:41.451252 18725 tablet_service.cc:1467] Processing CreateTablet for tablet a0d3c4ba64024bccad95ecc909d03ae0 (DEFAULT_TABLE table=default.Table_5f50cb33cf774b86b2380ee4a569dac9 [id=bf4dcbec74f946398c7fc3f6eec6d068]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:41.451732 18725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0d3c4ba64024bccad95ecc909d03ae0. 1 dirs total, 0 dirs full, 0 dirs failed
W20240430 20:11:41.451864 18699 leader_election.cc:341] T a9e0659564894b0e8cb7f600bb07a085 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Not found: Tablet not found: a9e0659564894b0e8cb7f600bb07a085
W20240430 20:11:41.451843 18557 leader_election.cc:341] T 3fd0e8c360734dcbb75dc2958b70c2b8 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:41.452188 20198 raft_consensus.cc:2781] T a9e0659564894b0e8cb7f600bb07a085 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:41.452291 20198 raft_consensus.cc:482] T a9e0659564894b0e8cb7f600bb07a085 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:41.452395 20198 raft_consensus.cc:3037] T a9e0659564894b0e8cb7f600bb07a085 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:41.455730 18588 tablet_service.cc:1467] Processing CreateTablet for tablet a0d3c4ba64024bccad95ecc909d03ae0 (DEFAULT_TABLE table=default.Table_5f50cb33cf774b86b2380ee4a569dac9 [id=bf4dcbec74f946398c7fc3f6eec6d068]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:41.456111 18588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0d3c4ba64024bccad95ecc909d03ae0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:41.456696 18733 tablet_service.cc:1467] Processing CreateTablet for tablet f8bfbf673ea346529d1b2b376a3da496 (DEFAULT_TABLE table=default.Table_1ebe0eb1a1984ee9bbb16d634ff2cedb [id=aa91ac92289a48b784e97eea9967192f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:41.457044 18733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8bfbf673ea346529d1b2b376a3da496. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:41.455024 20198 raft_consensus.cc:504] T a9e0659564894b0e8cb7f600bb07a085 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:41.458842 18591 tablet_service.cc:1467] Processing CreateTablet for tablet f8bfbf673ea346529d1b2b376a3da496 (DEFAULT_TABLE table=default.Table_1ebe0eb1a1984ee9bbb16d634ff2cedb [id=aa91ac92289a48b784e97eea9967192f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:41.458904 18724 tablet_service.cc:1467] Processing CreateTablet for tablet 3b7fdc6ec7e046c5af894f9a6eda17b0 (DEFAULT_TABLE table=default.Table_d441c3bc84ed42358d1bc6e717108efb [id=7b516b6eb3a240a680bf271eab0324f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:41.459210 18591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8bfbf673ea346529d1b2b376a3da496. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:41.459225 18724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b7fdc6ec7e046c5af894f9a6eda17b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:41.460067 20205 raft_consensus.cc:482] T 7542c7fe6cad452ab718f9ef54dbbbbe P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:41.460110 20198 leader_election.cc:288] T a9e0659564894b0e8cb7f600bb07a085 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Requested vote from peers 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:41.460850 20205 raft_consensus.cc:504] T 7542c7fe6cad452ab718f9ef54dbbbbe P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:41.461526 20205 leader_election.cc:288] T 7542c7fe6cad452ab718f9ef54dbbbbe P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
W20240430 20:11:41.462756 18699 leader_election.cc:341] T a9e0659564894b0e8cb7f600bb07a085 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Not found: Tablet not found: a9e0659564894b0e8cb7f600bb07a085
I20240430 20:11:41.463852 19980 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8 [term 4 FOLLOWER]: Advancing to term 5
I20240430 20:11:41.464102 18615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9e0659564894b0e8cb7f600bb07a085" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:41.464336 18615 raft_consensus.cc:3037] T a9e0659564894b0e8cb7f600bb07a085 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:41.447976 18478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd0e8c360734dcbb75dc2958b70c2b8" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:41.467370 18478 raft_consensus.cc:3037] T 3fd0e8c360734dcbb75dc2958b70c2b8 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:41.469295 19980 pending_rounds.cc:76] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: Aborting all ops after (but not including) 308
I20240430 20:11:41.469409 19980 pending_rounds.cc:97] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: Aborting uncommitted WRITE_OP operation due to leader change: 4.309
I20240430 20:11:41.470041 18478 raft_consensus.cc:2454] T 3fd0e8c360734dcbb75dc2958b70c2b8 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
I20240430 20:11:41.457754 18450 tablet_service.cc:1467] Processing CreateTablet for tablet a0d3c4ba64024bccad95ecc909d03ae0 (DEFAULT_TABLE table=default.Table_5f50cb33cf774b86b2380ee4a569dac9 [id=bf4dcbec74f946398c7fc3f6eec6d068]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:41.470666 18450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0d3c4ba64024bccad95ecc909d03ae0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:41.472230 18615 raft_consensus.cc:2454] T a9e0659564894b0e8cb7f600bb07a085 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:41.473824 18696 leader_election.cc:302] T a9e0659564894b0e8cb7f600bb07a085 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a
I20240430 20:11:41.458454 18454 tablet_service.cc:1467] Processing CreateTablet for tablet f8bfbf673ea346529d1b2b376a3da496 (DEFAULT_TABLE table=default.Table_1ebe0eb1a1984ee9bbb16d634ff2cedb [id=aa91ac92289a48b784e97eea9967192f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:41.462016 18474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7542c7fe6cad452ab718f9ef54dbbbbe" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:41.474226 18454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8bfbf673ea346529d1b2b376a3da496. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:41.474287 18474 raft_consensus.cc:2454] T 7542c7fe6cad452ab718f9ef54dbbbbe P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:41.475142 18449 tablet_service.cc:1467] Processing CreateTablet for tablet 3b7fdc6ec7e046c5af894f9a6eda17b0 (DEFAULT_TABLE table=default.Table_d441c3bc84ed42358d1bc6e717108efb [id=7b516b6eb3a240a680bf271eab0324f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:41.475502 18449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b7fdc6ec7e046c5af894f9a6eda17b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:41.478190 18446 tablet_service.cc:1467] Processing CreateTablet for tablet e5b85a066cdc46d48ccf8201e5423c29 (DEFAULT_TABLE table=default.Table_81547db3074240c98cdd95ed3b2dd813 [id=e0ea901452334b36a8493baed478cc1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:41.478564 18446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5b85a066cdc46d48ccf8201e5423c29. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:41.478866 18459 tablet_service.cc:1467] Processing CreateTablet for tablet 8668f7a285de43f48d83abecb4b2e96c (DEFAULT_TABLE table=default.Table_6d9a8c8bd13b42eca6834e707653a66b [id=71c1c0c9a8b544a0a9b1b1a414a6f749]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:41.479337 18734 tablet_service.cc:1467] Processing CreateTablet for tablet fd1b8407071e4e4287fc172009777414 (DEFAULT_TABLE table=default.Table_5572f0acadaa432e8570b159fd9b76bc [id=d0f3b1709ac94aed8ec568c0327eab34]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:41.479808 18734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd1b8407071e4e4287fc172009777414. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:41.480784 18699 leader_election.cc:302] T 7542c7fe6cad452ab718f9ef54dbbbbe P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters:
I20240430 20:11:41.481117 20198 raft_consensus.cc:2781] T a9e0659564894b0e8cb7f600bb07a085 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:41.481246 20198 raft_consensus.cc:686] T a9e0659564894b0e8cb7f600bb07a085 P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Becoming Leader. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Running, Role: LEADER
I20240430 20:11:41.481508 20198 consensus_queue.cc:237] T a9e0659564894b0e8cb7f600bb07a085 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:41.482530 20019 raft_consensus.cc:2781] T 7542c7fe6cad452ab718f9ef54dbbbbe P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:41.482643 20019 raft_consensus.cc:482] T 7542c7fe6cad452ab718f9ef54dbbbbe P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:41.482736 20019 raft_consensus.cc:3037] T 7542c7fe6cad452ab718f9ef54dbbbbe P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:41.482870 18447 tablet_service.cc:1467] Processing CreateTablet for tablet 347a8224114b49e5b7dcb75d9b726e0a (DEFAULT_TABLE table=default.Table_dd1d113759284183905a803b2733436d [id=9d581bf380f54b4dab56aa34b9c4b1ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:41.483465 18730 tablet_service.cc:1467] Processing CreateTablet for tablet 8668f7a285de43f48d83abecb4b2e96c (DEFAULT_TABLE table=default.Table_6d9a8c8bd13b42eca6834e707653a66b [id=71c1c0c9a8b544a0a9b1b1a414a6f749]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:41.483824 18730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8668f7a285de43f48d83abecb4b2e96c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:41.485545 20019 raft_consensus.cc:504] T 7542c7fe6cad452ab718f9ef54dbbbbe P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:41.486217 20019 leader_election.cc:288] T 7542c7fe6cad452ab718f9ef54dbbbbe P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Requested vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:41.486443 20019 raft_consensus.cc:482] T 5d09baed576e4136bf6d756ca39d08b2 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:41.488577 18474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7542c7fe6cad452ab718f9ef54dbbbbe" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:41.488770 18474 raft_consensus.cc:3037] T 7542c7fe6cad452ab718f9ef54dbbbbe P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:41.491210 18474 raft_consensus.cc:2454] T 7542c7fe6cad452ab718f9ef54dbbbbe P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:41.491765 18699 leader_election.cc:302] T 7542c7fe6cad452ab718f9ef54dbbbbe P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters:
I20240430 20:11:41.493048 20205 raft_consensus.cc:2781] T 7542c7fe6cad452ab718f9ef54dbbbbe P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:41.493180 20205 raft_consensus.cc:686] T 7542c7fe6cad452ab718f9ef54dbbbbe P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Becoming Leader. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Running, Role: LEADER
I20240430 20:11:41.493439 20205 consensus_queue.cc:237] T 7542c7fe6cad452ab718f9ef54dbbbbe P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:41.496078 18589 tablet_service.cc:1467] Processing CreateTablet for tablet 3b7fdc6ec7e046c5af894f9a6eda17b0 (DEFAULT_TABLE table=default.Table_d441c3bc84ed42358d1bc6e717108efb [id=7b516b6eb3a240a680bf271eab0324f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:41.496932 18592 tablet_service.cc:1467] Processing CreateTablet for tablet e5b85a066cdc46d48ccf8201e5423c29 (DEFAULT_TABLE table=default.Table_81547db3074240c98cdd95ed3b2dd813 [id=e0ea901452334b36a8493baed478cc1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:41.495898 18608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7542c7fe6cad452ab718f9ef54dbbbbe" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:41.500047 18592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5b85a066cdc46d48ccf8201e5423c29. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:41.502657 18600 tablet_service.cc:1467] Processing CreateTablet for tablet 347a8224114b49e5b7dcb75d9b726e0a (DEFAULT_TABLE table=default.Table_dd1d113759284183905a803b2733436d [id=9d581bf380f54b4dab56aa34b9c4b1ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:41.503064 18600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 347a8224114b49e5b7dcb75d9b726e0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:41.503099 18733 tablet_service.cc:1467] Processing CreateTablet for tablet e5b85a066cdc46d48ccf8201e5423c29 (DEFAULT_TABLE table=default.Table_81547db3074240c98cdd95ed3b2dd813 [id=e0ea901452334b36a8493baed478cc1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:41.503662 18733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5b85a066cdc46d48ccf8201e5423c29. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:41.498648 18558 leader_election.cc:302] T 3fd0e8c360734dcbb75dc2958b70c2b8 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters: 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:41.509411 18603 tablet_service.cc:1467] Processing CreateTablet for tablet 8668f7a285de43f48d83abecb4b2e96c (DEFAULT_TABLE table=default.Table_6d9a8c8bd13b42eca6834e707653a66b [id=71c1c0c9a8b544a0a9b1b1a414a6f749]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:41.509450 18724 tablet_service.cc:1467] Processing CreateTablet for tablet 347a8224114b49e5b7dcb75d9b726e0a (DEFAULT_TABLE table=default.Table_dd1d113759284183905a803b2733436d [id=9d581bf380f54b4dab56aa34b9c4b1ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:41.510056 18724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 347a8224114b49e5b7dcb75d9b726e0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:41.486583 20019 raft_consensus.cc:504] T 5d09baed576e4136bf6d756ca39d08b2 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:41.517139 20019 leader_election.cc:288] T 5d09baed576e4136bf6d756ca39d08b2 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:41.517621 18474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d09baed576e4136bf6d756ca39d08b2" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:41.517905 18474 raft_consensus.cc:2379] T 5d09baed576e4136bf6d756ca39d08b2 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c51c33d5616482cafb68e26688820f4 in current term 1: Already voted for candidate 5e1c6397f09747f091534faf93542710 in this term.
I20240430 20:11:41.518986 19023 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: Sending DeleteTablet for 3 replicas of tablet 95383d9d5a234a4d9d841a8b114428b0
I20240430 20:11:41.519609 19023 catalog_manager.cc:6554] ReplaceTablet: tablet 95383d9d5a234a4d9d841a8b114428b0 deleted and replaced by tablet 6f3996b7e29941e391ea141dd6cc44b2
I20240430 20:11:41.498450 18615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7542c7fe6cad452ab718f9ef54dbbbbe" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:41.520457 20024 raft_consensus.cc:482] T 43b5e2c005a34df08cc4335c6e014cb5 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:41.522433 18589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b7fdc6ec7e046c5af894f9a6eda17b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:41.523330 18449 tablet_service.cc:1514] Processing DeleteTablet for tablet 95383d9d5a234a4d9d841a8b114428b0 with delete_type TABLET_DATA_DELETED (replaced by tablet 6f3996b7e29941e391ea141dd6cc44b2) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:41.508723 19489 ts_tablet_manager.cc:1888] T 75ebf051383e42529dcb5f7dd0770a4b P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.1
I20240430 20:11:41.525775 19489 log.cc:1196] T 75ebf051383e42529dcb5f7dd0770a4b P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/75ebf051383e42529dcb5f7dd0770a4b
I20240430 20:11:41.526269 19489 ts_tablet_manager.cc:1909] T 75ebf051383e42529dcb5f7dd0770a4b P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:41.526727 18585 tablet_service.cc:1514] Processing DeleteTablet for tablet 95383d9d5a234a4d9d841a8b114428b0 with delete_type TABLET_DATA_DELETED (replaced by tablet 6f3996b7e29941e391ea141dd6cc44b2) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:41.528293 18985 tablet_bootstrap.cc:492] T ae87f5e92a934e2182164c231bc76df0 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:41.528503 18985 ts_tablet_manager.cc:1367] T ae87f5e92a934e2182164c231bc76df0 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.175s user 0.006s sys 0.007s
I20240430 20:11:41.529513 18985 raft_consensus.cc:348] T ae87f5e92a934e2182164c231bc76df0 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:41.529795 18985 raft_consensus.cc:374] T ae87f5e92a934e2182164c231bc76df0 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:41.529863 18985 raft_consensus.cc:729] T ae87f5e92a934e2182164c231bc76df0 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:41.530186 18985 consensus_queue.cc:260] T ae87f5e92a934e2182164c231bc76df0 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:41.530715 20096 raft_consensus.cc:2781] T 3fd0e8c360734dcbb75dc2958b70c2b8 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:41.530856 18985 ts_tablet_manager.cc:1398] T ae87f5e92a934e2182164c231bc76df0 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240430 20:11:41.530987 18617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d09baed576e4136bf6d756ca39d08b2" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:41.531287 18603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8668f7a285de43f48d83abecb4b2e96c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:41.534978 20024 raft_consensus.cc:504] T 43b5e2c005a34df08cc4335c6e014cb5 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:41.535303 18725 tablet_service.cc:1514] Processing DeleteTablet for tablet 95383d9d5a234a4d9d841a8b114428b0 with delete_type TABLET_DATA_DELETED (replaced by tablet 6f3996b7e29941e391ea141dd6cc44b2) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:41.535789 18696 leader_election.cc:302] T 5d09baed576e4136bf6d756ca39d08b2 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710
I20240430 20:11:41.544051 18988 tablet_bootstrap.cc:492] T 2f61e3f88a6448dfa323204bd8d14cc5 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:41.544236 18988 ts_tablet_manager.cc:1367] T 2f61e3f88a6448dfa323204bd8d14cc5 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.149s user 0.009s sys 0.002s
I20240430 20:11:41.547253 18988 raft_consensus.cc:348] T 2f61e3f88a6448dfa323204bd8d14cc5 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:41.547814 18988 raft_consensus.cc:374] T 2f61e3f88a6448dfa323204bd8d14cc5 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:41.539018 18608 raft_consensus.cc:3037] T 7542c7fe6cad452ab718f9ef54dbbbbe P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:41.550061 20198 raft_consensus.cc:3037] T 5d09baed576e4136bf6d756ca39d08b2 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:41.554548 20024 leader_election.cc:288] T 43b5e2c005a34df08cc4335c6e014cb5 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:41.554531 18756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43b5e2c005a34df08cc4335c6e014cb5" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:41.539129 20313 consensus_queue.cc:1035] T fd40967ab4a04001819e16c06a6c4300 P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.016s
W20240430 20:11:41.555334 18557 leader_election.cc:341] T 43b5e2c005a34df08cc4335c6e014cb5 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:41.556159 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 107) took 9429 ms. Trace:
I20240430 20:11:41.556325 19489 rpcz_store.cc:276] 0430 20:11:32.126903 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:32.126962 (+ 59us) service_pool.cc:225] Handling call
0430 20:11:41.122309 (+8995347us) ts_tablet_manager.cc:1120] Deleting tablet 75ebf051383e42529dcb5f7dd0770a4b
0430 20:11:41.122432 (+ 123us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:41.300567 (+178135us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:41.508598 (+208031us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:41.556120 (+ 47522us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":8989627}
I20240430 20:11:41.556581 20198 raft_consensus.cc:2726] T 5d09baed576e4136bf6d756ca39d08b2 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:41.556968 19001 catalog_manager.cc:4830] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): tablet 75ebf051383e42529dcb5f7dd0770a4b (table default.Table_e9655558850747968a52e5885a113f59 [id=9c272f6ad56440b682f12f027af57bc4]) successfully deleted
I20240430 20:11:41.547976 18988 raft_consensus.cc:729] T 2f61e3f88a6448dfa323204bd8d14cc5 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:41.558936 18474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43b5e2c005a34df08cc4335c6e014cb5" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
W20240430 20:11:41.559595 18558 leader_election.cc:341] T 43b5e2c005a34df08cc4335c6e014cb5 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:41.559720 18558 leader_election.cc:302] T 43b5e2c005a34df08cc4335c6e014cb5 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:41.562572 18449 tablet_service.cc:1467] Processing CreateTablet for tablet fd1b8407071e4e4287fc172009777414 (DEFAULT_TABLE table=default.Table_5572f0acadaa432e8570b159fd9b76bc [id=d0f3b1709ac94aed8ec568c0327eab34]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:41.562721 18461 tablet_service.cc:1467] Processing CreateTablet for tablet b7d8da6e2fe8499d9d53af84a25f2faf (DEFAULT_TABLE table=default.Table_5dd3db8b666a491e8b33cbb5a0b54ff5 [id=4b7dfcb0dd2b4e8ba3a9a4a2bce81287]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:41.564108 18448 tablet_service.cc:1467] Processing CreateTablet for tablet 00be6ddd91e84865a48743fb4d077eec (DEFAULT_TABLE table=default.Table_c5c549bbf87d46c7886dad5f93086cd5 [id=5db65c3521504fffbe2599dbb5337a6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:41.559764 18988 consensus_queue.cc:260] T 2f61e3f88a6448dfa323204bd8d14cc5 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:41.565388 18457 tablet_service.cc:1467] Processing CreateTablet for tablet 4391b5227fb64ef481542a962a5e21a5 (DEFAULT_TABLE table=default.Table_d82824c897a94dd0ba45955f9473b626 [id=d794640d21a843a185038673f7eb2b00]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:41.566561 20198 raft_consensus.cc:482] T e6eac92dfa5f4456bc49571878f0f8dd P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:41.566845 20198 raft_consensus.cc:504] T e6eac92dfa5f4456bc49571878f0f8dd P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:41.567137 18725 tablet_service.cc:1467] Processing CreateTablet for tablet b7d8da6e2fe8499d9d53af84a25f2faf (DEFAULT_TABLE table=default.Table_5dd3db8b666a491e8b33cbb5a0b54ff5 [id=4b7dfcb0dd2b4e8ba3a9a4a2bce81287]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:41.567376 18736 tablet_service.cc:1467] Processing CreateTablet for tablet dbb427aefa6b47b0805d1751fea4efcb (DEFAULT_TABLE table=default.Table_4130caf534004edfb413056d7f6cfd52 [id=da9ad991953a40c4b77b31317597ad1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:41.568133 18736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbb427aefa6b47b0805d1751fea4efcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:41.568953 20198 leader_election.cc:288] T e6eac92dfa5f4456bc49571878f0f8dd P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
W20240430 20:11:41.569671 18699 leader_election.cc:341] T e6eac92dfa5f4456bc49571878f0f8dd P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Not found: Tablet not found: e6eac92dfa5f4456bc49571878f0f8dd
I20240430 20:11:41.570137 18730 tablet_service.cc:1467] Processing CreateTablet for tablet 00be6ddd91e84865a48743fb4d077eec (DEFAULT_TABLE table=default.Table_c5c549bbf87d46c7886dad5f93086cd5 [id=5db65c3521504fffbe2599dbb5337a6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:41.570633 18730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00be6ddd91e84865a48743fb4d077eec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:41.567631 18734 tablet_service.cc:1467] Processing CreateTablet for tablet 4391b5227fb64ef481542a962a5e21a5 (DEFAULT_TABLE table=default.Table_d82824c897a94dd0ba45955f9473b626 [id=d794640d21a843a185038673f7eb2b00]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:41.578614 18734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4391b5227fb64ef481542a962a5e21a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:41.586900 19495 raft_consensus.cc:2229] T 5bde72e2791a497db7dd08686ca36d80 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:41.587136 19495 raft_consensus.cc:2258] T 5bde72e2791a497db7dd08686ca36d80 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:41.587533 20198 raft_consensus.cc:482] T 32c823d1700949ab8b22adcd83b7bfd1 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:41.587692 20198 raft_consensus.cc:504] T 32c823d1700949ab8b22adcd83b7bfd1 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:41.587826 19495 ts_tablet_manager.cc:1875] T 5bde72e2791a497db7dd08686ca36d80 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:41.588917 18725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7d8da6e2fe8499d9d53af84a25f2faf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:41.589071 20198 leader_election.cc:288] T 32c823d1700949ab8b22adcd83b7bfd1 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:41.596832 18447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 347a8224114b49e5b7dcb75d9b726e0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:41.597194 18459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8668f7a285de43f48d83abecb4b2e96c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:41.600612 19033 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_83b26c6b5e1942da864b8e60e882f5b4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:41.601156 19033 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_83b26c6b5e1942da864b8e60e882f5b4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:41.566687 19489 tablet_replica.cc:330] T 255c13ad8fae4ea4aa8e7b7aabb7602e P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:41.612398 19028 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_53b6a25ee2c242dcb83b4759648af9f3" } new_table_name: "default.Table_f1ad29d5986d4b799cfc67afb2b0dfdf" modify_external_catalogs: true
I20240430 20:11:41.618408 19024 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_41ccec912bfa439c888281ac81447a38"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:41.619280 19024 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_41ccec912bfa439c888281ac81447a38 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:41.623124 18474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32c823d1700949ab8b22adcd83b7bfd1" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:41.624497 18985 tablet_bootstrap.cc:492] T 34130f2820064fa0898ef1f9d7de1c6d P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:41.639474 19048 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet ae20f44df50343289b0a512a4ea53692 (table default.Table_1844f6b1bda84812825384e14a2fd94e [id=71da9acc3cff4d95bbac4a686b708fd5]). Expected version 1 got 0
W20240430 20:11:41.644264 18558 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.177.1:41217, user_credentials={real_user=slave}} blocked reactor thread for 84671.5us
I20240430 20:11:41.645506 18615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32c823d1700949ab8b22adcd83b7bfd1" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:41.645767 18615 raft_consensus.cc:2379] T 32c823d1700949ab8b22adcd83b7bfd1 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c51c33d5616482cafb68e26688820f4 in current term 1: Already voted for candidate 2efb542aa55e439b99ccb98d1c5c884a in this term.
I20240430 20:11:41.642156 20024 raft_consensus.cc:2726] T 43b5e2c005a34df08cc4335c6e014cb5 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:41.625217 19495 ts_tablet_manager.cc:1888] T 5bde72e2791a497db7dd08686ca36d80 P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:41.627424 18449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd1b8407071e4e4287fc172009777414. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:41.647214 19495 log.cc:1196] T 5bde72e2791a497db7dd08686ca36d80 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/5bde72e2791a497db7dd08686ca36d80
I20240430 20:11:41.647604 19495 ts_tablet_manager.cc:1909] T 5bde72e2791a497db7dd08686ca36d80 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:41.647933 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 78) took 10390 ms. Trace:
I20240430 20:11:41.648006 19495 rpcz_store.cc:276] 0430 20:11:31.257427 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:31.257606 (+ 179us) service_pool.cc:225] Handling call
0430 20:11:41.446222 (+10188616us) ts_tablet_manager.cc:1120] Deleting tablet 5bde72e2791a497db7dd08686ca36d80
0430 20:11:41.446235 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:41.600509 (+154274us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:41.625127 (+ 24618us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:41.647915 (+ 22788us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5497,"tablet-delete.queue_time_us":10080214}
I20240430 20:11:41.648378 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 81) took 10172 ms. Trace:
I20240430 20:11:41.648442 19495 rpcz_store.cc:276] 0430 20:11:31.475670 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:31.480920 (+ 5250us) service_pool.cc:225] Handling call
0430 20:11:41.648268 (+10167348us) ts_tablet_manager.cc:1120] Deleting tablet 5bd0362ff41b4faca0205d136d596737
0430 20:11:41.648276 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:41.648365 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":10116757}
I20240430 20:11:41.648689 19495 tablet_replica.cc:330] T 77285afcafe04a829e485998bbbe00f0 P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:41.648555 18617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6eac92dfa5f4456bc49571878f0f8dd" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:41.648864 18617 raft_consensus.cc:2379] T e6eac92dfa5f4456bc49571878f0f8dd P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c51c33d5616482cafb68e26688820f4 in current term 1: Already voted for candidate 2efb542aa55e439b99ccb98d1c5c884a in this term.
I20240430 20:11:41.648849 19495 raft_consensus.cc:2229] T 77285afcafe04a829e485998bbbe00f0 P 2efb542aa55e439b99ccb98d1c5c884a [term 2 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:41.649003 19495 raft_consensus.cc:2258] T 77285afcafe04a829e485998bbbe00f0 P 2efb542aa55e439b99ccb98d1c5c884a [term 2 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:41.650902 18448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00be6ddd91e84865a48743fb4d077eec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:41.651119 19495 ts_tablet_manager.cc:1875] T 77285afcafe04a829e485998bbbe00f0 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:41.654114 19004 catalog_manager.cc:4830] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): tablet 5bde72e2791a497db7dd08686ca36d80 (table default.Table_9c1e6182953140bb9cd594f08458c7d9 [id=5227206f44d04fa89b4842b899f33afb]) successfully deleted
W20240430 20:11:41.654495 19004 catalog_manager.cc:4810] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): delete failed for tablet 5bd0362ff41b4faca0205d136d596737 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5bd0362ff41b4faca0205d136d596737 already in progress: creating tablet
I20240430 20:11:41.668565 18457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4391b5227fb64ef481542a962a5e21a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:41.673637 18461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7d8da6e2fe8499d9d53af84a25f2faf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:41.679807 20312 raft_consensus.cc:482] T ca64de16d40f4a44b8b411f5526ba11a P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:41.679986 20312 raft_consensus.cc:504] T ca64de16d40f4a44b8b411f5526ba11a P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:41.682157 19495 ts_tablet_manager.cc:1888] T 77285afcafe04a829e485998bbbe00f0 P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240430 20:11:41.682291 19495 log.cc:1196] T 77285afcafe04a829e485998bbbe00f0 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/77285afcafe04a829e485998bbbe00f0
I20240430 20:11:41.685885 19495 ts_tablet_manager.cc:1909] T 77285afcafe04a829e485998bbbe00f0 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:41.686264 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 82) took 10110 ms. Trace:
I20240430 20:11:41.686352 19495 rpcz_store.cc:276] 0430 20:11:31.576321 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:31.576507 (+ 186us) service_pool.cc:225] Handling call
0430 20:11:41.648652 (+10072145us) ts_tablet_manager.cc:1120] Deleting tablet 77285afcafe04a829e485998bbbe00f0
0430 20:11:41.648659 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:41.654042 (+ 5383us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:41.682031 (+ 27989us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:41.686240 (+ 4209us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":10071922}
I20240430 20:11:41.686841 19495 tablet_replica.cc:330] T 802336602d8f4ac4bd7e0872d6f3dae9 P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:41.687263 19004 catalog_manager.cc:4830] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): tablet 77285afcafe04a829e485998bbbe00f0 (table default.Table_e63f64a411094f8db2ef641f2257ca79 [id=040dec3a3d804cdfbdada359bbe5123b]) successfully deleted
I20240430 20:11:41.693491 18586 tablet_service.cc:1467] Processing CreateTablet for tablet 4391b5227fb64ef481542a962a5e21a5 (DEFAULT_TABLE table=default.Table_d82824c897a94dd0ba45955f9473b626 [id=d794640d21a843a185038673f7eb2b00]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:41.687062 19495 raft_consensus.cc:2229] T 802336602d8f4ac4bd7e0872d6f3dae9 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Raft consensus shutting down.
I20240430 20:11:41.694114 18586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4391b5227fb64ef481542a962a5e21a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:41.694219 19495 raft_consensus.cc:2258] T 802336602d8f4ac4bd7e0872d6f3dae9 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:41.697249 19495 ts_tablet_manager.cc:1875] T 802336602d8f4ac4bd7e0872d6f3dae9 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:41.701344 18592 tablet_service.cc:1467] Processing CreateTablet for tablet fd1b8407071e4e4287fc172009777414 (DEFAULT_TABLE table=default.Table_5572f0acadaa432e8570b159fd9b76bc [id=d0f3b1709ac94aed8ec568c0327eab34]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:41.701967 18592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd1b8407071e4e4287fc172009777414. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:41.698874 20024 consensus_queue.cc:1035] T 76d6e9d547ca4a848eae852944b1d117 P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20240430 20:11:41.708873 20313 raft_consensus.cc:482] T 34130f2820064fa0898ef1f9d7de1c6d P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:41.709092 20313 raft_consensus.cc:504] T 34130f2820064fa0898ef1f9d7de1c6d P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:41.705668 18585 tablet_service.cc:1467] Processing CreateTablet for tablet 00be6ddd91e84865a48743fb4d077eec (DEFAULT_TABLE table=default.Table_c5c549bbf87d46c7886dad5f93086cd5 [id=5db65c3521504fffbe2599dbb5337a6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:41.719597 18585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00be6ddd91e84865a48743fb4d077eec. 1 dirs total, 0 dirs full, 0 dirs failed
W20240430 20:11:41.727218 18999 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 34130f2820064fa0898ef1f9d7de1c6d (table default.Table_69892a8264a4477db48ee6248758e046 [id=5a1c4a7be2fc465fac89a50854eb3322], current schema version=3) failed: Not found: failed to reset TS proxy: No leader found: Tablet 34130f2820064fa0898ef1f9d7de1c6d consensus state has no leader
I20240430 20:11:41.727885 19495 ts_tablet_manager.cc:1888] T 802336602d8f4ac4bd7e0872d6f3dae9 P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:41.728000 19495 log.cc:1196] T 802336602d8f4ac4bd7e0872d6f3dae9 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/802336602d8f4ac4bd7e0872d6f3dae9
I20240430 20:11:41.728340 19495 ts_tablet_manager.cc:1909] T 802336602d8f4ac4bd7e0872d6f3dae9 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:41.728637 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 84) took 10147 ms. Trace:
I20240430 20:11:41.728711 19495 rpcz_store.cc:276] 0430 20:11:31.581776 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:31.581972 (+ 196us) service_pool.cc:225] Handling call
0430 20:11:41.686790 (+10104818us) ts_tablet_manager.cc:1120] Deleting tablet 802336602d8f4ac4bd7e0872d6f3dae9
0430 20:11:41.686801 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:41.708661 (+ 21860us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:41.727764 (+ 19103us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:41.728621 (+ 857us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":10104560}
I20240430 20:11:41.729044 19495 tablet_replica.cc:330] T 26ed12f498644d15bf45e884dcad172e P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:41.729300 19495 raft_consensus.cc:2229] T 26ed12f498644d15bf45e884dcad172e P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:41.729533 19495 raft_consensus.cc:2258] T 26ed12f498644d15bf45e884dcad172e P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:41.731129 19004 catalog_manager.cc:4830] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): tablet 802336602d8f4ac4bd7e0872d6f3dae9 (table default.Table_d997b723a0dd48f6bc2d42c9bfd6bc14 [id=7737ec4807974640aa8556cb1c3ca3ef]) successfully deleted
I20240430 20:11:41.734484 20235 raft_consensus.cc:482] T 87ac32a0ddb84676adc83038ef664591 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:41.734694 20313 leader_election.cc:288] T 34130f2820064fa0898ef1f9d7de1c6d P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:41.734648 20235 raft_consensus.cc:504] T 87ac32a0ddb84676adc83038ef664591 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:41.735234 20235 leader_election.cc:288] T 87ac32a0ddb84676adc83038ef664591 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:41.736549 20312 leader_election.cc:288] T ca64de16d40f4a44b8b411f5526ba11a P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:41.736773 19495 ts_tablet_manager.cc:1875] T 26ed12f498644d15bf45e884dcad172e P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:41.744423 19471 ts_tablet_manager.cc:1888] T fcc9698abda4491b98e1e879589913a8 P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:41.744588 19471 log.cc:1196] T fcc9698abda4491b98e1e879589913a8 P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/fcc9698abda4491b98e1e879589913a8
I20240430 20:11:41.745770 20096 raft_consensus.cc:686] T 3fd0e8c360734dcbb75dc2958b70c2b8 P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Becoming Leader. State: Replica: 5e1c6397f09747f091534faf93542710, State: Running, Role: LEADER
I20240430 20:11:41.750377 18608 raft_consensus.cc:2454] T 7542c7fe6cad452ab718f9ef54dbbbbe P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:41.750972 19495 ts_tablet_manager.cc:1888] T 26ed12f498644d15bf45e884dcad172e P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:41.751124 19495 log.cc:1196] T 26ed12f498644d15bf45e884dcad172e P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/26ed12f498644d15bf45e884dcad172e
I20240430 20:11:41.751505 19495 ts_tablet_manager.cc:1909] T 26ed12f498644d15bf45e884dcad172e P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:41.751868 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 85) took 10109 ms. Trace:
I20240430 20:11:41.751964 19495 rpcz_store.cc:276] 0430 20:11:31.642682 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:31.642884 (+ 202us) service_pool.cc:225] Handling call
0430 20:11:41.728995 (+10086111us) ts_tablet_manager.cc:1120] Deleting tablet 26ed12f498644d15bf45e884dcad172e
0430 20:11:41.729005 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:41.747709 (+ 18704us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:41.750842 (+ 3133us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:41.751834 (+ 992us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":10085867}
I20240430 20:11:41.752228 18475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca64de16d40f4a44b8b411f5526ba11a" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:41.752375 19495 tablet_replica.cc:330] T 7dfdecba959543a795dfcd3eeda29e58 P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:41.752427 18475 raft_consensus.cc:2454] T ca64de16d40f4a44b8b411f5526ba11a P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 0.
I20240430 20:11:41.752537 19004 catalog_manager.cc:4830] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): tablet 26ed12f498644d15bf45e884dcad172e (table default.Table_3f42877693b04aaa837c89b8983d5a72 [id=6d499a625cee4df2a4f40ee189cc6220]) successfully deleted
I20240430 20:11:41.752606 19495 raft_consensus.cc:2229] T 7dfdecba959543a795dfcd3eeda29e58 P 2efb542aa55e439b99ccb98d1c5c884a [term 2 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:41.752691 18474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34130f2820064fa0898ef1f9d7de1c6d" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:41.752805 19495 raft_consensus.cc:2258] T 7dfdecba959543a795dfcd3eeda29e58 P 2efb542aa55e439b99ccb98d1c5c884a [term 2 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:41.753443 19495 ts_tablet_manager.cc:1875] T 7dfdecba959543a795dfcd3eeda29e58 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:41.746260 20096 consensus_queue.cc:237] T 3fd0e8c360734dcbb75dc2958b70c2b8 P 5e1c6397f09747f091534faf93542710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:41.754319 18617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87ac32a0ddb84676adc83038ef664591" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
W20240430 20:11:41.759457 18415 leader_election.cc:341] T 87ac32a0ddb84676adc83038ef664591 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:41.764307 19471 ts_tablet_manager.cc:1909] T fcc9698abda4491b98e1e879589913a8 P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:41.765846 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 89) took 9975 ms. Trace:
I20240430 20:11:41.765983 19471 rpcz_store.cc:276] 0430 20:11:31.789995 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:31.832652 (+ 42657us) service_pool.cc:225] Handling call
0430 20:11:41.354996 (+9522344us) ts_tablet_manager.cc:1120] Deleting tablet fcc9698abda4491b98e1e879589913a8
0430 20:11:41.355012 (+ 16us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:41.644997 (+289985us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:41.744282 (+ 99285us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:41.765813 (+ 21531us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":9511381}
I20240430 20:11:41.766408 19471 tablet_replica.cc:330] T 5d09baed576e4136bf6d756ca39d08b2 P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:41.767495 20311 consensus_queue.cc:1035] T acf3b6fead41483a82c43dc0286fab7c P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:41.770496 18558 leader_election.cc:302] T ca64de16d40f4a44b8b411f5526ba11a P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:41.770958 19495 ts_tablet_manager.cc:1888] T 7dfdecba959543a795dfcd3eeda29e58 P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.2
W20240430 20:11:41.770968 18558 leader_election.cc:341] T 34130f2820064fa0898ef1f9d7de1c6d P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:41.771072 19495 log.cc:1196] T 7dfdecba959543a795dfcd3eeda29e58 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/7dfdecba959543a795dfcd3eeda29e58
I20240430 20:11:41.771406 19495 ts_tablet_manager.cc:1909] T 7dfdecba959543a795dfcd3eeda29e58 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:41.771512 20312 raft_consensus.cc:2781] T ca64de16d40f4a44b8b411f5526ba11a P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:41.771615 20312 raft_consensus.cc:482] T ca64de16d40f4a44b8b411f5526ba11a P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:41.771698 20312 raft_consensus.cc:3037] T ca64de16d40f4a44b8b411f5526ba11a P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:41.775700 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 86) took 10116 ms. Trace:
I20240430 20:11:41.775828 19495 rpcz_store.cc:276] 0430 20:11:31.658800 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:31.658880 (+ 80us) service_pool.cc:225] Handling call
0430 20:11:41.752319 (+10093439us) ts_tablet_manager.cc:1120] Deleting tablet 7dfdecba959543a795dfcd3eeda29e58
0430 20:11:41.752332 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:41.766345 (+ 14013us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:41.770850 (+ 4505us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:41.775670 (+ 4820us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":10093149}
I20240430 20:11:41.776229 19495 tablet_replica.cc:330] T fcc9698abda4491b98e1e879589913a8 P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:41.776420 19495 raft_consensus.cc:2229] T fcc9698abda4491b98e1e879589913a8 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Raft consensus shutting down.
I20240430 20:11:41.776479 19004 catalog_manager.cc:4830] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): tablet 7dfdecba959543a795dfcd3eeda29e58 (table default.Table_da5216d025844afda5a83ada555bc120 [id=dc55b25c3ca14863a21042e805e3bf9a]) successfully deleted
I20240430 20:11:41.776744 19495 raft_consensus.cc:2258] T fcc9698abda4491b98e1e879589913a8 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:41.777328 19495 ts_tablet_manager.cc:1875] T fcc9698abda4491b98e1e879589913a8 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:41.781071 20235 raft_consensus.cc:482] T ae87f5e92a934e2182164c231bc76df0 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:41.781240 20235 raft_consensus.cc:504] T ae87f5e92a934e2182164c231bc76df0 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:41.781857 20235 leader_election.cc:288] T ae87f5e92a934e2182164c231bc76df0 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:41.793334 18617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae87f5e92a934e2182164c231bc76df0" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:41.793576 18617 raft_consensus.cc:2454] T ae87f5e92a934e2182164c231bc76df0 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 0.
I20240430 20:11:41.794210 18415 leader_election.cc:302] T ae87f5e92a934e2182164c231bc76df0 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:41.794624 20235 raft_consensus.cc:2781] T ae87f5e92a934e2182164c231bc76df0 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:41.794703 20235 raft_consensus.cc:482] T ae87f5e92a934e2182164c231bc76df0 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:41.794757 20235 raft_consensus.cc:3037] T ae87f5e92a934e2182164c231bc76df0 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:41.796325 19471 raft_consensus.cc:2229] T 5d09baed576e4136bf6d756ca39d08b2 P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Raft consensus shutting down.
I20240430 20:11:41.796698 19471 pending_rounds.cc:62] T 5d09baed576e4136bf6d756ca39d08b2 P 5e1c6397f09747f091534faf93542710: Trying to abort 1 pending ops.
I20240430 20:11:41.796742 19471 pending_rounds.cc:68] T 5d09baed576e4136bf6d756ca39d08b2 P 5e1c6397f09747f091534faf93542710: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7022624363246829568 op_type: NO_OP noop_request { }
I20240430 20:11:41.796833 19471 raft_consensus.cc:2866] T 5d09baed576e4136bf6d756ca39d08b2 P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240430 20:11:41.796897 19471 raft_consensus.cc:2258] T 5d09baed576e4136bf6d756ca39d08b2 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:41.797272 20235 raft_consensus.cc:504] T ae87f5e92a934e2182164c231bc76df0 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:41.798439 20235 leader_election.cc:288] T ae87f5e92a934e2182164c231bc76df0 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:41.801340 18617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae87f5e92a934e2182164c231bc76df0" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:41.802081 18617 raft_consensus.cc:3037] T ae87f5e92a934e2182164c231bc76df0 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:41.802863 20312 raft_consensus.cc:504] T ca64de16d40f4a44b8b411f5526ba11a P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:41.805505 19471 ts_tablet_manager.cc:1875] T 5d09baed576e4136bf6d756ca39d08b2 P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:41.812242 19495 ts_tablet_manager.cc:1888] T fcc9698abda4491b98e1e879589913a8 P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:41.812410 19495 log.cc:1196] T fcc9698abda4491b98e1e879589913a8 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/fcc9698abda4491b98e1e879589913a8
I20240430 20:11:41.812806 19495 ts_tablet_manager.cc:1909] T fcc9698abda4491b98e1e879589913a8 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:41.813264 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 91) took 10054 ms. Trace:
I20240430 20:11:41.813375 19495 rpcz_store.cc:276] 0430 20:11:31.758949 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:31.759008 (+ 59us) service_pool.cc:225] Handling call
0430 20:11:41.776181 (+10017173us) ts_tablet_manager.cc:1120] Deleting tablet fcc9698abda4491b98e1e879589913a8
0430 20:11:41.776191 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:41.780221 (+ 4030us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:41.790583 (+ 10362us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:41.813237 (+ 22654us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":10016902}
I20240430 20:11:41.813838 19495 tablet_replica.cc:330] T 5d09baed576e4136bf6d756ca39d08b2 P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:41.813839 20312 leader_election.cc:288] T ca64de16d40f4a44b8b411f5526ba11a P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:41.814100 19495 raft_consensus.cc:2229] T 5d09baed576e4136bf6d756ca39d08b2 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:41.814294 19495 raft_consensus.cc:2258] T 5d09baed576e4136bf6d756ca39d08b2 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:41.814730 18474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca64de16d40f4a44b8b411f5526ba11a" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:41.815826 18474 raft_consensus.cc:3037] T ca64de16d40f4a44b8b411f5526ba11a P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:41.820519 19495 ts_tablet_manager.cc:1875] T 5d09baed576e4136bf6d756ca39d08b2 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:41.833490 18617 raft_consensus.cc:2454] T ae87f5e92a934e2182164c231bc76df0 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 1.
I20240430 20:11:41.834229 18415 leader_election.cc:302] T ae87f5e92a934e2182164c231bc76df0 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:41.834690 20235 raft_consensus.cc:2781] T ae87f5e92a934e2182164c231bc76df0 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:41.834836 20235 raft_consensus.cc:686] T ae87f5e92a934e2182164c231bc76df0 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Becoming Leader. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Running, Role: LEADER
I20240430 20:11:41.835079 20235 consensus_queue.cc:237] T ae87f5e92a934e2182164c231bc76df0 P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:41.846017 18601 tablet_service.cc:1467] Processing CreateTablet for tablet b7d8da6e2fe8499d9d53af84a25f2faf (DEFAULT_TABLE table=default.Table_5dd3db8b666a491e8b33cbb5a0b54ff5 [id=4b7dfcb0dd2b4e8ba3a9a4a2bce81287]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:41.846632 18983 tablet_bootstrap.cc:492] T 84568f90b9c54cf292099808d4cbf5dc P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:41.850922 18983 ts_tablet_manager.cc:1367] T 84568f90b9c54cf292099808d4cbf5dc P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.411s user 0.028s sys 0.018s
I20240430 20:11:41.851773 18601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7d8da6e2fe8499d9d53af84a25f2faf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:41.820992 18474 raft_consensus.cc:2454] T ca64de16d40f4a44b8b411f5526ba11a P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
I20240430 20:11:41.852548 18558 leader_election.cc:302] T ca64de16d40f4a44b8b411f5526ba11a P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:41.854144 20096 raft_consensus.cc:2781] T ca64de16d40f4a44b8b411f5526ba11a P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:41.854419 20096 raft_consensus.cc:686] T ca64de16d40f4a44b8b411f5526ba11a P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Becoming Leader. State: Replica: 5e1c6397f09747f091534faf93542710, State: Running, Role: LEADER
I20240430 20:11:41.856832 19495 ts_tablet_manager.cc:1888] T 5d09baed576e4136bf6d756ca39d08b2 P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:41.857187 19495 log.cc:1196] T 5d09baed576e4136bf6d756ca39d08b2 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/5d09baed576e4136bf6d756ca39d08b2
I20240430 20:11:41.857599 19495 ts_tablet_manager.cc:1909] T 5d09baed576e4136bf6d756ca39d08b2 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:41.858018 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 100) took 9803 ms. Trace:
I20240430 20:11:41.858533 18985 tablet_bootstrap.cc:654] T 34130f2820064fa0898ef1f9d7de1c6d P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:41.864594 18592 tablet_service.cc:1467] Processing CreateTablet for tablet dbb427aefa6b47b0805d1751fea4efcb (DEFAULT_TABLE table=default.Table_4130caf534004edfb413056d7f6cfd52 [id=da9ad991953a40c4b77b31317597ad1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:41.865149 18592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbb427aefa6b47b0805d1751fea4efcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:41.868547 18696 leader_election.cc:302] T 32c823d1700949ab8b22adcd83b7bfd1 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710
I20240430 20:11:41.870177 18760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca64de16d40f4a44b8b411f5526ba11a" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:41.871814 18983 raft_consensus.cc:348] T 84568f90b9c54cf292099808d4cbf5dc P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:41.872056 18983 raft_consensus.cc:374] T 84568f90b9c54cf292099808d4cbf5dc P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:41.872113 18983 raft_consensus.cc:729] T 84568f90b9c54cf292099808d4cbf5dc P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:41.872309 18983 consensus_queue.cc:260] T 84568f90b9c54cf292099808d4cbf5dc P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:41.872767 18983 ts_tablet_manager.cc:1398] T 84568f90b9c54cf292099808d4cbf5dc P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.022s user 0.001s sys 0.001s
I20240430 20:11:41.874526 18983 tablet_bootstrap.cc:492] T 1462ecf1176c4e66be42abddbe36b00c P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:41.880077 18983 tablet_bootstrap.cc:654] T 1462ecf1176c4e66be42abddbe36b00c P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:41.882323 19048 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 6e754af64cb64fd2a082adae7d5f24da (table default.Table_208b12b8e9bd4f3e8bc9c6d422ada649 [id=1df4e95702cd46f59ab21b101049a9ed]). Expected version 1 got 0
I20240430 20:11:41.882735 19048 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet acf3b6fead41483a82c43dc0286fab7c (table default.Table_01450a1e661d450a8cbe185e16e8250c [id=ce9911e0a91949b1997ee47321800255]). Expected version 1 got 0
I20240430 20:11:41.883159 19048 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet fbdeb4714b8c4b93ab72822c43329aae (table default.Table_c12ca6938d424fbfa2e1df5cdb00fdbc [id=0fb93f36bc17408ea56bc4cafcfd8580]). Expected version 1 got 0
I20240430 20:11:41.883630 19048 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet aa9998f4e17147aa8f2b35c01980463a (table default.Table_ffb3192f443d437ab63f56f628e596ad [id=608890834da843fa9af0cfc0d548029e]). Expected version 2 got 0
I20240430 20:11:41.854740 20096 consensus_queue.cc:237] T ca64de16d40f4a44b8b411f5526ba11a P 5e1c6397f09747f091534faf93542710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:41.858659 19495 rpcz_store.cc:276] 0430 20:11:32.054964 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:32.055045 (+ 81us) service_pool.cc:225] Handling call
0430 20:11:41.813778 (+9758733us) ts_tablet_manager.cc:1120] Deleting tablet 5d09baed576e4136bf6d756ca39d08b2
0430 20:11:41.813790 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:41.825105 (+ 11315us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:41.856678 (+ 31573us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:41.857998 (+ 1320us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":9729005}
I20240430 20:11:41.886107 19495 tablet_replica.cc:330] T 7542c7fe6cad452ab718f9ef54dbbbbe P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:41.886372 19495 raft_consensus.cc:2229] T 7542c7fe6cad452ab718f9ef54dbbbbe P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:41.886543 19495 raft_consensus.cc:2258] T 7542c7fe6cad452ab718f9ef54dbbbbe P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:41.887184 19495 ts_tablet_manager.cc:1875] T 7542c7fe6cad452ab718f9ef54dbbbbe P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:41.888551 19004 catalog_manager.cc:4830] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): tablet 5d09baed576e4136bf6d756ca39d08b2 (table default.Table_772ce91c27ce4119b6d1b1993e417f21 [id=29b41dafb4334b14a09602eabac2f031]) successfully deleted
I20240430 20:11:41.888866 18988 ts_tablet_manager.cc:1398] T 2f61e3f88a6448dfa323204bd8d14cc5 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.318s user 0.004s sys 0.000s
I20240430 20:11:41.891127 20024 consensus_queue.cc:1035] T 76d6e9d547ca4a848eae852944b1d117 P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:41.895504 19495 ts_tablet_manager.cc:1888] T 7542c7fe6cad452ab718f9ef54dbbbbe P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:41.895759 19495 log.cc:1196] T 7542c7fe6cad452ab718f9ef54dbbbbe P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/7542c7fe6cad452ab718f9ef54dbbbbe
W20240430 20:11:41.897243 18696 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.177.2:46027, user_credentials={real_user=slave}} blocked reactor thread for 174441us
I20240430 20:11:41.896193 19495 ts_tablet_manager.cc:1909] T 7542c7fe6cad452ab718f9ef54dbbbbe P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:41.865617 18457 tablet_service.cc:1467] Processing CreateTablet for tablet dbb427aefa6b47b0805d1751fea4efcb (DEFAULT_TABLE table=default.Table_4130caf534004edfb413056d7f6cfd52 [id=da9ad991953a40c4b77b31317597ad1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:41.897269 18762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca64de16d40f4a44b8b411f5526ba11a" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:41.897111 20309 raft_consensus.cc:3037] T 32c823d1700949ab8b22adcd83b7bfd1 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:41.898504 18756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34130f2820064fa0898ef1f9d7de1c6d" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:41.898910 18763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae87f5e92a934e2182164c231bc76df0" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:41.898937 18750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae87f5e92a934e2182164c231bc76df0" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:41.899441 18988 tablet_bootstrap.cc:492] T ca64de16d40f4a44b8b411f5526ba11a P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:41.898804 18457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbb427aefa6b47b0805d1751fea4efcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:41.900012 18696 leader_election.cc:302] T e6eac92dfa5f4456bc49571878f0f8dd P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710
I20240430 20:11:41.898905 18764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87ac32a0ddb84676adc83038ef664591" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:41.901162 20198 raft_consensus.cc:3037] T e6eac92dfa5f4456bc49571878f0f8dd P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:41.902979 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 102) took 9810 ms. Trace:
W20240430 20:11:41.904601 18557 leader_election.cc:341] T ca64de16d40f4a44b8b411f5526ba11a P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:41.904744 18988 tablet_bootstrap.cc:654] T ca64de16d40f4a44b8b411f5526ba11a P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:41.903831 19495 rpcz_store.cc:276] 0430 20:11:32.092438 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:32.092848 (+ 410us) service_pool.cc:225] Handling call
0430 20:11:41.886034 (+9793186us) ts_tablet_manager.cc:1120] Deleting tablet 7542c7fe6cad452ab718f9ef54dbbbbe
0430 20:11:41.886049 (+ 15us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:41.892030 (+ 5981us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:41.895365 (+ 3335us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:41.902960 (+ 7595us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":9792898}
W20240430 20:11:41.905470 18557 leader_election.cc:341] T ca64de16d40f4a44b8b411f5526ba11a P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:41.905834 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 105) took 9813 ms. Trace:
I20240430 20:11:41.905850 19004 catalog_manager.cc:4830] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): tablet 7542c7fe6cad452ab718f9ef54dbbbbe (table default.Table_8af52c0f1d404b42bc53287f64e20c2d [id=8e590e0e404344658b5cee90132d91d1]) successfully deleted
W20240430 20:11:41.907756 18557 leader_election.cc:341] T 34130f2820064fa0898ef1f9d7de1c6d P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:41.907878 18557 leader_election.cc:302] T 34130f2820064fa0898ef1f9d7de1c6d P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:41.908257 20311 raft_consensus.cc:2726] T 34130f2820064fa0898ef1f9d7de1c6d P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240430 20:11:41.908831 18417 leader_election.cc:341] T ae87f5e92a934e2182164c231bc76df0 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
W20240430 20:11:41.909145 18417 leader_election.cc:341] T ae87f5e92a934e2182164c231bc76df0 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:41.908192 19495 rpcz_store.cc:276] 0430 20:11:32.092718 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:32.093720 (+ 1002us) service_pool.cc:225] Handling call
0430 20:11:41.905733 (+9812013us) ts_tablet_manager.cc:1120] Deleting tablet aacabead3c874e77a47d2ecf19dbfa82
0430 20:11:41.905744 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:41.905816 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":9811809}
I20240430 20:11:41.884166 19048 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet a59ef6c545dd41a28265a4c1b8ce10e0 (table default.Table_07e40af8f1c546048975c8edf88170d7 [id=86df5d161d8641c89070e7eed9a39360]). Expected version 2 got 0
W20240430 20:11:41.909418 18417 leader_election.cc:341] T 87ac32a0ddb84676adc83038ef664591 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:41.909503 18417 leader_election.cc:302] T 87ac32a0ddb84676adc83038ef664591 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a; no voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710
I20240430 20:11:41.909722 20235 raft_consensus.cc:2726] T 87ac32a0ddb84676adc83038ef664591 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240430 20:11:41.909984 19004 catalog_manager.cc:4797] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): delete failed for tablet aacabead3c874e77a47d2ecf19dbfa82 because the tablet was not found. No further retry: Not found: Tablet not found: aacabead3c874e77a47d2ecf19dbfa82
I20240430 20:11:41.910027 20019 raft_consensus.cc:482] T 2a1e50f99c5d4b44bdbb51b58a92bc75 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:41.909960 19033 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_1840b0e89a9840fe92bd1ca13c915806" } new_table_name: "default.Table_744b5c96ea014bdb8feae740356a790e" modify_external_catalogs: true
I20240430 20:11:41.910187 20019 raft_consensus.cc:504] T 2a1e50f99c5d4b44bdbb51b58a92bc75 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:41.910898 20019 leader_election.cc:288] T 2a1e50f99c5d4b44bdbb51b58a92bc75 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:41.911263 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 101) took 9817 ms. Trace:
I20240430 20:11:41.911376 19495 rpcz_store.cc:276] 0430 20:11:32.092304 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:32.094209 (+ 1905us) service_pool.cc:225] Handling call
0430 20:11:41.909463 (+9815254us) ts_tablet_manager.cc:1120] Deleting tablet 47c37876694c4a4abb1091c631dab7a5
0430 20:11:41.909473 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:41.909536 (+ 63us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":9815050}
I20240430 20:11:41.912109 19024 master_service.cc:946] ReplaceTablet: received request to replace tablet 32c823d1700949ab8b22adcd83b7bfd1 from {username='slave'} at 127.0.0.1:51664
W20240430 20:11:41.912492 19004 catalog_manager.cc:4797] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): delete failed for tablet 47c37876694c4a4abb1091c631dab7a5 because the tablet was not found. No further retry: Not found: Tablet not found: 47c37876694c4a4abb1091c631dab7a5
I20240430 20:11:41.914731 19048 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet a05e32a33aa7488eb287cc6314c09c44 (table default.Table_2d1cfc78363a47aa8dba7ef7d80f8c9a [id=955d833ff8f14bf6957abadf9ef669e2]). Expected version 2 got 0
I20240430 20:11:41.915083 19048 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet c9e7660eccc448dbb99d135f82bfd261 (table default.Table_685a26737bb046fb85ced15df7ae3423 [id=51d177e8498848b38c43e3f8d11f092c]). Expected version 1 got 0
I20240430 20:11:41.915961 19048 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet f5eaf2413e1d4e69b56fae61f768f0ce (table default.Table_1280cfe32d5b41abb4d3eb1bd70641bf [id=8489b872074a42b6abb713e420968141]). Expected version 1 got 0
I20240430 20:11:41.915969 20198 raft_consensus.cc:2726] T e6eac92dfa5f4456bc49571878f0f8dd P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:41.916596 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 107) took 9820 ms. Trace:
I20240430 20:11:41.916643 19048 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 488d0e609b224674932f890939664df2 (table default.Table_58f35a2120874472978ca0e5f816e86c [id=ff18001c6d814afc8c1768d0c84a6bac]). Expected version 1 got 0
I20240430 20:11:41.916755 19495 rpcz_store.cc:276] 0430 20:11:32.095947 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:32.096002 (+ 55us) service_pool.cc:225] Handling call
0430 20:11:41.914883 (+9818881us) ts_tablet_manager.cc:1120] Deleting tablet 1398d4d2305b48e08fdb12432bd6c4b0
0430 20:11:41.914896 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:41.916574 (+ 1678us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":9818704}
W20240430 20:11:41.917258 19004 catalog_manager.cc:4797] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): delete failed for tablet 1398d4d2305b48e08fdb12432bd6c4b0 because the tablet was not found. No further retry: Not found: Tablet not found: 1398d4d2305b48e08fdb12432bd6c4b0
I20240430 20:11:41.917910 19048 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 22fef8bb6b034db098c3372ae992f7a7 (table default.Table_4440df38610340758047115845b42b95 [id=7d186ca20c094cc19aef792fe1c094e9]). Expected version 1 got 0
I20240430 20:11:41.918655 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 108) took 9817 ms. Trace:
I20240430 20:11:41.919088 18750 raft_consensus.cc:3037] T acf3b6fead41483a82c43dc0286fab7c P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:41.920704 20309 raft_consensus.cc:2726] T 32c823d1700949ab8b22adcd83b7bfd1 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:41.923086 19048 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 66dd767bd3f8494f94061fb89e5e5680 (table default.Table_9c1e6182953140bb9cd594f08458c7d9 [id=5227206f44d04fa89b4842b899f33afb]). Expected version 1 got 0
W20240430 20:11:41.918077 18696 leader_election.cc:341] T 2a1e50f99c5d4b44bdbb51b58a92bc75 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Not found: Tablet not found: 2a1e50f99c5d4b44bdbb51b58a92bc75
I20240430 20:11:41.924468 20312 consensus_queue.cc:1035] T acf3b6fead41483a82c43dc0286fab7c P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:41.925546 18985 tablet_bootstrap.cc:492] T 34130f2820064fa0898ef1f9d7de1c6d P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:41.925946 19495 rpcz_store.cc:276] 0430 20:11:32.100714 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:32.100762 (+ 48us) service_pool.cc:225] Handling call
0430 20:11:41.918536 (+9817774us) ts_tablet_manager.cc:1120] Deleting tablet 68ca48351cd4412381f5195d81109016
0430 20:11:41.918551 (+ 15us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:41.918636 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":9816386}
I20240430 20:11:41.926573 19495 tablet_replica.cc:330] T 2f61e3f88a6448dfa323204bd8d14cc5 P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
W20240430 20:11:41.926563 19004 catalog_manager.cc:4797] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): delete failed for tablet 68ca48351cd4412381f5195d81109016 because the tablet was not found. No further retry: Not found: Tablet not found: 68ca48351cd4412381f5195d81109016
I20240430 20:11:41.926928 19495 raft_consensus.cc:2229] T 2f61e3f88a6448dfa323204bd8d14cc5 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Raft consensus shutting down.
I20240430 20:11:41.927202 19495 pending_rounds.cc:62] T 2f61e3f88a6448dfa323204bd8d14cc5 P 2efb542aa55e439b99ccb98d1c5c884a: Trying to abort 1 pending ops.
I20240430 20:11:41.927321 19495 pending_rounds.cc:68] T 2f61e3f88a6448dfa323204bd8d14cc5 P 2efb542aa55e439b99ccb98d1c5c884a: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7022624364173729792 op_type: NO_OP noop_request { }
I20240430 20:11:41.928436 19048 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 2e79df8744e74b5991a79ce978ff0202 (table default.Table_276bbdc7cfe542d1a2936485a2aad1ec [id=f146a15e6e8a4acbb4c734c1c863da57]). Expected version 1 got 0
I20240430 20:11:41.929682 19495 raft_consensus.cc:2866] T 2f61e3f88a6448dfa323204bd8d14cc5 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240430 20:11:41.929862 19495 raft_consensus.cc:2258] T 2f61e3f88a6448dfa323204bd8d14cc5 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:41.926100 18985 ts_tablet_manager.cc:1367] T 34130f2820064fa0898ef1f9d7de1c6d P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.302s user 0.004s sys 0.000s
I20240430 20:11:41.930186 19048 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 76d6e9d547ca4a848eae852944b1d117 (table default.Table_104ae015924a48fd8d27220792f07470 [id=ff86c0357c164dd0991098c416087773]). Expected version 2 got 0
I20240430 20:11:41.931275 18985 raft_consensus.cc:348] T 34130f2820064fa0898ef1f9d7de1c6d P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:41.931672 19048 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet acd23f0a63d5448aaa7ed78422e67bad (table default.Table_972f13be4dca4cd19b9a04534dc59f51 [id=69f37db4de634a3185613e40013e0ed7]). Expected version 1 got 0
I20240430 20:11:41.932164 18985 raft_consensus.cc:374] T 34130f2820064fa0898ef1f9d7de1c6d P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:41.932483 18985 raft_consensus.cc:729] T 34130f2820064fa0898ef1f9d7de1c6d P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:41.931972 19495 ts_tablet_manager.cc:1875] T 2f61e3f88a6448dfa323204bd8d14cc5 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:41.933688 18985 consensus_queue.cc:260] T 34130f2820064fa0898ef1f9d7de1c6d P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:41.935886 18985 ts_tablet_manager.cc:1398] T 34130f2820064fa0898ef1f9d7de1c6d P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240430 20:11:41.937711 19048 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 69f88532952a4591bf1577541c3430e6 (table default.Table_1177828cad1f4a24bd397619dad96170 [id=a95fed1bc3fa4865bf9a35dfc51b659c]). Expected version 2 got 0
I20240430 20:11:41.938841 18985 tablet_bootstrap.cc:492] T ebf74809b52f47f3b0ed21b6ac0f541a P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
W20240430 20:11:41.939095 18699 leader_election.cc:341] T 2a1e50f99c5d4b44bdbb51b58a92bc75 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Not found: Tablet not found: 2a1e50f99c5d4b44bdbb51b58a92bc75
I20240430 20:11:41.939232 18699 leader_election.cc:302] T 2a1e50f99c5d4b44bdbb51b58a92bc75 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710
I20240430 20:11:41.939733 19048 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 32c823d1700949ab8b22adcd83b7bfd1 (table default.Table_e8c73c89623d4b49b8d7f0a01d729bdd [id=5cd10682ce52414cb5a103e4433a6a0d]). Expected version 1 got 0
I20240430 20:11:41.939767 20308 raft_consensus.cc:2726] T 2a1e50f99c5d4b44bdbb51b58a92bc75 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:41.942981 19048 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet c6e3d09402574f1e8fe9e4160c70a3ee (table default.Table_656db97355f64d7c8aa29b1352dfe9c8 [id=450f5d1c7e084a1bb88d035c0ccfcf82]). Expected version 1 got 0
I20240430 20:11:41.898375 19489 raft_consensus.cc:2229] T 255c13ad8fae4ea4aa8e7b7aabb7602e P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:41.945186 19495 ts_tablet_manager.cc:1888] T 2f61e3f88a6448dfa323204bd8d14cc5 P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:41.945636 18985 tablet_bootstrap.cc:654] T ebf74809b52f47f3b0ed21b6ac0f541a P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:41.945822 20105 consensus_queue.cc:1035] T 32c823d1700949ab8b22adcd83b7bfd1 P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:41.946702 19495 log.cc:1196] T 2f61e3f88a6448dfa323204bd8d14cc5 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/2f61e3f88a6448dfa323204bd8d14cc5
I20240430 20:11:41.947131 19495 ts_tablet_manager.cc:1909] T 2f61e3f88a6448dfa323204bd8d14cc5 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:41.947729 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 109) took 9842 ms. Trace:
I20240430 20:11:41.948189 19495 rpcz_store.cc:276] 0430 20:11:32.104853 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:32.104924 (+ 71us) service_pool.cc:225] Handling call
0430 20:11:41.926524 (+9821600us) ts_tablet_manager.cc:1120] Deleting tablet 2f61e3f88a6448dfa323204bd8d14cc5
0430 20:11:41.926533 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:41.942239 (+ 15706us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:41.945096 (+ 2857us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:41.947711 (+ 2615us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":9821357}
I20240430 20:11:41.949019 19004 catalog_manager.cc:4830] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): tablet 2f61e3f88a6448dfa323204bd8d14cc5 (table default.Table_db56d63aef494592b69246a0c13006f2 [id=cde04507ee1d464f9fffc44176da8d16]) successfully deleted
I20240430 20:11:41.949111 19495 tablet_replica.cc:330] T fae9a35a2ac844b4bc2ebdf214aa7171 P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:41.949685 19495 raft_consensus.cc:2229] T fae9a35a2ac844b4bc2ebdf214aa7171 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:41.950162 19495 raft_consensus.cc:2258] T fae9a35a2ac844b4bc2ebdf214aa7171 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:41.951000 19495 ts_tablet_manager.cc:1875] T fae9a35a2ac844b4bc2ebdf214aa7171 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:41.952425 19489 raft_consensus.cc:2258] T 255c13ad8fae4ea4aa8e7b7aabb7602e P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:41.953825 19489 ts_tablet_manager.cc:1875] T 255c13ad8fae4ea4aa8e7b7aabb7602e P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:41.966703 18985 tablet_bootstrap.cc:492] T ebf74809b52f47f3b0ed21b6ac0f541a P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:41.966950 18985 ts_tablet_manager.cc:1367] T ebf74809b52f47f3b0ed21b6ac0f541a P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.028s user 0.003s sys 0.006s
I20240430 20:11:41.967929 18985 raft_consensus.cc:348] T ebf74809b52f47f3b0ed21b6ac0f541a P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:41.968293 18985 raft_consensus.cc:374] T ebf74809b52f47f3b0ed21b6ac0f541a P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:41.968410 18985 raft_consensus.cc:729] T ebf74809b52f47f3b0ed21b6ac0f541a P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:41.968762 18985 consensus_queue.cc:260] T ebf74809b52f47f3b0ed21b6ac0f541a P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:41.969352 18985 ts_tablet_manager.cc:1398] T ebf74809b52f47f3b0ed21b6ac0f541a P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240430 20:11:41.969657 18985 tablet_bootstrap.cc:492] T 46251c08277e498da04a41d009310c62 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:41.971112 18985 tablet_bootstrap.cc:654] T 46251c08277e498da04a41d009310c62 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:41.972340 19495 ts_tablet_manager.cc:1888] T fae9a35a2ac844b4bc2ebdf214aa7171 P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240430 20:11:41.973321 19495 log.cc:1196] T fae9a35a2ac844b4bc2ebdf214aa7171 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/fae9a35a2ac844b4bc2ebdf214aa7171
I20240430 20:11:41.973776 19495 ts_tablet_manager.cc:1909] T fae9a35a2ac844b4bc2ebdf214aa7171 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:41.974148 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 116) took 9701 ms. Trace:
I20240430 20:11:41.974288 19495 rpcz_store.cc:276] 0430 20:11:32.272953 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:32.273417 (+ 464us) service_pool.cc:225] Handling call
0430 20:11:41.949063 (+9675646us) ts_tablet_manager.cc:1120] Deleting tablet fae9a35a2ac844b4bc2ebdf214aa7171
0430 20:11:41.949074 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:41.960520 (+ 11446us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:41.972211 (+ 11691us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:41.974128 (+ 1917us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":9675364}
I20240430 20:11:41.974864 19495 tablet_replica.cc:330] T 44fa6917a85c46f3aaa363df46ea586b P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:41.974823 19004 catalog_manager.cc:4830] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): tablet fae9a35a2ac844b4bc2ebdf214aa7171 (table default.Table_730a215d7af74a5bb753adca39dea676 [id=a5d580cffa2447d78fc9f2c8e31264f1]) successfully deleted
I20240430 20:11:41.975114 19495 raft_consensus.cc:2229] T 44fa6917a85c46f3aaa363df46ea586b P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:41.975323 19495 raft_consensus.cc:2258] T 44fa6917a85c46f3aaa363df46ea586b P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:41.975904 19495 ts_tablet_manager.cc:1875] T 44fa6917a85c46f3aaa363df46ea586b P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:41.984504 18985 tablet_bootstrap.cc:492] T 46251c08277e498da04a41d009310c62 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:41.984728 18985 ts_tablet_manager.cc:1367] T 46251c08277e498da04a41d009310c62 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.001s
I20240430 20:11:41.985584 18985 raft_consensus.cc:348] T 46251c08277e498da04a41d009310c62 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:41.985893 18985 raft_consensus.cc:374] T 46251c08277e498da04a41d009310c62 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:41.986002 18985 raft_consensus.cc:729] T 46251c08277e498da04a41d009310c62 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:41.986290 18985 consensus_queue.cc:260] T 46251c08277e498da04a41d009310c62 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:41.986969 18985 ts_tablet_manager.cc:1398] T 46251c08277e498da04a41d009310c62 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240430 20:11:41.987274 18985 tablet_bootstrap.cc:492] T 84568f90b9c54cf292099808d4cbf5dc P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:41.988669 18985 tablet_bootstrap.cc:654] T 84568f90b9c54cf292099808d4cbf5dc P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:41.995630 19471 ts_tablet_manager.cc:1888] T 5d09baed576e4136bf6d756ca39d08b2 P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:41.995890 19471 log.cc:1196] T 5d09baed576e4136bf6d756ca39d08b2 P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/5d09baed576e4136bf6d756ca39d08b2
I20240430 20:11:41.996425 19471 ts_tablet_manager.cc:1909] T 5d09baed576e4136bf6d756ca39d08b2 P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:41.996799 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 98) took 9934 ms. Trace:
I20240430 20:11:41.996938 19471 rpcz_store.cc:276] 0430 20:11:32.061963 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:32.062011 (+ 48us) service_pool.cc:225] Handling call
0430 20:11:41.766344 (+9704333us) ts_tablet_manager.cc:1120] Deleting tablet 5d09baed576e4136bf6d756ca39d08b2
0430 20:11:41.766358 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:41.950658 (+184300us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:41.995538 (+ 44880us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:41.996775 (+ 1237us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":9704066}
I20240430 20:11:41.997490 18999 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet 5d09baed576e4136bf6d756ca39d08b2 (table default.Table_772ce91c27ce4119b6d1b1993e417f21 [id=29b41dafb4334b14a09602eabac2f031]) successfully deleted
I20240430 20:11:41.997948 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 107) took 9850 ms. Trace:
I20240430 20:11:41.998106 19495 ts_tablet_manager.cc:1888] T 44fa6917a85c46f3aaa363df46ea586b P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:41.998118 19471 rpcz_store.cc:276] 0430 20:11:32.147200 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:32.147252 (+ 52us) service_pool.cc:225] Handling call
0430 20:11:41.997837 (+9850585us) ts_tablet_manager.cc:1120] Deleting tablet 93d158de64994f3c9fa45a1250a4f533
0430 20:11:41.997850 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:41.997930 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":9849491}
I20240430 20:11:41.998265 19495 log.cc:1196] T 44fa6917a85c46f3aaa363df46ea586b P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/44fa6917a85c46f3aaa363df46ea586b
I20240430 20:11:41.998728 19495 ts_tablet_manager.cc:1909] T 44fa6917a85c46f3aaa363df46ea586b P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
W20240430 20:11:41.998752 18999 catalog_manager.cc:4797] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): delete failed for tablet 93d158de64994f3c9fa45a1250a4f533 because the tablet was not found. No further retry: Not found: Tablet not found: 93d158de64994f3c9fa45a1250a4f533
I20240430 20:11:41.999106 19471 tablet_replica.cc:330] T 2f61e3f88a6448dfa323204bd8d14cc5 P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:41.999107 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 122) took 9645 ms. Trace:
I20240430 20:11:41.999230 19495 rpcz_store.cc:276] 0430 20:11:32.353784 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:32.353876 (+ 92us) service_pool.cc:225] Handling call
0430 20:11:41.974812 (+9620936us) ts_tablet_manager.cc:1120] Deleting tablet 44fa6917a85c46f3aaa363df46ea586b
0430 20:11:41.974821 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:41.980706 (+ 5885us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:41.997999 (+ 17293us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:41.999086 (+ 1087us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":9620615}
I20240430 20:11:41.999763 19004 catalog_manager.cc:4830] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): tablet 44fa6917a85c46f3aaa363df46ea586b (table default.Table_5924e4520c6c49988ccef21ca04d25d4 [id=68c1eb7252974258aabd9873e6654b10]) successfully deleted
I20240430 20:11:41.999819 19495 tablet_replica.cc:330] T b72fe1f0713b4076a81a3fc1c74f7fa7 P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:42.000125 19495 raft_consensus.cc:2229] T b72fe1f0713b4076a81a3fc1c74f7fa7 P 2efb542aa55e439b99ccb98d1c5c884a [term 2 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:42.000347 19495 raft_consensus.cc:2258] T b72fe1f0713b4076a81a3fc1c74f7fa7 P 2efb542aa55e439b99ccb98d1c5c884a [term 2 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:42.000939 19495 ts_tablet_manager.cc:1875] T b72fe1f0713b4076a81a3fc1c74f7fa7 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:41.944061 19048 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 3ec3993042f5498b818bdf32d5087928 (table default.Table_ba82d0fed0264b818de8278e1d142e5d [id=27217d463b9c40ec926a1862a31f1f24]). Expected version 1 got 0
I20240430 20:11:42.007671 19048 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 1462ecf1176c4e66be42abddbe36b00c (table default.Table_b49a7942aa8c4f58acb94c35d1cdbb70 [id=6257559b0e9545c58848a76af0c5a5f7]). Expected version 1 got 0
I20240430 20:11:42.008025 19048 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 78ba48dadef548708153878810cc46f8 (table default.Table_b9983e101a6547649eb44ee04b5e1147 [id=29769be2b9ec4ecba9de5c0a65692f85]). Expected version 1 got 0
I20240430 20:11:42.008451 19048 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet ebf74809b52f47f3b0ed21b6ac0f541a (table default.Table_b2d07520b9944a89ad62fca70396174e [id=1c8e0a3a44ec4a798770c0165ed3dffa]). Expected version 1 got 0
I20240430 20:11:42.008708 19048 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 0368d37dd9e2452982fb81f539c6805b (table default.Table_404b186c544e4b30bb36cb77d1a14120 [id=4330626e342a48699ea939add5c91b11]). Expected version 1 got 0
I20240430 20:11:42.008886 19048 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 331c031d44854d679a27d10d9151d0ea (table default.Table_763f0800e8bb4fa69976886416250f42 [id=61c8a5b99ef34a0baeeb343fef7cfd05]). Expected version 1 got 0
I20240430 20:11:42.009325 19048 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 47bf64ecc41b445da4ff697e141243dd (table default.Table_53b6a25ee2c242dcb83b4759648af9f3 [id=838fc8b61a864dea8877a6f144e4cb4f]). Expected version 1 got 0
I20240430 20:11:42.009649 19048 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 2da900b6bbfa4fa99fe8494fd2e68da8 (table default.Table_19a91a7f17ff456ebb145e8dd89c0940 [id=67283dd3995a44c6a7a7bc8722e4f955]). Expected version 1 got 0
I20240430 20:11:42.009959 19048 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 8e91920e7e36418d8c27d1f0bfd9b981 (table default.Table_e860191f401c434e8cf0085ef41952b9 [id=f393bcc9669c4ea985a2d40deb161f0c]). Expected version 1 got 0
I20240430 20:11:42.010538 19048 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 99203cc522de424184c6f97453f166fd (table default.Table_cb2a4b9ceaf4431facfff2c9bcf94c5d [id=3f8690a437ae49a3beb009fa7899d5d0]). Expected version 1 got 0
I20240430 20:11:42.010982 18985 tablet_bootstrap.cc:492] T 84568f90b9c54cf292099808d4cbf5dc P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:42.011183 18985 ts_tablet_manager.cc:1367] T 84568f90b9c54cf292099808d4cbf5dc P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.003s
I20240430 20:11:42.011283 19048 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet a482f4622f774d60b4efd6827b8be7d7 (table default.Table_e686cdfcbcc046749fa5b51578a64794 [id=c3dc9f05316d46d6a3ef4d03f40877ec]). Expected version 1 got 0
I20240430 20:11:42.011431 19471 raft_consensus.cc:2229] T 2f61e3f88a6448dfa323204bd8d14cc5 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:42.011678 19471 raft_consensus.cc:2258] T 2f61e3f88a6448dfa323204bd8d14cc5 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:42.012069 18985 raft_consensus.cc:348] T 84568f90b9c54cf292099808d4cbf5dc P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:42.012272 18985 raft_consensus.cc:374] T 84568f90b9c54cf292099808d4cbf5dc P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:42.012333 18985 raft_consensus.cc:729] T 84568f90b9c54cf292099808d4cbf5dc P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:42.012665 19471 ts_tablet_manager.cc:1875] T 2f61e3f88a6448dfa323204bd8d14cc5 P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:42.012570 18985 consensus_queue.cc:260] T 84568f90b9c54cf292099808d4cbf5dc P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:42.013203 18983 tablet_bootstrap.cc:492] T 1462ecf1176c4e66be42abddbe36b00c P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:42.013396 18983 ts_tablet_manager.cc:1367] T 1462ecf1176c4e66be42abddbe36b00c P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.139s user 0.007s sys 0.007s
I20240430 20:11:42.013391 18985 ts_tablet_manager.cc:1398] T 84568f90b9c54cf292099808d4cbf5dc P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240430 20:11:42.014057 18985 tablet_bootstrap.cc:492] T 43b5e2c005a34df08cc4335c6e014cb5 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:42.017280 18983 raft_consensus.cc:348] T 1462ecf1176c4e66be42abddbe36b00c P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:42.017511 18983 raft_consensus.cc:374] T 1462ecf1176c4e66be42abddbe36b00c P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:42.017588 18983 raft_consensus.cc:729] T 1462ecf1176c4e66be42abddbe36b00c P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:42.017781 18983 consensus_queue.cc:260] T 1462ecf1176c4e66be42abddbe36b00c P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:42.018293 18983 ts_tablet_manager.cc:1398] T 1462ecf1176c4e66be42abddbe36b00c P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240430 20:11:42.015681 18985 tablet_bootstrap.cc:654] T 43b5e2c005a34df08cc4335c6e014cb5 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:42.019297 18983 tablet_bootstrap.cc:492] T c2b019f529a54228b804437bb025e31d P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:42.021188 19495 ts_tablet_manager.cc:1888] T b72fe1f0713b4076a81a3fc1c74f7fa7 P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:42.021328 19495 log.cc:1196] T b72fe1f0713b4076a81a3fc1c74f7fa7 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/b72fe1f0713b4076a81a3fc1c74f7fa7
I20240430 20:11:42.021692 19495 ts_tablet_manager.cc:1909] T b72fe1f0713b4076a81a3fc1c74f7fa7 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:42.022471 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 124) took 9659 ms. Trace:
I20240430 20:11:42.022636 19495 rpcz_store.cc:276] 0430 20:11:32.362941 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:32.363013 (+ 72us) service_pool.cc:225] Handling call
0430 20:11:41.999775 (+9636762us) ts_tablet_manager.cc:1120] Deleting tablet b72fe1f0713b4076a81a3fc1c74f7fa7
0430 20:11:41.999785 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:42.004551 (+ 4766us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:42.021079 (+ 16528us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:42.022453 (+ 1374us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":9636488}
I20240430 20:11:42.023240 19495 tablet_replica.cc:330] T 75ebf051383e42529dcb5f7dd0770a4b P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:42.023687 19495 raft_consensus.cc:2229] T 75ebf051383e42529dcb5f7dd0770a4b P 2efb542aa55e439b99ccb98d1c5c884a [term 2 LEADER]: Raft consensus shutting down.
I20240430 20:11:42.024267 19495 raft_consensus.cc:2258] T 75ebf051383e42529dcb5f7dd0770a4b P 2efb542aa55e439b99ccb98d1c5c884a [term 2 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:42.025177 18983 tablet_bootstrap.cc:654] T c2b019f529a54228b804437bb025e31d P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:42.025193 19495 ts_tablet_manager.cc:1875] T 75ebf051383e42529dcb5f7dd0770a4b P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:42.029664 18983 tablet_bootstrap.cc:492] T c2b019f529a54228b804437bb025e31d P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:42.029904 18983 ts_tablet_manager.cc:1367] T c2b019f529a54228b804437bb025e31d P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.002s
I20240430 20:11:42.030737 18983 raft_consensus.cc:348] T c2b019f529a54228b804437bb025e31d P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:42.031000 18983 raft_consensus.cc:374] T c2b019f529a54228b804437bb025e31d P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:42.031083 18983 raft_consensus.cc:729] T c2b019f529a54228b804437bb025e31d P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:42.031333 18983 consensus_queue.cc:260] T c2b019f529a54228b804437bb025e31d P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:42.031940 18983 ts_tablet_manager.cc:1398] T c2b019f529a54228b804437bb025e31d P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240430 20:11:42.032243 18983 tablet_bootstrap.cc:492] T 46251c08277e498da04a41d009310c62 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:42.033636 18983 tablet_bootstrap.cc:654] T 46251c08277e498da04a41d009310c62 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:42.044041 19495 ts_tablet_manager.cc:1888] T 75ebf051383e42529dcb5f7dd0770a4b P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.1
I20240430 20:11:42.052670 18668 heartbeater.cc:347] Connected to a master server at 127.17.177.62:36441
I20240430 20:11:42.049535 20236 raft_consensus.cc:482] T 7bde1d6934784c9aa5911181e8df9eeb P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3c51c33d5616482cafb68e26688820f4)
I20240430 20:11:42.053527 20236 raft_consensus.cc:504] T 7bde1d6934784c9aa5911181e8df9eeb P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:42.054204 20236 leader_election.cc:288] T 7bde1d6934784c9aa5911181e8df9eeb P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:42.054338 20236 raft_consensus.cc:482] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5e1c6397f09747f091534faf93542710)
I20240430 20:11:42.054450 20236 raft_consensus.cc:504] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:42.055501 18608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bde1d6934784c9aa5911181e8df9eeb" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:42.055743 18608 raft_consensus.cc:2379] T 7bde1d6934784c9aa5911181e8df9eeb P 5e1c6397f09747f091534faf93542710 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2efb542aa55e439b99ccb98d1c5c884a in current term 2: Already voted for candidate 5e1c6397f09747f091534faf93542710 in this term.
I20240430 20:11:42.055918 18810 heartbeater.cc:503] Master 127.17.177.60:37225 was elected leader, sending a full tablet report...
I20240430 20:11:42.056941 20236 leader_election.cc:288] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:42.053493 19495 log.cc:1196] T 75ebf051383e42529dcb5f7dd0770a4b P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/75ebf051383e42529dcb5f7dd0770a4b
I20240430 20:11:42.057816 18608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd6083f12c7e4d1e8a2006b3ddf9aa3a" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:42.057906 19495 ts_tablet_manager.cc:1909] T 75ebf051383e42529dcb5f7dd0770a4b P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:42.058001 18608 raft_consensus.cc:2454] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 5e1c6397f09747f091534faf93542710 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 2.
I20240430 20:11:42.058508 18415 leader_election.cc:302] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:42.058643 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 125) took 9688 ms. Trace:
I20240430 20:11:42.058730 19495 rpcz_store.cc:276] 0430 20:11:32.370715 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:32.370791 (+ 76us) service_pool.cc:225] Handling call
0430 20:11:42.023183 (+9652392us) ts_tablet_manager.cc:1120] Deleting tablet 75ebf051383e42529dcb5f7dd0770a4b
0430 20:11:42.023196 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:42.040820 (+ 17624us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:42.043922 (+ 3102us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:42.058623 (+ 14701us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":9652083}
I20240430 20:11:42.059216 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 127) took 9687 ms. Trace:
I20240430 20:11:42.059353 19495 rpcz_store.cc:276] 0430 20:11:32.371422 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:32.371551 (+ 129us) service_pool.cc:225] Handling call
0430 20:11:42.059106 (+9687555us) ts_tablet_manager.cc:1120] Deleting tablet 115c4a12440647a4bb46eb6824f7ea98
0430 20:11:42.059115 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:42.059202 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":9687267}
I20240430 20:11:42.059696 20235 raft_consensus.cc:2781] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240430 20:11:42.059969 20235 raft_consensus.cc:482] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 5e1c6397f09747f091534faf93542710)
I20240430 20:11:42.060060 19471 ts_tablet_manager.cc:1888] T 2f61e3f88a6448dfa323204bd8d14cc5 P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:42.060159 19471 log.cc:1196] T 2f61e3f88a6448dfa323204bd8d14cc5 P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/2f61e3f88a6448dfa323204bd8d14cc5
I20240430 20:11:42.060271 20235 raft_consensus.cc:3037] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Advancing to term 2
I20240430 20:11:42.060462 19471 ts_tablet_manager.cc:1909] T 2f61e3f88a6448dfa323204bd8d14cc5 P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:42.060694 18983 tablet_bootstrap.cc:492] T 46251c08277e498da04a41d009310c62 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:42.060885 18983 ts_tablet_manager.cc:1367] T 46251c08277e498da04a41d009310c62 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.029s user 0.002s sys 0.008s
I20240430 20:11:42.061211 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 101) took 9914 ms. Trace:
I20240430 20:11:42.061300 19471 rpcz_store.cc:276] 0430 20:11:32.146574 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:32.147304 (+ 730us) service_pool.cc:225] Handling call
0430 20:11:41.999058 (+9851754us) ts_tablet_manager.cc:1120] Deleting tablet 2f61e3f88a6448dfa323204bd8d14cc5
0430 20:11:41.999067 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:42.048265 (+ 49198us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:42.059952 (+ 11687us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:42.061195 (+ 1243us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":9850178}
I20240430 20:11:42.061903 18983 raft_consensus.cc:348] T 46251c08277e498da04a41d009310c62 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:42.062124 18983 raft_consensus.cc:374] T 46251c08277e498da04a41d009310c62 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:42.062213 18983 raft_consensus.cc:729] T 46251c08277e498da04a41d009310c62 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:42.062476 18983 consensus_queue.cc:260] T 46251c08277e498da04a41d009310c62 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:42.063066 18983 ts_tablet_manager.cc:1398] T 46251c08277e498da04a41d009310c62 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240430 20:11:42.063369 18983 tablet_bootstrap.cc:492] T 87ac32a0ddb84676adc83038ef664591 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:42.064553 20235 raft_consensus.cc:504] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 2efb542aa55e439b99ccb98d1c5c884a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:42.064954 18983 tablet_bootstrap.cc:654] T 87ac32a0ddb84676adc83038ef664591 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:42.065543 18608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd6083f12c7e4d1e8a2006b3ddf9aa3a" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:42.066780 18985 tablet_bootstrap.cc:492] T 43b5e2c005a34df08cc4335c6e014cb5 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:42.066897 18985 ts_tablet_manager.cc:1367] T 43b5e2c005a34df08cc4335c6e014cb5 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.053s user 0.003s sys 0.008s
I20240430 20:11:42.067593 18985 raft_consensus.cc:348] T 43b5e2c005a34df08cc4335c6e014cb5 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:42.067816 18985 raft_consensus.cc:374] T 43b5e2c005a34df08cc4335c6e014cb5 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:42.067870 18985 raft_consensus.cc:729] T 43b5e2c005a34df08cc4335c6e014cb5 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:42.068076 18985 consensus_queue.cc:260] T 43b5e2c005a34df08cc4335c6e014cb5 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:42.068573 18985 ts_tablet_manager.cc:1398] T 43b5e2c005a34df08cc4335c6e014cb5 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240430 20:11:42.068552 20236 consensus_queue.cc:1035] T 32c823d1700949ab8b22adcd83b7bfd1 P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:42.068943 18985 tablet_bootstrap.cc:492] T 1462ecf1176c4e66be42abddbe36b00c P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:42.069864 18608 raft_consensus.cc:2454] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 5e1c6397f09747f091534faf93542710 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 2.
I20240430 20:11:42.070523 18985 tablet_bootstrap.cc:654] T 1462ecf1176c4e66be42abddbe36b00c P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:42.070664 18415 leader_election.cc:302] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:42.070811 20235 leader_election.cc:288] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 election: Requested vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:42.070958 20235 raft_consensus.cc:2781] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 2efb542aa55e439b99ccb98d1c5c884a [term 2 FOLLOWER]: Leader election won for term 2
I20240430 20:11:42.071501 18763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bde1d6934784c9aa5911181e8df9eeb" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:42.073055 18985 tablet_bootstrap.cc:492] T 1462ecf1176c4e66be42abddbe36b00c P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:42.073263 18985 ts_tablet_manager.cc:1367] T 1462ecf1176c4e66be42abddbe36b00c P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240430 20:11:42.074113 20235 raft_consensus.cc:686] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 2efb542aa55e439b99ccb98d1c5c884a [term 2 LEADER]: Becoming Leader. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Running, Role: LEADER
I20240430 20:11:42.074169 18985 raft_consensus.cc:348] T 1462ecf1176c4e66be42abddbe36b00c P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:42.074707 18756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd6083f12c7e4d1e8a2006b3ddf9aa3a" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:42.075628 18985 raft_consensus.cc:374] T 1462ecf1176c4e66be42abddbe36b00c P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:42.075718 18985 raft_consensus.cc:729] T 1462ecf1176c4e66be42abddbe36b00c P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:42.023176 19004 catalog_manager.cc:4830] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): tablet b72fe1f0713b4076a81a3fc1c74f7fa7 (table default.Table_16b082d140234e1c9a1e70654839b4cd [id=c0d8dc87d1f9426a85f6176f161595a1]) successfully deleted
I20240430 20:11:42.061743 18999 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet 2f61e3f88a6448dfa323204bd8d14cc5 (table default.Table_db56d63aef494592b69246a0c13006f2 [id=cde04507ee1d464f9fffc44176da8d16]) successfully deleted
W20240430 20:11:42.076388 19004 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.17.177.1:41217, user_credentials={real_user=slave}} blocked reactor thread for 53205.7us
I20240430 20:11:42.076658 19004 catalog_manager.cc:4830] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): tablet 75ebf051383e42529dcb5f7dd0770a4b (table default.Table_e9655558850747968a52e5885a113f59 [id=9c272f6ad56440b682f12f027af57bc4]) successfully deleted
I20240430 20:11:42.076750 18985 consensus_queue.cc:260] T 1462ecf1176c4e66be42abddbe36b00c P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
W20240430 20:11:42.076968 19004 catalog_manager.cc:4810] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): delete failed for tablet 115c4a12440647a4bb46eb6824f7ea98 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 115c4a12440647a4bb46eb6824f7ea98 already in progress: creating tablet
I20240430 20:11:42.077379 18985 ts_tablet_manager.cc:1398] T 1462ecf1176c4e66be42abddbe36b00c P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240430 20:11:42.074502 20235 consensus_queue.cc:237] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:42.079247 18985 tablet_bootstrap.cc:492] T a59ef6c545dd41a28265a4c1b8ce10e0 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:42.079754 19048 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet fdf53061efd74fa29a9d8208d1fd210e (table default.Table_3c8f9f7b22f64fb1bdc3f1550987233a [id=7fd6f8576f4c4cbc8ffcdb4ab95e15dc]). Expected version 1 got 0
I20240430 20:11:42.081133 18985 tablet_bootstrap.cc:654] T a59ef6c545dd41a28265a4c1b8ce10e0 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:42.082216 19048 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 12795670f21447a8964fe1efc6e92398 (table default.Table_896bbae1f6944005b0272ca8ebc97218 [id=d5cea90cbc2c48df9d8b80da1eccc949]). Expected version 3 got 0
I20240430 20:11:42.082546 19048 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 46251c08277e498da04a41d009310c62 (table default.Table_4e56923d440944978f0aacc140a0dd8e [id=b1312a6cd1a54003b1db727a2a9d125d]). Expected version 1 got 0
I20240430 20:11:42.083220 18983 tablet_bootstrap.cc:492] T 87ac32a0ddb84676adc83038ef664591 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:42.083278 18763 raft_consensus.cc:2454] T 7bde1d6934784c9aa5911181e8df9eeb P 3c51c33d5616482cafb68e26688820f4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 2.
I20240430 20:11:42.083407 18983 ts_tablet_manager.cc:1367] T 87ac32a0ddb84676adc83038ef664591 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.004s
I20240430 20:11:42.084389 18756 raft_consensus.cc:2379] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 3c51c33d5616482cafb68e26688820f4 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2efb542aa55e439b99ccb98d1c5c884a in current term 2: Already voted for candidate 3c51c33d5616482cafb68e26688820f4 in this term.
I20240430 20:11:42.084262 18983 raft_consensus.cc:348] T 87ac32a0ddb84676adc83038ef664591 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:42.084453 18983 raft_consensus.cc:374] T 87ac32a0ddb84676adc83038ef664591 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:42.084499 18983 raft_consensus.cc:729] T 87ac32a0ddb84676adc83038ef664591 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:42.084714 18762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd6083f12c7e4d1e8a2006b3ddf9aa3a" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:42.084718 18983 consensus_queue.cc:260] T 87ac32a0ddb84676adc83038ef664591 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:42.085316 18762 raft_consensus.cc:2379] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 3c51c33d5616482cafb68e26688820f4 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 2efb542aa55e439b99ccb98d1c5c884a in current term 2: Already voted for candidate 3c51c33d5616482cafb68e26688820f4 in this term.
I20240430 20:11:42.085397 18983 ts_tablet_manager.cc:1398] T 87ac32a0ddb84676adc83038ef664591 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240430 20:11:42.083747 18417 leader_election.cc:302] T 7bde1d6934784c9aa5911181e8df9eeb P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters: 5e1c6397f09747f091534faf93542710
W20240430 20:11:42.085536 18699 consensus_peers.cc:485] T 44fa6917a85c46f3aaa363df46ea586b P 3c51c33d5616482cafb68e26688820f4 -> Peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Couldn't send request to peer 2efb542aa55e439b99ccb98d1c5c884a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 44fa6917a85c46f3aaa363df46ea586b. This is attempt 1: this message will repeat every 5th retry.
I20240430 20:11:42.085711 20236 raft_consensus.cc:2781] T 7bde1d6934784c9aa5911181e8df9eeb P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240430 20:11:42.085812 20236 raft_consensus.cc:482] T 7bde1d6934784c9aa5911181e8df9eeb P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3c51c33d5616482cafb68e26688820f4)
I20240430 20:11:42.085898 20236 raft_consensus.cc:3037] T 7bde1d6934784c9aa5911181e8df9eeb P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Advancing to term 2
I20240430 20:11:42.059836 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 128) took 9687 ms. Trace:
I20240430 20:11:42.086231 19495 rpcz_store.cc:276] 0430 20:11:32.372075 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:32.372738 (+ 663us) service_pool.cc:225] Handling call
0430 20:11:42.059732 (+9686994us) ts_tablet_manager.cc:1120] Deleting tablet 2cdccf4be7094ca5b863dd5c3432d55d
0430 20:11:42.059741 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:42.059820 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":9686760}
I20240430 20:11:42.087924 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 130) took 9715 ms. Trace:
I20240430 20:11:42.088033 19495 rpcz_store.cc:276] 0430 20:11:32.372697 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:32.373125 (+ 428us) service_pool.cc:225] Handling call
0430 20:11:42.087811 (+9714686us) ts_tablet_manager.cc:1120] Deleting tablet a1aeee1b02214975a376cfd36957afe8
0430 20:11:42.087826 (+ 15us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:42.087906 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":9714470}
I20240430 20:11:42.089069 19495 tablet_replica.cc:330] T 73453a7fa30342e29db552b17b62f0c1 P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:42.088964 20236 raft_consensus.cc:504] T 7bde1d6934784c9aa5911181e8df9eeb P 2efb542aa55e439b99ccb98d1c5c884a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:42.090354 19495 raft_consensus.cc:2229] T 73453a7fa30342e29db552b17b62f0c1 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:42.090553 19495 raft_consensus.cc:2258] T 73453a7fa30342e29db552b17b62f0c1 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:42.091074 18608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bde1d6934784c9aa5911181e8df9eeb" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:42.091270 19495 ts_tablet_manager.cc:1875] T 73453a7fa30342e29db552b17b62f0c1 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:42.091369 18608 raft_consensus.cc:2379] T 7bde1d6934784c9aa5911181e8df9eeb P 5e1c6397f09747f091534faf93542710 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 2efb542aa55e439b99ccb98d1c5c884a in current term 2: Already voted for candidate 5e1c6397f09747f091534faf93542710 in this term.
I20240430 20:11:42.092077 20236 leader_election.cc:288] T 7bde1d6934784c9aa5911181e8df9eeb P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:42.097435 19495 ts_tablet_manager.cc:1888] T 73453a7fa30342e29db552b17b62f0c1 P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20240430 20:11:42.086719 19004 catalog_manager.cc:4810] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): delete failed for tablet 2cdccf4be7094ca5b863dd5c3432d55d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2cdccf4be7094ca5b863dd5c3432d55d already in progress: creating tablet
I20240430 20:11:42.100729 19495 log.cc:1196] T 73453a7fa30342e29db552b17b62f0c1 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/73453a7fa30342e29db552b17b62f0c1
W20240430 20:11:42.100953 19004 catalog_manager.cc:4797] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): delete failed for tablet a1aeee1b02214975a376cfd36957afe8 because the tablet was not found. No further retry: Not found: Tablet not found: a1aeee1b02214975a376cfd36957afe8
I20240430 20:11:42.101447 19495 ts_tablet_manager.cc:1909] T 73453a7fa30342e29db552b17b62f0c1 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:42.101672 19048 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet b22aaedb0a6c4fd9ad0ea8fa019001e1 (table default.Table_730a215d7af74a5bb753adca39dea676 [id=a5d580cffa2447d78fc9f2c8e31264f1]). Expected version 5 got 0
I20240430 20:11:42.102775 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 123) took 9740 ms. Trace:
I20240430 20:11:42.103655 18762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bde1d6934784c9aa5911181e8df9eeb" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:42.104121 19495 rpcz_store.cc:276] 0430 20:11:32.362823 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:32.387045 (+ 24222us) service_pool.cc:225] Handling call
0430 20:11:42.089006 (+9701961us) ts_tablet_manager.cc:1120] Deleting tablet 73453a7fa30342e29db552b17b62f0c1
0430 20:11:42.089020 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:42.094531 (+ 5511us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:42.097355 (+ 2824us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:42.102746 (+ 5391us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1015,"tablet-delete.queue_time_us":9701346}
I20240430 20:11:42.104700 19489 ts_tablet_manager.cc:1888] T 255c13ad8fae4ea4aa8e7b7aabb7602e P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:42.104876 19489 log.cc:1196] T 255c13ad8fae4ea4aa8e7b7aabb7602e P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/255c13ad8fae4ea4aa8e7b7aabb7602e
I20240430 20:11:42.105201 19495 tablet_replica.cc:330] T 87ac32a0ddb84676adc83038ef664591 P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:42.105568 19495 raft_consensus.cc:2229] T 87ac32a0ddb84676adc83038ef664591 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:42.105608 19489 ts_tablet_manager.cc:1909] T 255c13ad8fae4ea4aa8e7b7aabb7602e P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:42.106002 19495 raft_consensus.cc:2258] T 87ac32a0ddb84676adc83038ef664591 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:42.105995 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 112) took 9972 ms. Trace:
I20240430 20:11:42.106099 19489 rpcz_store.cc:276] 0430 20:11:32.133839 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:32.134239 (+ 400us) service_pool.cc:225] Handling call
0430 20:11:41.566561 (+9432322us) ts_tablet_manager.cc:1120] Deleting tablet 255c13ad8fae4ea4aa8e7b7aabb7602e
0430 20:11:41.566579 (+ 18us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:41.992738 (+426159us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:42.097202 (+104464us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:42.105950 (+ 8748us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":262126,"tablet-delete.queue_time_us":9432017}
I20240430 20:11:42.106827 19495 ts_tablet_manager.cc:1875] T 87ac32a0ddb84676adc83038ef664591 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:42.107139 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 110) took 9973 ms. Trace:
I20240430 20:11:42.107249 19489 rpcz_store.cc:276] 0430 20:11:32.133562 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:32.134572 (+ 1010us) service_pool.cc:225] Handling call
0430 20:11:42.107009 (+9972437us) ts_tablet_manager.cc:1120] Deleting tablet 93d158de64994f3c9fa45a1250a4f533
0430 20:11:42.107018 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:42.107095 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":9972201}
I20240430 20:11:42.107751 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 114) took 9972 ms. Trace:
I20240430 20:11:42.107856 19489 rpcz_store.cc:276] 0430 20:11:32.135391 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:32.135460 (+ 69us) service_pool.cc:225] Handling call
0430 20:11:42.107597 (+9972137us) ts_tablet_manager.cc:1120] Deleting tablet 02176eee8e044a7db8cac48439e9f594
0430 20:11:42.107610 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:42.107710 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":9971920}
I20240430 20:11:42.106815 19001 catalog_manager.cc:4830] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): tablet 255c13ad8fae4ea4aa8e7b7aabb7602e (table default.Table_a5ab8fc47d4e4d128ba4370b3249d8e5 [id=1238beeef04b476b8f4dc5af2fb68012]) successfully deleted
I20240430 20:11:42.107756 18762 raft_consensus.cc:2454] T 7bde1d6934784c9aa5911181e8df9eeb P 3c51c33d5616482cafb68e26688820f4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 2.
W20240430 20:11:42.108477 19001 catalog_manager.cc:4797] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): delete failed for tablet 93d158de64994f3c9fa45a1250a4f533 because the tablet was not found. No further retry: Not found: Tablet not found: 93d158de64994f3c9fa45a1250a4f533
W20240430 20:11:42.108752 19001 catalog_manager.cc:4797] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): delete failed for tablet 02176eee8e044a7db8cac48439e9f594 because the tablet was not found. No further retry: Not found: Tablet not found: 02176eee8e044a7db8cac48439e9f594
I20240430 20:11:42.107380 19004 catalog_manager.cc:4830] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): tablet 73453a7fa30342e29db552b17b62f0c1 (table default.Table_7a9f9a8c27bb4392acddd2a7ea3cc3b2 [id=cfad6f932073488b8097ef7381bc9fd1]) successfully deleted
I20240430 20:11:42.108891 18417 leader_election.cc:302] T 7bde1d6934784c9aa5911181e8df9eeb P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters: 5e1c6397f09747f091534faf93542710
I20240430 20:11:42.109237 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 113) took 9974 ms. Trace:
I20240430 20:11:42.109340 20236 raft_consensus.cc:2781] T 7bde1d6934784c9aa5911181e8df9eeb P 2efb542aa55e439b99ccb98d1c5c884a [term 2 FOLLOWER]: Leader election won for term 2
I20240430 20:11:42.109577 20236 raft_consensus.cc:686] T 7bde1d6934784c9aa5911181e8df9eeb P 2efb542aa55e439b99ccb98d1c5c884a [term 2 LEADER]: Becoming Leader. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Running, Role: LEADER
I20240430 20:11:42.109839 20236 consensus_queue.cc:237] T 7bde1d6934784c9aa5911181e8df9eeb P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:42.109448 18988 tablet_bootstrap.cc:492] T ca64de16d40f4a44b8b411f5526ba11a P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:42.110561 18988 ts_tablet_manager.cc:1367] T ca64de16d40f4a44b8b411f5526ba11a P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.221s user 0.004s sys 0.012s
I20240430 20:11:42.111338 18988 raft_consensus.cc:348] T ca64de16d40f4a44b8b411f5526ba11a P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:42.112432 18988 raft_consensus.cc:374] T ca64de16d40f4a44b8b411f5526ba11a P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:42.113009 18988 raft_consensus.cc:729] T ca64de16d40f4a44b8b411f5526ba11a P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:42.113267 18988 consensus_queue.cc:260] T ca64de16d40f4a44b8b411f5526ba11a P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:42.108505 19048 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 3e5225f1f5f544ecbf8dfa470af3daca (table default.Table_c7c87a8e019944cab3ec99e9563ac01a [id=abf3765074eb4d7d9d2bcc4ab721e6bd]). Expected version 1 got 0
I20240430 20:11:42.114182 19048 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet ed0c37a71ede4bfc9eb5ad1a4365f25e (table default.Table_a2e85efc6b454524b0b122102ec07002 [id=86d795a82e674d859104741ebd04f4e8]). Expected version 1 got 0
I20240430 20:11:42.114359 19048 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet a0baee91f5b44945b595d0e234af018e (table default.Table_6a30a0abb67d4a298f87216515a92153 [id=b3d1c2f97c4648518dbea83a46cadd85]). Expected version 1 got 0
I20240430 20:11:42.114533 19495 ts_tablet_manager.cc:1888] T 87ac32a0ddb84676adc83038ef664591 P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:42.114094 19489 rpcz_store.cc:276] 0430 20:11:32.135226 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:32.135707 (+ 481us) service_pool.cc:225] Handling call
0430 20:11:42.109128 (+9973421us) ts_tablet_manager.cc:1120] Deleting tablet c5fd0730133c472299c93f4b0830e527
0430 20:11:42.109136 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:42.109196 (+ 60us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":9973254}
W20240430 20:11:42.114986 19001 catalog_manager.cc:4797] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): delete failed for tablet c5fd0730133c472299c93f4b0830e527 because the tablet was not found. No further retry: Not found: Tablet not found: c5fd0730133c472299c93f4b0830e527
I20240430 20:11:42.117547 18988 ts_tablet_manager.cc:1398] T ca64de16d40f4a44b8b411f5526ba11a P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240430 20:11:42.118829 19489 tablet_replica.cc:330] T 7542c7fe6cad452ab718f9ef54dbbbbe P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:42.119074 19048 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 62732af95c6a4c749c5b726f0460dc72 (table default.Table_be2ebe77f859489f9cd3e912cb1edc8b [id=2d44ec94131148048021b2ee6366ba30]). Expected version 2 got 0
I20240430 20:11:42.119230 19048 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet eb6c96047b114ebba73db7a371f2da8e (table default.Table_7d5f6f18cbd64c869739872a7dff9be2 [id=e67d619e36c0485f8f22b4a81cb4cb6c]). Expected version 1 got 0
W20240430 20:11:42.119541 19049 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 130007acc8ba4e709c3eb85fabe75903 consensus state has no leader
I20240430 20:11:42.120087 19049 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_7956b5f5fb284c749d82c1980caa8828" } new_table_name: "default.Table_cf60b652e9dc49e7a948ce66fd1cbffe" modify_external_catalogs: true
W20240430 20:11:42.121526 19028 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 47bf64ecc41b445da4ff697e141243dd consensus state has no leader
I20240430 20:11:42.121951 19028 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_4d14d7a9609346baa18fdee2e3a2abfa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:42.122462 19028 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4d14d7a9609346baa18fdee2e3a2abfa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:42.122967 19495 log.cc:1196] T 87ac32a0ddb84676adc83038ef664591 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/87ac32a0ddb84676adc83038ef664591
I20240430 20:11:42.123307 19495 ts_tablet_manager.cc:1909] T 87ac32a0ddb84676adc83038ef664591 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:42.123591 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 131) took 9685 ms. Trace:
I20240430 20:11:42.123652 19495 rpcz_store.cc:276] 0430 20:11:32.438047 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:32.438102 (+ 55us) service_pool.cc:225] Handling call
0430 20:11:42.105148 (+9667046us) ts_tablet_manager.cc:1120] Deleting tablet 87ac32a0ddb84676adc83038ef664591
0430 20:11:42.105160 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:42.111586 (+ 6426us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:42.114450 (+ 2864us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:42.123574 (+ 9124us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":9666803}
W20240430 20:11:42.124864 19033 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3a03b6997921408cb5a65f4cae223e9c consensus state has no leader
I20240430 20:11:42.125221 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 132) took 9666 ms. Trace:
I20240430 20:11:42.125262 19033 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_a2f72cd314144523b378f2c5b324eed9" } modify_external_catalogs: true
I20240430 20:11:42.125298 19495 rpcz_store.cc:276] 0430 20:11:32.459165 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:32.519751 (+ 60586us) service_pool.cc:225] Handling call
0430 20:11:42.125096 (+9605345us) ts_tablet_manager.cc:1120] Deleting tablet f940d17b02b541289121463e05b0bb5f
0430 20:11:42.125106 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:42.125204 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":9605048}
I20240430 20:11:42.125438 19033 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_a2f72cd314144523b378f2c5b324eed9" } modify_external_catalogs: true
I20240430 20:11:42.126058 19026 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: Sending DeleteTablet for 3 replicas of tablet 283e5e73cbc04daba0de19f4541f9fd0
I20240430 20:11:42.126124 19495 tablet_replica.cc:330] T 20d166756819406ba85044f4fee9c4ce P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:42.126058 19004 catalog_manager.cc:4830] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): tablet 87ac32a0ddb84676adc83038ef664591 (table default.Table_763f0800e8bb4fa69976886416250f42 [id=61c8a5b99ef34a0baeeb343fef7cfd05]) successfully deleted
I20240430 20:11:42.126365 19495 raft_consensus.cc:2229] T 20d166756819406ba85044f4fee9c4ce P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus shutting down.
W20240430 20:11:42.126374 19004 catalog_manager.cc:4810] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): delete failed for tablet f940d17b02b541289121463e05b0bb5f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f940d17b02b541289121463e05b0bb5f already in progress: creating tablet
I20240430 20:11:42.126539 19495 raft_consensus.cc:2258] T 20d166756819406ba85044f4fee9c4ce P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:42.126722 19026 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:51664 (request call id 241) took 1438 ms. Trace:
I20240430 20:11:42.127148 19495 ts_tablet_manager.cc:1875] T 20d166756819406ba85044f4fee9c4ce P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:42.126816 19026 rpcz_store.cc:276] 0430 20:11:40.687786 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:40.747665 (+ 59879us) service_pool.cc:225] Handling call
0430 20:11:40.770459 (+ 22794us) spinlock_profiling.cc:243] Waited 22.5 ms on lock 0x616000040654. stack: 00007feb16c74756 00007feb16c74510 00007feb15ca748d 0000000000955804 00000000009557cd 00007feb257b315d 00007feb257b3253 00007feb224e5d41 00007feb2a0d134d 00007feb2a25eca6 00007feb21047f66 00007feb21047c95 00007feb1d97c04e 00007feb1d97af2e 00007feb1d981405 00007feb1d981d2f
0430 20:11:40.770889 (+ 430us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0430 20:11:40.771012 (+ 123us) catalog_manager.cc:2749] Modifying in-memory table state
0430 20:11:40.771088 (+ 76us) catalog_manager.cc:2758] Locking tablets
0430 20:11:41.202573 (+431485us) catalog_manager.cc:2772] Removing table and tablets from system table
0430 20:11:41.307003 (+104430us) catalog_manager.cc:2790] Removing table from by-name map
0430 20:11:41.307096 (+ 93us) catalog_manager.cc:2806] Committing in-memory state
0430 20:11:42.126005 (+818909us) catalog_manager.cc:2810] Aborting table tasks
0430 20:11:42.126660 (+ 655us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:41.263108 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:41.263375 (+ 267us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:41.263383 (+ 8us) write_op.cc:427] Acquired schema lock
0430 20:11:41.263385 (+ 2us) tablet.cc:660] Decoding operations
0430 20:11:41.263653 (+ 268us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:41.263699 (+ 46us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:41.263706 (+ 7us) tablet.cc:683] Acquiring locks for 2 operations
0430 20:11:41.263976 (+ 270us) tablet.cc:699] Row locks acquired
0430 20:11:41.263979 (+ 3us) write_op.cc:260] PREPARE: finished
0430 20:11:41.264051 (+ 72us) write_op.cc:270] Start()
0430 20:11:41.264124 (+ 73us) write_op.cc:276] Timestamp: P: 1714507901263954 usec, L: 0
0430 20:11:41.264128 (+ 4us) op_driver.cc:331] REPLICATION: starting
0430 20:11:41.264552 (+ 424us) log.cc:842] Serialized 728 byte log entry
0430 20:11:41.305880 (+ 41328us) op_driver.cc:444] REPLICATION: finished
0430 20:11:41.306494 (+ 614us) write_op.cc:301] APPLY: starting
0430 20:11:41.306514 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:41.306567 (+ 53us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:41.306616 (+ 49us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:41.306689 (+ 73us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:41.306725 (+ 36us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0430 20:11:41.306733 (+ 8us) write_op.cc:312] APPLY: finished
0430 20:11:41.306816 (+ 83us) log.cc:842] Serialized 29 byte log entry
0430 20:11:41.306878 (+ 62us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:41.306901 (+ 23us) write_op.cc:446] Released schema lock
0430 20:11:41.306919 (+ 18us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:41.306986 (+ 67us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:41.306990 (+ 4us) write_op.cc:446] Released schema lock
Metrics: {"spinlock_wait_cycles":51774592,"child_traces":[["op",{"apply.queue_time_us":562,"apply.run_cpu_time_us":549,"apply.run_wall_time_us":548,"num_ops":2,"prepare.queue_time_us":59258,"prepare.run_cpu_time_us":1256,"prepare.run_wall_time_us":2900,"raft.queue_time_us":50,"raft.run_cpu_time_us":265,"raft.run_wall_time_us":265,"replication_time_us":41718}]]}
I20240430 20:11:42.127686 19026 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_605f23ec0a58457f87356d7f45d3b45c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:42.128172 19026 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_605f23ec0a58457f87356d7f45d3b45c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:42.129509 18601 tablet_service.cc:1514] Processing DeleteTablet for tablet 283e5e73cbc04daba0de19f4541f9fd0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:40 UTC) from {username='slave'} at 127.0.0.1:40218
W20240430 20:11:42.130491 19049 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 266fd23786bc435db5083370f3330ed0 has no consensus state
I20240430 20:11:42.130730 19027 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: Sending DeleteTablet for 3 replicas of tablet 2aa2a489116940dea3173cad16d94fa5
I20240430 20:11:42.130924 19049 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_69622a6c03f14409a34ad4d98f24c5a1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240430 20:11:42.131191 19027 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:51664 (request call id 253) took 1209 ms. Trace:
W20240430 20:11:42.131420 19049 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_69622a6c03f14409a34ad4d98f24c5a1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:42.131299 19027 rpcz_store.cc:276] 0430 20:11:40.922170 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:40.949071 (+ 26901us) service_pool.cc:225] Handling call
0430 20:11:40.949596 (+ 525us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0430 20:11:40.949747 (+ 151us) catalog_manager.cc:2749] Modifying in-memory table state
0430 20:11:40.949819 (+ 72us) catalog_manager.cc:2758] Locking tablets
0430 20:11:41.202602 (+252783us) catalog_manager.cc:2772] Removing table and tablets from system table
0430 20:11:41.309269 (+106667us) catalog_manager.cc:2790] Removing table from by-name map
0430 20:11:41.309345 (+ 76us) catalog_manager.cc:2806] Committing in-memory state
0430 20:11:42.130667 (+821322us) catalog_manager.cc:2810] Aborting table tasks
0430 20:11:42.131153 (+ 486us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:41.266093 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:41.266255 (+ 162us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:41.266289 (+ 34us) write_op.cc:427] Acquired schema lock
0430 20:11:41.266291 (+ 2us) tablet.cc:660] Decoding operations
0430 20:11:41.266515 (+ 224us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:41.266527 (+ 12us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:41.266559 (+ 32us) tablet.cc:683] Acquiring locks for 2 operations
0430 20:11:41.267148 (+ 589us) tablet.cc:699] Row locks acquired
0430 20:11:41.267152 (+ 4us) write_op.cc:260] PREPARE: finished
0430 20:11:41.267224 (+ 72us) write_op.cc:270] Start()
0430 20:11:41.267287 (+ 63us) write_op.cc:276] Timestamp: P: 1714507901267157 usec, L: 0
0430 20:11:41.267291 (+ 4us) op_driver.cc:331] REPLICATION: starting
0430 20:11:41.304270 (+ 36979us) log.cc:842] Serialized 728 byte log entry
0430 20:11:41.306389 (+ 2119us) op_driver.cc:444] REPLICATION: finished
0430 20:11:41.307070 (+ 681us) write_op.cc:301] APPLY: starting
0430 20:11:41.307082 (+ 12us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:41.307118 (+ 36us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:41.307154 (+ 36us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:41.307221 (+ 67us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:41.307247 (+ 26us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0430 20:11:41.307253 (+ 6us) write_op.cc:312] APPLY: finished
0430 20:11:41.307308 (+ 55us) log.cc:842] Serialized 29 byte log entry
0430 20:11:41.307346 (+ 38us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:41.307361 (+ 15us) write_op.cc:446] Released schema lock
0430 20:11:41.307367 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:41.307416 (+ 49us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:41.307419 (+ 3us) write_op.cc:446] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":648,"apply.run_cpu_time_us":387,"apply.run_wall_time_us":386,"mutex_wait_us":38,"num_ops":2,"prepare.queue_time_us":62298,"prepare.run_cpu_time_us":1174,"prepare.run_wall_time_us":49430,"raft.queue_time_us":395,"raft.run_cpu_time_us":112,"raft.run_wall_time_us":113,"replication_time_us":39065}]]}
I20240430 20:11:42.132071 19027 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_ca129cca71d34045824d5f945570c3bd" } new_table_name: "default.Table_edc4c27ed846454288f73f6a0d88cb69" modify_external_catalogs: true
I20240430 20:11:42.132484 19051 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: Sending DeleteTablet for 3 replicas of tablet 9c73d0a6bc414d3b9900570848d19a51
I20240430 20:11:42.133530 19051 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:51664 (request call id 264) took 1065 ms. Trace:
I20240430 20:11:42.133617 19051 rpcz_store.cc:276] 0430 20:11:41.067789 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:41.175654 (+107865us) service_pool.cc:225] Handling call
0430 20:11:41.176834 (+ 1180us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0430 20:11:41.177115 (+ 281us) catalog_manager.cc:2749] Modifying in-memory table state
0430 20:11:41.177235 (+ 120us) catalog_manager.cc:2758] Locking tablets
0430 20:11:41.210575 (+ 33340us) catalog_manager.cc:2772] Removing table and tablets from system table
0430 20:11:41.540814 (+330239us) catalog_manager.cc:2790] Removing table from by-name map
0430 20:11:41.540946 (+ 132us) catalog_manager.cc:2806] Committing in-memory state
0430 20:11:42.132427 (+591481us) catalog_manager.cc:2810] Aborting table tasks
0430 20:11:42.133468 (+ 1041us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:41.507113 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:41.507282 (+ 169us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:41.507293 (+ 11us) write_op.cc:427] Acquired schema lock
0430 20:11:41.507298 (+ 5us) tablet.cc:660] Decoding operations
0430 20:11:41.507461 (+ 163us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:41.524324 (+ 16863us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:41.524337 (+ 13us) tablet.cc:683] Acquiring locks for 2 operations
0430 20:11:41.524520 (+ 183us) tablet.cc:699] Row locks acquired
0430 20:11:41.524525 (+ 5us) write_op.cc:260] PREPARE: finished
0430 20:11:41.524578 (+ 53us) write_op.cc:270] Start()
0430 20:11:41.524630 (+ 52us) write_op.cc:276] Timestamp: P: 1714507901524539 usec, L: 0
0430 20:11:41.524636 (+ 6us) op_driver.cc:331] REPLICATION: starting
0430 20:11:41.524942 (+ 306us) log.cc:842] Serialized 734 byte log entry
0430 20:11:41.539962 (+ 15020us) op_driver.cc:444] REPLICATION: finished
0430 20:11:41.540131 (+ 169us) write_op.cc:301] APPLY: starting
0430 20:11:41.540150 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:41.540212 (+ 62us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:41.540255 (+ 43us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:41.540343 (+ 88us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:41.540392 (+ 49us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0430 20:11:41.540403 (+ 11us) write_op.cc:312] APPLY: finished
0430 20:11:41.540496 (+ 93us) log.cc:842] Serialized 29 byte log entry
0430 20:11:41.540570 (+ 74us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:41.540599 (+ 29us) write_op.cc:446] Released schema lock
0430 20:11:41.540609 (+ 10us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:41.540675 (+ 66us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:41.540678 (+ 3us) write_op.cc:446] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":89,"apply.run_cpu_time_us":611,"apply.run_wall_time_us":610,"num_ops":2,"prepare.queue_time_us":295954,"prepare.run_cpu_time_us":1208,"prepare.run_wall_time_us":29570,"raft.queue_time_us":3571,"raft.run_cpu_time_us":354,"raft.run_wall_time_us":355,"replication_time_us":15292}]]}
I20240430 20:11:42.134510 19051 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_4839b72b4f2143b2b23e8f3865d3e203"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:42.135035 19051 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4839b72b4f2143b2b23e8f3865d3e203 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:42.140439 19495 ts_tablet_manager.cc:1888] T 20d166756819406ba85044f4fee9c4ce P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:42.141291 18592 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c73d0a6bc414d3b9900570848d19a51 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:41 UTC) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:42.141577 18601 tablet_service.cc:1514] Processing DeleteTablet for tablet 2aa2a489116940dea3173cad16d94fa5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:40 UTC) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:42.141672 19495 log.cc:1196] T 20d166756819406ba85044f4fee9c4ce P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/20d166756819406ba85044f4fee9c4ce
I20240430 20:11:42.142117 19495 ts_tablet_manager.cc:1909] T 20d166756819406ba85044f4fee9c4ce P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:42.142450 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 145) took 9561 ms. Trace:
I20240430 20:11:42.142532 19495 rpcz_store.cc:276] 0430 20:11:32.581024 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:32.581084 (+ 60us) service_pool.cc:225] Handling call
0430 20:11:42.126079 (+9544995us) ts_tablet_manager.cc:1120] Deleting tablet 20d166756819406ba85044f4fee9c4ce
0430 20:11:42.126088 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:42.130070 (+ 3982us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:42.140314 (+ 10244us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:42.142428 (+ 2114us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":9544760}
I20240430 20:11:42.144191 19495 tablet_replica.cc:330] T 1555924b5f1444398f12ee82c3cb5b13 P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:42.144595 19495 raft_consensus.cc:2229] T 1555924b5f1444398f12ee82c3cb5b13 P 2efb542aa55e439b99ccb98d1c5c884a [term 2 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:42.145023 19495 raft_consensus.cc:2258] T 1555924b5f1444398f12ee82c3cb5b13 P 2efb542aa55e439b99ccb98d1c5c884a [term 2 FOLLOWER]: Raft consensus is shut down!
W20240430 20:11:42.145823 19027 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7ce09fcbe1ac457a9c05efaf000c3263 consensus state has no leader
I20240430 20:11:42.146144 19027 master_service.cc:946] ReplaceTablet: received request to replace tablet f7b0132c17d44a9ba17288e0321a7f11 from {username='slave'} at 127.0.0.1:51664
I20240430 20:11:42.146447 19495 ts_tablet_manager.cc:1875] T 1555924b5f1444398f12ee82c3cb5b13 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:42.149632 19004 catalog_manager.cc:4830] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): tablet 20d166756819406ba85044f4fee9c4ce (table default.Table_b331bf8aeaae4911ac01ed3fbd2627bd [id=1bf5c29b08134670984b6569db298129]) successfully deleted
I20240430 20:11:42.149936 18457 tablet_service.cc:1514] Processing DeleteTablet for tablet 283e5e73cbc04daba0de19f4541f9fd0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:40 UTC) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:42.149946 18449 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c73d0a6bc414d3b9900570848d19a51 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:41 UTC) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:42.149957 18461 tablet_service.cc:1514] Processing DeleteTablet for tablet 2aa2a489116940dea3173cad16d94fa5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:40 UTC) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:42.153066 19049 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_2f6a85489f2e49db8974ff86813c85ee"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240430 20:11:42.153484 19028 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_730a215d7af74a5bb753adca39dea676" } modify_external_catalogs: true
I20240430 20:11:42.153637 19028 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_730a215d7af74a5bb753adca39dea676" } modify_external_catalogs: true
W20240430 20:11:42.153993 19049 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2f6a85489f2e49db8974ff86813c85ee in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:42.154798 19026 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_304521bb46d9434481340ff5ce7ca994"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:42.155550 19026 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_304521bb46d9434481340ff5ce7ca994 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:42.157265 19495 ts_tablet_manager.cc:1888] T 1555924b5f1444398f12ee82c3cb5b13 P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.2
I20240430 20:11:42.162817 19495 log.cc:1196] T 1555924b5f1444398f12ee82c3cb5b13 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/1555924b5f1444398f12ee82c3cb5b13
I20240430 20:11:42.163575 19495 ts_tablet_manager.cc:1909] T 1555924b5f1444398f12ee82c3cb5b13 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:42.163947 20096 raft_consensus.cc:482] T 87ac32a0ddb84676adc83038ef664591 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:42.164094 20096 raft_consensus.cc:504] T 87ac32a0ddb84676adc83038ef664591 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:42.164331 20105 raft_consensus.cc:482] T 1462ecf1176c4e66be42abddbe36b00c P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:42.164464 20105 raft_consensus.cc:504] T 1462ecf1176c4e66be42abddbe36b00c P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:42.164348 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 146) took 9575 ms. Trace:
I20240430 20:11:42.164786 20096 leader_election.cc:288] T 87ac32a0ddb84676adc83038ef664591 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:42.164733 19495 rpcz_store.cc:276] 0430 20:11:32.589224 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:32.589278 (+ 54us) service_pool.cc:225] Handling call
0430 20:11:42.144144 (+9554866us) ts_tablet_manager.cc:1120] Deleting tablet 1555924b5f1444398f12ee82c3cb5b13
0430 20:11:42.144154 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:42.153571 (+ 9417us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:42.157136 (+ 3565us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:42.164328 (+ 7192us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":9554640}
I20240430 20:11:42.165164 20105 leader_election.cc:288] T 1462ecf1176c4e66be42abddbe36b00c P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:42.165791 19004 catalog_manager.cc:4830] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): tablet 1555924b5f1444398f12ee82c3cb5b13 (table default.Table_388d774462294cab8342c6de331f9712 [id=aca48e01301e4f5f9562df8c87829b9d]) successfully deleted
I20240430 20:11:42.165815 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 147) took 9550 ms. Trace:
I20240430 20:11:42.165861 18608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1462ecf1176c4e66be42abddbe36b00c" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:42.165902 19495 rpcz_store.cc:276] 0430 20:11:32.615656 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:32.615711 (+ 55us) service_pool.cc:225] Handling call
0430 20:11:42.165692 (+9549981us) ts_tablet_manager.cc:1120] Deleting tablet eb76f60448774fe1967fffe7c6873cfc
0430 20:11:42.165703 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:42.165800 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":9549726}
I20240430 20:11:42.166046 18608 raft_consensus.cc:2454] T 1462ecf1176c4e66be42abddbe36b00c P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 0.
W20240430 20:11:42.166132 18558 leader_election.cc:341] T 87ac32a0ddb84676adc83038ef664591 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Not found: Tablet not found: 87ac32a0ddb84676adc83038ef664591
W20240430 20:11:42.166366 19004 catalog_manager.cc:4810] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): delete failed for tablet eb76f60448774fe1967fffe7c6873cfc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet eb76f60448774fe1967fffe7c6873cfc already in progress: creating tablet
I20240430 20:11:42.166666 18415 leader_election.cc:302] T 1462ecf1176c4e66be42abddbe36b00c P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:42.166744 19495 tablet_replica.cc:330] T 6e826459c3f4437da81c9e6e09f0c22f P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:42.166944 19495 raft_consensus.cc:2229] T 6e826459c3f4437da81c9e6e09f0c22f P 2efb542aa55e439b99ccb98d1c5c884a [term 2 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:42.167093 19495 raft_consensus.cc:2258] T 6e826459c3f4437da81c9e6e09f0c22f P 2efb542aa55e439b99ccb98d1c5c884a [term 2 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:42.167714 20096 raft_consensus.cc:482] T e3e74e76ab2b4fe492ffcc1838656fd1 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3c51c33d5616482cafb68e26688820f4)
I20240430 20:11:42.167857 20096 raft_consensus.cc:504] T e3e74e76ab2b4fe492ffcc1838656fd1 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:42.168308 19495 ts_tablet_manager.cc:1875] T 6e826459c3f4437da81c9e6e09f0c22f P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:42.168362 20096 leader_election.cc:288] T e3e74e76ab2b4fe492ffcc1838656fd1 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:42.167136 20105 raft_consensus.cc:2781] T 1462ecf1176c4e66be42abddbe36b00c P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:42.168773 20105 raft_consensus.cc:482] T 1462ecf1176c4e66be42abddbe36b00c P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:42.168854 20105 raft_consensus.cc:3037] T 1462ecf1176c4e66be42abddbe36b00c P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:42.171209 18478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3e74e76ab2b4fe492ffcc1838656fd1" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:42.171473 18478 raft_consensus.cc:2379] T e3e74e76ab2b4fe492ffcc1838656fd1 P 2efb542aa55e439b99ccb98d1c5c884a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e1c6397f09747f091534faf93542710 in current term 2: Already voted for candidate 2efb542aa55e439b99ccb98d1c5c884a in this term.
I20240430 20:11:42.174883 20105 raft_consensus.cc:504] T 1462ecf1176c4e66be42abddbe36b00c P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:42.175935 20105 leader_election.cc:288] T 1462ecf1176c4e66be42abddbe36b00c P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:42.176442 18983 tablet_bootstrap.cc:492] T a59ef6c545dd41a28265a4c1b8ce10e0 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:42.177140 19471 tablet_replica.cc:330] T 7542c7fe6cad452ab718f9ef54dbbbbe P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:42.177340 19471 raft_consensus.cc:2229] T 7542c7fe6cad452ab718f9ef54dbbbbe P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:42.177484 19471 raft_consensus.cc:2258] T 7542c7fe6cad452ab718f9ef54dbbbbe P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:42.177464 19027 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: Sending DeleteTablet for 3 replicas of tablet f7b0132c17d44a9ba17288e0321a7f11
I20240430 20:11:42.177978 19471 ts_tablet_manager.cc:1875] T 7542c7fe6cad452ab718f9ef54dbbbbe P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:42.177997 19027 catalog_manager.cc:6554] ReplaceTablet: tablet f7b0132c17d44a9ba17288e0321a7f11 deleted and replaced by tablet bea2ca3d09164afd9941cb7e35f5ae24
I20240430 20:11:42.177501 19033 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: Sending DeleteTablet for 3 replicas of tablet 4f6f9ae722da49e58c9b72107f6da2fa
I20240430 20:11:42.178642 19024 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: Sending DeleteTablet for 3 replicas of tablet 32c823d1700949ab8b22adcd83b7bfd1
I20240430 20:11:42.179071 19024 catalog_manager.cc:6554] ReplaceTablet: tablet 32c823d1700949ab8b22adcd83b7bfd1 deleted and replaced by tablet 024069ab786749028c020abd1c836cf5
I20240430 20:11:42.179315 19024 master_service.cc:946] ReplaceTablet: received request to replace tablet ae87f5e92a934e2182164c231bc76df0 from {username='slave'} at 127.0.0.1:51664
I20240430 20:11:42.179066 19033 master_service.cc:946] ReplaceTablet: received request to replace tablet fdf53061efd74fa29a9d8208d1fd210e from {username='slave'} at 127.0.0.1:51664
I20240430 20:11:42.180527 19028 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: Sending DeleteTablet for 3 replicas of tablet b22aaedb0a6c4fd9ad0ea8fa019001e1
I20240430 20:11:42.181181 19028 master_service.cc:946] ReplaceTablet: received request to replace tablet fae9a35a2ac844b4bc2ebdf214aa7171 from {username='slave'} at 127.0.0.1:51664
I20240430 20:11:42.181991 19028 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_fd465030bff1492cb52f47c7e4d84403"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240430 20:11:42.182677 18461 tablet_service.cc:1514] Processing DeleteTablet for tablet f7b0132c17d44a9ba17288e0321a7f11 with delete_type TABLET_DATA_DELETED (replaced by tablet bea2ca3d09164afd9941cb7e35f5ae24) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:42.182905 18449 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f6f9ae722da49e58c9b72107f6da2fa with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:42 UTC) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:42.182958 18457 tablet_service.cc:1514] Processing DeleteTablet for tablet 32c823d1700949ab8b22adcd83b7bfd1 with delete_type TABLET_DATA_DELETED (replaced by tablet 024069ab786749028c020abd1c836cf5) from {username='slave'} at 127.0.0.1:41680
W20240430 20:11:42.183238 19028 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fd465030bff1492cb52f47c7e4d84403 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:42.183506 19027 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_2eed103638364cde94cd8af68d3cdf9b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:42.184016 19027 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2eed103638364cde94cd8af68d3cdf9b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:42.187203 18457 tablet_service.cc:1514] Processing DeleteTablet for tablet b22aaedb0a6c4fd9ad0ea8fa019001e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:42 UTC) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:42.187283 19051 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_7390a5615e234bf38ddf41eefe70bdf3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:42.188035 19051 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7390a5615e234bf38ddf41eefe70bdf3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:42.189566 19026 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_1407edea18d440e7adacadcfc8d89054"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240430 20:11:42.190140 19495 ts_tablet_manager.cc:1888] T 6e826459c3f4437da81c9e6e09f0c22f P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.3
W20240430 20:11:42.190163 19026 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1407edea18d440e7adacadcfc8d89054 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:42.193203 18589 tablet_service.cc:1514] Processing DeleteTablet for tablet f7b0132c17d44a9ba17288e0321a7f11 with delete_type TABLET_DATA_DELETED (replaced by tablet bea2ca3d09164afd9941cb7e35f5ae24) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:42.193308 18585 tablet_service.cc:1514] Processing DeleteTablet for tablet b22aaedb0a6c4fd9ad0ea8fa019001e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:42 UTC) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:42.193527 18592 tablet_service.cc:1514] Processing DeleteTablet for tablet 32c823d1700949ab8b22adcd83b7bfd1 with delete_type TABLET_DATA_DELETED (replaced by tablet 024069ab786749028c020abd1c836cf5) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:42.193648 18617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1462ecf1176c4e66be42abddbe36b00c" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:42.193827 18617 raft_consensus.cc:3037] T 1462ecf1176c4e66be42abddbe36b00c P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:42.195652 18601 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f6f9ae722da49e58c9b72107f6da2fa with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:42 UTC) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:42.197253 18983 tablet_bootstrap.cc:654] T a59ef6c545dd41a28265a4c1b8ce10e0 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:42.199363 18617 raft_consensus.cc:2454] T 1462ecf1176c4e66be42abddbe36b00c P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 1.
I20240430 20:11:42.204075 19024 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: Sending DeleteTablet for 3 replicas of tablet ae87f5e92a934e2182164c231bc76df0
I20240430 20:11:42.204638 19024 catalog_manager.cc:6554] ReplaceTablet: tablet ae87f5e92a934e2182164c231bc76df0 deleted and replaced by tablet 4fa9575d96a84bef876467e21dada803
I20240430 20:11:42.205782 18601 tablet_service.cc:1514] Processing DeleteTablet for tablet ae87f5e92a934e2182164c231bc76df0 with delete_type TABLET_DATA_DELETED (replaced by tablet 4fa9575d96a84bef876467e21dada803) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:42.206219 19033 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: Sending DeleteTablet for 3 replicas of tablet fdf53061efd74fa29a9d8208d1fd210e
I20240430 20:11:42.206732 19033 catalog_manager.cc:6554] ReplaceTablet: tablet fdf53061efd74fa29a9d8208d1fd210e deleted and replaced by tablet 5df364628561475391c9a5e10ff5a05f
I20240430 20:11:42.206962 18601 tablet_service.cc:1514] Processing DeleteTablet for tablet fdf53061efd74fa29a9d8208d1fd210e with delete_type TABLET_DATA_DELETED (replaced by tablet 5df364628561475391c9a5e10ff5a05f) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:42.212010 18988 tablet_bootstrap.cc:492] T 46251c08277e498da04a41d009310c62 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:42.212201 19471 ts_tablet_manager.cc:1888] T 7542c7fe6cad452ab718f9ef54dbbbbe P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:42.213338 19471 log.cc:1196] T 7542c7fe6cad452ab718f9ef54dbbbbe P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/7542c7fe6cad452ab718f9ef54dbbbbe
I20240430 20:11:42.213881 19471 ts_tablet_manager.cc:1909] T 7542c7fe6cad452ab718f9ef54dbbbbe P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:42.215207 18744 tablet_service.cc:1514] Processing DeleteTablet for tablet b22aaedb0a6c4fd9ad0ea8fa019001e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:42 UTC) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:42.215318 18726 tablet_service.cc:1514] Processing DeleteTablet for tablet ae87f5e92a934e2182164c231bc76df0 with delete_type TABLET_DATA_DELETED (replaced by tablet 4fa9575d96a84bef876467e21dada803) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:42.215394 18763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1462ecf1176c4e66be42abddbe36b00c" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:42.215417 18764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1462ecf1176c4e66be42abddbe36b00c" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:42.215710 20198 raft_consensus.cc:482] T 2f61e3f88a6448dfa323204bd8d14cc5 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:42.215925 20198 raft_consensus.cc:504] T 2f61e3f88a6448dfa323204bd8d14cc5 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:42.216569 18725 tablet_service.cc:1514] Processing DeleteTablet for tablet 2aa2a489116940dea3173cad16d94fa5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:40 UTC) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:42.216881 20309 consensus_queue.cc:1035] T 6e754af64cb64fd2a082adae7d5f24da P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.049s
I20240430 20:11:42.217244 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 104) took 10070 ms. Trace:
I20240430 20:11:42.217280 18724 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f6f9ae722da49e58c9b72107f6da2fa with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:42 UTC) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:42.217394 18734 tablet_service.cc:1514] Processing DeleteTablet for tablet 283e5e73cbc04daba0de19f4541f9fd0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:40 UTC) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:42.217324 19471 rpcz_store.cc:276] 0430 20:11:32.146901 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:32.147461 (+ 560us) service_pool.cc:225] Handling call
0430 20:11:42.109658 (+9962197us) ts_tablet_manager.cc:1120] Deleting tablet 7542c7fe6cad452ab718f9ef54dbbbbe
0430 20:11:42.177074 (+ 67416us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:42.195372 (+ 18298us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:42.202461 (+ 7089us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:42.217224 (+ 14763us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":9943855}
I20240430 20:11:42.217500 18730 tablet_service.cc:1514] Processing DeleteTablet for tablet f7b0132c17d44a9ba17288e0321a7f11 with delete_type TABLET_DATA_DELETED (replaced by tablet bea2ca3d09164afd9941cb7e35f5ae24) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:42.217700 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 109) took 10065 ms. Trace:
I20240430 20:11:42.217768 19471 rpcz_store.cc:276] 0430 20:11:32.165221 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:32.165627 (+ 406us) service_pool.cc:225] Handling call
0430 20:11:42.217598 (+10051971us) ts_tablet_manager.cc:1120] Deleting tablet 02176eee8e044a7db8cac48439e9f594
0430 20:11:42.217606 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:42.217682 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":10051760}
I20240430 20:11:42.206389 18457 tablet_service.cc:1514] Processing DeleteTablet for tablet ae87f5e92a934e2182164c231bc76df0 with delete_type TABLET_DATA_DELETED (replaced by tablet 4fa9575d96a84bef876467e21dada803) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:42.207803 18449 tablet_service.cc:1514] Processing DeleteTablet for tablet fdf53061efd74fa29a9d8208d1fd210e with delete_type TABLET_DATA_DELETED (replaced by tablet 5df364628561475391c9a5e10ff5a05f) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:42.218180 19495 log.cc:1196] T 6e826459c3f4437da81c9e6e09f0c22f P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/6e826459c3f4437da81c9e6e09f0c22f
I20240430 20:11:42.218892 19495 ts_tablet_manager.cc:1909] T 6e826459c3f4437da81c9e6e09f0c22f P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:42.220244 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 148) took 9597 ms. Trace:
I20240430 20:11:42.220613 18999 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet 7542c7fe6cad452ab718f9ef54dbbbbe (table default.Table_8af52c0f1d404b42bc53287f64e20c2d [id=8e590e0e404344658b5cee90132d91d1]) successfully deleted
W20240430 20:11:42.220943 18999 catalog_manager.cc:4797] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): delete failed for tablet 02176eee8e044a7db8cac48439e9f594 because the tablet was not found. No further retry: Not found: Tablet not found: 02176eee8e044a7db8cac48439e9f594
W20240430 20:11:42.219769 18417 leader_election.cc:341] T 1462ecf1176c4e66be42abddbe36b00c P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
W20240430 20:11:42.221455 18417 leader_election.cc:341] T 1462ecf1176c4e66be42abddbe36b00c P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:42.221192 19495 rpcz_store.cc:276] 0430 20:11:32.623171 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:32.623227 (+ 56us) service_pool.cc:225] Handling call
0430 20:11:42.166697 (+9543470us) ts_tablet_manager.cc:1120] Deleting tablet 6e826459c3f4437da81c9e6e09f0c22f
0430 20:11:42.166707 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:42.175508 (+ 8801us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:42.190032 (+ 14524us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:42.220225 (+ 30193us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":9543202}
I20240430 20:11:42.222239 19004 catalog_manager.cc:4830] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): tablet 6e826459c3f4437da81c9e6e09f0c22f (table default.Table_873c21f6dffd4d47a742b3699ab85268 [id=d7e2c4300ff94fa2acc65d6d6c932c65]) successfully deleted
I20240430 20:11:42.222730 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 149) took 9587 ms. Trace:
I20240430 20:11:42.224278 20235 raft_consensus.cc:482] T 84568f90b9c54cf292099808d4cbf5dc P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:42.224092 19495 rpcz_store.cc:276] 0430 20:11:32.635220 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:32.635279 (+ 59us) service_pool.cc:225] Handling call
0430 20:11:42.222620 (+9587341us) ts_tablet_manager.cc:1120] Deleting tablet 2873af6da928463f9809864b8eaf0c4a
0430 20:11:42.222630 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:42.222715 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":9587093}
W20240430 20:11:42.225773 19004 catalog_manager.cc:4810] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): delete failed for tablet 2873af6da928463f9809864b8eaf0c4a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2873af6da928463f9809864b8eaf0c4a already in progress: creating tablet
I20240430 20:11:42.225929 20096 raft_consensus.cc:482] T 84568f90b9c54cf292099808d4cbf5dc P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:42.224684 20235 raft_consensus.cc:504] T 84568f90b9c54cf292099808d4cbf5dc P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:42.226063 20096 raft_consensus.cc:504] T 84568f90b9c54cf292099808d4cbf5dc P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:42.226650 20235 leader_election.cc:288] T 84568f90b9c54cf292099808d4cbf5dc P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:42.226667 20096 leader_election.cc:288] T 84568f90b9c54cf292099808d4cbf5dc P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:42.227262 19495 tablet_replica.cc:330] T c2b019f529a54228b804437bb025e31d P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:42.227463 18478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84568f90b9c54cf292099808d4cbf5dc" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:42.227798 18478 raft_consensus.cc:2454] T 84568f90b9c54cf292099808d4cbf5dc P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 0.
I20240430 20:11:42.227864 19495 raft_consensus.cc:2229] T c2b019f529a54228b804437bb025e31d P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:42.228022 19495 raft_consensus.cc:2258] T c2b019f529a54228b804437bb025e31d P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:42.228368 18558 leader_election.cc:302] T 84568f90b9c54cf292099808d4cbf5dc P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:42.228554 19495 ts_tablet_manager.cc:1875] T c2b019f529a54228b804437bb025e31d P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:42.228721 20096 raft_consensus.cc:2781] T 84568f90b9c54cf292099808d4cbf5dc P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:42.228802 20096 raft_consensus.cc:482] T 84568f90b9c54cf292099808d4cbf5dc P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:42.228865 20096 raft_consensus.cc:3037] T 84568f90b9c54cf292099808d4cbf5dc P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
W20240430 20:11:42.228816 18699 leader_election.cc:341] T 2f61e3f88a6448dfa323204bd8d14cc5 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Not found: Tablet not found: 2f61e3f88a6448dfa323204bd8d14cc5
W20240430 20:11:42.229318 18696 leader_election.cc:341] T 2f61e3f88a6448dfa323204bd8d14cc5 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Not found: Tablet not found: 2f61e3f88a6448dfa323204bd8d14cc5
I20240430 20:11:42.229833 18733 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c73d0a6bc414d3b9900570848d19a51 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:41 UTC) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:42.230082 18756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3e74e76ab2b4fe492ffcc1838656fd1" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:42.230314 18756 raft_consensus.cc:2454] T e3e74e76ab2b4fe492ffcc1838656fd1 P 3c51c33d5616482cafb68e26688820f4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 2.
I20240430 20:11:42.230711 18557 leader_election.cc:302] T e3e74e76ab2b4fe492ffcc1838656fd1 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a
I20240430 20:11:42.230947 20047 raft_consensus.cc:2781] T e3e74e76ab2b4fe492ffcc1838656fd1 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240430 20:11:42.231007 20047 raft_consensus.cc:482] T e3e74e76ab2b4fe492ffcc1838656fd1 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3c51c33d5616482cafb68e26688820f4)
I20240430 20:11:42.231069 20047 raft_consensus.cc:3037] T e3e74e76ab2b4fe492ffcc1838656fd1 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Advancing to term 2
I20240430 20:11:42.231185 18729 tablet_service.cc:1514] Processing DeleteTablet for tablet fdf53061efd74fa29a9d8208d1fd210e with delete_type TABLET_DATA_DELETED (replaced by tablet 5df364628561475391c9a5e10ff5a05f) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:42.231626 19489 raft_consensus.cc:2229] T 7542c7fe6cad452ab718f9ef54dbbbbe P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Raft consensus shutting down.
I20240430 20:11:42.233449 20047 raft_consensus.cc:504] T e3e74e76ab2b4fe492ffcc1838656fd1 P 5e1c6397f09747f091534faf93542710 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:42.234303 20047 leader_election.cc:288] T e3e74e76ab2b4fe492ffcc1838656fd1 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 2 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:42.234333 20198 leader_election.cc:288] T 2f61e3f88a6448dfa323204bd8d14cc5 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:42.234297 18762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87ac32a0ddb84676adc83038ef664591" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:42.234544 18736 tablet_service.cc:1514] Processing DeleteTablet for tablet 32c823d1700949ab8b22adcd83b7bfd1 with delete_type TABLET_DATA_DELETED (replaced by tablet 024069ab786749028c020abd1c836cf5) from {username='slave'} at 127.0.0.1:59830
W20240430 20:11:42.234934 18557 leader_election.cc:341] T 87ac32a0ddb84676adc83038ef664591 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:42.235039 18557 leader_election.cc:302] T 87ac32a0ddb84676adc83038ef664591 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:42.235131 18988 tablet_bootstrap.cc:654] T 46251c08277e498da04a41d009310c62 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:42.235350 18750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84568f90b9c54cf292099808d4cbf5dc" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:42.235495 18608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84568f90b9c54cf292099808d4cbf5dc" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:42.235580 19489 pending_rounds.cc:62] T 7542c7fe6cad452ab718f9ef54dbbbbe P 3c51c33d5616482cafb68e26688820f4: Trying to abort 1 pending ops.
I20240430 20:11:42.235836 20047 raft_consensus.cc:2726] T 87ac32a0ddb84676adc83038ef664591 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:42.235965 18415 leader_election.cc:302] T 1462ecf1176c4e66be42abddbe36b00c P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters: 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:42.235996 18762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3e74e76ab2b4fe492ffcc1838656fd1" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:42.236025 18696 leader_election.cc:302] T 2f61e3f88a6448dfa323204bd8d14cc5 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710
I20240430 20:11:42.236308 20105 raft_consensus.cc:2781] T 1462ecf1176c4e66be42abddbe36b00c P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:42.236487 20105 raft_consensus.cc:686] T 1462ecf1176c4e66be42abddbe36b00c P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Becoming Leader. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Running, Role: LEADER
I20240430 20:11:42.236562 20309 raft_consensus.cc:2726] T 2f61e3f88a6448dfa323204bd8d14cc5 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:42.236829 18478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3e74e76ab2b4fe492ffcc1838656fd1" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:42.236761 20105 consensus_queue.cc:237] T 1462ecf1176c4e66be42abddbe36b00c P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:42.237107 18478 raft_consensus.cc:2379] T e3e74e76ab2b4fe492ffcc1838656fd1 P 2efb542aa55e439b99ccb98d1c5c884a [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 5e1c6397f09747f091534faf93542710 in current term 2: Already voted for candidate 2efb542aa55e439b99ccb98d1c5c884a in this term.
W20240430 20:11:42.238247 18417 leader_election.cc:341] T 84568f90b9c54cf292099808d4cbf5dc P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:42.239426 20096 raft_consensus.cc:504] T 84568f90b9c54cf292099808d4cbf5dc P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:42.239862 18608 raft_consensus.cc:2379] T 84568f90b9c54cf292099808d4cbf5dc P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2efb542aa55e439b99ccb98d1c5c884a in current term 1: Already voted for candidate 5e1c6397f09747f091534faf93542710 in this term.
I20240430 20:11:42.241050 20096 leader_election.cc:288] T 84568f90b9c54cf292099808d4cbf5dc P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:42.241266 18985 tablet_bootstrap.cc:492] T a59ef6c545dd41a28265a4c1b8ce10e0 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:42.241444 18985 ts_tablet_manager.cc:1367] T a59ef6c545dd41a28265a4c1b8ce10e0 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.162s user 0.015s sys 0.036s
I20240430 20:11:42.241744 18478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84568f90b9c54cf292099808d4cbf5dc" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:42.241941 18478 raft_consensus.cc:3037] T 84568f90b9c54cf292099808d4cbf5dc P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:42.242288 18415 leader_election.cc:302] T 84568f90b9c54cf292099808d4cbf5dc P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a; no voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710
I20240430 20:11:42.242333 18985 raft_consensus.cc:348] T a59ef6c545dd41a28265a4c1b8ce10e0 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:42.243016 18985 raft_consensus.cc:374] T a59ef6c545dd41a28265a4c1b8ce10e0 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:42.243147 18985 raft_consensus.cc:729] T a59ef6c545dd41a28265a4c1b8ce10e0 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:42.243533 18985 consensus_queue.cc:260] T a59ef6c545dd41a28265a4c1b8ce10e0 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:42.245026 18985 ts_tablet_manager.cc:1398] T a59ef6c545dd41a28265a4c1b8ce10e0 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240430 20:11:42.245301 18478 raft_consensus.cc:2454] T 84568f90b9c54cf292099808d4cbf5dc P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
I20240430 20:11:42.245406 18985 tablet_bootstrap.cc:492] T 7ce09fcbe1ac457a9c05efaf000c3263 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:42.246093 18558 leader_election.cc:302] T 84568f90b9c54cf292099808d4cbf5dc P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:42.246855 20096 raft_consensus.cc:2781] T 84568f90b9c54cf292099808d4cbf5dc P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:42.248890 18985 tablet_bootstrap.cc:654] T 7ce09fcbe1ac457a9c05efaf000c3263 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:42.249089 20096 raft_consensus.cc:686] T 84568f90b9c54cf292099808d4cbf5dc P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Becoming Leader. State: Replica: 5e1c6397f09747f091534faf93542710, State: Running, Role: LEADER
I20240430 20:11:42.249435 20096 consensus_queue.cc:237] T 84568f90b9c54cf292099808d4cbf5dc P 5e1c6397f09747f091534faf93542710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:42.218009 19471 tablet_replica.cc:330] T 47c37876694c4a4abb1091c631dab7a5 P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:42.250995 19471 raft_consensus.cc:2229] T 47c37876694c4a4abb1091c631dab7a5 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:42.251412 19471 raft_consensus.cc:2258] T 47c37876694c4a4abb1091c631dab7a5 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:42.252133 19471 ts_tablet_manager.cc:1875] T 47c37876694c4a4abb1091c631dab7a5 P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:42.245752 20105 raft_consensus.cc:2726] T 84568f90b9c54cf292099808d4cbf5dc P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:42.251267 20362 raft_consensus.cc:482] T a59ef6c545dd41a28265a4c1b8ce10e0 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:42.252575 20362 raft_consensus.cc:504] T a59ef6c545dd41a28265a4c1b8ce10e0 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:42.253309 20362 leader_election.cc:288] T a59ef6c545dd41a28265a4c1b8ce10e0 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:42.253954 18617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a59ef6c545dd41a28265a4c1b8ce10e0" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
W20240430 20:11:42.255230 18415 leader_election.cc:341] T a59ef6c545dd41a28265a4c1b8ce10e0 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:42.245563 19495 ts_tablet_manager.cc:1888] T c2b019f529a54228b804437bb025e31d P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:42.258690 19495 log.cc:1196] T c2b019f529a54228b804437bb025e31d P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/c2b019f529a54228b804437bb025e31d
I20240430 20:11:42.259040 19495 ts_tablet_manager.cc:1909] T c2b019f529a54228b804437bb025e31d P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:42.259747 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 153) took 9211 ms. Trace:
I20240430 20:11:42.260396 19489 pending_rounds.cc:68] T 7542c7fe6cad452ab718f9ef54dbbbbe P 3c51c33d5616482cafb68e26688820f4: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7022624364519804928 op_type: NO_OP noop_request { }
I20240430 20:11:42.260588 19489 raft_consensus.cc:2866] T 7542c7fe6cad452ab718f9ef54dbbbbe P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240430 20:11:42.261437 19495 rpcz_store.cc:276] 0430 20:11:33.048421 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:33.050071 (+ 1650us) service_pool.cc:225] Handling call
0430 20:11:42.227217 (+9177146us) ts_tablet_manager.cc:1120] Deleting tablet c2b019f529a54228b804437bb025e31d
0430 20:11:42.227226 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:42.231846 (+ 4620us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:42.240702 (+ 8856us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:42.259726 (+ 19024us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":9176913}
I20240430 20:11:42.262707 19004 catalog_manager.cc:4830] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): tablet c2b019f529a54228b804437bb025e31d (table default.Table_8028554bc993419e80dbf7d526b63bf4 [id=27ef5b3429fb4353afe80dc3089e9914]) successfully deleted
I20240430 20:11:42.263248 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 154) took 9203 ms. Trace:
I20240430 20:11:42.264221 19495 rpcz_store.cc:276] 0430 20:11:33.059576 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:33.059650 (+ 74us) service_pool.cc:225] Handling call
0430 20:11:42.263126 (+9203476us) ts_tablet_manager.cc:1120] Deleting tablet 246fb63a5e9842df8263800e28a5935c
0430 20:11:42.263136 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:42.263233 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":9203245}
I20240430 20:11:42.265157 18750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a59ef6c545dd41a28265a4c1b8ce10e0" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:42.265480 20198 consensus_queue.cc:1035] T 6e754af64cb64fd2a082adae7d5f24da P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
W20240430 20:11:42.265616 19004 catalog_manager.cc:4810] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): delete failed for tablet 246fb63a5e9842df8263800e28a5935c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 246fb63a5e9842df8263800e28a5935c already in progress: creating tablet
I20240430 20:11:42.267205 19495 tablet_replica.cc:330] T a745334c12c545c1ba825c7552c60359 P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:42.268693 19495 raft_consensus.cc:2229] T a745334c12c545c1ba825c7552c60359 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:42.268947 19495 raft_consensus.cc:2258] T a745334c12c545c1ba825c7552c60359 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:42.269656 19495 ts_tablet_manager.cc:1875] T a745334c12c545c1ba825c7552c60359 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240430 20:11:42.270977 18417 leader_election.cc:341] T a59ef6c545dd41a28265a4c1b8ce10e0 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:42.271093 18417 leader_election.cc:302] T a59ef6c545dd41a28265a4c1b8ce10e0 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a; no voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710
I20240430 20:11:42.272572 20362 raft_consensus.cc:2726] T a59ef6c545dd41a28265a4c1b8ce10e0 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:42.274494 18983 tablet_bootstrap.cc:492] T a59ef6c545dd41a28265a4c1b8ce10e0 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:42.274729 18983 ts_tablet_manager.cc:1367] T a59ef6c545dd41a28265a4c1b8ce10e0 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.098s user 0.002s sys 0.008s
I20240430 20:11:42.276109 18985 tablet_bootstrap.cc:492] T 7ce09fcbe1ac457a9c05efaf000c3263 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:42.276073 18983 raft_consensus.cc:348] T a59ef6c545dd41a28265a4c1b8ce10e0 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:42.276312 18983 raft_consensus.cc:374] T a59ef6c545dd41a28265a4c1b8ce10e0 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:42.276365 18985 ts_tablet_manager.cc:1367] T 7ce09fcbe1ac457a9c05efaf000c3263 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.031s user 0.002s sys 0.003s
I20240430 20:11:42.276413 18983 raft_consensus.cc:729] T a59ef6c545dd41a28265a4c1b8ce10e0 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:42.276679 18983 consensus_queue.cc:260] T a59ef6c545dd41a28265a4c1b8ce10e0 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:42.277406 19471 ts_tablet_manager.cc:1888] T 47c37876694c4a4abb1091c631dab7a5 P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:42.277467 18983 ts_tablet_manager.cc:1398] T a59ef6c545dd41a28265a4c1b8ce10e0 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240430 20:11:42.277537 19471 log.cc:1196] T 47c37876694c4a4abb1091c631dab7a5 P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/47c37876694c4a4abb1091c631dab7a5
I20240430 20:11:42.277771 18983 tablet_bootstrap.cc:492] T ebf74809b52f47f3b0ed21b6ac0f541a P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:42.277861 19471 ts_tablet_manager.cc:1909] T 47c37876694c4a4abb1091c631dab7a5 P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:42.278162 18764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84568f90b9c54cf292099808d4cbf5dc" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:42.278256 18985 raft_consensus.cc:348] T 7ce09fcbe1ac457a9c05efaf000c3263 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
W20240430 20:11:42.278692 18557 leader_election.cc:341] T 84568f90b9c54cf292099808d4cbf5dc P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:42.278769 18985 raft_consensus.cc:374] T 7ce09fcbe1ac457a9c05efaf000c3263 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:42.278998 18985 raft_consensus.cc:729] T 7ce09fcbe1ac457a9c05efaf000c3263 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:42.279402 18983 tablet_bootstrap.cc:654] T ebf74809b52f47f3b0ed21b6ac0f541a P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:42.279373 18985 consensus_queue.cc:260] T 7ce09fcbe1ac457a9c05efaf000c3263 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:42.280429 18985 ts_tablet_manager.cc:1398] T 7ce09fcbe1ac457a9c05efaf000c3263 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240430 20:11:42.281339 18985 tablet_bootstrap.cc:492] T abdaf243e5be4ad58036b23151b14fe4 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:42.281729 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 103) took 10134 ms. Trace:
I20240430 20:11:42.281841 19471 rpcz_store.cc:276] 0430 20:11:32.146801 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:32.147436 (+ 635us) service_pool.cc:225] Handling call
0430 20:11:42.217971 (+10070535us) ts_tablet_manager.cc:1120] Deleting tablet 47c37876694c4a4abb1091c631dab7a5
0430 20:11:42.217978 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:42.257914 (+ 39936us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:42.277240 (+ 19326us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:42.281702 (+ 4462us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":10049942}
I20240430 20:11:42.282213 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 113) took 10113 ms. Trace:
I20240430 20:11:42.282306 18999 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet 47c37876694c4a4abb1091c631dab7a5 (table default.Table_85072f20d90d4470bf3f65189b19fd51 [id=0fd2754282af4b2b9fcc7952be704cd3]) successfully deleted
I20240430 20:11:42.282290 19471 rpcz_store.cc:276] 0430 20:11:32.168850 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:32.168918 (+ 68us) service_pool.cc:225] Handling call
0430 20:11:42.282113 (+10113195us) ts_tablet_manager.cc:1120] Deleting tablet d4508e4f6dbd42109d951c648b3dc7df
0430 20:11:42.282124 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:42.282196 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":10112938}
I20240430 20:11:42.282732 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 114) took 10113 ms. Trace:
I20240430 20:11:42.282806 19471 rpcz_store.cc:276] 0430 20:11:32.169023 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:32.169428 (+ 405us) service_pool.cc:225] Handling call
0430 20:11:42.282657 (+10113229us) ts_tablet_manager.cc:1120] Deleting tablet 68ca48351cd4412381f5195d81109016
0430 20:11:42.282665 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:42.282717 (+ 52us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":10113045}
W20240430 20:11:42.283063 18999 catalog_manager.cc:4797] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): delete failed for tablet d4508e4f6dbd42109d951c648b3dc7df because the tablet was not found. No further retry: Not found: Tablet not found: d4508e4f6dbd42109d951c648b3dc7df
W20240430 20:11:42.283592 18999 catalog_manager.cc:4797] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): delete failed for tablet 68ca48351cd4412381f5195d81109016 because the tablet was not found. No further retry: Not found: Tablet not found: 68ca48351cd4412381f5195d81109016
I20240430 20:11:42.286152 18985 tablet_bootstrap.cc:654] T abdaf243e5be4ad58036b23151b14fe4 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:42.286727 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 105) took 10139 ms. Trace:
I20240430 20:11:42.286828 19471 rpcz_store.cc:276] 0430 20:11:32.147024 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:32.147489 (+ 465us) service_pool.cc:225] Handling call
0430 20:11:42.283014 (+10135525us) ts_tablet_manager.cc:1120] Deleting tablet d9e578c99058424a9b86390641f9c34f
0430 20:11:42.283022 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:42.286703 (+ 3681us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":10113250}
I20240430 20:11:42.287168 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 108) took 10139 ms. Trace:
W20240430 20:11:42.287304 18999 catalog_manager.cc:4797] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): delete failed for tablet d9e578c99058424a9b86390641f9c34f because the tablet was not found. No further retry: Not found: Tablet not found: d9e578c99058424a9b86390641f9c34f
I20240430 20:11:42.287243 19471 rpcz_store.cc:276] 0430 20:11:32.147317 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:32.147361 (+ 44us) service_pool.cc:225] Handling call
0430 20:11:42.287093 (+10139732us) ts_tablet_manager.cc:1120] Deleting tablet c5fd0730133c472299c93f4b0830e527
0430 20:11:42.287102 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:42.287152 (+ 50us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":10116736}
I20240430 20:11:42.287516 19471 tablet_replica.cc:330] T 0a9066d895744c5bb7d9532b2940603c P 5e1c6397f09747f091534faf93542710: stopping tablet replica
W20240430 20:11:42.287793 18999 catalog_manager.cc:4797] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): delete failed for tablet c5fd0730133c472299c93f4b0830e527 because the tablet was not found. No further retry: Not found: Tablet not found: c5fd0730133c472299c93f4b0830e527
I20240430 20:11:42.291568 19489 raft_consensus.cc:2258] T 7542c7fe6cad452ab718f9ef54dbbbbe P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:42.291699 19471 raft_consensus.cc:2229] T 0a9066d895744c5bb7d9532b2940603c P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:42.291904 19471 raft_consensus.cc:2258] T 0a9066d895744c5bb7d9532b2940603c P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:42.293241 19471 ts_tablet_manager.cc:1875] T 0a9066d895744c5bb7d9532b2940603c P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:42.294072 18762 raft_consensus.cc:2454] T e3e74e76ab2b4fe492ffcc1838656fd1 P 3c51c33d5616482cafb68e26688820f4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 2.
I20240430 20:11:42.294410 18750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84568f90b9c54cf292099808d4cbf5dc" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:42.295076 18557 leader_election.cc:302] T e3e74e76ab2b4fe492ffcc1838656fd1 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a
I20240430 20:11:42.295192 19489 ts_tablet_manager.cc:1875] T 7542c7fe6cad452ab718f9ef54dbbbbe P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:42.295243 18985 tablet_bootstrap.cc:492] T abdaf243e5be4ad58036b23151b14fe4 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:42.295382 18985 ts_tablet_manager.cc:1367] T abdaf243e5be4ad58036b23151b14fe4 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.002s
W20240430 20:11:42.295500 18557 leader_election.cc:341] T 84568f90b9c54cf292099808d4cbf5dc P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:42.295876 20362 raft_consensus.cc:482] T 34130f2820064fa0898ef1f9d7de1c6d P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:42.295913 20047 raft_consensus.cc:2781] T e3e74e76ab2b4fe492ffcc1838656fd1 P 5e1c6397f09747f091534faf93542710 [term 2 FOLLOWER]: Leader election won for term 2
I20240430 20:11:42.296018 20047 raft_consensus.cc:686] T e3e74e76ab2b4fe492ffcc1838656fd1 P 5e1c6397f09747f091534faf93542710 [term 2 LEADER]: Becoming Leader. State: Replica: 5e1c6397f09747f091534faf93542710, State: Running, Role: LEADER
I20240430 20:11:42.296010 20362 raft_consensus.cc:504] T 34130f2820064fa0898ef1f9d7de1c6d P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:42.296255 20047 consensus_queue.cc:237] T e3e74e76ab2b4fe492ffcc1838656fd1 P 5e1c6397f09747f091534faf93542710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:42.296605 20362 leader_election.cc:288] T 34130f2820064fa0898ef1f9d7de1c6d P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:42.297636 18617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34130f2820064fa0898ef1f9d7de1c6d" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:42.297864 18617 raft_consensus.cc:2454] T 34130f2820064fa0898ef1f9d7de1c6d P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 0.
I20240430 20:11:42.297819 18750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34130f2820064fa0898ef1f9d7de1c6d" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:42.298306 18415 leader_election.cc:302] T 34130f2820064fa0898ef1f9d7de1c6d P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
W20240430 20:11:42.298446 18417 leader_election.cc:341] T 34130f2820064fa0898ef1f9d7de1c6d P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:42.298619 20362 raft_consensus.cc:2781] T 34130f2820064fa0898ef1f9d7de1c6d P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:42.298684 20362 raft_consensus.cc:482] T 34130f2820064fa0898ef1f9d7de1c6d P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:42.298745 20362 raft_consensus.cc:3037] T 34130f2820064fa0898ef1f9d7de1c6d P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:42.301380 20362 raft_consensus.cc:504] T 34130f2820064fa0898ef1f9d7de1c6d P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:42.302150 20362 leader_election.cc:288] T 34130f2820064fa0898ef1f9d7de1c6d P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:42.303009 18608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34130f2820064fa0898ef1f9d7de1c6d" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:42.303200 18608 raft_consensus.cc:3037] T 34130f2820064fa0898ef1f9d7de1c6d P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:42.305903 18608 raft_consensus.cc:2454] T 34130f2820064fa0898ef1f9d7de1c6d P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 1.
I20240430 20:11:42.306455 18415 leader_election.cc:302] T 34130f2820064fa0898ef1f9d7de1c6d P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:42.307255 20362 raft_consensus.cc:2781] T 34130f2820064fa0898ef1f9d7de1c6d P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:42.307413 20362 raft_consensus.cc:686] T 34130f2820064fa0898ef1f9d7de1c6d P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Becoming Leader. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Running, Role: LEADER
I20240430 20:11:42.307410 18750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34130f2820064fa0898ef1f9d7de1c6d" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:42.307857 19471 ts_tablet_manager.cc:1888] T 0a9066d895744c5bb7d9532b2940603c P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:42.307783 20362 consensus_queue.cc:237] T 34130f2820064fa0898ef1f9d7de1c6d P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:42.307978 19471 log.cc:1196] T 0a9066d895744c5bb7d9532b2940603c P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/0a9066d895744c5bb7d9532b2940603c
I20240430 20:11:42.308372 19471 ts_tablet_manager.cc:1909] T 0a9066d895744c5bb7d9532b2940603c P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
W20240430 20:11:42.308868 18417 leader_election.cc:341] T 34130f2820064fa0898ef1f9d7de1c6d P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
W20240430 20:11:42.308908 18696 consensus_peers.cc:485] T 0a9066d895744c5bb7d9532b2940603c P 3c51c33d5616482cafb68e26688820f4 -> Peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Couldn't send request to peer 5e1c6397f09747f091534faf93542710. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240430 20:11:42.309110 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 106) took 10162 ms. Trace:
I20240430 20:11:42.309247 20198 raft_consensus.cc:482] T b72fe1f0713b4076a81a3fc1c74f7fa7 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2efb542aa55e439b99ccb98d1c5c884a)
I20240430 20:11:42.309406 20198 raft_consensus.cc:504] T b72fe1f0713b4076a81a3fc1c74f7fa7 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } }
I20240430 20:11:42.310024 20198 leader_election.cc:288] T b72fe1f0713b4076a81a3fc1c74f7fa7 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:42.311118 18617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b72fe1f0713b4076a81a3fc1c74f7fa7" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:42.311451 18617 raft_consensus.cc:2379] T b72fe1f0713b4076a81a3fc1c74f7fa7 P 5e1c6397f09747f091534faf93542710 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c51c33d5616482cafb68e26688820f4 in current term 2: Already voted for candidate 5e1c6397f09747f091534faf93542710 in this term.
W20240430 20:11:42.311877 18699 leader_election.cc:341] T b72fe1f0713b4076a81a3fc1c74f7fa7 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Not found: Tablet not found: b72fe1f0713b4076a81a3fc1c74f7fa7
I20240430 20:11:42.312217 18696 leader_election.cc:302] T b72fe1f0713b4076a81a3fc1c74f7fa7 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710
I20240430 20:11:42.312794 20198 raft_consensus.cc:3037] T b72fe1f0713b4076a81a3fc1c74f7fa7 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Advancing to term 2
I20240430 20:11:42.315745 20198 raft_consensus.cc:2726] T b72fe1f0713b4076a81a3fc1c74f7fa7 P 3c51c33d5616482cafb68e26688820f4 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240430 20:11:42.317520 19495 ts_tablet_manager.cc:1888] T a745334c12c545c1ba825c7552c60359 P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:42.319918 19495 log.cc:1196] T a745334c12c545c1ba825c7552c60359 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/a745334c12c545c1ba825c7552c60359
I20240430 20:11:42.320504 19495 ts_tablet_manager.cc:1909] T a745334c12c545c1ba825c7552c60359 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:42.321241 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 163) took 8115 ms. Trace:
I20240430 20:11:42.321738 19495 rpcz_store.cc:276] 0430 20:11:34.205401 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:34.205521 (+ 120us) service_pool.cc:225] Handling call
0430 20:11:42.267157 (+8061636us) ts_tablet_manager.cc:1120] Deleting tablet a745334c12c545c1ba825c7552c60359
0430 20:11:42.267165 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:42.273604 (+ 6439us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:42.317401 (+ 43797us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:42.321219 (+ 3818us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7476870}
I20240430 20:11:42.322546 19004 catalog_manager.cc:4830] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): tablet a745334c12c545c1ba825c7552c60359 (table default.Table_36adeb442f1245aba00549559bb329f3 [id=4ab2e4b038744db8b2cc94c26dbcc35f]) successfully deleted
I20240430 20:11:42.318759 18985 raft_consensus.cc:348] T abdaf243e5be4ad58036b23151b14fe4 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:42.322959 18985 raft_consensus.cc:374] T abdaf243e5be4ad58036b23151b14fe4 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:42.323057 18985 raft_consensus.cc:729] T abdaf243e5be4ad58036b23151b14fe4 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:42.323340 18985 consensus_queue.cc:260] T abdaf243e5be4ad58036b23151b14fe4 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:42.323936 18985 ts_tablet_manager.cc:1398] T abdaf243e5be4ad58036b23151b14fe4 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.028s user 0.002s sys 0.000s
I20240430 20:11:42.324270 18985 tablet_bootstrap.cc:492] T 2cdccf4be7094ca5b863dd5c3432d55d P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:42.325675 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 181) took 8082 ms. Trace:
I20240430 20:11:42.326400 19026 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_575286687a4e4020a315d78080c2e28e" } new_table_name: "default.Table_7fb60e4b56f444009ab5da219238eec7" modify_external_catalogs: true
I20240430 20:11:42.326577 19051 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_cb067ab866334a64a0186737511a9855" } new_table_name: "default.Table_4493a13725004bf5a2996bc9369a0892" modify_external_catalogs: true
I20240430 20:11:42.326486 19495 rpcz_store.cc:276] 0430 20:11:34.242860 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:34.790319 (+547459us) service_pool.cc:225] Handling call
0430 20:11:42.325545 (+7535226us) ts_tablet_manager.cc:1120] Deleting tablet 50486644ec234933b9a0b45ee39a6685
0430 20:11:42.325555 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:42.325658 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7535031}
W20240430 20:11:42.327316 19004 catalog_manager.cc:4810] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): delete failed for tablet 50486644ec234933b9a0b45ee39a6685 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 50486644ec234933b9a0b45ee39a6685 already in progress: creating tablet
I20240430 20:11:42.309206 19471 rpcz_store.cc:276] 0430 20:11:32.147111 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:32.147571 (+ 460us) service_pool.cc:225] Handling call
0430 20:11:42.287470 (+10139899us) ts_tablet_manager.cc:1120] Deleting tablet 0a9066d895744c5bb7d9532b2940603c
0430 20:11:42.287481 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:42.304847 (+ 17366us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:42.307762 (+ 2915us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:42.309094 (+ 1332us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3988,"tablet-delete.queue_time_us":10116881}
I20240430 20:11:42.328235 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 179) took 8085 ms. Trace:
I20240430 20:11:42.328348 19033 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_6f80b003810342a6b03c6dbb3c955772" } new_table_name: "default.Table_df19cb8a6f4d46699350ec5b69c14e5a" modify_external_catalogs: true
I20240430 20:11:42.328765 19471 tablet_replica.cc:330] T 75ebf051383e42529dcb5f7dd0770a4b P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:42.329026 19471 raft_consensus.cc:2229] T 75ebf051383e42529dcb5f7dd0770a4b P 5e1c6397f09747f091534faf93542710 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:42.329262 19471 raft_consensus.cc:2258] T 75ebf051383e42529dcb5f7dd0770a4b P 5e1c6397f09747f091534faf93542710 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:42.328624 19495 rpcz_store.cc:276] 0430 20:11:34.242435 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:34.246645 (+ 4210us) service_pool.cc:225] Handling call
0430 20:11:42.327831 (+8081186us) ts_tablet_manager.cc:1120] Deleting tablet 06d28b6f48a34b17b0d7fb3102df2f4c
0430 20:11:42.327843 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:42.328215 (+ 372us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6496567}
I20240430 20:11:42.330035 19471 ts_tablet_manager.cc:1875] T 75ebf051383e42529dcb5f7dd0770a4b P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:42.330390 18999 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet 0a9066d895744c5bb7d9532b2940603c (table default.Table_7b72c19b4765456dba543151abaf1d44 [id=ec7831570fea4ff2944bc6304fc17bc9]) successfully deleted
I20240430 20:11:42.330554 19495 tablet_replica.cc:330] T 0b4d9e860d7c45fd91d7a566605c9283 P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:42.331508 19495 raft_consensus.cc:2229] T 0b4d9e860d7c45fd91d7a566605c9283 P 2efb542aa55e439b99ccb98d1c5c884a [term 2 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:42.332271 18985 tablet_bootstrap.cc:654] T 2cdccf4be7094ca5b863dd5c3432d55d P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:42.332340 19495 raft_consensus.cc:2258] T 0b4d9e860d7c45fd91d7a566605c9283 P 2efb542aa55e439b99ccb98d1c5c884a [term 2 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:42.333565 19027 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_2ec9639eca4a47439b7859929cd705bc" } modify_external_catalogs: true
I20240430 20:11:42.333762 19027 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_2ec9639eca4a47439b7859929cd705bc" } modify_external_catalogs: true
I20240430 20:11:42.335268 19495 ts_tablet_manager.cc:1875] T 0b4d9e860d7c45fd91d7a566605c9283 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240430 20:11:42.340087 19004 catalog_manager.cc:4810] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): delete failed for tablet 06d28b6f48a34b17b0d7fb3102df2f4c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 06d28b6f48a34b17b0d7fb3102df2f4c already in progress: creating tablet
I20240430 20:11:42.340214 20198 consensus_queue.cc:1035] T a9e0659564894b0e8cb7f600bb07a085 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20240430 20:11:42.344488 19024 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_fd465030bff1492cb52f47c7e4d84403" } new_table_name: "default.Table_4d68c38d9ba7452a829f8318b2b267b8" modify_external_catalogs: true
I20240430 20:11:42.344602 20236 raft_consensus.cc:482] T 46251c08277e498da04a41d009310c62 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:42.344753 20236 raft_consensus.cc:504] T 46251c08277e498da04a41d009310c62 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:42.346009 19049 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_8a6b5152d20b4890918aedb454764ee9" } modify_external_catalogs: true
I20240430 20:11:42.346185 19049 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_8a6b5152d20b4890918aedb454764ee9" } modify_external_catalogs: true
I20240430 20:11:42.347350 20236 leader_election.cc:288] T 46251c08277e498da04a41d009310c62 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:42.347035 18608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46251c08277e498da04a41d009310c62" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:42.348217 18608 raft_consensus.cc:2454] T 46251c08277e498da04a41d009310c62 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 0.
I20240430 20:11:42.348608 18415 leader_election.cc:302] T 46251c08277e498da04a41d009310c62 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:42.348871 20236 raft_consensus.cc:2781] T 46251c08277e498da04a41d009310c62 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:42.348946 20236 raft_consensus.cc:482] T 46251c08277e498da04a41d009310c62 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:42.348999 20236 raft_consensus.cc:3037] T 46251c08277e498da04a41d009310c62 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:42.350373 18750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46251c08277e498da04a41d009310c62" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
W20240430 20:11:42.352505 18558 consensus_peers.cc:485] T fae9a35a2ac844b4bc2ebdf214aa7171 P 5e1c6397f09747f091534faf93542710 -> Peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Couldn't send request to peer 2efb542aa55e439b99ccb98d1c5c884a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fae9a35a2ac844b4bc2ebdf214aa7171. This is attempt 1: this message will repeat every 5th retry.
I20240430 20:11:42.353615 20236 raft_consensus.cc:504] T 46251c08277e498da04a41d009310c62 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
W20240430 20:11:42.355067 18417 leader_election.cc:341] T 46251c08277e498da04a41d009310c62 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:42.358691 18608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46251c08277e498da04a41d009310c62" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:42.358700 18750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46251c08277e498da04a41d009310c62" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:42.358919 18608 raft_consensus.cc:3037] T 46251c08277e498da04a41d009310c62 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:42.361588 18608 raft_consensus.cc:2454] T 46251c08277e498da04a41d009310c62 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 1.
I20240430 20:11:42.362094 18415 leader_election.cc:302] T 46251c08277e498da04a41d009310c62 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:42.362401 20105 raft_consensus.cc:2781] T 46251c08277e498da04a41d009310c62 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:42.362522 20105 raft_consensus.cc:686] T 46251c08277e498da04a41d009310c62 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Becoming Leader. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Running, Role: LEADER
I20240430 20:11:42.362803 20105 consensus_queue.cc:237] T 46251c08277e498da04a41d009310c62 P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
W20240430 20:11:42.363584 19024 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 821c55edb3414d72b1489eea97d3f73a has no consensus state
I20240430 20:11:42.364023 19024 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_2d1cfc78363a47aa8dba7ef7d80f8c9a" } modify_external_catalogs: true
I20240430 20:11:42.364199 19024 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_2d1cfc78363a47aa8dba7ef7d80f8c9a" } modify_external_catalogs: true
W20240430 20:11:42.364349 18417 leader_election.cc:341] T 46251c08277e498da04a41d009310c62 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
W20240430 20:11:42.368263 18699 consensus_peers.cc:485] T a9e0659564894b0e8cb7f600bb07a085 P 3c51c33d5616482cafb68e26688820f4 -> Peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Couldn't send request to peer 2efb542aa55e439b99ccb98d1c5c884a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a9e0659564894b0e8cb7f600bb07a085. This is attempt 1: this message will repeat every 5th retry.
I20240430 20:11:42.371615 20236 leader_election.cc:288] T 46251c08277e498da04a41d009310c62 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:42.381693 18985 tablet_bootstrap.cc:492] T 2cdccf4be7094ca5b863dd5c3432d55d P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:42.382031 18983 tablet_bootstrap.cc:492] T ebf74809b52f47f3b0ed21b6ac0f541a P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:42.382267 18983 ts_tablet_manager.cc:1367] T ebf74809b52f47f3b0ed21b6ac0f541a P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.105s user 0.003s sys 0.000s
I20240430 20:11:42.382717 18985 ts_tablet_manager.cc:1367] T 2cdccf4be7094ca5b863dd5c3432d55d P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.059s user 0.005s sys 0.000s
I20240430 20:11:42.383121 20096 raft_consensus.cc:482] T c2b019f529a54228b804437bb025e31d P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:42.383268 20096 raft_consensus.cc:504] T c2b019f529a54228b804437bb025e31d P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:42.383985 20096 leader_election.cc:288] T c2b019f529a54228b804437bb025e31d P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:42.384721 18983 raft_consensus.cc:348] T ebf74809b52f47f3b0ed21b6ac0f541a P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
W20240430 20:11:42.385039 18558 leader_election.cc:341] T c2b019f529a54228b804437bb025e31d P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Not found: Tablet not found: c2b019f529a54228b804437bb025e31d
I20240430 20:11:42.385018 18983 raft_consensus.cc:374] T ebf74809b52f47f3b0ed21b6ac0f541a P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:42.385315 18983 raft_consensus.cc:729] T ebf74809b52f47f3b0ed21b6ac0f541a P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:42.385466 19471 ts_tablet_manager.cc:1888] T 75ebf051383e42529dcb5f7dd0770a4b P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.1
I20240430 20:11:42.388520 19471 log.cc:1196] T 75ebf051383e42529dcb5f7dd0770a4b P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/75ebf051383e42529dcb5f7dd0770a4b
I20240430 20:11:42.389124 19495 ts_tablet_manager.cc:1888] T 0b4d9e860d7c45fd91d7a566605c9283 P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.3
I20240430 20:11:42.389230 19495 log.cc:1196] T 0b4d9e860d7c45fd91d7a566605c9283 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/0b4d9e860d7c45fd91d7a566605c9283
I20240430 20:11:42.389568 19495 ts_tablet_manager.cc:1909] T 0b4d9e860d7c45fd91d7a566605c9283 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:42.385663 18983 consensus_queue.cc:260] T ebf74809b52f47f3b0ed21b6ac0f541a P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:42.389911 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 180) took 8147 ms. Trace:
I20240430 20:11:42.389992 19495 rpcz_store.cc:276] 0430 20:11:34.242702 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:34.643025 (+400323us) service_pool.cc:225] Handling call
0430 20:11:42.330001 (+7686976us) ts_tablet_manager.cc:1120] Deleting tablet 0b4d9e860d7c45fd91d7a566605c9283
0430 20:11:42.330166 (+ 165us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:42.385668 (+ 55502us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:42.389037 (+ 3369us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:42.389884 (+ 847us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6498459}
I20240430 20:11:42.390466 19004 catalog_manager.cc:4830] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): tablet 0b4d9e860d7c45fd91d7a566605c9283 (table default.Table_896bbae1f6944005b0272ca8ebc97218 [id=d5cea90cbc2c48df9d8b80da1eccc949]) successfully deleted
I20240430 20:11:42.391247 18985 raft_consensus.cc:348] T 2cdccf4be7094ca5b863dd5c3432d55d P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:42.391481 18983 ts_tablet_manager.cc:1398] T ebf74809b52f47f3b0ed21b6ac0f541a P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240430 20:11:42.391584 18985 raft_consensus.cc:374] T 2cdccf4be7094ca5b863dd5c3432d55d P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:42.391659 18985 raft_consensus.cc:729] T 2cdccf4be7094ca5b863dd5c3432d55d P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:42.391666 18758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2b019f529a54228b804437bb025e31d" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:42.391882 18983 tablet_bootstrap.cc:492] T 73453a7fa30342e29db552b17b62f0c1 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:42.391909 18985 consensus_queue.cc:260] T 2cdccf4be7094ca5b863dd5c3432d55d P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
W20240430 20:11:42.392169 18557 leader_election.cc:341] T c2b019f529a54228b804437bb025e31d P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:42.392333 18557 leader_election.cc:302] T c2b019f529a54228b804437bb025e31d P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:42.392498 18985 ts_tablet_manager.cc:1398] T 2cdccf4be7094ca5b863dd5c3432d55d P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20240430 20:11:42.392735 20096 raft_consensus.cc:2726] T c2b019f529a54228b804437bb025e31d P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:42.392768 18985 tablet_bootstrap.cc:492] T 50486644ec234933b9a0b45ee39a6685 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:42.394119 18985 tablet_bootstrap.cc:654] T 50486644ec234933b9a0b45ee39a6685 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:42.394119 18983 tablet_bootstrap.cc:654] T 73453a7fa30342e29db552b17b62f0c1 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:42.395191 19471 ts_tablet_manager.cc:1909] T 75ebf051383e42529dcb5f7dd0770a4b P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:42.395645 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 102) took 10248 ms. Trace:
I20240430 20:11:42.395785 19471 rpcz_store.cc:276] 0430 20:11:32.146706 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:32.147403 (+ 697us) service_pool.cc:225] Handling call
0430 20:11:42.328695 (+10181292us) ts_tablet_manager.cc:1120] Deleting tablet 75ebf051383e42529dcb5f7dd0770a4b
0430 20:11:42.328706 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:42.363721 (+ 35015us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:42.385351 (+ 21630us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:42.395625 (+ 10274us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":10157747}
I20240430 20:11:42.396466 18999 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet 75ebf051383e42529dcb5f7dd0770a4b (table default.Table_e9655558850747968a52e5885a113f59 [id=9c272f6ad56440b682f12f027af57bc4]) successfully deleted
I20240430 20:11:42.396837 19471 tablet_replica.cc:330] T f5546f9edeea44d7a6c5cb954f5157ba P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:42.397046 19471 raft_consensus.cc:2229] T f5546f9edeea44d7a6c5cb954f5157ba P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:42.398397 19471 raft_consensus.cc:2258] T f5546f9edeea44d7a6c5cb954f5157ba P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:42.399403 18985 tablet_bootstrap.cc:492] T 50486644ec234933b9a0b45ee39a6685 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:42.399538 18985 ts_tablet_manager.cc:1367] T 50486644ec234933b9a0b45ee39a6685 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240430 20:11:42.399997 19471 ts_tablet_manager.cc:1875] T f5546f9edeea44d7a6c5cb954f5157ba P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:42.400681 18736 tablet_service.cc:1467] Processing CreateTablet for tablet 9e741b00c74d43b5b6243ba7003f8804 (DEFAULT_TABLE table=default.Table_f2a19db5098448f79bf95cccb1abd368 [id=fcaf7a6481644ae8990aa4e3d473bd50]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:42.400830 18601 tablet_service.cc:1467] Processing CreateTablet for tablet 9e741b00c74d43b5b6243ba7003f8804 (DEFAULT_TABLE table=default.Table_f2a19db5098448f79bf95cccb1abd368 [id=fcaf7a6481644ae8990aa4e3d473bd50]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:42.400974 18449 tablet_service.cc:1467] Processing CreateTablet for tablet 9e741b00c74d43b5b6243ba7003f8804 (DEFAULT_TABLE table=default.Table_f2a19db5098448f79bf95cccb1abd368 [id=fcaf7a6481644ae8990aa4e3d473bd50]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:42.401093 18736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e741b00c74d43b5b6243ba7003f8804. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:42.401168 18601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e741b00c74d43b5b6243ba7003f8804. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:42.401289 18449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e741b00c74d43b5b6243ba7003f8804. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:42.402963 18733 tablet_service.cc:1467] Processing CreateTablet for tablet d34a450a91d542c98261456d1f635e9c (DEFAULT_TABLE table=default.Table_f30868d759f64cceb682278f0dbb51ae [id=3291bc6bb4f146ee8f90d89029a70aef]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:42.403366 18733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d34a450a91d542c98261456d1f635e9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:42.407409 18457 tablet_service.cc:1467] Processing CreateTablet for tablet fcc16db4d6be42edb7bd1ce18fe0190d (DEFAULT_TABLE table=default.Table_74b89f328a354c9a937566cdaf2a3377 [id=68c6513144ed425aa70cce409b50e176]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:42.407765 18457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcc16db4d6be42edb7bd1ce18fe0190d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:42.407941 18729 tablet_service.cc:1467] Processing CreateTablet for tablet fcc16db4d6be42edb7bd1ce18fe0190d (DEFAULT_TABLE table=default.Table_74b89f328a354c9a937566cdaf2a3377 [id=68c6513144ed425aa70cce409b50e176]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:42.408259 18729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcc16db4d6be42edb7bd1ce18fe0190d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:42.409566 18592 tablet_service.cc:1467] Processing CreateTablet for tablet fcc16db4d6be42edb7bd1ce18fe0190d (DEFAULT_TABLE table=default.Table_74b89f328a354c9a937566cdaf2a3377 [id=68c6513144ed425aa70cce409b50e176]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:42.410028 18592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcc16db4d6be42edb7bd1ce18fe0190d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:42.413097 19023 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet c9e7660eccc448dbb99d135f82bfd261 (table default.Table_685a26737bb046fb85ced15df7ae3423 [id=51d177e8498848b38c43e3f8d11f092c]). Expected version 1 got 0
I20240430 20:11:42.419078 19495 tablet_replica.cc:330] T c66c142419ca4e1399ecd1c8d1e62a59 P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:42.419324 19495 raft_consensus.cc:2229] T c66c142419ca4e1399ecd1c8d1e62a59 P 2efb542aa55e439b99ccb98d1c5c884a [term 2 LEADER]: Raft consensus shutting down.
I20240430 20:11:42.420361 19495 raft_consensus.cc:2258] T c66c142419ca4e1399ecd1c8d1e62a59 P 2efb542aa55e439b99ccb98d1c5c884a [term 2 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:42.421070 19495 ts_tablet_manager.cc:1875] T c66c142419ca4e1399ecd1c8d1e62a59 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:42.400261 18985 raft_consensus.cc:348] T 50486644ec234933b9a0b45ee39a6685 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:42.425185 18985 raft_consensus.cc:374] T 50486644ec234933b9a0b45ee39a6685 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:42.425262 18985 raft_consensus.cc:729] T 50486644ec234933b9a0b45ee39a6685 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:42.425722 18985 consensus_queue.cc:260] T 50486644ec234933b9a0b45ee39a6685 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:42.426963 18983 tablet_bootstrap.cc:492] T 73453a7fa30342e29db552b17b62f0c1 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:42.426970 18985 ts_tablet_manager.cc:1398] T 50486644ec234933b9a0b45ee39a6685 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.027s user 0.002s sys 0.000s
I20240430 20:11:42.427152 18983 ts_tablet_manager.cc:1367] T 73453a7fa30342e29db552b17b62f0c1 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.035s user 0.001s sys 0.003s
I20240430 20:11:42.427229 18985 tablet_bootstrap.cc:492] T 5bd0362ff41b4faca0205d136d596737 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:42.427809 18983 raft_consensus.cc:348] T 73453a7fa30342e29db552b17b62f0c1 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:42.428004 18983 raft_consensus.cc:374] T 73453a7fa30342e29db552b17b62f0c1 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:42.428056 18983 raft_consensus.cc:729] T 73453a7fa30342e29db552b17b62f0c1 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:42.428284 18983 consensus_queue.cc:260] T 73453a7fa30342e29db552b17b62f0c1 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:42.428720 18985 tablet_bootstrap.cc:654] T 5bd0362ff41b4faca0205d136d596737 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:42.428895 18983 ts_tablet_manager.cc:1398] T 73453a7fa30342e29db552b17b62f0c1 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240430 20:11:42.429159 18983 tablet_bootstrap.cc:492] T 5f02d8862e1f4ccfa9dbda73dbe18615 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:42.436275 19029 catalog_manager.cc:5484] T 0eb4f3e7ece6496293e01b63620d644b P 5e1c6397f09747f091534faf93542710 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e1c6397f09747f091534faf93542710 (127.17.177.2). New cstate: current_term: 1 leader_uuid: "5e1c6397f09747f091534faf93542710" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } health_report { overall_health: UNKNOWN } } }
I20240430 20:11:42.436952 19029 catalog_manager.cc:5484] T acf3b6fead41483a82c43dc0286fab7c P 5e1c6397f09747f091534faf93542710 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e1c6397f09747f091534faf93542710 (127.17.177.2). New cstate: current_term: 1 leader_uuid: "5e1c6397f09747f091534faf93542710" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } health_report { overall_health: UNKNOWN } } }
I20240430 20:11:42.437340 19029 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet acf3b6fead41483a82c43dc0286fab7c (table default.Table_01450a1e661d450a8cbe185e16e8250c [id=ce9911e0a91949b1997ee47321800255]). Expected version 1 got 0
I20240430 20:11:42.437580 19029 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 6e754af64cb64fd2a082adae7d5f24da (table default.Table_208b12b8e9bd4f3e8bc9c6d422ada649 [id=1df4e95702cd46f59ab21b101049a9ed]). Expected version 1 got 0
I20240430 20:11:42.439409 19495 ts_tablet_manager.cc:1888] T c66c142419ca4e1399ecd1c8d1e62a59 P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.3
I20240430 20:11:42.439543 19495 log.cc:1196] T c66c142419ca4e1399ecd1c8d1e62a59 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/c66c142419ca4e1399ecd1c8d1e62a59
I20240430 20:11:42.439922 19495 ts_tablet_manager.cc:1909] T c66c142419ca4e1399ecd1c8d1e62a59 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:42.440105 18983 tablet_bootstrap.cc:654] T 5f02d8862e1f4ccfa9dbda73dbe18615 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:42.440229 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 190) took 6321 ms. Trace:
I20240430 20:11:42.440866 18457 tablet_service.cc:1514] Processing DeleteTablet for tablet 73453a7fa30342e29db552b17b62f0c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:32 UTC) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:42.441318 18449 tablet_service.cc:1514] Processing DeleteTablet for tablet fcc9698abda4491b98e1e879589913a8 with delete_type TABLET_DATA_DELETED (replaced by tablet de1dfe6c82d34a4d8d02e466c55327ff) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:42.441493 18457 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e826459c3f4437da81c9e6e09f0c22f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:32 UTC) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:42.441715 19471 ts_tablet_manager.cc:1888] T f5546f9edeea44d7a6c5cb954f5157ba P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:42.441812 19471 log.cc:1196] T f5546f9edeea44d7a6c5cb954f5157ba P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/f5546f9edeea44d7a6c5cb954f5157ba
I20240430 20:11:42.441932 18449 tablet_service.cc:1514] Processing DeleteTablet for tablet 75ebf051383e42529dcb5f7dd0770a4b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:27 UTC) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:42.442078 18457 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f61e3f88a6448dfa323204bd8d14cc5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:31 UTC) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:42.442389 18457 tablet_service.cc:1514] Processing DeleteTablet for tablet 7542c7fe6cad452ab718f9ef54dbbbbe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:29 UTC) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:42.442678 18457 tablet_service.cc:1514] Processing DeleteTablet for tablet 87ac32a0ddb84676adc83038ef664591 with delete_type TABLET_DATA_DELETED (replaced by tablet 331c031d44854d679a27d10d9151d0ea) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:42.442945 19471 ts_tablet_manager.cc:1909] T f5546f9edeea44d7a6c5cb954f5157ba P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:42.443078 18457 tablet_service.cc:1514] Processing DeleteTablet for tablet c2b019f529a54228b804437bb025e31d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:32 UTC) from {username='slave'} at 127.0.0.1:41680
W20240430 20:11:42.443360 19023 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.17.177.1:50353 (request call id 91) took 1039 ms (client timeout 1000 ms). Trace:
W20240430 20:11:42.443537 19023 rpcz_store.cc:269] 0430 20:11:41.403795 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:41.519797 (+116002us) service_pool.cc:225] Handling call
0430 20:11:42.443290 (+923493us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1546,"reported_tablets":15}
I20240430 20:11:42.443899 18985 tablet_bootstrap.cc:492] T 5bd0362ff41b4faca0205d136d596737 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:42.443905 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 111) took 10276 ms. Trace:
I20240430 20:11:42.443975 19471 rpcz_store.cc:276] 0430 20:11:32.167758 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:32.168162 (+ 404us) service_pool.cc:225] Handling call
0430 20:11:42.396790 (+10228628us) ts_tablet_manager.cc:1120] Deleting tablet f5546f9edeea44d7a6c5cb954f5157ba
0430 20:11:42.396799 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:42.424326 (+ 27527us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:42.441600 (+ 17274us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:42.443884 (+ 2284us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":938,"tablet-delete.queue_time_us":10223044}
I20240430 20:11:42.444540 18985 ts_tablet_manager.cc:1367] T 5bd0362ff41b4faca0205d136d596737 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.002s
I20240430 20:11:42.445200 20324 raft_consensus.cc:482] T ebf74809b52f47f3b0ed21b6ac0f541a P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:42.445334 20324 raft_consensus.cc:504] T ebf74809b52f47f3b0ed21b6ac0f541a P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:42.445983 20324 leader_election.cc:288] T ebf74809b52f47f3b0ed21b6ac0f541a P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:42.446910 18478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebf74809b52f47f3b0ed21b6ac0f541a" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:42.447081 18758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebf74809b52f47f3b0ed21b6ac0f541a" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:42.447153 18478 raft_consensus.cc:2454] T ebf74809b52f47f3b0ed21b6ac0f541a P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 0.
W20240430 20:11:42.447639 18557 leader_election.cc:341] T ebf74809b52f47f3b0ed21b6ac0f541a P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:42.448318 18558 leader_election.cc:302] T ebf74809b52f47f3b0ed21b6ac0f541a P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters: 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:42.448915 18985 raft_consensus.cc:348] T 5bd0362ff41b4faca0205d136d596737 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:42.449488 18729 tablet_service.cc:1467] Processing CreateTablet for tablet 17ddcaf971154a52b606bbf29a539bf0 (DEFAULT_TABLE table=default.Table_57208e9f76f14782a806f0a2971a3a58 [id=872c2faa4a464edd98a51df2e1b3f817]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:42.449568 20324 raft_consensus.cc:2781] T ebf74809b52f47f3b0ed21b6ac0f541a P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:42.449642 20324 raft_consensus.cc:482] T ebf74809b52f47f3b0ed21b6ac0f541a P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:42.449707 20324 raft_consensus.cc:3037] T ebf74809b52f47f3b0ed21b6ac0f541a P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:42.449657 18457 tablet_service.cc:1467] Processing CreateTablet for tablet d34a450a91d542c98261456d1f635e9c (DEFAULT_TABLE table=default.Table_f30868d759f64cceb682278f0dbb51ae [id=3291bc6bb4f146ee8f90d89029a70aef]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:42.449899 18729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17ddcaf971154a52b606bbf29a539bf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:42.450018 18457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d34a450a91d542c98261456d1f635e9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:42.452241 20324 raft_consensus.cc:504] T ebf74809b52f47f3b0ed21b6ac0f541a P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:42.452597 18999 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet f5546f9edeea44d7a6c5cb954f5157ba (table default.Table_aa42ff1beec14dd6baf999b8962907e0 [id=836a4da2ef3a455db26af8149ca5415b]) successfully deleted
I20240430 20:11:42.453259 18601 tablet_service.cc:1467] Processing CreateTablet for tablet d34a450a91d542c98261456d1f635e9c (DEFAULT_TABLE table=default.Table_f30868d759f64cceb682278f0dbb51ae [id=3291bc6bb4f146ee8f90d89029a70aef]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:42.453505 20324 leader_election.cc:288] T ebf74809b52f47f3b0ed21b6ac0f541a P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:42.453600 18601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d34a450a91d542c98261456d1f635e9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:42.453811 18758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebf74809b52f47f3b0ed21b6ac0f541a" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
W20240430 20:11:42.454289 18557 leader_election.cc:341] T ebf74809b52f47f3b0ed21b6ac0f541a P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:42.445983 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 112) took 10277 ms. Trace:
I20240430 20:11:42.454819 19471 rpcz_store.cc:276] 0430 20:11:32.168727 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:32.170020 (+ 1293us) service_pool.cc:225] Handling call
0430 20:11:42.445870 (+10275850us) ts_tablet_manager.cc:1120] Deleting tablet 1398d4d2305b48e08fdb12432bd6c4b0
0430 20:11:42.445881 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:42.445965 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":10271386}
I20240430 20:11:42.455219 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 110) took 10289 ms. Trace:
I20240430 20:11:42.455338 19471 rpcz_store.cc:276] 0430 20:11:32.166142 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:32.166201 (+ 59us) service_pool.cc:225] Handling call
0430 20:11:42.455137 (+10288936us) ts_tablet_manager.cc:1120] Deleting tablet aacabead3c874e77a47d2ecf19dbfa82
0430 20:11:42.455147 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:42.455205 (+ 58us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":10279880}
I20240430 20:11:42.455667 19471 tablet_replica.cc:330] T fae9a35a2ac844b4bc2ebdf214aa7171 P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:42.455926 19471 raft_consensus.cc:2229] T fae9a35a2ac844b4bc2ebdf214aa7171 P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Raft consensus shutting down.
I20240430 20:11:42.456198 18985 raft_consensus.cc:374] T 5bd0362ff41b4faca0205d136d596737 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:42.456283 18985 raft_consensus.cc:729] T 5bd0362ff41b4faca0205d136d596737 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:42.456723 19471 raft_consensus.cc:2258] T fae9a35a2ac844b4bc2ebdf214aa7171 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:42.456708 18985 consensus_queue.cc:260] T 5bd0362ff41b4faca0205d136d596737 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:42.457690 18985 ts_tablet_manager.cc:1398] T 5bd0362ff41b4faca0205d136d596737 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.013s user 0.001s sys 0.001s
I20240430 20:11:42.457970 18985 tablet_bootstrap.cc:492] T f7b0132c17d44a9ba17288e0321a7f11 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:42.458400 19471 ts_tablet_manager.cc:1875] T fae9a35a2ac844b4bc2ebdf214aa7171 P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:42.459519 18985 tablet_bootstrap.cc:654] T f7b0132c17d44a9ba17288e0321a7f11 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:42.460037 18730 tablet_service.cc:1467] Processing CreateTablet for tablet e041756c962b4a9f97c2cb314e7d8c05 (DEFAULT_TABLE table=default.Table_de88cafd92b149f9b545369bff71033c [id=c4bf4806f3d84579b16a3038d8d982c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:42.460446 18730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e041756c962b4a9f97c2cb314e7d8c05. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:42.461380 18983 tablet_bootstrap.cc:492] T 5f02d8862e1f4ccfa9dbda73dbe18615 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:42.461608 18734 tablet_service.cc:1467] Processing CreateTablet for tablet 94fbd4a57a1e438891fcb456f6b40be3 (DEFAULT_TABLE table=default.Table_43f360b4669c4bfd8d6849258825fac5 [id=4fa623ffdcf347639150a355806038af]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:42.461848 18983 ts_tablet_manager.cc:1367] T 5f02d8862e1f4ccfa9dbda73dbe18615 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.033s user 0.000s sys 0.005s
I20240430 20:11:42.461966 18734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94fbd4a57a1e438891fcb456f6b40be3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:42.462949 18983 raft_consensus.cc:348] T 5f02d8862e1f4ccfa9dbda73dbe18615 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:42.463375 18983 raft_consensus.cc:374] T 5f02d8862e1f4ccfa9dbda73dbe18615 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:42.463539 18983 raft_consensus.cc:729] T 5f02d8862e1f4ccfa9dbda73dbe18615 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:42.463642 18725 tablet_service.cc:1467] Processing CreateTablet for tablet cdeef95b54ca42d38d25b57738d84ef1 (DEFAULT_TABLE table=default.Table_8039a17f6e4b4e9d90222f99d53d90a1 [id=e05d019ab2ad44c8af71fb04b4a27fdd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:42.463965 18725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdeef95b54ca42d38d25b57738d84ef1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:42.463950 18983 consensus_queue.cc:260] T 5f02d8862e1f4ccfa9dbda73dbe18615 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:42.440316 19495 rpcz_store.cc:276] 0430 20:11:36.118826 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:37.456813 (+1337987us) service_pool.cc:225] Handling call
0430 20:11:42.419017 (+4962204us) ts_tablet_manager.cc:1120] Deleting tablet c66c142419ca4e1399ecd1c8d1e62a59
0430 20:11:42.419029 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:42.424332 (+ 5303us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:42.439301 (+ 14969us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:42.440208 (+ 907us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4961811}
I20240430 20:11:42.464990 18983 ts_tablet_manager.cc:1398] T 5f02d8862e1f4ccfa9dbda73dbe18615 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240430 20:11:42.465471 19495 tablet_replica.cc:330] T 84568f90b9c54cf292099808d4cbf5dc P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:42.465499 18983 tablet_bootstrap.cc:492] T e87fb2f0361d4aa3ada27bbdfa88e94c P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:42.465664 19495 raft_consensus.cc:2229] T 84568f90b9c54cf292099808d4cbf5dc P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:42.465811 19495 raft_consensus.cc:2258] T 84568f90b9c54cf292099808d4cbf5dc P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:42.466858 18724 tablet_service.cc:1467] Processing CreateTablet for tablet 8e7f08e9ac274bfa86763eec895bbfda (DEFAULT_TABLE table=default.Table_5721e7a6b1ef44a780ff57d59fa76f64 [id=01a774990824472a87414d89d98e1863]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:42.467207 18724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e7f08e9ac274bfa86763eec895bbfda. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:42.467219 19495 ts_tablet_manager.cc:1875] T 84568f90b9c54cf292099808d4cbf5dc P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:42.470768 19004 catalog_manager.cc:4830] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): tablet c66c142419ca4e1399ecd1c8d1e62a59 (table default.Table_0443359dbeff437fb6cffb0db33cbb30 [id=93c44abdf6b147228224388398cd7aeb]) successfully deleted
I20240430 20:11:42.471062 18448 tablet_service.cc:1467] Processing CreateTablet for tablet 94fbd4a57a1e438891fcb456f6b40be3 (DEFAULT_TABLE table=default.Table_43f360b4669c4bfd8d6849258825fac5 [id=4fa623ffdcf347639150a355806038af]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:42.471239 18457 tablet_service.cc:1467] Processing CreateTablet for tablet 17ddcaf971154a52b606bbf29a539bf0 (DEFAULT_TABLE table=default.Table_57208e9f76f14782a806f0a2971a3a58 [id=872c2faa4a464edd98a51df2e1b3f817]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:42.471299 18461 tablet_service.cc:1467] Processing CreateTablet for tablet e041756c962b4a9f97c2cb314e7d8c05 (DEFAULT_TABLE table=default.Table_de88cafd92b149f9b545369bff71033c [id=c4bf4806f3d84579b16a3038d8d982c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:42.471431 18448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94fbd4a57a1e438891fcb456f6b40be3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:42.471690 18461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e041756c962b4a9f97c2cb314e7d8c05. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:42.471885 18457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17ddcaf971154a52b606bbf29a539bf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:42.471904 18446 tablet_service.cc:1467] Processing CreateTablet for tablet 8e7f08e9ac274bfa86763eec895bbfda (DEFAULT_TABLE table=default.Table_5721e7a6b1ef44a780ff57d59fa76f64 [id=01a774990824472a87414d89d98e1863]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:42.472239 18446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e7f08e9ac274bfa86763eec895bbfda. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:42.471062 18449 tablet_service.cc:1467] Processing CreateTablet for tablet f9bf604e6c234349a9f72f45a4c4aef6 (DEFAULT_TABLE table=default.Table_9049c9eeb10540d3815bab49caa1e82e [id=619746ec6b3244ee9ba511745c38bc25]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:42.475551 18449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9bf604e6c234349a9f72f45a4c4aef6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:42.478724 18985 tablet_bootstrap.cc:492] T f7b0132c17d44a9ba17288e0321a7f11 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:42.478857 18985 ts_tablet_manager.cc:1367] T f7b0132c17d44a9ba17288e0321a7f11 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.000s
W20240430 20:11:42.479629 18558 consensus_peers.cc:485] T 0b4d9e860d7c45fd91d7a566605c9283 P 5e1c6397f09747f091534faf93542710 -> Peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Couldn't send request to peer 2efb542aa55e439b99ccb98d1c5c884a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0b4d9e860d7c45fd91d7a566605c9283. This is attempt 1: this message will repeat every 5th retry.
I20240430 20:11:42.481061 18985 raft_consensus.cc:348] T f7b0132c17d44a9ba17288e0321a7f11 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:42.481240 18985 raft_consensus.cc:374] T f7b0132c17d44a9ba17288e0321a7f11 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:42.481281 18985 raft_consensus.cc:729] T f7b0132c17d44a9ba17288e0321a7f11 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:42.482308 18985 consensus_queue.cc:260] T f7b0132c17d44a9ba17288e0321a7f11 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:42.483758 18478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebf74809b52f47f3b0ed21b6ac0f541a" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:42.483940 18478 raft_consensus.cc:3037] T ebf74809b52f47f3b0ed21b6ac0f541a P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:42.486348 18478 raft_consensus.cc:2454] T ebf74809b52f47f3b0ed21b6ac0f541a P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
I20240430 20:11:42.486739 18558 leader_election.cc:302] T ebf74809b52f47f3b0ed21b6ac0f541a P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters: 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:42.487064 20324 raft_consensus.cc:2781] T ebf74809b52f47f3b0ed21b6ac0f541a P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:42.487217 20324 raft_consensus.cc:686] T ebf74809b52f47f3b0ed21b6ac0f541a P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Becoming Leader. State: Replica: 5e1c6397f09747f091534faf93542710, State: Running, Role: LEADER
W20240430 20:11:42.490639 18999 catalog_manager.cc:4797] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): delete failed for tablet 1398d4d2305b48e08fdb12432bd6c4b0 because the tablet was not found. No further retry: Not found: Tablet not found: 1398d4d2305b48e08fdb12432bd6c4b0
W20240430 20:11:42.490954 18999 catalog_manager.cc:4797] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): delete failed for tablet aacabead3c874e77a47d2ecf19dbfa82 because the tablet was not found. No further retry: Not found: Tablet not found: aacabead3c874e77a47d2ecf19dbfa82
I20240430 20:11:42.491124 18729 tablet_service.cc:1467] Processing CreateTablet for tablet f9bf604e6c234349a9f72f45a4c4aef6 (DEFAULT_TABLE table=default.Table_9049c9eeb10540d3815bab49caa1e82e [id=619746ec6b3244ee9ba511745c38bc25]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:42.492172 19495 ts_tablet_manager.cc:1888] T 84568f90b9c54cf292099808d4cbf5dc P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:42.492283 19495 log.cc:1196] T 84568f90b9c54cf292099808d4cbf5dc P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/84568f90b9c54cf292099808d4cbf5dc
I20240430 20:11:42.493283 18448 tablet_service.cc:1467] Processing CreateTablet for tablet cdeef95b54ca42d38d25b57738d84ef1 (DEFAULT_TABLE table=default.Table_8039a17f6e4b4e9d90222f99d53d90a1 [id=e05d019ab2ad44c8af71fb04b4a27fdd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:42.493717 18448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdeef95b54ca42d38d25b57738d84ef1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:42.487493 20324 consensus_queue.cc:237] T ebf74809b52f47f3b0ed21b6ac0f541a P 5e1c6397f09747f091534faf93542710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:42.495368 18589 tablet_service.cc:1467] Processing CreateTablet for tablet f9bf604e6c234349a9f72f45a4c4aef6 (DEFAULT_TABLE table=default.Table_9049c9eeb10540d3815bab49caa1e82e [id=619746ec6b3244ee9ba511745c38bc25]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:42.495574 18586 tablet_service.cc:1467] Processing CreateTablet for tablet e041756c962b4a9f97c2cb314e7d8c05 (DEFAULT_TABLE table=default.Table_de88cafd92b149f9b545369bff71033c [id=c4bf4806f3d84579b16a3038d8d982c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:42.495723 18589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9bf604e6c234349a9f72f45a4c4aef6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:42.496034 18591 tablet_service.cc:1467] Processing CreateTablet for tablet 8e7f08e9ac274bfa86763eec895bbfda (DEFAULT_TABLE table=default.Table_5721e7a6b1ef44a780ff57d59fa76f64 [id=01a774990824472a87414d89d98e1863]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:42.496199 19495 ts_tablet_manager.cc:1909] T 84568f90b9c54cf292099808d4cbf5dc P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:42.496327 18591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e7f08e9ac274bfa86763eec895bbfda. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:42.496508 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 191) took 6377 ms. Trace:
I20240430 20:11:42.496605 19495 rpcz_store.cc:276] 0430 20:11:36.118992 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:37.457245 (+1338253us) service_pool.cc:225] Handling call
0430 20:11:42.465421 (+5008176us) ts_tablet_manager.cc:1120] Deleting tablet 84568f90b9c54cf292099808d4cbf5dc
0430 20:11:42.465431 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:42.484856 (+ 19425us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:42.492075 (+ 7219us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:42.496492 (+ 4417us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5007935}
I20240430 20:11:42.498072 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 192) took 6379 ms. Trace:
I20240430 20:11:42.498169 19495 rpcz_store.cc:276] 0430 20:11:36.119244 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:37.457498 (+1338254us) service_pool.cc:225] Handling call
0430 20:11:42.497914 (+5040416us) ts_tablet_manager.cc:1120] Deleting tablet 6db55bb510b24b0cae897936211e15f0
0430 20:11:42.497928 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:42.498051 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5040245}
I20240430 20:11:42.496335 18585 tablet_service.cc:1467] Processing CreateTablet for tablet 17ddcaf971154a52b606bbf29a539bf0 (DEFAULT_TABLE table=default.Table_57208e9f76f14782a806f0a2971a3a58 [id=872c2faa4a464edd98a51df2e1b3f817]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:42.498759 18603 tablet_service.cc:1467] Processing CreateTablet for tablet 94fbd4a57a1e438891fcb456f6b40be3 (DEFAULT_TABLE table=default.Table_43f360b4669c4bfd8d6849258825fac5 [id=4fa623ffdcf347639150a355806038af]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:42.498972 18585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17ddcaf971154a52b606bbf29a539bf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:42.499176 19495 tablet_replica.cc:330] T 680553d4d0bc4fcb839235c1832757e1 P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:42.499173 18588 tablet_service.cc:1467] Processing CreateTablet for tablet cdeef95b54ca42d38d25b57738d84ef1 (DEFAULT_TABLE table=default.Table_8039a17f6e4b4e9d90222f99d53d90a1 [id=e05d019ab2ad44c8af71fb04b4a27fdd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:42.499411 19495 raft_consensus.cc:2229] T 680553d4d0bc4fcb839235c1832757e1 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:42.499488 18588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdeef95b54ca42d38d25b57738d84ef1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:42.499559 19495 raft_consensus.cc:2258] T 680553d4d0bc4fcb839235c1832757e1 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:42.500057 19495 ts_tablet_manager.cc:1875] T 680553d4d0bc4fcb839235c1832757e1 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:42.502750 18586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e041756c962b4a9f97c2cb314e7d8c05. 1 dirs total, 0 dirs full, 0 dirs failed
W20240430 20:11:42.502818 18558 consensus_peers.cc:485] T a745334c12c545c1ba825c7552c60359 P 5e1c6397f09747f091534faf93542710 -> Peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Couldn't send request to peer 2efb542aa55e439b99ccb98d1c5c884a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a745334c12c545c1ba825c7552c60359. This is attempt 1: this message will repeat every 5th retry.
I20240430 20:11:42.503787 18603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94fbd4a57a1e438891fcb456f6b40be3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:42.506865 20096 consensus_queue.cc:1035] T 0eb4f3e7ece6496293e01b63620d644b P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:42.508803 18983 tablet_bootstrap.cc:654] T e87fb2f0361d4aa3ada27bbdfa88e94c P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:42.510294 19004 catalog_manager.cc:4830] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): tablet 84568f90b9c54cf292099808d4cbf5dc (table default.Table_0034641cfa4d42a68ff4341e87fcb396 [id=53c496365b43473fa7e61ca706cdd984]) successfully deleted
W20240430 20:11:42.510635 19004 catalog_manager.cc:4810] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): delete failed for tablet 6db55bb510b24b0cae897936211e15f0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6db55bb510b24b0cae897936211e15f0 already in progress: creating tablet
I20240430 20:11:42.516114 19023 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_4e56923d440944978f0aacc140a0dd8e" } new_table_name: "default.Table_59297b42e8cf4ab3b9e95e2e70d3c1fe" modify_external_catalogs: true
I20240430 20:11:42.523180 19495 ts_tablet_manager.cc:1888] T 680553d4d0bc4fcb839235c1832757e1 P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240430 20:11:42.523315 19495 log.cc:1196] T 680553d4d0bc4fcb839235c1832757e1 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/680553d4d0bc4fcb839235c1832757e1
I20240430 20:11:42.523659 19495 ts_tablet_manager.cc:1909] T 680553d4d0bc4fcb839235c1832757e1 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:42.524407 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 198) took 6392 ms. Trace:
I20240430 20:11:42.524510 19495 rpcz_store.cc:276] 0430 20:11:36.132031 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:38.488352 (+2356321us) service_pool.cc:225] Handling call
0430 20:11:42.499118 (+4010766us) ts_tablet_manager.cc:1120] Deleting tablet 680553d4d0bc4fcb839235c1832757e1
0430 20:11:42.499131 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:42.503023 (+ 3892us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:42.523056 (+ 20033us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:42.524381 (+ 1325us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4010540}
I20240430 20:11:42.535487 18988 tablet_bootstrap.cc:492] T 46251c08277e498da04a41d009310c62 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:42.535753 18988 ts_tablet_manager.cc:1367] T 46251c08277e498da04a41d009310c62 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.418s user 0.003s sys 0.013s
I20240430 20:11:42.536604 18988 raft_consensus.cc:348] T 46251c08277e498da04a41d009310c62 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:42.536895 18988 raft_consensus.cc:374] T 46251c08277e498da04a41d009310c62 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:42.537000 18988 raft_consensus.cc:729] T 46251c08277e498da04a41d009310c62 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:42.538550 20198 raft_consensus.cc:482] T aec9c09cc0454acab6006bb3e10982b7 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2efb542aa55e439b99ccb98d1c5c884a)
I20240430 20:11:42.538738 20198 raft_consensus.cc:504] T aec9c09cc0454acab6006bb3e10982b7 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:42.539541 20198 leader_election.cc:288] T aec9c09cc0454acab6006bb3e10982b7 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:42.540069 18725 tablet_service.cc:1467] Processing CreateTablet for tablet a37d917bcc45468f9e3ee805640fba98 (DEFAULT_TABLE table=default.Table_94da8b9b6d284bd28c2310417f57e095 [id=d42cd820755946d8928380af5476a908]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:42.540303 18478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aec9c09cc0454acab6006bb3e10982b7" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:42.540515 18478 raft_consensus.cc:2454] T aec9c09cc0454acab6006bb3e10982b7 P 2efb542aa55e439b99ccb98d1c5c884a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 2.
I20240430 20:11:42.541050 18699 leader_election.cc:302] T aec9c09cc0454acab6006bb3e10982b7 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters:
I20240430 20:11:42.541296 20198 raft_consensus.cc:2781] T aec9c09cc0454acab6006bb3e10982b7 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240430 20:11:42.541373 20198 raft_consensus.cc:482] T aec9c09cc0454acab6006bb3e10982b7 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 2efb542aa55e439b99ccb98d1c5c884a)
I20240430 20:11:42.541440 20198 raft_consensus.cc:3037] T aec9c09cc0454acab6006bb3e10982b7 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Advancing to term 2
I20240430 20:11:42.543663 18726 tablet_service.cc:1467] Processing CreateTablet for tablet 9a76db2c7ff146758c54908e428a0e02 (DEFAULT_TABLE table=default.Table_4991bbcb43c540629cbc285f0484d79d [id=79162f889e6340248053e0814ebb387a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:42.544148 20198 raft_consensus.cc:504] T aec9c09cc0454acab6006bb3e10982b7 P 3c51c33d5616482cafb68e26688820f4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:42.545675 18744 tablet_service.cc:1467] Processing CreateTablet for tablet 9cc9251abd4e40d59a2bd2d0fada6165 (DEFAULT_TABLE table=default.Table_77b10083620349b0bcbb86e5de19f2ce [id=756ff53a80c842c5b62c972f542a7f34]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:42.545693 18474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aec9c09cc0454acab6006bb3e10982b7" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:42.548781 18474 raft_consensus.cc:2454] T aec9c09cc0454acab6006bb3e10982b7 P 2efb542aa55e439b99ccb98d1c5c884a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 2.
I20240430 20:11:42.549415 18735 tablet_service.cc:1467] Processing CreateTablet for tablet de546e2d6a5142159f23bd50d78ff735 (DEFAULT_TABLE table=default.Table_401b6bfce5ed4b1bb5643b8a53269b60 [id=4a9083e60fa248c28b61b9a6ee49e9d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:42.550300 18699 leader_election.cc:302] T aec9c09cc0454acab6006bb3e10982b7 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters:
I20240430 20:11:42.550614 20019 raft_consensus.cc:2781] T aec9c09cc0454acab6006bb3e10982b7 P 3c51c33d5616482cafb68e26688820f4 [term 2 FOLLOWER]: Leader election won for term 2
I20240430 20:11:42.550676 20198 leader_election.cc:288] T aec9c09cc0454acab6006bb3e10982b7 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 2 election: Requested vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:42.550742 20019 raft_consensus.cc:686] T aec9c09cc0454acab6006bb3e10982b7 P 3c51c33d5616482cafb68e26688820f4 [term 2 LEADER]: Becoming Leader. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Running, Role: LEADER
I20240430 20:11:42.550998 20019 consensus_queue.cc:237] T aec9c09cc0454acab6006bb3e10982b7 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:42.551280 18617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aec9c09cc0454acab6006bb3e10982b7" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:42.551501 18617 raft_consensus.cc:2379] T aec9c09cc0454acab6006bb3e10982b7 P 5e1c6397f09747f091534faf93542710 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c51c33d5616482cafb68e26688820f4 in current term 2: Already voted for candidate 5e1c6397f09747f091534faf93542710 in this term.
I20240430 20:11:42.551736 18608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aec9c09cc0454acab6006bb3e10982b7" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:42.551990 18608 raft_consensus.cc:2379] T aec9c09cc0454acab6006bb3e10982b7 P 5e1c6397f09747f091534faf93542710 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 3c51c33d5616482cafb68e26688820f4 in current term 2: Already voted for candidate 5e1c6397f09747f091534faf93542710 in this term.
I20240430 20:11:42.553081 20096 consensus_queue.cc:1035] T 3fd0e8c360734dcbb75dc2958b70c2b8 P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:42.554577 18446 tablet_service.cc:1467] Processing CreateTablet for tablet 9a76db2c7ff146758c54908e428a0e02 (DEFAULT_TABLE table=default.Table_4991bbcb43c540629cbc285f0484d79d [id=79162f889e6340248053e0814ebb387a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:42.555166 18446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a76db2c7ff146758c54908e428a0e02. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:42.555467 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 204) took 3792 ms. Trace:
I20240430 20:11:42.556051 19495 rpcz_store.cc:276] 0430 20:11:38.762966 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:38.763018 (+ 52us) service_pool.cc:225] Handling call
0430 20:11:42.555349 (+3792331us) ts_tablet_manager.cc:1120] Deleting tablet 558e71ec370d46ca8fbdcec5b57eb2ec
0430 20:11:42.555357 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:42.555451 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3792111}
I20240430 20:11:42.556825 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 203) took 3794 ms. Trace:
I20240430 20:11:42.556913 19495 rpcz_store.cc:276] 0430 20:11:38.762793 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:38.766651 (+ 3858us) service_pool.cc:225] Handling call
0430 20:11:42.556725 (+3790074us) ts_tablet_manager.cc:1120] Deleting tablet e87fb2f0361d4aa3ada27bbdfa88e94c
0430 20:11:42.556734 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:42.556811 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3781442}
I20240430 20:11:42.557637 19495 tablet_replica.cc:330] T b76a4105a2974fe280e759349037d027 P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:42.554577 18449 tablet_service.cc:1467] Processing CreateTablet for tablet a37d917bcc45468f9e3ee805640fba98 (DEFAULT_TABLE table=default.Table_94da8b9b6d284bd28c2310417f57e095 [id=d42cd820755946d8928380af5476a908]), partition=RANGE (key) PARTITION UNBOUNDED
W20240430 20:11:42.558048 19004 catalog_manager.cc:4810] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): delete failed for tablet 558e71ec370d46ca8fbdcec5b57eb2ec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 558e71ec370d46ca8fbdcec5b57eb2ec already in progress: creating tablet
I20240430 20:11:42.558179 18449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a37d917bcc45468f9e3ee805640fba98. 1 dirs total, 0 dirs full, 0 dirs failed
W20240430 20:11:42.558387 19004 catalog_manager.cc:4810] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): delete failed for tablet e87fb2f0361d4aa3ada27bbdfa88e94c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e87fb2f0361d4aa3ada27bbdfa88e94c already in progress: creating tablet
I20240430 20:11:42.560398 18588 tablet_service.cc:1467] Processing CreateTablet for tablet 9a76db2c7ff146758c54908e428a0e02 (DEFAULT_TABLE table=default.Table_4991bbcb43c540629cbc285f0484d79d [id=79162f889e6340248053e0814ebb387a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:42.560827 18588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a76db2c7ff146758c54908e428a0e02. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:42.566869 20362 raft_consensus.cc:482] T e3e74e76ab2b4fe492ffcc1838656fd1 P 2efb542aa55e439b99ccb98d1c5c884a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:42.568015 20362 raft_consensus.cc:504] T e3e74e76ab2b4fe492ffcc1838656fd1 P 2efb542aa55e439b99ccb98d1c5c884a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:42.569214 18608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3e74e76ab2b4fe492ffcc1838656fd1" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:42.570456 20362 leader_election.cc:288] T e3e74e76ab2b4fe492ffcc1838656fd1 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:42.572084 18592 tablet_service.cc:1467] Processing CreateTablet for tablet 9cc9251abd4e40d59a2bd2d0fada6165 (DEFAULT_TABLE table=default.Table_77b10083620349b0bcbb86e5de19f2ce [id=756ff53a80c842c5b62c972f542a7f34]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:42.572670 18592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cc9251abd4e40d59a2bd2d0fada6165. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:42.572652 19495 raft_consensus.cc:2229] T b76a4105a2974fe280e759349037d027 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:42.572840 19495 raft_consensus.cc:2258] T b76a4105a2974fe280e759349037d027 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:42.574010 19495 ts_tablet_manager.cc:1875] T b76a4105a2974fe280e759349037d027 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:42.579173 18735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de546e2d6a5142159f23bd50d78ff735. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:42.579224 18461 tablet_service.cc:1467] Processing CreateTablet for tablet 9cc9251abd4e40d59a2bd2d0fada6165 (DEFAULT_TABLE table=default.Table_77b10083620349b0bcbb86e5de19f2ce [id=756ff53a80c842c5b62c972f542a7f34]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:42.579654 18461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cc9251abd4e40d59a2bd2d0fada6165. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:42.580293 18726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a76db2c7ff146758c54908e428a0e02. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:42.580210 18750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3e74e76ab2b4fe492ffcc1838656fd1" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:42.580475 18750 raft_consensus.cc:2454] T e3e74e76ab2b4fe492ffcc1838656fd1 P 3c51c33d5616482cafb68e26688820f4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 2.
I20240430 20:11:42.581101 18744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cc9251abd4e40d59a2bd2d0fada6165. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:42.581517 18725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a37d917bcc45468f9e3ee805640fba98. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:42.582768 18417 leader_election.cc:302] T e3e74e76ab2b4fe492ffcc1838656fd1 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters: 5e1c6397f09747f091534faf93542710
I20240430 20:11:42.587360 20105 raft_consensus.cc:2781] T e3e74e76ab2b4fe492ffcc1838656fd1 P 2efb542aa55e439b99ccb98d1c5c884a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240430 20:11:42.587447 20105 raft_consensus.cc:482] T e3e74e76ab2b4fe492ffcc1838656fd1 P 2efb542aa55e439b99ccb98d1c5c884a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:42.587522 20105 raft_consensus.cc:3037] T e3e74e76ab2b4fe492ffcc1838656fd1 P 2efb542aa55e439b99ccb98d1c5c884a [term 2 FOLLOWER]: Advancing to term 3
I20240430 20:11:42.589615 18729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9bf604e6c234349a9f72f45a4c4aef6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:42.590164 20105 raft_consensus.cc:504] T e3e74e76ab2b4fe492ffcc1838656fd1 P 2efb542aa55e439b99ccb98d1c5c884a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:42.591483 18608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3e74e76ab2b4fe492ffcc1838656fd1" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:42.592272 20105 leader_election.cc:288] T e3e74e76ab2b4fe492ffcc1838656fd1 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 3 election: Requested vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:42.592396 20105 raft_consensus.cc:482] T 7ce09fcbe1ac457a9c05efaf000c3263 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:42.592516 20105 raft_consensus.cc:504] T 7ce09fcbe1ac457a9c05efaf000c3263 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:42.593009 18601 tablet_service.cc:1467] Processing CreateTablet for tablet a37d917bcc45468f9e3ee805640fba98 (DEFAULT_TABLE table=default.Table_94da8b9b6d284bd28c2310417f57e095 [id=d42cd820755946d8928380af5476a908]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:42.593400 18601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a37d917bcc45468f9e3ee805640fba98. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:42.593708 18608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ce09fcbe1ac457a9c05efaf000c3263" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
W20240430 20:11:42.594264 18415 leader_election.cc:341] T 7ce09fcbe1ac457a9c05efaf000c3263 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:42.595360 18758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3e74e76ab2b4fe492ffcc1838656fd1" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:42.595552 18758 raft_consensus.cc:3037] T e3e74e76ab2b4fe492ffcc1838656fd1 P 3c51c33d5616482cafb68e26688820f4 [term 2 FOLLOWER]: Advancing to term 3
I20240430 20:11:42.595378 18750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ce09fcbe1ac457a9c05efaf000c3263" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
W20240430 20:11:42.599084 18417 leader_election.cc:341] T 7ce09fcbe1ac457a9c05efaf000c3263 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:42.599225 18417 leader_election.cc:302] T 7ce09fcbe1ac457a9c05efaf000c3263 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a; no voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710
I20240430 20:11:42.599526 20236 raft_consensus.cc:2726] T 7ce09fcbe1ac457a9c05efaf000c3263 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:42.600935 20105 leader_election.cc:288] T 7ce09fcbe1ac457a9c05efaf000c3263 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:42.600991 18988 consensus_queue.cc:260] T 46251c08277e498da04a41d009310c62 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:42.601406 18758 raft_consensus.cc:2454] T e3e74e76ab2b4fe492ffcc1838656fd1 P 3c51c33d5616482cafb68e26688820f4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 3.
I20240430 20:11:42.601747 18417 leader_election.cc:302] T e3e74e76ab2b4fe492ffcc1838656fd1 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters: 5e1c6397f09747f091534faf93542710
I20240430 20:11:42.602183 20105 raft_consensus.cc:2781] T e3e74e76ab2b4fe492ffcc1838656fd1 P 2efb542aa55e439b99ccb98d1c5c884a [term 3 FOLLOWER]: Leader election won for term 3
I20240430 20:11:42.602407 20105 raft_consensus.cc:686] T e3e74e76ab2b4fe492ffcc1838656fd1 P 2efb542aa55e439b99ccb98d1c5c884a [term 3 LEADER]: Becoming Leader. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Running, Role: LEADER
I20240430 20:11:42.602651 20105 consensus_queue.cc:237] T e3e74e76ab2b4fe492ffcc1838656fd1 P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:42.605180 18988 ts_tablet_manager.cc:1398] T 46251c08277e498da04a41d009310c62 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.069s user 0.001s sys 0.001s
I20240430 20:11:42.605485 18988 tablet_bootstrap.cc:492] T 87ac32a0ddb84676adc83038ef664591 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:42.483768 18985 ts_tablet_manager.cc:1398] T f7b0132c17d44a9ba17288e0321a7f11 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240430 20:11:42.613139 18985 tablet_bootstrap.cc:492] T 115c4a12440647a4bb46eb6824f7ea98 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:42.622668 18985 tablet_bootstrap.cc:654] T 115c4a12440647a4bb46eb6824f7ea98 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:42.626410 19028 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 3a03b6997921408cb5a65f4cae223e9c (table default.Table_744b5c96ea014bdb8feae740356a790e [id=83b1ac2e9c3d4a10b02f96ebb17faf44]). Expected version 1 got 0
I20240430 20:11:42.629359 19028 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet ae20f44df50343289b0a512a4ea53692 (table default.Table_1844f6b1bda84812825384e14a2fd94e [id=71da9acc3cff4d95bbac4a686b708fd5]). Expected version 1 got 0
I20240430 20:11:42.629949 19028 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet eb6c96047b114ebba73db7a371f2da8e (table default.Table_7d5f6f18cbd64c869739872a7dff9be2 [id=e67d619e36c0485f8f22b4a81cb4cb6c]). Expected version 1 got 0
I20240430 20:11:42.630360 19028 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 62732af95c6a4c749c5b726f0460dc72 (table default.Table_be2ebe77f859489f9cd3e912cb1edc8b [id=2d44ec94131148048021b2ee6366ba30]). Expected version 2 got 0
I20240430 20:11:42.631002 19028 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet a0baee91f5b44945b595d0e234af018e (table default.Table_6a30a0abb67d4a298f87216515a92153 [id=b3d1c2f97c4648518dbea83a46cadd85]). Expected version 1 got 0
I20240430 20:11:42.633003 18983 tablet_bootstrap.cc:492] T e87fb2f0361d4aa3ada27bbdfa88e94c P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:42.633143 18983 ts_tablet_manager.cc:1367] T e87fb2f0361d4aa3ada27bbdfa88e94c P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.168s user 0.002s sys 0.002s
I20240430 20:11:42.634124 19028 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet ed0c37a71ede4bfc9eb5ad1a4365f25e (table default.Table_a2e85efc6b454524b0b122102ec07002 [id=86d795a82e674d859104741ebd04f4e8]). Expected version 1 got 0
I20240430 20:11:42.634754 19028 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 3e5225f1f5f544ecbf8dfa470af3daca (table default.Table_c7c87a8e019944cab3ec99e9563ac01a [id=abf3765074eb4d7d9d2bcc4ab721e6bd]). Expected version 1 got 0
I20240430 20:11:42.635215 19028 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet a59ef6c545dd41a28265a4c1b8ce10e0 (table default.Table_07e40af8f1c546048975c8edf88170d7 [id=86df5d161d8641c89070e7eed9a39360]). Expected version 2 got 0
I20240430 20:11:42.636123 19028 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet c9e7660eccc448dbb99d135f82bfd261 (table default.Table_685a26737bb046fb85ced15df7ae3423 [id=51d177e8498848b38c43e3f8d11f092c]). Expected version 1 got 0
I20240430 20:11:42.636070 18983 raft_consensus.cc:348] T e87fb2f0361d4aa3ada27bbdfa88e94c P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:42.636297 18983 raft_consensus.cc:374] T e87fb2f0361d4aa3ada27bbdfa88e94c P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:42.636369 18983 raft_consensus.cc:729] T e87fb2f0361d4aa3ada27bbdfa88e94c P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:42.636569 18983 consensus_queue.cc:260] T e87fb2f0361d4aa3ada27bbdfa88e94c P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:42.637146 18983 ts_tablet_manager.cc:1398] T e87fb2f0361d4aa3ada27bbdfa88e94c P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240430 20:11:42.637440 18983 tablet_bootstrap.cc:492] T 2cdccf4be7094ca5b863dd5c3432d55d P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:42.637147 19028 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet f5eaf2413e1d4e69b56fae61f768f0ce (table default.Table_1280cfe32d5b41abb4d3eb1bd70641bf [id=8489b872074a42b6abb713e420968141]). Expected version 1 got 0
I20240430 20:11:42.637872 19028 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 66dd767bd3f8494f94061fb89e5e5680 (table default.Table_9c1e6182953140bb9cd594f08458c7d9 [id=5227206f44d04fa89b4842b899f33afb]). Expected version 1 got 0
I20240430 20:11:42.638911 18983 tablet_bootstrap.cc:654] T 2cdccf4be7094ca5b863dd5c3432d55d P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:42.640019 19028 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 22fef8bb6b034db098c3372ae992f7a7 (table default.Table_4440df38610340758047115845b42b95 [id=7d186ca20c094cc19aef792fe1c094e9]). Expected version 1 got 0
I20240430 20:11:42.640673 19028 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 086b3dfa84c045a18c8c4f177dce606a (table default.Table_6f80b003810342a6b03c6dbb3c955772 [id=197923633fc54a4582207942b51530e2]). Expected version 1 got 0
I20240430 20:11:42.641867 19028 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet dbb427aefa6b47b0805d1751fea4efcb (table default.Table_4130caf534004edfb413056d7f6cfd52 [id=da9ad991953a40c4b77b31317597ad1d]). Expected version 4 got 0
I20240430 20:11:42.642990 19028 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet acd23f0a63d5448aaa7ed78422e67bad (table default.Table_972f13be4dca4cd19b9a04534dc59f51 [id=69f37db4de634a3185613e40013e0ed7]). Expected version 1 got 0
I20240430 20:11:42.643837 19028 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 2e79df8744e74b5991a79ce978ff0202 (table default.Table_276bbdc7cfe542d1a2936485a2aad1ec [id=f146a15e6e8a4acbb4c734c1c863da57]). Expected version 1 got 0
I20240430 20:11:42.644357 19028 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 1050ab528c7d41559831652a1437d358 (table default.Table_acddc4cbf6334546b5fb3e4752fef2a5 [id=274cb15fd751410e83327ac4e8dc4629]). Expected version 2 got 0
I20240430 20:11:42.645165 19028 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 488d0e609b224674932f890939664df2 (table default.Table_58f35a2120874472978ca0e5f816e86c [id=ff18001c6d814afc8c1768d0c84a6bac]). Expected version 1 got 0
I20240430 20:11:42.645776 19028 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 69f88532952a4591bf1577541c3430e6 (table default.Table_1177828cad1f4a24bd397619dad96170 [id=a95fed1bc3fa4865bf9a35dfc51b659c]). Expected version 2 got 0
I20240430 20:11:42.646749 20205 raft_consensus.cc:482] T e2d90ff17e574f88b2061b1c117d4b7d P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2efb542aa55e439b99ccb98d1c5c884a)
I20240430 20:11:42.646894 20205 raft_consensus.cc:504] T e2d90ff17e574f88b2061b1c117d4b7d P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:42.646996 19028 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 76d6e9d547ca4a848eae852944b1d117 (table default.Table_104ae015924a48fd8d27220792f07470 [id=ff86c0357c164dd0991098c416087773]). Expected version 2 got 0
I20240430 20:11:42.647316 19028 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet fbdeb4714b8c4b93ab72822c43329aae (table default.Table_c12ca6938d424fbfa2e1df5cdb00fdbc [id=0fb93f36bc17408ea56bc4cafcfd8580]). Expected version 1 got 0
I20240430 20:11:42.647483 19028 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet acf3b6fead41483a82c43dc0286fab7c (table default.Table_01450a1e661d450a8cbe185e16e8250c [id=ce9911e0a91949b1997ee47321800255]). Expected version 1 got 0
I20240430 20:11:42.647543 20205 leader_election.cc:288] T e2d90ff17e574f88b2061b1c117d4b7d P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:42.647573 18983 tablet_bootstrap.cc:492] T 2cdccf4be7094ca5b863dd5c3432d55d P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:42.647709 18983 ts_tablet_manager.cc:1367] T 2cdccf4be7094ca5b863dd5c3432d55d P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.000s
I20240430 20:11:42.648512 18608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2d90ff17e574f88b2061b1c117d4b7d" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:42.648783 18608 raft_consensus.cc:2379] T e2d90ff17e574f88b2061b1c117d4b7d P 5e1c6397f09747f091534faf93542710 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c51c33d5616482cafb68e26688820f4 in current term 2: Already voted for candidate 5e1c6397f09747f091534faf93542710 in this term.
I20240430 20:11:42.647692 19028 catalog_manager.cc:5484] T 6e754af64cb64fd2a082adae7d5f24da P 3c51c33d5616482cafb68e26688820f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c51c33d5616482cafb68e26688820f4 (127.17.177.3). New cstate: current_term: 1 leader_uuid: "3c51c33d5616482cafb68e26688820f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } health_report { overall_health: UNKNOWN } } }
I20240430 20:11:42.649096 18482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2d90ff17e574f88b2061b1c117d4b7d" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:42.649327 18482 raft_consensus.cc:2454] T e2d90ff17e574f88b2061b1c117d4b7d P 2efb542aa55e439b99ccb98d1c5c884a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 2.
I20240430 20:11:42.649463 19028 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 6e754af64cb64fd2a082adae7d5f24da (table default.Table_208b12b8e9bd4f3e8bc9c6d422ada649 [id=1df4e95702cd46f59ab21b101049a9ed]). Expected version 1 got 0
W20240430 20:11:42.649669 18557 consensus_peers.cc:485] T 3fd0e8c360734dcbb75dc2958b70c2b8 P 5e1c6397f09747f091534faf93542710 -> Peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Couldn't send request to peer 3c51c33d5616482cafb68e26688820f4. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240430 20:11:42.649716 19028 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet aa9998f4e17147aa8f2b35c01980463a (table default.Table_ffb3192f443d437ab63f56f628e596ad [id=608890834da843fa9af0cfc0d548029e]). Expected version 2 got 0
I20240430 20:11:42.649981 19028 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet c6e3d09402574f1e8fe9e4160c70a3ee (table default.Table_656db97355f64d7c8aa29b1352dfe9c8 [id=450f5d1c7e084a1bb88d035c0ccfcf82]). Expected version 1 got 0
I20240430 20:11:42.649995 20324 consensus_queue.cc:1035] T 0eb4f3e7ece6496293e01b63620d644b P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:42.651051 18699 leader_election.cc:302] T e2d90ff17e574f88b2061b1c117d4b7d P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters: 5e1c6397f09747f091534faf93542710
I20240430 20:11:42.651408 19028 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 3ec3993042f5498b818bdf32d5087928 (table default.Table_ba82d0fed0264b818de8278e1d142e5d [id=27217d463b9c40ec926a1862a31f1f24]). Expected version 1 got 0
I20240430 20:11:42.651640 20205 raft_consensus.cc:2781] T e2d90ff17e574f88b2061b1c117d4b7d P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240430 20:11:42.651748 20205 raft_consensus.cc:482] T e2d90ff17e574f88b2061b1c117d4b7d P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 2efb542aa55e439b99ccb98d1c5c884a)
I20240430 20:11:42.651818 20205 raft_consensus.cc:3037] T e2d90ff17e574f88b2061b1c117d4b7d P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Advancing to term 2
I20240430 20:11:42.651984 19028 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 1462ecf1176c4e66be42abddbe36b00c (table default.Table_b49a7942aa8c4f58acb94c35d1cdbb70 [id=6257559b0e9545c58848a76af0c5a5f7]). Expected version 1 got 0
I20240430 20:11:42.652499 19028 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 78ba48dadef548708153878810cc46f8 (table default.Table_b9983e101a6547649eb44ee04b5e1147 [id=29769be2b9ec4ecba9de5c0a65692f85]). Expected version 1 got 0
I20240430 20:11:42.653110 19028 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 47bf64ecc41b445da4ff697e141243dd (table default.Table_f1ad29d5986d4b799cfc67afb2b0dfdf [id=838fc8b61a864dea8877a6f144e4cb4f]). Expected version 2 got 0
I20240430 20:11:42.653702 19028 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet ebf74809b52f47f3b0ed21b6ac0f541a (table default.Table_b2d07520b9944a89ad62fca70396174e [id=1c8e0a3a44ec4a798770c0165ed3dffa]). Expected version 1 got 0
I20240430 20:11:42.654462 19028 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 0368d37dd9e2452982fb81f539c6805b (table default.Table_404b186c544e4b30bb36cb77d1a14120 [id=4330626e342a48699ea939add5c91b11]). Expected version 1 got 0
I20240430 20:11:42.655160 19028 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 331c031d44854d679a27d10d9151d0ea (table default.Table_763f0800e8bb4fa69976886416250f42 [id=61c8a5b99ef34a0baeeb343fef7cfd05]). Expected version 1 got 0
I20240430 20:11:42.655629 19028 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 7ce09fcbe1ac457a9c05efaf000c3263 (table default.Table_edc4c27ed846454288f73f6a0d88cb69 [id=78fd3f516083489ab3b42d90dcef7e4d]). Expected version 1 got 0
I20240430 20:11:42.656443 19028 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 2da900b6bbfa4fa99fe8494fd2e68da8 (table default.Table_19a91a7f17ff456ebb145e8dd89c0940 [id=67283dd3995a44c6a7a7bc8722e4f955]). Expected version 1 got 0
I20240430 20:11:42.657348 19028 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 8e91920e7e36418d8c27d1f0bfd9b981 (table default.Table_e860191f401c434e8cf0085ef41952b9 [id=f393bcc9669c4ea985a2d40deb161f0c]). Expected version 1 got 0
I20240430 20:11:42.657742 19028 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 99203cc522de424184c6f97453f166fd (table default.Table_cb2a4b9ceaf4431facfff2c9bcf94c5d [id=3f8690a437ae49a3beb009fa7899d5d0]). Expected version 1 got 0
I20240430 20:11:42.658444 19028 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet a482f4622f774d60b4efd6827b8be7d7 (table default.Table_e686cdfcbcc046749fa5b51578a64794 [id=c3dc9f05316d46d6a3ef4d03f40877ec]). Expected version 1 got 0
I20240430 20:11:42.659289 19028 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 130007acc8ba4e709c3eb85fabe75903 (table default.Table_78987875650940c293247bc43db345ab [id=36f478a20a1f4a53a52fc29ade62b7f6]). Expected version 1 got 0
I20240430 20:11:42.660806 19028 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 12795670f21447a8964fe1efc6e92398 (table default.Table_896bbae1f6944005b0272ca8ebc97218 [id=d5cea90cbc2c48df9d8b80da1eccc949]). Expected version 3 got 0
I20240430 20:11:42.664749 19471 ts_tablet_manager.cc:1888] T fae9a35a2ac844b4bc2ebdf214aa7171 P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240430 20:11:42.665407 18448 tablet_service.cc:1467] Processing CreateTablet for tablet de546e2d6a5142159f23bd50d78ff735 (DEFAULT_TABLE table=default.Table_401b6bfce5ed4b1bb5643b8a53269b60 [id=4a9083e60fa248c28b61b9a6ee49e9d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:42.668030 18988 tablet_bootstrap.cc:654] T 87ac32a0ddb84676adc83038ef664591 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:42.672173 19028 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 46251c08277e498da04a41d009310c62 (table default.Table_4e56923d440944978f0aacc140a0dd8e [id=b1312a6cd1a54003b1db727a2a9d125d]). Expected version 1 got 0
I20240430 20:11:42.672262 19495 ts_tablet_manager.cc:1888] T b76a4105a2974fe280e759349037d027 P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:42.672364 19495 log.cc:1196] T b76a4105a2974fe280e759349037d027 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/b76a4105a2974fe280e759349037d027
I20240430 20:11:42.672742 19495 ts_tablet_manager.cc:1909] T b76a4105a2974fe280e759349037d027 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:42.673175 19471 log.cc:1196] T fae9a35a2ac844b4bc2ebdf214aa7171 P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/fae9a35a2ac844b4bc2ebdf214aa7171
I20240430 20:11:42.673640 19471 ts_tablet_manager.cc:1909] T fae9a35a2ac844b4bc2ebdf214aa7171 P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:42.666888 20397 tablet.cc:1722] T c9e7660eccc448dbb99d135f82bfd261 P 5e1c6397f09747f091534faf93542710: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240430 20:11:42.675989 19049 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: Sending DeleteTablet for 3 replicas of tablet dad16ae02b8b4e29a724d8eec67bb4b8
I20240430 20:11:42.667182 20205 raft_consensus.cc:504] T e2d90ff17e574f88b2061b1c117d4b7d P 3c51c33d5616482cafb68e26688820f4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:42.677450 20205 leader_election.cc:288] T e2d90ff17e574f88b2061b1c117d4b7d P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 2 election: Requested vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:42.677958 18482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2d90ff17e574f88b2061b1c117d4b7d" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:42.679630 18461 tablet_service.cc:1514] Processing DeleteTablet for tablet dad16ae02b8b4e29a724d8eec67bb4b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:42 UTC) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:42.680292 18736 tablet_service.cc:1514] Processing DeleteTablet for tablet dad16ae02b8b4e29a724d8eec67bb4b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:42 UTC) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:42.680771 18601 tablet_service.cc:1514] Processing DeleteTablet for tablet dad16ae02b8b4e29a724d8eec67bb4b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:42 UTC) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:42.680953 18482 raft_consensus.cc:2454] T e2d90ff17e574f88b2061b1c117d4b7d P 2efb542aa55e439b99ccb98d1c5c884a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 2.
I20240430 20:11:42.681442 18699 leader_election.cc:302] T e2d90ff17e574f88b2061b1c117d4b7d P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters:
I20240430 20:11:42.681643 18608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2d90ff17e574f88b2061b1c117d4b7d" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:42.681752 18461 tablet_service.cc:1467] Processing CreateTablet for tablet 798a13f545fd4d8f8c654eaa2452eda1 (DEFAULT_TABLE table=default.Table_fd33fc89e9d84f5c87a6572acdfe2c94 [id=2ab17149d0444085b85e5f0959ebb2ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:42.681856 18608 raft_consensus.cc:2379] T e2d90ff17e574f88b2061b1c117d4b7d P 5e1c6397f09747f091534faf93542710 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 3c51c33d5616482cafb68e26688820f4 in current term 2: Already voted for candidate 5e1c6397f09747f091534faf93542710 in this term.
I20240430 20:11:42.682195 20205 raft_consensus.cc:2781] T e2d90ff17e574f88b2061b1c117d4b7d P 3c51c33d5616482cafb68e26688820f4 [term 2 FOLLOWER]: Leader election won for term 2
I20240430 20:11:42.682341 20205 raft_consensus.cc:686] T e2d90ff17e574f88b2061b1c117d4b7d P 3c51c33d5616482cafb68e26688820f4 [term 2 LEADER]: Becoming Leader. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Running, Role: LEADER
I20240430 20:11:42.682621 18736 tablet_service.cc:1467] Processing CreateTablet for tablet 798a13f545fd4d8f8c654eaa2452eda1 (DEFAULT_TABLE table=default.Table_fd33fc89e9d84f5c87a6572acdfe2c94 [id=2ab17149d0444085b85e5f0959ebb2ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:42.682647 20205 consensus_queue.cc:237] T e2d90ff17e574f88b2061b1c117d4b7d P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:42.683046 18736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 798a13f545fd4d8f8c654eaa2452eda1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:42.685276 18983 raft_consensus.cc:348] T 2cdccf4be7094ca5b863dd5c3432d55d P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:42.685657 18448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de546e2d6a5142159f23bd50d78ff735. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:42.686681 18983 raft_consensus.cc:374] T 2cdccf4be7094ca5b863dd5c3432d55d P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:42.686760 18983 raft_consensus.cc:729] T 2cdccf4be7094ca5b863dd5c3432d55d P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:42.686972 18983 consensus_queue.cc:260] T 2cdccf4be7094ca5b863dd5c3432d55d P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:42.687620 18983 ts_tablet_manager.cc:1398] T 2cdccf4be7094ca5b863dd5c3432d55d P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.040s user 0.002s sys 0.000s
I20240430 20:11:42.688306 18588 tablet_service.cc:1467] Processing CreateTablet for tablet 266fd23786bc435db5083370f3330ed0 (DEFAULT_TABLE table=default.Table_cf60b652e9dc49e7a948ce66fd1cbffe [id=9734bb8dd05a400ca4244b3829f01dee]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:42.688709 18603 tablet_service.cc:1467] Processing CreateTablet for tablet ef40bfdb5bf84f54a0068f40c7d1617a (DEFAULT_TABLE table=default.Table_91c01cce23fe4f4182952019c19875c0 [id=b7251a2a9c91439db19f9b910ebb5a3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:42.669302 18592 tablet_service.cc:1467] Processing CreateTablet for tablet de546e2d6a5142159f23bd50d78ff735 (DEFAULT_TABLE table=default.Table_401b6bfce5ed4b1bb5643b8a53269b60 [id=4a9083e60fa248c28b61b9a6ee49e9d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:42.689416 18601 tablet_service.cc:1467] Processing CreateTablet for tablet 798a13f545fd4d8f8c654eaa2452eda1 (DEFAULT_TABLE table=default.Table_fd33fc89e9d84f5c87a6572acdfe2c94 [id=2ab17149d0444085b85e5f0959ebb2ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:42.690110 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 115) took 10370 ms. Trace:
I20240430 20:11:42.690192 19471 rpcz_store.cc:276] 0430 20:11:32.319530 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:32.320086 (+ 556us) service_pool.cc:225] Handling call
0430 20:11:42.455626 (+10135540us) ts_tablet_manager.cc:1120] Deleting tablet fae9a35a2ac844b4bc2ebdf214aa7171
0430 20:11:42.455633 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:42.513838 (+ 58205us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:42.664633 (+150795us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:42.690089 (+ 25456us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":10131563}
I20240430 20:11:42.690764 18457 tablet_service.cc:1467] Processing CreateTablet for tablet ef40bfdb5bf84f54a0068f40c7d1617a (DEFAULT_TABLE table=default.Table_91c01cce23fe4f4182952019c19875c0 [id=b7251a2a9c91439db19f9b910ebb5a3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:42.691160 18457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef40bfdb5bf84f54a0068f40c7d1617a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:42.691689 18734 tablet_service.cc:1467] Processing CreateTablet for tablet 266fd23786bc435db5083370f3330ed0 (DEFAULT_TABLE table=default.Table_cf60b652e9dc49e7a948ce66fd1cbffe [id=9734bb8dd05a400ca4244b3829f01dee]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:42.691772 18591 tablet_service.cc:1467] Processing CreateTablet for tablet 6f3996b7e29941e391ea141dd6cc44b2 (DEFAULT_TABLE table=default.Table_da5216d025844afda5a83ada555bc120 [id=dc55b25c3ca14863a21042e805e3bf9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:42.692087 18734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 266fd23786bc435db5083370f3330ed0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:42.692116 18585 tablet_service.cc:1467] Processing CreateTablet for tablet 772eba5f24c549c791cad75565858285 (DEFAULT_TABLE table=default.Table_5fc77d5f60ae44f4bb4ca9f7f183672c [id=6ca223ca6354449c959d85ed2ac6a9c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:42.694896 18724 tablet_service.cc:1467] Processing CreateTablet for tablet 5b26319488644d6a8b6be60460a5e7af (DEFAULT_TABLE table=default.Table_69892a8264a4477db48ee6248758e046 [id=5a1c4a7be2fc465fac89a50854eb3322]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:42.695298 18724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b26319488644d6a8b6be60460a5e7af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:42.695355 18733 tablet_service.cc:1467] Processing CreateTablet for tablet ef40bfdb5bf84f54a0068f40c7d1617a (DEFAULT_TABLE table=default.Table_91c01cce23fe4f4182952019c19875c0 [id=b7251a2a9c91439db19f9b910ebb5a3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:42.695660 18733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef40bfdb5bf84f54a0068f40c7d1617a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:42.699008 18601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 798a13f545fd4d8f8c654eaa2452eda1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:42.700178 18457 tablet_service.cc:1467] Processing CreateTablet for tablet 772eba5f24c549c791cad75565858285 (DEFAULT_TABLE table=default.Table_5fc77d5f60ae44f4bb4ca9f7f183672c [id=6ca223ca6354449c959d85ed2ac6a9c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:42.700273 18586 tablet_service.cc:1467] Processing CreateTablet for tablet 5b26319488644d6a8b6be60460a5e7af (DEFAULT_TABLE table=default.Table_69892a8264a4477db48ee6248758e046 [id=5a1c4a7be2fc465fac89a50854eb3322]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:42.700557 18457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 772eba5f24c549c791cad75565858285. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:42.705497 18809 heartbeater.cc:347] Connected to a master server at 127.17.177.62:36441
I20240430 20:11:42.705744 18725 tablet_service.cc:1467] Processing CreateTablet for tablet 772eba5f24c549c791cad75565858285 (DEFAULT_TABLE table=default.Table_5fc77d5f60ae44f4bb4ca9f7f183672c [id=6ca223ca6354449c959d85ed2ac6a9c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:42.708601 18446 tablet_service.cc:1467] Processing CreateTablet for tablet 5b26319488644d6a8b6be60460a5e7af (DEFAULT_TABLE table=default.Table_69892a8264a4477db48ee6248758e046 [id=5a1c4a7be2fc465fac89a50854eb3322]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:42.709010 18446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b26319488644d6a8b6be60460a5e7af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:42.710944 18586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b26319488644d6a8b6be60460a5e7af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:42.711143 18449 tablet_service.cc:1467] Processing CreateTablet for tablet 266fd23786bc435db5083370f3330ed0 (DEFAULT_TABLE table=default.Table_cf60b652e9dc49e7a948ce66fd1cbffe [id=9734bb8dd05a400ca4244b3829f01dee]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:42.711838 18449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 266fd23786bc435db5083370f3330ed0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:42.714823 18603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef40bfdb5bf84f54a0068f40c7d1617a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:42.714952 20398 tablet.cc:1722] T c9e7660eccc448dbb99d135f82bfd261 P 2efb542aa55e439b99ccb98d1c5c884a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240430 20:11:42.716552 20395 raft_consensus.cc:482] T aec9c09cc0454acab6006bb3e10982b7 P 5e1c6397f09747f091534faf93542710 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:42.716714 20395 raft_consensus.cc:504] T aec9c09cc0454acab6006bb3e10982b7 P 5e1c6397f09747f091534faf93542710 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:42.717353 20395 leader_election.cc:288] T aec9c09cc0454acab6006bb3e10982b7 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:42.717990 18482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aec9c09cc0454acab6006bb3e10982b7" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:42.718189 18482 raft_consensus.cc:2454] T aec9c09cc0454acab6006bb3e10982b7 P 2efb542aa55e439b99ccb98d1c5c884a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 2.
I20240430 20:11:42.718634 18592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de546e2d6a5142159f23bd50d78ff735. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:42.718977 18983 tablet_bootstrap.cc:492] T 115c4a12440647a4bb46eb6824f7ea98 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:42.720297 18983 tablet_bootstrap.cc:654] T 115c4a12440647a4bb46eb6824f7ea98 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:42.721179 19471 tablet_replica.cc:330] T 44fa6917a85c46f3aaa363df46ea586b P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:42.721359 19471 raft_consensus.cc:2229] T 44fa6917a85c46f3aaa363df46ea586b P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:42.721531 19471 raft_consensus.cc:2258] T 44fa6917a85c46f3aaa363df46ea586b P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:42.722046 19471 ts_tablet_manager.cc:1875] T 44fa6917a85c46f3aaa363df46ea586b P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:42.721359 18585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 772eba5f24c549c791cad75565858285. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:42.722374 18591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f3996b7e29941e391ea141dd6cc44b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:42.718626 18558 leader_election.cc:302] T aec9c09cc0454acab6006bb3e10982b7 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:42.722843 20396 raft_consensus.cc:2781] T aec9c09cc0454acab6006bb3e10982b7 P 5e1c6397f09747f091534faf93542710 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240430 20:11:42.722936 20396 raft_consensus.cc:482] T aec9c09cc0454acab6006bb3e10982b7 P 5e1c6397f09747f091534faf93542710 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:42.723001 20396 raft_consensus.cc:3037] T aec9c09cc0454acab6006bb3e10982b7 P 5e1c6397f09747f091534faf93542710 [term 2 FOLLOWER]: Advancing to term 3
I20240430 20:11:42.725365 20396 raft_consensus.cc:504] T aec9c09cc0454acab6006bb3e10982b7 P 5e1c6397f09747f091534faf93542710 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:42.725873 18588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 266fd23786bc435db5083370f3330ed0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:42.725962 20396 leader_election.cc:288] T aec9c09cc0454acab6006bb3e10982b7 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 3 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
W20240430 20:11:42.735625 18696 consensus_peers.cc:485] T 44fa6917a85c46f3aaa363df46ea586b P 3c51c33d5616482cafb68e26688820f4 -> Peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Couldn't send request to peer 5e1c6397f09747f091534faf93542710. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
W20240430 20:11:42.736438 19033 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 086b3dfa84c045a18c8c4f177dce606a consensus state has no leader
I20240430 20:11:42.736594 18725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 772eba5f24c549c791cad75565858285. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:42.739053 19471 ts_tablet_manager.cc:1888] T 44fa6917a85c46f3aaa363df46ea586b P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:42.739172 19471 log.cc:1196] T 44fa6917a85c46f3aaa363df46ea586b P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/44fa6917a85c46f3aaa363df46ea586b
I20240430 20:11:42.739503 19471 ts_tablet_manager.cc:1909] T 44fa6917a85c46f3aaa363df46ea586b P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:42.740600 19027 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: Sending DeleteTablet for 3 replicas of tablet 7683caba80594627900a0185f772c131
W20240430 20:11:42.740685 19051 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 489dc838744a49eba542009c86688aa5 consensus state has no leader
I20240430 20:11:42.741354 19051 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_fc68663ab2d74c34bedefcbad2d98a69"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240430 20:11:42.741593 19027 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_2c1b15dc75504997924b1f36ca1f69a8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:42.742183 19027 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2c1b15dc75504997924b1f36ca1f69a8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:42.742218 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 121) took 10357 ms. Trace:
I20240430 20:11:42.742290 19471 rpcz_store.cc:276] 0430 20:11:32.389077 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:32.392449 (+ 3372us) service_pool.cc:225] Handling call
0430 20:11:42.692992 (+10300543us) ts_tablet_manager.cc:1120] Deleting tablet 44fa6917a85c46f3aaa363df46ea586b
0430 20:11:42.721117 (+ 28125us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:42.734494 (+ 13377us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:42.738951 (+ 4457us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:42.742197 (+ 3246us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":10300283}
I20240430 20:11:42.742661 19471 tablet_replica.cc:330] T 73453a7fa30342e29db552b17b62f0c1 P 5e1c6397f09747f091534faf93542710: stopping tablet replica
W20240430 20:11:42.742751 19051 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fc68663ab2d74c34bedefcbad2d98a69 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:42.742864 19471 raft_consensus.cc:2229] T 73453a7fa30342e29db552b17b62f0c1 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:42.743026 19471 raft_consensus.cc:2258] T 73453a7fa30342e29db552b17b62f0c1 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:42.745198 19471 ts_tablet_manager.cc:1875] T 73453a7fa30342e29db552b17b62f0c1 P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:42.745302 18446 tablet_service.cc:1514] Processing DeleteTablet for tablet 7683caba80594627900a0185f772c131 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:42 UTC) from {username='slave'} at 127.0.0.1:41680
W20240430 20:11:42.745642 19026 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b285e3eb16d148779c74e057458d8461 consensus state has no leader
I20240430 20:11:42.745816 18478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aec9c09cc0454acab6006bb3e10982b7" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:42.746023 18478 raft_consensus.cc:3037] T aec9c09cc0454acab6006bb3e10982b7 P 2efb542aa55e439b99ccb98d1c5c884a [term 2 FOLLOWER]: Advancing to term 3
I20240430 20:11:42.746071 19026 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_bab588d4539e4155a9018d0db939e71f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240430 20:11:42.746801 18733 tablet_service.cc:1514] Processing DeleteTablet for tablet 7683caba80594627900a0185f772c131 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:42 UTC) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:42.748677 18478 raft_consensus.cc:2454] T aec9c09cc0454acab6006bb3e10982b7 P 2efb542aa55e439b99ccb98d1c5c884a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 3.
W20240430 20:11:42.749727 19026 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bab588d4539e4155a9018d0db939e71f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:42.756467 18988 tablet_bootstrap.cc:492] T 87ac32a0ddb84676adc83038ef664591 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:42.756669 18988 ts_tablet_manager.cc:1367] T 87ac32a0ddb84676adc83038ef664591 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.151s user 0.002s sys 0.002s
I20240430 20:11:42.757566 18988 raft_consensus.cc:348] T 87ac32a0ddb84676adc83038ef664591 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:42.758635 20105 raft_consensus.cc:482] T 43b5e2c005a34df08cc4335c6e014cb5 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:42.758781 20105 raft_consensus.cc:504] T 43b5e2c005a34df08cc4335c6e014cb5 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:42.759465 20105 leader_election.cc:288] T 43b5e2c005a34df08cc4335c6e014cb5 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:42.757824 18988 raft_consensus.cc:374] T 87ac32a0ddb84676adc83038ef664591 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20240430 20:11:42.760562 18999 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 47bf64ecc41b445da4ff697e141243dd (table default.Table_f1ad29d5986d4b799cfc67afb2b0dfdf [id=838fc8b61a864dea8877a6f144e4cb4f], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 47bf64ecc41b445da4ff697e141243dd consensus state has no leader
I20240430 20:11:42.760439 18758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43b5e2c005a34df08cc4335c6e014cb5" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:42.760902 18988 raft_consensus.cc:729] T 87ac32a0ddb84676adc83038ef664591 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:42.761004 19471 ts_tablet_manager.cc:1888] T 73453a7fa30342e29db552b17b62f0c1 P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:42.761116 19471 log.cc:1196] T 73453a7fa30342e29db552b17b62f0c1 P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/73453a7fa30342e29db552b17b62f0c1
I20240430 20:11:42.761418 19471 ts_tablet_manager.cc:1909] T 73453a7fa30342e29db552b17b62f0c1 P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
W20240430 20:11:42.761525 18417 leader_election.cc:341] T 43b5e2c005a34df08cc4335c6e014cb5 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:42.762075 18558 leader_election.cc:302] T aec9c09cc0454acab6006bb3e10982b7 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:42.762049 18608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43b5e2c005a34df08cc4335c6e014cb5" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:42.762271 18608 raft_consensus.cc:2454] T 43b5e2c005a34df08cc4335c6e014cb5 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 0.
I20240430 20:11:42.762333 20395 raft_consensus.cc:2781] T aec9c09cc0454acab6006bb3e10982b7 P 5e1c6397f09747f091534faf93542710 [term 3 FOLLOWER]: Leader election won for term 3
I20240430 20:11:42.762461 20395 raft_consensus.cc:686] T aec9c09cc0454acab6006bb3e10982b7 P 5e1c6397f09747f091534faf93542710 [term 3 LEADER]: Becoming Leader. State: Replica: 5e1c6397f09747f091534faf93542710, State: Running, Role: LEADER
I20240430 20:11:42.762742 20395 consensus_queue.cc:237] T aec9c09cc0454acab6006bb3e10982b7 P 5e1c6397f09747f091534faf93542710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:42.762928 18415 leader_election.cc:302] T 43b5e2c005a34df08cc4335c6e014cb5 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters: 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:42.763288 18999 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet 44fa6917a85c46f3aaa363df46ea586b (table default.Table_5924e4520c6c49988ccef21ca04d25d4 [id=68c1eb7252974258aabd9873e6654b10]) successfully deleted
I20240430 20:11:42.763373 20105 raft_consensus.cc:2781] T 43b5e2c005a34df08cc4335c6e014cb5 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:42.763455 20105 raft_consensus.cc:482] T 43b5e2c005a34df08cc4335c6e014cb5 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:42.763516 20105 raft_consensus.cc:3037] T 43b5e2c005a34df08cc4335c6e014cb5 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:42.763974 19027 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_9110c0cb4db441dbb8bdce422eadc189"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240430 20:11:42.764101 18585 tablet_service.cc:1514] Processing DeleteTablet for tablet 7683caba80594627900a0185f772c131 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:42 UTC) from {username='slave'} at 127.0.0.1:40218
W20240430 20:11:42.764663 19027 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9110c0cb4db441dbb8bdce422eadc189 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:42.766119 20105 raft_consensus.cc:504] T 43b5e2c005a34df08cc4335c6e014cb5 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:42.680326 19489 ts_tablet_manager.cc:1888] T 7542c7fe6cad452ab718f9ef54dbbbbe P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:42.767052 19489 log.cc:1196] T 7542c7fe6cad452ab718f9ef54dbbbbe P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/7542c7fe6cad452ab718f9ef54dbbbbe
I20240430 20:11:42.767439 18758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43b5e2c005a34df08cc4335c6e014cb5" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:42.767974 18608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43b5e2c005a34df08cc4335c6e014cb5" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:42.768198 18608 raft_consensus.cc:3037] T 43b5e2c005a34df08cc4335c6e014cb5 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
W20240430 20:11:42.768153 18417 leader_election.cc:341] T 43b5e2c005a34df08cc4335c6e014cb5 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:42.770164 19489 ts_tablet_manager.cc:1909] T 7542c7fe6cad452ab718f9ef54dbbbbe P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:42.770695 19026 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_4d14d7a9609346baa18fdee2e3a2abfa" } modify_external_catalogs: true
I20240430 20:11:42.770779 20105 leader_election.cc:288] T 43b5e2c005a34df08cc4335c6e014cb5 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:42.770890 19026 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_4d14d7a9609346baa18fdee2e3a2abfa" } modify_external_catalogs: true
I20240430 20:11:42.771112 18608 raft_consensus.cc:2454] T 43b5e2c005a34df08cc4335c6e014cb5 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 1.
I20240430 20:11:42.771179 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 111) took 10637 ms. Trace:
I20240430 20:11:42.771278 19489 rpcz_store.cc:276] 0430 20:11:32.133710 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:32.136250 (+ 2540us) service_pool.cc:225] Handling call
0430 20:11:42.118751 (+9982501us) ts_tablet_manager.cc:1120] Deleting tablet 7542c7fe6cad452ab718f9ef54dbbbbe
0430 20:11:42.118759 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:42.298338 (+179579us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:42.680153 (+381815us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:42.771150 (+ 90997us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":93409,"tablet-delete.queue_time_us":9982285}
I20240430 20:11:42.771564 18415 leader_election.cc:302] T 43b5e2c005a34df08cc4335c6e014cb5 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters: 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:42.771800 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 115) took 10633 ms. Trace:
I20240430 20:11:42.771960 20105 raft_consensus.cc:2781] T 43b5e2c005a34df08cc4335c6e014cb5 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:42.771888 19489 rpcz_store.cc:276] 0430 20:11:32.138172 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:32.138226 (+ 54us) service_pool.cc:225] Handling call
0430 20:11:42.771683 (+10633457us) ts_tablet_manager.cc:1120] Deleting tablet a1aeee1b02214975a376cfd36957afe8
0430 20:11:42.771696 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:42.771782 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":10633217}
I20240430 20:11:42.772194 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 116) took 10633 ms. Trace:
I20240430 20:11:42.772220 20105 raft_consensus.cc:686] T 43b5e2c005a34df08cc4335c6e014cb5 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Becoming Leader. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Running, Role: LEADER
I20240430 20:11:42.772280 19489 rpcz_store.cc:276] 0430 20:11:32.139201 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:32.139254 (+ 53us) service_pool.cc:225] Handling call
0430 20:11:42.772118 (+10632864us) ts_tablet_manager.cc:1120] Deleting tablet aacabead3c874e77a47d2ecf19dbfa82
0430 20:11:42.772128 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:42.772180 (+ 52us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":10632648}
I20240430 20:11:42.772594 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 117) took 10632 ms. Trace:
I20240430 20:11:42.772728 19001 catalog_manager.cc:4830] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): tablet 7542c7fe6cad452ab718f9ef54dbbbbe (table default.Table_8af52c0f1d404b42bc53287f64e20c2d [id=8e590e0e404344658b5cee90132d91d1]) successfully deleted
I20240430 20:11:42.772675 19489 rpcz_store.cc:276] 0430 20:11:32.139678 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:32.139727 (+ 49us) service_pool.cc:225] Handling call
0430 20:11:42.772521 (+10632794us) ts_tablet_manager.cc:1120] Deleting tablet 1398d4d2305b48e08fdb12432bd6c4b0
0430 20:11:42.772531 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:42.772580 (+ 49us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":10632613}
I20240430 20:11:42.772974 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 118) took 10629 ms. Trace:
I20240430 20:11:42.772656 20105 consensus_queue.cc:237] T 43b5e2c005a34df08cc4335c6e014cb5 P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
W20240430 20:11:42.773036 19001 catalog_manager.cc:4797] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): delete failed for tablet a1aeee1b02214975a376cfd36957afe8 because the tablet was not found. No further retry: Not found: Tablet not found: a1aeee1b02214975a376cfd36957afe8
I20240430 20:11:42.773041 19489 rpcz_store.cc:276] 0430 20:11:32.143413 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:32.143469 (+ 56us) service_pool.cc:225] Handling call
0430 20:11:42.772900 (+10629431us) ts_tablet_manager.cc:1120] Deleting tablet 68ca48351cd4412381f5195d81109016
0430 20:11:42.772909 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:42.772960 (+ 51us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":10629236}
W20240430 20:11:42.773249 19001 catalog_manager.cc:4797] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): delete failed for tablet aacabead3c874e77a47d2ecf19dbfa82 because the tablet was not found. No further retry: Not found: Tablet not found: aacabead3c874e77a47d2ecf19dbfa82
I20240430 20:11:42.773304 19489 tablet_replica.cc:330] T fae9a35a2ac844b4bc2ebdf214aa7171 P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:42.773532 19489 raft_consensus.cc:2229] T fae9a35a2ac844b4bc2ebdf214aa7171 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:42.773705 19489 raft_consensus.cc:2258] T fae9a35a2ac844b4bc2ebdf214aa7171 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:42.773687 20236 raft_consensus.cc:482] T abdaf243e5be4ad58036b23151b14fe4 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:42.773805 20236 raft_consensus.cc:504] T abdaf243e5be4ad58036b23151b14fe4 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:42.774227 19489 ts_tablet_manager.cc:1875] T fae9a35a2ac844b4bc2ebdf214aa7171 P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:42.774365 20236 leader_election.cc:288] T abdaf243e5be4ad58036b23151b14fe4 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:42.774484 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 122) took 10384 ms. Trace:
W20240430 20:11:42.774502 19001 catalog_manager.cc:4797] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): delete failed for tablet 1398d4d2305b48e08fdb12432bd6c4b0 because the tablet was not found. No further retry: Not found: Tablet not found: 1398d4d2305b48e08fdb12432bd6c4b0
W20240430 20:11:42.774760 19001 catalog_manager.cc:4797] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): delete failed for tablet 68ca48351cd4412381f5195d81109016 because the tablet was not found. No further retry: Not found: Tablet not found: 68ca48351cd4412381f5195d81109016
I20240430 20:11:42.774631 19471 rpcz_store.cc:276] 0430 20:11:32.389562 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:32.392738 (+ 3176us) service_pool.cc:225] Handling call
0430 20:11:42.742617 (+10349879us) ts_tablet_manager.cc:1120] Deleting tablet 73453a7fa30342e29db552b17b62f0c1
0430 20:11:42.742627 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:42.752437 (+ 9810us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:42.760873 (+ 8436us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:42.774463 (+ 13590us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":10349683}
I20240430 20:11:42.775048 19471 tablet_replica.cc:330] T b72fe1f0713b4076a81a3fc1c74f7fa7 P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:42.775046 18758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abdaf243e5be4ad58036b23151b14fe4" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:42.775221 19471 raft_consensus.cc:2229] T b72fe1f0713b4076a81a3fc1c74f7fa7 P 5e1c6397f09747f091534faf93542710 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:42.775300 18999 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet 73453a7fa30342e29db552b17b62f0c1 (table default.Table_7a9f9a8c27bb4392acddd2a7ea3cc3b2 [id=cfad6f932073488b8097ef7381bc9fd1]) successfully deleted
I20240430 20:11:42.775352 19471 pending_rounds.cc:62] T b72fe1f0713b4076a81a3fc1c74f7fa7 P 5e1c6397f09747f091534faf93542710: Trying to abort 1 pending ops.
I20240430 20:11:42.775393 19471 pending_rounds.cc:68] T b72fe1f0713b4076a81a3fc1c74f7fa7 P 5e1c6397f09747f091534faf93542710: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7022624341382275072 op_type: NO_OP noop_request { }
W20240430 20:11:42.775808 18417 leader_election.cc:341] T abdaf243e5be4ad58036b23151b14fe4 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:42.775827 18608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abdaf243e5be4ad58036b23151b14fe4" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:42.776448 19471 raft_consensus.cc:2866] T b72fe1f0713b4076a81a3fc1c74f7fa7 P 5e1c6397f09747f091534faf93542710 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
W20240430 20:11:42.776465 18415 leader_election.cc:341] T abdaf243e5be4ad58036b23151b14fe4 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:42.776556 18415 leader_election.cc:302] T abdaf243e5be4ad58036b23151b14fe4 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a; no voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710
I20240430 20:11:42.776712 19471 raft_consensus.cc:2258] T b72fe1f0713b4076a81a3fc1c74f7fa7 P 5e1c6397f09747f091534faf93542710 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:42.776934 20236 raft_consensus.cc:2726] T abdaf243e5be4ad58036b23151b14fe4 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:42.777403 19471 ts_tablet_manager.cc:1875] T b72fe1f0713b4076a81a3fc1c74f7fa7 P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240430 20:11:42.778714 18557 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.177.3:44867, user_credentials={real_user=slave}} blocked reactor thread for 128906us
I20240430 20:11:42.779260 19051 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_88b0eaf3d0a1461ab5e2130cb69f0dee" } modify_external_catalogs: true
I20240430 20:11:42.779448 19051 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_88b0eaf3d0a1461ab5e2130cb69f0dee" } modify_external_catalogs: true
W20240430 20:11:42.783005 18558 consensus_peers.cc:485] T 680553d4d0bc4fcb839235c1832757e1 P 5e1c6397f09747f091534faf93542710 -> Peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Couldn't send request to peer 2efb542aa55e439b99ccb98d1c5c884a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 680553d4d0bc4fcb839235c1832757e1. This is attempt 1: this message will repeat every 5th retry.
I20240430 20:11:42.783586 18461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 798a13f545fd4d8f8c654eaa2452eda1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:42.783840 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 205) took 4020 ms. Trace:
I20240430 20:11:42.783957 19495 rpcz_store.cc:276] 0430 20:11:38.763407 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:38.766761 (+ 3354us) service_pool.cc:225] Handling call
0430 20:11:42.557599 (+3790838us) ts_tablet_manager.cc:1120] Deleting tablet b76a4105a2974fe280e759349037d027
0430 20:11:42.557607 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:42.576926 (+ 19319us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:42.672141 (+ 95215us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:42.783812 (+111671us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3773412}
I20240430 20:11:42.784494 19004 catalog_manager.cc:4830] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): tablet b76a4105a2974fe280e759349037d027 (table default.Table_95220daaf6d44c08beff4ed6abdbea79 [id=75f0256acc45402d8b6809d92c38a1d8]) successfully deleted
I20240430 20:11:42.784854 19495 tablet_replica.cc:330] T 24dcc20832804229a2664873bf6ddaa7 P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:42.785096 19495 raft_consensus.cc:2229] T 24dcc20832804229a2664873bf6ddaa7 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:42.785344 19495 raft_consensus.cc:2258] T 24dcc20832804229a2664873bf6ddaa7 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:42.786890 19495 ts_tablet_manager.cc:1875] T 24dcc20832804229a2664873bf6ddaa7 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:42.787104 20236 raft_consensus.cc:482] T 724e51ff05ec4baebc2bb55e05c8eddc P 2efb542aa55e439b99ccb98d1c5c884a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:42.787350 20236 raft_consensus.cc:504] T 724e51ff05ec4baebc2bb55e05c8eddc P 2efb542aa55e439b99ccb98d1c5c884a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:42.787955 20236 leader_election.cc:288] T 724e51ff05ec4baebc2bb55e05c8eddc P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:42.788492 18608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "724e51ff05ec4baebc2bb55e05c8eddc" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:42.788682 18608 raft_consensus.cc:2454] T 724e51ff05ec4baebc2bb55e05c8eddc P 5e1c6397f09747f091534faf93542710 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 2.
I20240430 20:11:42.789124 18415 leader_election.cc:302] T 724e51ff05ec4baebc2bb55e05c8eddc P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:42.789404 20236 raft_consensus.cc:2781] T 724e51ff05ec4baebc2bb55e05c8eddc P 2efb542aa55e439b99ccb98d1c5c884a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240430 20:11:42.789471 20236 raft_consensus.cc:482] T 724e51ff05ec4baebc2bb55e05c8eddc P 2efb542aa55e439b99ccb98d1c5c884a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:42.789549 20236 raft_consensus.cc:3037] T 724e51ff05ec4baebc2bb55e05c8eddc P 2efb542aa55e439b99ccb98d1c5c884a [term 2 FOLLOWER]: Advancing to term 3
I20240430 20:11:42.790159 18750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "724e51ff05ec4baebc2bb55e05c8eddc" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:42.790349 18750 raft_consensus.cc:2454] T 724e51ff05ec4baebc2bb55e05c8eddc P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 1.
I20240430 20:11:42.791584 18985 tablet_bootstrap.cc:492] T 115c4a12440647a4bb46eb6824f7ea98 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:42.791692 18985 ts_tablet_manager.cc:1367] T 115c4a12440647a4bb46eb6824f7ea98 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.179s user 0.010s sys 0.010s
I20240430 20:11:42.792007 20236 raft_consensus.cc:504] T 724e51ff05ec4baebc2bb55e05c8eddc P 2efb542aa55e439b99ccb98d1c5c884a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:42.792583 20236 leader_election.cc:288] T 724e51ff05ec4baebc2bb55e05c8eddc P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 3 election: Requested vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:42.793179 18608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "724e51ff05ec4baebc2bb55e05c8eddc" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:42.793180 18750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "724e51ff05ec4baebc2bb55e05c8eddc" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:42.793360 18608 raft_consensus.cc:3037] T 724e51ff05ec4baebc2bb55e05c8eddc P 5e1c6397f09747f091534faf93542710 [term 2 FOLLOWER]: Advancing to term 3
I20240430 20:11:42.793359 18750 raft_consensus.cc:3037] T 724e51ff05ec4baebc2bb55e05c8eddc P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Advancing to term 3
I20240430 20:11:42.796000 18750 raft_consensus.cc:2454] T 724e51ff05ec4baebc2bb55e05c8eddc P 3c51c33d5616482cafb68e26688820f4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 3.
I20240430 20:11:42.796000 18608 raft_consensus.cc:2454] T 724e51ff05ec4baebc2bb55e05c8eddc P 5e1c6397f09747f091534faf93542710 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 3.
I20240430 20:11:42.796439 18417 leader_election.cc:302] T 724e51ff05ec4baebc2bb55e05c8eddc P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters:
I20240430 20:11:42.796687 20235 raft_consensus.cc:482] T 50486644ec234933b9a0b45ee39a6685 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:42.796806 20235 raft_consensus.cc:504] T 50486644ec234933b9a0b45ee39a6685 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:42.797364 20235 leader_election.cc:288] T 50486644ec234933b9a0b45ee39a6685 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:42.797505 20235 raft_consensus.cc:2781] T 724e51ff05ec4baebc2bb55e05c8eddc P 2efb542aa55e439b99ccb98d1c5c884a [term 3 FOLLOWER]: Leader election won for term 3
I20240430 20:11:42.797590 20235 raft_consensus.cc:686] T 724e51ff05ec4baebc2bb55e05c8eddc P 2efb542aa55e439b99ccb98d1c5c884a [term 3 LEADER]: Becoming Leader. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Running, Role: LEADER
I20240430 20:11:42.797798 20235 consensus_queue.cc:237] T 724e51ff05ec4baebc2bb55e05c8eddc P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:42.799413 18608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50486644ec234933b9a0b45ee39a6685" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:42.799777 18750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50486644ec234933b9a0b45ee39a6685" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
W20240430 20:11:42.799953 18415 leader_election.cc:341] T 50486644ec234933b9a0b45ee39a6685 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Illegal state: must be running to vote when last-logged opid is not known
W20240430 20:11:42.800272 18417 leader_election.cc:341] T 50486644ec234933b9a0b45ee39a6685 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:42.800423 19026 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet 293874dda34742ff9228c9f6078614b8
I20240430 20:11:42.800375 18417 leader_election.cc:302] T 50486644ec234933b9a0b45ee39a6685 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a; no voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710
I20240430 20:11:42.800876 19026 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_2e5bcbca76d345fea6d7b7dac08a7afb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:42.801607 19026 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2e5bcbca76d345fea6d7b7dac08a7afb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:42.802920 19471 ts_tablet_manager.cc:1888] T b72fe1f0713b4076a81a3fc1c74f7fa7 P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:42.792474 18985 raft_consensus.cc:348] T 115c4a12440647a4bb46eb6824f7ea98 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:42.804203 18985 raft_consensus.cc:374] T 115c4a12440647a4bb46eb6824f7ea98 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:42.804327 18985 raft_consensus.cc:729] T 115c4a12440647a4bb46eb6824f7ea98 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:42.804620 18985 consensus_queue.cc:260] T 115c4a12440647a4bb46eb6824f7ea98 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
W20240430 20:11:42.805284 18558 consensus_peers.cc:485] T 1555924b5f1444398f12ee82c3cb5b13 P 5e1c6397f09747f091534faf93542710 -> Peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Couldn't send request to peer 2efb542aa55e439b99ccb98d1c5c884a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1555924b5f1444398f12ee82c3cb5b13. This is attempt 1: this message will repeat every 5th retry.
I20240430 20:11:42.805681 19471 log.cc:1196] T b72fe1f0713b4076a81a3fc1c74f7fa7 P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/b72fe1f0713b4076a81a3fc1c74f7fa7
I20240430 20:11:42.806327 19471 ts_tablet_manager.cc:1909] T b72fe1f0713b4076a81a3fc1c74f7fa7 P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:42.806676 19051 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: Sending DeleteTablet for 3 replicas of tablet 2797eed593fb4b06a3c4d6a28e8075a2
I20240430 20:11:42.807977 19051 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_6e886b67708e4dc1b9e1753b1dd00800"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240430 20:11:42.808650 20324 raft_consensus.cc:482] T 5f02d8862e1f4ccfa9dbda73dbe18615 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240430 20:11:42.808750 19051 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6e886b67708e4dc1b9e1753b1dd00800 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:42.808804 20324 raft_consensus.cc:504] T 5f02d8862e1f4ccfa9dbda73dbe18615 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:42.809510 20324 leader_election.cc:288] T 5f02d8862e1f4ccfa9dbda73dbe18615 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:42.810274 18478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f02d8862e1f4ccfa9dbda73dbe18615" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:42.810523 18478 raft_consensus.cc:2454] T 5f02d8862e1f4ccfa9dbda73dbe18615 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 0.
I20240430 20:11:42.810875 18585 tablet_service.cc:1514] Processing DeleteTablet for tablet 2797eed593fb4b06a3c4d6a28e8075a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:42 UTC) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:42.811445 18558 leader_election.cc:302] T 5f02d8862e1f4ccfa9dbda73dbe18615 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:42.811748 20324 raft_consensus.cc:2781] T 5f02d8862e1f4ccfa9dbda73dbe18615 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:42.811846 20324 raft_consensus.cc:482] T 5f02d8862e1f4ccfa9dbda73dbe18615 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:42.811918 20324 raft_consensus.cc:3037] T 5f02d8862e1f4ccfa9dbda73dbe18615 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:42.814553 20324 raft_consensus.cc:504] T 5f02d8862e1f4ccfa9dbda73dbe18615 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:42.816709 20324 leader_election.cc:288] T 5f02d8862e1f4ccfa9dbda73dbe18615 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:42.816810 18585 tablet_service.cc:1467] Processing CreateTablet for tablet e28529e1c2ae4c7c9b9b0e4bd709d132 (DEFAULT_TABLE table=default.Table_d8df23edb8ce480bb59a2ab0f1fc7521 [id=4c9193fcff2f41768d578803372434e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:42.817250 18585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e28529e1c2ae4c7c9b9b0e4bd709d132. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:42.817512 18729 tablet_service.cc:1467] Processing CreateTablet for tablet e28529e1c2ae4c7c9b9b0e4bd709d132 (DEFAULT_TABLE table=default.Table_d8df23edb8ce480bb59a2ab0f1fc7521 [id=4c9193fcff2f41768d578803372434e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:42.817618 18733 tablet_service.cc:1467] Processing CreateTablet for tablet eb7554e115d84e6ca4f05509c5f33e2a (DEFAULT_TABLE table=default.Table_4bb81ccd60a148d197b8a4ebd9c9cc49 [id=ee0542402cc1426681e9583fbb7340e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:42.819470 18724 tablet_service.cc:1467] Processing CreateTablet for tablet 6f3996b7e29941e391ea141dd6cc44b2 (DEFAULT_TABLE table=default.Table_da5216d025844afda5a83ada555bc120 [id=dc55b25c3ca14863a21042e805e3bf9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:42.819882 18734 tablet_service.cc:1514] Processing DeleteTablet for tablet 2797eed593fb4b06a3c4d6a28e8075a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:42 UTC) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:42.819917 18478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f02d8862e1f4ccfa9dbda73dbe18615" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:42.820120 18478 raft_consensus.cc:3037] T 5f02d8862e1f4ccfa9dbda73dbe18615 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:42.825707 18601 tablet_service.cc:1467] Processing CreateTablet for tablet eb7554e115d84e6ca4f05509c5f33e2a (DEFAULT_TABLE table=default.Table_4bb81ccd60a148d197b8a4ebd9c9cc49 [id=ee0542402cc1426681e9583fbb7340e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:42.828516 18478 raft_consensus.cc:2454] T 5f02d8862e1f4ccfa9dbda73dbe18615 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
I20240430 20:11:42.830171 20362 raft_consensus.cc:2726] T 50486644ec234933b9a0b45ee39a6685 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:42.831470 19033 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet c9e7660eccc448dbb99d135f82bfd261 (table default.Table_685a26737bb046fb85ced15df7ae3423 [id=51d177e8498848b38c43e3f8d11f092c]). Expected version 1 got 0
I20240430 20:11:42.831701 19033 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 6e754af64cb64fd2a082adae7d5f24da (table default.Table_208b12b8e9bd4f3e8bc9c6d422ada649 [id=1df4e95702cd46f59ab21b101049a9ed]). Expected version 1 got 0
I20240430 20:11:42.832078 19033 catalog_manager.cc:5484] T 7bde1d6934784c9aa5911181e8df9eeb P 2efb542aa55e439b99ccb98d1c5c884a reported cstate change: term changed from 1 to 2, leader changed from 3c51c33d5616482cafb68e26688820f4 (127.17.177.3) to 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1). New cstate: current_term: 2 leader_uuid: "2efb542aa55e439b99ccb98d1c5c884a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } health_report { overall_health: UNKNOWN } } }
I20240430 20:11:42.832687 19033 catalog_manager.cc:5484] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 2efb542aa55e439b99ccb98d1c5c884a reported cstate change: term changed from 1 to 2, leader changed from 5e1c6397f09747f091534faf93542710 (127.17.177.2) to 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1). New cstate: current_term: 2 leader_uuid: "2efb542aa55e439b99ccb98d1c5c884a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } health_report { overall_health: HEALTHY } } }
I20240430 20:11:42.829039 18558 leader_election.cc:302] T 5f02d8862e1f4ccfa9dbda73dbe18615 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:42.833187 19033 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 7ce09fcbe1ac457a9c05efaf000c3263 (table default.Table_edc4c27ed846454288f73f6a0d88cb69 [id=78fd3f516083489ab3b42d90dcef7e4d]). Expected version 1 got 0
I20240430 20:11:42.833487 19033 catalog_manager.cc:5484] T 46251c08277e498da04a41d009310c62 P 2efb542aa55e439b99ccb98d1c5c884a reported cstate change: term changed from 0 to 1, leader changed from <none> to 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1). New cstate: current_term: 1 leader_uuid: "2efb542aa55e439b99ccb98d1c5c884a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } health_report { overall_health: HEALTHY } } }
I20240430 20:11:42.833891 20324 raft_consensus.cc:2781] T 5f02d8862e1f4ccfa9dbda73dbe18615 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:42.833954 19033 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 46251c08277e498da04a41d009310c62 (table default.Table_4e56923d440944978f0aacc140a0dd8e [id=b1312a6cd1a54003b1db727a2a9d125d]). Expected version 1 got 0
I20240430 20:11:42.834107 20324 raft_consensus.cc:686] T 5f02d8862e1f4ccfa9dbda73dbe18615 P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Becoming Leader. State: Replica: 5e1c6397f09747f091534faf93542710, State: Running, Role: LEADER
I20240430 20:11:42.834205 19033 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet a59ef6c545dd41a28265a4c1b8ce10e0 (table default.Table_07e40af8f1c546048975c8edf88170d7 [id=86df5d161d8641c89070e7eed9a39360]). Expected version 2 got 0
I20240430 20:11:42.834450 19033 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet ebf74809b52f47f3b0ed21b6ac0f541a (table default.Table_b2d07520b9944a89ad62fca70396174e [id=1c8e0a3a44ec4a798770c0165ed3dffa]). Expected version 1 got 0
I20240430 20:11:42.834460 20324 consensus_queue.cc:237] T 5f02d8862e1f4ccfa9dbda73dbe18615 P 5e1c6397f09747f091534faf93542710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:42.834800 19033 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 76d6e9d547ca4a848eae852944b1d117 (table default.Table_104ae015924a48fd8d27220792f07470 [id=ff86c0357c164dd0991098c416087773]). Expected version 2 got 0
I20240430 20:11:42.835170 19033 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet acf3b6fead41483a82c43dc0286fab7c (table default.Table_01450a1e661d450a8cbe185e16e8250c [id=ce9911e0a91949b1997ee47321800255]). Expected version 1 got 0
I20240430 20:11:42.828634 18601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb7554e115d84e6ca4f05509c5f33e2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:42.835734 19033 catalog_manager.cc:5484] T 1462ecf1176c4e66be42abddbe36b00c P 2efb542aa55e439b99ccb98d1c5c884a reported cstate change: term changed from 0 to 1, leader changed from <none> to 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1). New cstate: current_term: 1 leader_uuid: "2efb542aa55e439b99ccb98d1c5c884a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } health_report { overall_health: HEALTHY } } }
I20240430 20:11:42.836432 19033 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 1462ecf1176c4e66be42abddbe36b00c (table default.Table_b49a7942aa8c4f58acb94c35d1cdbb70 [id=6257559b0e9545c58848a76af0c5a5f7]). Expected version 1 got 0
I20240430 20:11:42.761128 18988 consensus_queue.cc:260] T 87ac32a0ddb84676adc83038ef664591 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:42.857100 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 123) took 10466 ms. Trace:
I20240430 20:11:42.857242 19471 rpcz_store.cc:276] 0430 20:11:32.390821 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:32.392935 (+ 2114us) service_pool.cc:225] Handling call
0430 20:11:42.775000 (+10382065us) ts_tablet_manager.cc:1120] Deleting tablet b72fe1f0713b4076a81a3fc1c74f7fa7
0430 20:11:42.775010 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:42.786346 (+ 11336us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:42.802793 (+ 16447us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:42.856960 (+ 54167us) spinlock_profiling.cc:243] Waited 49.5 ms on lock 0x6160000b6854. stack: 00007ffa8b97e756 00007ffa8b97e510 00007ffa8a9b148d 0000000000955804 00000000009557cd 000000000095519a 00007ffa971ec88a 00007ffa971ac1c4 00007ffa984c86a6 00007ffa9af598a8 00007ffa9af8b814 00007ffa9af6cb00 00007ffa9af6c88d 0000000000a60bcc 00007ffa8b9c4b74 00007ffa8b9c819b
0430 20:11:42.857059 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":113877376,"tablet-delete.queue_time_us":10381565}
I20240430 20:11:42.857595 19471 tablet_replica.cc:330] T 87ac32a0ddb84676adc83038ef664591 P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:42.857787 19471 raft_consensus.cc:2229] T 87ac32a0ddb84676adc83038ef664591 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:42.857992 19471 raft_consensus.cc:2258] T 87ac32a0ddb84676adc83038ef664591 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:42.858686 19471 ts_tablet_manager.cc:1875] T 87ac32a0ddb84676adc83038ef664591 P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:42.859261 18528 heartbeater.cc:347] Connected to a master server at 127.17.177.62:36441
I20240430 20:11:42.862643 18999 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet b72fe1f0713b4076a81a3fc1c74f7fa7 (table default.Table_16b082d140234e1c9a1e70654839b4cd [id=c0d8dc87d1f9426a85f6176f161595a1]) successfully deleted
I20240430 20:11:42.862874 19049 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet c9e7660eccc448dbb99d135f82bfd261 (table default.Table_685a26737bb046fb85ced15df7ae3423 [id=51d177e8498848b38c43e3f8d11f092c]). Expected version 1 got 0
I20240430 20:11:42.863090 19049 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 6e754af64cb64fd2a082adae7d5f24da (table default.Table_208b12b8e9bd4f3e8bc9c6d422ada649 [id=1df4e95702cd46f59ab21b101049a9ed]). Expected version 1 got 0
I20240430 20:11:42.863271 19049 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet a59ef6c545dd41a28265a4c1b8ce10e0 (table default.Table_07e40af8f1c546048975c8edf88170d7 [id=86df5d161d8641c89070e7eed9a39360]). Expected version 2 got 0
I20240430 20:11:42.863517 19049 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 76d6e9d547ca4a848eae852944b1d117 (table default.Table_104ae015924a48fd8d27220792f07470 [id=ff86c0357c164dd0991098c416087773]). Expected version 2 got 0
I20240430 20:11:42.863749 19049 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet acf3b6fead41483a82c43dc0286fab7c (table default.Table_01450a1e661d450a8cbe185e16e8250c [id=ce9911e0a91949b1997ee47321800255]). Expected version 1 got 0
I20240430 20:11:42.863983 19049 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet ebf74809b52f47f3b0ed21b6ac0f541a (table default.Table_b2d07520b9944a89ad62fca70396174e [id=1c8e0a3a44ec4a798770c0165ed3dffa]). Expected version 1 got 0
I20240430 20:11:42.864213 19049 catalog_manager.cc:5484] T 3fd0e8c360734dcbb75dc2958b70c2b8 P 5e1c6397f09747f091534faf93542710 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e1c6397f09747f091534faf93542710 (127.17.177.2). New cstate: current_term: 1 leader_uuid: "5e1c6397f09747f091534faf93542710" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } health_report { overall_health: UNKNOWN } } }
I20240430 20:11:42.864771 19049 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 1462ecf1176c4e66be42abddbe36b00c (table default.Table_b49a7942aa8c4f58acb94c35d1cdbb70 [id=6257559b0e9545c58848a76af0c5a5f7]). Expected version 1 got 0
I20240430 20:11:42.864995 19049 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet acd23f0a63d5448aaa7ed78422e67bad (table default.Table_972f13be4dca4cd19b9a04534dc59f51 [id=69f37db4de634a3185613e40013e0ed7]). Expected version 1 got 0
I20240430 20:11:42.865259 19049 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 46251c08277e498da04a41d009310c62 (table default.Table_4e56923d440944978f0aacc140a0dd8e [id=b1312a6cd1a54003b1db727a2a9d125d]). Expected version 1 got 0
I20240430 20:11:42.865413 19049 catalog_manager.cc:5484] T e3e74e76ab2b4fe492ffcc1838656fd1 P 5e1c6397f09747f091534faf93542710 reported cstate change: term changed from 1 to 2, leader changed from 3c51c33d5616482cafb68e26688820f4 (127.17.177.3) to 5e1c6397f09747f091534faf93542710 (127.17.177.2). New cstate: current_term: 2 leader_uuid: "5e1c6397f09747f091534faf93542710" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } health_report { overall_health: UNKNOWN } } }
I20240430 20:11:42.866034 19049 catalog_manager.cc:5484] T ca64de16d40f4a44b8b411f5526ba11a P 5e1c6397f09747f091534faf93542710 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e1c6397f09747f091534faf93542710 (127.17.177.2). New cstate: current_term: 1 leader_uuid: "5e1c6397f09747f091534faf93542710" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } health_report { overall_health: UNKNOWN } } }
I20240430 20:11:42.866618 19024 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: Sending DeleteTablet for 3 replicas of tablet a05e32a33aa7488eb287cc6314c09c44
I20240430 20:11:42.867435 19024 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_e3165b7ce52a44bf9d2bed43f9c5d6c7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:42.868031 19024 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e3165b7ce52a44bf9d2bed43f9c5d6c7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:42.868923 19471 ts_tablet_manager.cc:1888] T 87ac32a0ddb84676adc83038ef664591 P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:42.869048 19471 log.cc:1196] T 87ac32a0ddb84676adc83038ef664591 P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/87ac32a0ddb84676adc83038ef664591
I20240430 20:11:42.869422 19471 ts_tablet_manager.cc:1909] T 87ac32a0ddb84676adc83038ef664591 P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:42.870328 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 125) took 10403 ms. Trace:
W20240430 20:11:42.870373 19029 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.17.177.2:41749 (request call id 107) took 1421 ms (client timeout 1000 ms). Trace:
I20240430 20:11:42.870417 19471 rpcz_store.cc:276] 0430 20:11:32.466422 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:32.466805 (+ 383us) service_pool.cc:225] Handling call
0430 20:11:42.857544 (+10390739us) ts_tablet_manager.cc:1120] Deleting tablet 87ac32a0ddb84676adc83038ef664591
0430 20:11:42.857553 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:42.863770 (+ 6217us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:42.868784 (+ 5014us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:42.870308 (+ 1524us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":10390508}
I20240430 20:11:42.871002 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 126) took 10404 ms. Trace:
I20240430 20:11:42.871073 19471 rpcz_store.cc:276] 0430 20:11:32.466579 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:32.467069 (+ 490us) service_pool.cc:225] Handling call
0430 20:11:42.870868 (+10403799us) ts_tablet_manager.cc:1120] Deleting tablet f940d17b02b541289121463e05b0bb5f
0430 20:11:42.870878 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:42.870983 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":10403524}
I20240430 20:11:42.871479 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 127) took 10380 ms. Trace:
I20240430 20:11:42.871593 19471 rpcz_store.cc:276] 0430 20:11:32.491002 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:32.491057 (+ 55us) service_pool.cc:225] Handling call
0430 20:11:42.871393 (+10380336us) ts_tablet_manager.cc:1120] Deleting tablet 115c4a12440647a4bb46eb6824f7ea98
0430 20:11:42.871401 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:42.871466 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":10380074}
I20240430 20:11:42.872011 19471 tablet_replica.cc:330] T 2cdccf4be7094ca5b863dd5c3432d55d P 5e1c6397f09747f091534faf93542710: stopping tablet replica
W20240430 20:11:42.870505 19029 rpcz_store.cc:269] 0430 20:11:41.449089 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:41.545086 (+ 95997us) service_pool.cc:225] Handling call
0430 20:11:42.870334 (+1325248us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:42.525698 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:42.525861 (+ 163us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:42.525871 (+ 10us) write_op.cc:427] Acquired schema lock
0430 20:11:42.525875 (+ 4us) tablet.cc:660] Decoding operations
0430 20:11:42.526048 (+ 173us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:42.526067 (+ 19us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:42.526076 (+ 9us) tablet.cc:683] Acquiring locks for 2 operations
0430 20:11:42.526210 (+ 134us) tablet.cc:699] Row locks acquired
0430 20:11:42.526214 (+ 4us) write_op.cc:260] PREPARE: finished
0430 20:11:42.526254 (+ 40us) write_op.cc:270] Start()
0430 20:11:42.526296 (+ 42us) write_op.cc:276] Timestamp: P: 1714507902526224 usec, L: 0
0430 20:11:42.526302 (+ 6us) op_driver.cc:331] REPLICATION: starting
0430 20:11:42.526547 (+ 245us) log.cc:842] Serialized 907 byte log entry
0430 20:11:42.530225 (+ 3678us) op_driver.cc:444] REPLICATION: finished
0430 20:11:42.530319 (+ 94us) write_op.cc:301] APPLY: starting
0430 20:11:42.530334 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:42.530381 (+ 47us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:42.530413 (+ 32us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:42.530479 (+ 66us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:42.530516 (+ 37us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0430 20:11:42.530524 (+ 8us) write_op.cc:312] APPLY: finished
0430 20:11:42.530614 (+ 90us) log.cc:842] Serialized 29 byte log entry
0430 20:11:42.530679 (+ 65us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:42.530702 (+ 23us) write_op.cc:446] Released schema lock
0430 20:11:42.530727 (+ 25us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:42.530789 (+ 62us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:42.530793 (+ 4us) write_op.cc:446] Released schema lock
Metrics: {"reported_tablets":13,"child_traces":[["op",{"apply.queue_time_us":50,"apply.run_cpu_time_us":522,"apply.run_wall_time_us":520,"num_ops":2,"prepare.queue_time_us":87197,"prepare.run_cpu_time_us":996,"prepare.run_wall_time_us":2457,"raft.queue_time_us":39,"raft.run_cpu_time_us":264,"raft.run_wall_time_us":264,"replication_time_us":3900}]]}
I20240430 20:11:42.872473 19471 raft_consensus.cc:2229] T 2cdccf4be7094ca5b863dd5c3432d55d P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:42.872396 19029 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_650b1b52ac7a453a95434b798c408f6e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240430 20:11:42.872774 19471 raft_consensus.cc:2258] T 2cdccf4be7094ca5b863dd5c3432d55d P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Raft consensus is shut down!
W20240430 20:11:42.873050 19029 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_650b1b52ac7a453a95434b798c408f6e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:42.873821 18983 tablet_bootstrap.cc:492] T 115c4a12440647a4bb46eb6824f7ea98 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:42.874019 18983 ts_tablet_manager.cc:1367] T 115c4a12440647a4bb46eb6824f7ea98 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.155s user 0.027s sys 0.024s
I20240430 20:11:42.875288 18585 tablet_service.cc:1514] Processing DeleteTablet for tablet a05e32a33aa7488eb287cc6314c09c44 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:42 UTC) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:42.876200 18591 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f61e3f88a6448dfa323204bd8d14cc5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:31 UTC) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:42.876188 18999 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet 87ac32a0ddb84676adc83038ef664591 (table default.Table_763f0800e8bb4fa69976886416250f42 [id=61c8a5b99ef34a0baeeb343fef7cfd05]) successfully deleted
I20240430 20:11:42.876237 18588 tablet_service.cc:1514] Processing DeleteTablet for tablet 7542c7fe6cad452ab718f9ef54dbbbbe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:29 UTC) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:42.876466 18585 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d09baed576e4136bf6d756ca39d08b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:31 UTC) from {username='slave'} at 127.0.0.1:40218
W20240430 20:11:42.876626 18999 catalog_manager.cc:4810] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): delete failed for tablet f940d17b02b541289121463e05b0bb5f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f940d17b02b541289121463e05b0bb5f already in progress: creating tablet
I20240430 20:11:42.876685 18586 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a9066d895744c5bb7d9532b2940603c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:29 UTC) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:42.874981 18983 raft_consensus.cc:348] T 115c4a12440647a4bb46eb6824f7ea98 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:42.877005 18983 raft_consensus.cc:374] T 115c4a12440647a4bb46eb6824f7ea98 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:42.877120 18983 raft_consensus.cc:729] T 115c4a12440647a4bb46eb6824f7ea98 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
W20240430 20:11:42.877067 18999 catalog_manager.cc:4810] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): delete failed for tablet 115c4a12440647a4bb46eb6824f7ea98 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 115c4a12440647a4bb46eb6824f7ea98 already in progress: creating tablet
W20240430 20:11:42.877317 19048 catalog_manager.cc:5676] Failed to send AlterTable RPC for tablet aa9998f4e17147aa8f2b35c01980463a (table default.Table_ffb3192f443d437ab63f56f628e596ad [id=608890834da843fa9af0cfc0d548029e], current schema version=2): Not found: failed to reset TS proxy: No leader found: Tablet aa9998f4e17147aa8f2b35c01980463a consensus state has no leader
I20240430 20:11:42.877431 18983 consensus_queue.cc:260] T 115c4a12440647a4bb46eb6824f7ea98 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:42.878193 18983 ts_tablet_manager.cc:1398] T 115c4a12440647a4bb46eb6824f7ea98 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240430 20:11:42.878515 18983 tablet_bootstrap.cc:492] T 2e79df8744e74b5991a79ce978ff0202 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
W20240430 20:11:42.879393 19048 catalog_manager.cc:5676] Failed to send AlterTable RPC for tablet 69f88532952a4591bf1577541c3430e6 (table default.Table_1177828cad1f4a24bd397619dad96170 [id=a95fed1bc3fa4865bf9a35dfc51b659c], current schema version=2): Not found: failed to reset TS proxy: No leader found: Tablet 69f88532952a4591bf1577541c3430e6 consensus state has no leader
I20240430 20:11:42.880159 18983 tablet_bootstrap.cc:654] T 2e79df8744e74b5991a79ce978ff0202 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
W20240430 20:11:42.881057 19048 catalog_manager.cc:5676] Failed to send AlterTable RPC for tablet b22aaedb0a6c4fd9ad0ea8fa019001e1 (table default.Table_730a215d7af74a5bb753adca39dea676 [id=a5d580cffa2447d78fc9f2c8e31264f1], current schema version=5): Not found: failed to reset TS proxy: No leader found: Tablet b22aaedb0a6c4fd9ad0ea8fa019001e1 consensus state has no leader
I20240430 20:11:42.880934 19471 ts_tablet_manager.cc:1875] T 2cdccf4be7094ca5b863dd5c3432d55d P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240430 20:11:42.882858 19048 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.17.177.3:42157 (request call id 84) took 1829 ms (client timeout 1000 ms). Trace:
W20240430 20:11:42.884224 19048 rpcz_store.cc:269] 0430 20:11:41.053022 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:41.059079 (+ 6057us) service_pool.cc:225] Handling call
0430 20:11:42.882806 (+1823727us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":26,"reported_tablets":172}
I20240430 20:11:42.885740 20019 raft_consensus.cc:482] T 87ac32a0ddb84676adc83038ef664591 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:42.885897 20019 raft_consensus.cc:504] T 87ac32a0ddb84676adc83038ef664591 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:42.887869 19048 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_50a4836df84543a888fcdb06ccc5ec41"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:42.889272 19048 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_50a4836df84543a888fcdb06ccc5ec41 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240430 20:11:42.897827 18417 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.177.3:44867, user_credentials={real_user=slave}} blocked reactor thread for 97548.5us
I20240430 20:11:42.899029 18608 raft_consensus.cc:1261] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 5e1c6397f09747f091534faf93542710 [term 2 FOLLOWER]: Refusing update from remote peer 2efb542aa55e439b99ccb98d1c5c884a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240430 20:11:42.899922 20235 consensus_queue.cc:1035] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240430 20:11:42.900880 19033 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_dfb59f35018441fd9a72ea860cdd01f6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240430 20:11:42.903936 19471 ts_tablet_manager.cc:1888] T 2cdccf4be7094ca5b863dd5c3432d55d P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:42.905200 19471 log.cc:1196] T 2cdccf4be7094ca5b863dd5c3432d55d P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/2cdccf4be7094ca5b863dd5c3432d55d
I20240430 20:11:42.905709 19471 ts_tablet_manager.cc:1909] T 2cdccf4be7094ca5b863dd5c3432d55d P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:42.909750 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 129) took 10386 ms. Trace:
I20240430 20:11:42.911127 19471 rpcz_store.cc:276] 0430 20:11:32.522843 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:32.522927 (+ 84us) service_pool.cc:225] Handling call
0430 20:11:42.871969 (+10349042us) ts_tablet_manager.cc:1120] Deleting tablet 2cdccf4be7094ca5b863dd5c3432d55d
0430 20:11:42.871977 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:42.885493 (+ 13516us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:42.903808 (+ 18315us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:42.909727 (+ 5919us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":10348240}
I20240430 20:11:42.911916 20395 raft_consensus.cc:482] T 115c4a12440647a4bb46eb6824f7ea98 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:42.912725 20395 raft_consensus.cc:504] T 115c4a12440647a4bb46eb6824f7ea98 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:42.913262 18461 tablet_service.cc:1514] Processing DeleteTablet for tablet 2797eed593fb4b06a3c4d6a28e8075a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:42 UTC) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:42.913337 20395 leader_election.cc:288] T 115c4a12440647a4bb46eb6824f7ea98 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:42.912554 18983 tablet_bootstrap.cc:492] T 2e79df8744e74b5991a79ce978ff0202 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:42.914144 18983 ts_tablet_manager.cc:1367] T 2e79df8744e74b5991a79ce978ff0202 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.036s user 0.004s sys 0.002s
I20240430 20:11:42.914474 18446 tablet_service.cc:1467] Processing CreateTablet for tablet 6f3996b7e29941e391ea141dd6cc44b2 (DEFAULT_TABLE table=default.Table_da5216d025844afda5a83ada555bc120 [id=dc55b25c3ca14863a21042e805e3bf9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:42.914942 18983 raft_consensus.cc:348] T 2e79df8744e74b5991a79ce978ff0202 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:42.915132 18983 raft_consensus.cc:374] T 2e79df8744e74b5991a79ce978ff0202 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:42.915185 18983 raft_consensus.cc:729] T 2e79df8744e74b5991a79ce978ff0202 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:42.915426 18983 consensus_queue.cc:260] T 2e79df8744e74b5991a79ce978ff0202 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:42.916018 18983 ts_tablet_manager.cc:1398] T 2e79df8744e74b5991a79ce978ff0202 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240430 20:11:42.916297 18983 tablet_bootstrap.cc:492] T 30a521916bda46678be20a066bd1d3f3 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:42.917704 18983 tablet_bootstrap.cc:654] T 30a521916bda46678be20a066bd1d3f3 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:42.918567 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 128) took 10426 ms. Trace:
I20240430 20:11:42.918684 19471 rpcz_store.cc:276] 0430 20:11:32.491816 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:32.491870 (+ 54us) service_pool.cc:225] Handling call
0430 20:11:42.918450 (+10426580us) ts_tablet_manager.cc:1120] Deleting tablet 29974dc2c7dd45b99b6cc0712378fc30
0430 20:11:42.918461 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:42.918549 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":10361867}
I20240430 20:11:42.918635 18999 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet 2cdccf4be7094ca5b863dd5c3432d55d (table default.Table_f33a9e03642c44efbf1c27f1c7e70fb9 [id=c64cf189816b46cdb25574e5fddd9684]) successfully deleted
W20240430 20:11:42.919554 18999 catalog_manager.cc:4797] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): delete failed for tablet 29974dc2c7dd45b99b6cc0712378fc30 because the tablet was not found. No further retry: Not found: Tablet not found: 29974dc2c7dd45b99b6cc0712378fc30
I20240430 20:11:42.920989 19471 tablet_replica.cc:330] T 1555924b5f1444398f12ee82c3cb5b13 P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:42.921259 19471 raft_consensus.cc:2229] T 1555924b5f1444398f12ee82c3cb5b13 P 5e1c6397f09747f091534faf93542710 [term 2 LEADER]: Raft consensus shutting down.
I20240430 20:11:42.921633 19471 raft_consensus.cc:2258] T 1555924b5f1444398f12ee82c3cb5b13 P 5e1c6397f09747f091534faf93542710 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:42.926858 18478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "115c4a12440647a4bb46eb6824f7ea98" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:42.927099 18478 raft_consensus.cc:2454] T 115c4a12440647a4bb46eb6824f7ea98 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 0.
I20240430 20:11:42.927930 18558 leader_election.cc:302] T 115c4a12440647a4bb46eb6824f7ea98 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:42.928484 20395 raft_consensus.cc:2781] T 115c4a12440647a4bb46eb6824f7ea98 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:42.928560 20395 raft_consensus.cc:482] T 115c4a12440647a4bb46eb6824f7ea98 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:42.928620 20395 raft_consensus.cc:3037] T 115c4a12440647a4bb46eb6824f7ea98 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:42.930408 19471 ts_tablet_manager.cc:1875] T 1555924b5f1444398f12ee82c3cb5b13 P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:42.931818 20395 raft_consensus.cc:504] T 115c4a12440647a4bb46eb6824f7ea98 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:42.932121 19023 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_0fcc91de96a946018c72ba03c71bcc05" } new_table_name: "default.Table_3776a4b9d4f24aa6b7f882bb4d0116ef" modify_external_catalogs: true
I20240430 20:11:42.933454 18478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "115c4a12440647a4bb46eb6824f7ea98" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:42.933670 18478 raft_consensus.cc:3037] T 115c4a12440647a4bb46eb6824f7ea98 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:42.934213 19051 master_service.cc:946] ReplaceTablet: received request to replace tablet 8901327b3efb4201adce61a9266142ea from {username='slave'} at 127.0.0.1:51664
I20240430 20:11:42.935947 18668 heartbeater.cc:347] Connected to a master server at 127.17.177.62:36441
I20240430 20:11:42.936725 18478 raft_consensus.cc:2454] T 115c4a12440647a4bb46eb6824f7ea98 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
I20240430 20:11:42.936770 19026 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_ffb3192f443d437ab63f56f628e596ad" } modify_external_catalogs: true
I20240430 20:11:42.936978 19026 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_ffb3192f443d437ab63f56f628e596ad" } modify_external_catalogs: true
I20240430 20:11:42.938143 18558 leader_election.cc:302] T 115c4a12440647a4bb46eb6824f7ea98 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:42.938421 20324 raft_consensus.cc:2781] T 115c4a12440647a4bb46eb6824f7ea98 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:42.938560 20324 raft_consensus.cc:686] T 115c4a12440647a4bb46eb6824f7ea98 P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Becoming Leader. State: Replica: 5e1c6397f09747f091534faf93542710, State: Running, Role: LEADER
W20240430 20:11:42.938678 19033 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dfb59f35018441fd9a72ea860cdd01f6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:42.938839 20324 consensus_queue.cc:237] T 115c4a12440647a4bb46eb6824f7ea98 P 5e1c6397f09747f091534faf93542710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:42.940572 18733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb7554e115d84e6ca4f05509c5f33e2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:42.941007 20395 leader_election.cc:288] T 115c4a12440647a4bb46eb6824f7ea98 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:42.941135 18724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f3996b7e29941e391ea141dd6cc44b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:42.942380 18729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e28529e1c2ae4c7c9b9b0e4bd709d132. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:42.945741 18461 tablet_service.cc:1467] Processing CreateTablet for tablet e28529e1c2ae4c7c9b9b0e4bd709d132 (DEFAULT_TABLE table=default.Table_d8df23edb8ce480bb59a2ab0f1fc7521 [id=4c9193fcff2f41768d578803372434e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:42.945789 18457 tablet_service.cc:1467] Processing CreateTablet for tablet eb7554e115d84e6ca4f05509c5f33e2a (DEFAULT_TABLE table=default.Table_4bb81ccd60a148d197b8a4ebd9c9cc49 [id=ee0542402cc1426681e9583fbb7340e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:42.947234 18447 tablet_service.cc:1514] Processing DeleteTablet for tablet a05e32a33aa7488eb287cc6314c09c44 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:42 UTC) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:42.948910 18447 tablet_service.cc:1514] Processing DeleteTablet for tablet 84568f90b9c54cf292099808d4cbf5dc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:34 UTC) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:42.949551 18447 tablet_service.cc:1514] Processing DeleteTablet for tablet 2cdccf4be7094ca5b863dd5c3432d55d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:30 UTC) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:42.950203 18617 raft_consensus.cc:1261] T 46251c08277e498da04a41d009310c62 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Refusing update from remote peer 2efb542aa55e439b99ccb98d1c5c884a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240430 20:11:42.950664 18985 ts_tablet_manager.cc:1398] T 115c4a12440647a4bb46eb6824f7ea98 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.159s user 0.005s sys 0.001s
I20240430 20:11:42.950950 20409 consensus_queue.cc:1035] T 46251c08277e498da04a41d009310c62 P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:42.951886 19027 catalog_manager.cc:5484] T 724e51ff05ec4baebc2bb55e05c8eddc P 2efb542aa55e439b99ccb98d1c5c884a reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "2efb542aa55e439b99ccb98d1c5c884a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } health_report { overall_health: HEALTHY } } }
I20240430 20:11:42.952642 19027 catalog_manager.cc:5484] T e3e74e76ab2b4fe492ffcc1838656fd1 P 2efb542aa55e439b99ccb98d1c5c884a reported cstate change: term changed from 2 to 3, leader changed from 5e1c6397f09747f091534faf93542710 (127.17.177.2) to 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1). New cstate: current_term: 3 leader_uuid: "2efb542aa55e439b99ccb98d1c5c884a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } health_report { overall_health: HEALTHY } } }
I20240430 20:11:42.954488 19027 catalog_manager.cc:5484] T 43b5e2c005a34df08cc4335c6e014cb5 P 2efb542aa55e439b99ccb98d1c5c884a reported cstate change: term changed from 0 to 1, leader changed from <none> to 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1). New cstate: current_term: 1 leader_uuid: "2efb542aa55e439b99ccb98d1c5c884a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } health_report { overall_health: HEALTHY } } }
I20240430 20:11:42.955649 20105 consensus_queue.cc:1035] T ae87f5e92a934e2182164c231bc76df0 P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:42.961298 18734 tablet_service.cc:1514] Processing DeleteTablet for tablet a05e32a33aa7488eb287cc6314c09c44 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:42 UTC) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:42.961585 18736 tablet_service.cc:1514] Processing DeleteTablet for tablet 7542c7fe6cad452ab718f9ef54dbbbbe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:29 UTC) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:42.961769 18734 tablet_service.cc:1514] Processing DeleteTablet for tablet 75ebf051383e42529dcb5f7dd0770a4b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:27 UTC) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:42.961920 18736 tablet_service.cc:1514] Processing DeleteTablet for tablet 47c37876694c4a4abb1091c631dab7a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:27 UTC) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:42.962061 18734 tablet_service.cc:1514] Processing DeleteTablet for tablet 255c13ad8fae4ea4aa8e7b7aabb7602e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:27 UTC) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:42.962548 18601 tablet_service.cc:1514] Processing DeleteTablet for tablet 73453a7fa30342e29db552b17b62f0c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:32 UTC) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:42.964157 18601 tablet_service.cc:1514] Processing DeleteTablet for tablet 87ac32a0ddb84676adc83038ef664591 with delete_type TABLET_DATA_DELETED (replaced by tablet 331c031d44854d679a27d10d9151d0ea) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:42.965040 20019 leader_election.cc:288] T 87ac32a0ddb84676adc83038ef664591 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:42.968253 18735 tablet_service.cc:1514] Processing DeleteTablet for tablet a663030d37614053a7c0369841403422 with delete_type TABLET_DATA_DELETED (Sending initial creation of tablet) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:42.968482 20019 raft_consensus.cc:482] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 3c51c33d5616482cafb68e26688820f4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:42.968590 20019 raft_consensus.cc:504] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 3c51c33d5616482cafb68e26688820f4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:42.968997 20398 tablet.cc:1722] T 46251c08277e498da04a41d009310c62 P 2efb542aa55e439b99ccb98d1c5c884a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240430 20:11:42.969770 18988 ts_tablet_manager.cc:1398] T 87ac32a0ddb84676adc83038ef664591 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.213s user 0.002s sys 0.001s
I20240430 20:11:42.971501 20397 tablet.cc:1722] T 46251c08277e498da04a41d009310c62 P 5e1c6397f09747f091534faf93542710: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240430 20:11:42.972152 18750 raft_consensus.cc:1261] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 3c51c33d5616482cafb68e26688820f4 [term 2 FOLLOWER]: Refusing update from remote peer 2efb542aa55e439b99ccb98d1c5c884a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240430 20:11:42.972594 18616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd6083f12c7e4d1e8a2006b3ddf9aa3a" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
W20240430 20:11:42.973222 18696 leader_election.cc:341] T 87ac32a0ddb84676adc83038ef664591 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Not found: Tablet not found: 87ac32a0ddb84676adc83038ef664591
I20240430 20:11:42.974939 19049 master_service.cc:946] ReplaceTablet: received request to replace tablet 6d1d2193fd424c91b143f5256df5fcd0 from {username='slave'} at 127.0.0.1:51664
I20240430 20:11:42.976119 20395 raft_consensus.cc:482] T a59ef6c545dd41a28265a4c1b8ce10e0 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:42.976250 20395 raft_consensus.cc:504] T a59ef6c545dd41a28265a4c1b8ce10e0 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:42.976843 20395 leader_election.cc:288] T a59ef6c545dd41a28265a4c1b8ce10e0 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:42.973029 18750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aec9c09cc0454acab6006bb3e10982b7" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:42.979306 18750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aec9c09cc0454acab6006bb3e10982b7" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:42.979399 20019 leader_election.cc:288] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:42.990775 18988 tablet_bootstrap.cc:492] T c2b019f529a54228b804437bb025e31d P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:42.992182 18988 tablet_bootstrap.cc:654] T c2b019f529a54228b804437bb025e31d P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
W20240430 20:11:42.993973 19023 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1701a15b115741aea13f8327e3d26b34 consensus state has no leader
I20240430 20:11:42.995064 19023 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_208b12b8e9bd4f3e8bc9c6d422ada649" } new_table_name: "default.Table_938c56d9ae2f4ff6b3264a53b65d0977" modify_external_catalogs: true
I20240430 20:11:42.997675 18478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a59ef6c545dd41a28265a4c1b8ce10e0" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:42.997903 18478 raft_consensus.cc:2454] T a59ef6c545dd41a28265a4c1b8ce10e0 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 0.
I20240430 20:11:42.998144 18750 raft_consensus.cc:3037] T ca64de16d40f4a44b8b411f5526ba11a P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:42.998535 18558 leader_election.cc:302] T a59ef6c545dd41a28265a4c1b8ce10e0 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:42.998813 20324 raft_consensus.cc:2781] T a59ef6c545dd41a28265a4c1b8ce10e0 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:42.998898 20324 raft_consensus.cc:482] T a59ef6c545dd41a28265a4c1b8ce10e0 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:42.998970 20324 raft_consensus.cc:3037] T a59ef6c545dd41a28265a4c1b8ce10e0 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:43.001551 20324 raft_consensus.cc:504] T a59ef6c545dd41a28265a4c1b8ce10e0 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:43.002174 20324 leader_election.cc:288] T a59ef6c545dd41a28265a4c1b8ce10e0 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:43.002902 18985 tablet_bootstrap.cc:492] T 681cdd85ac6847a0a1aaded310456582 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:43.003754 18478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a59ef6c545dd41a28265a4c1b8ce10e0" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:43.003939 18478 raft_consensus.cc:3037] T a59ef6c545dd41a28265a4c1b8ce10e0 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:43.006915 18764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f02d8862e1f4ccfa9dbda73dbe18615" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:42.997123 18758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f02d8862e1f4ccfa9dbda73dbe18615" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:43.007944 19471 ts_tablet_manager.cc:1888] T 1555924b5f1444398f12ee82c3cb5b13 P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.2
I20240430 20:11:43.008440 19471 log.cc:1196] T 1555924b5f1444398f12ee82c3cb5b13 P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/1555924b5f1444398f12ee82c3cb5b13
I20240430 20:11:43.010728 18985 tablet_bootstrap.cc:654] T 681cdd85ac6847a0a1aaded310456582 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:43.012324 18764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "115c4a12440647a4bb46eb6824f7ea98" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:43.012498 18756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a59ef6c545dd41a28265a4c1b8ce10e0" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:43.012715 18762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "115c4a12440647a4bb46eb6824f7ea98" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
W20240430 20:11:43.013420 18699 leader_election.cc:341] T 87ac32a0ddb84676adc83038ef664591 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Not found: Tablet not found: 87ac32a0ddb84676adc83038ef664591
I20240430 20:11:43.013537 18699 leader_election.cc:302] T 87ac32a0ddb84676adc83038ef664591 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710
I20240430 20:11:43.013854 18754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a59ef6c545dd41a28265a4c1b8ce10e0" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:43.014494 20205 raft_consensus.cc:2726] T 87ac32a0ddb84676adc83038ef664591 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:43.014544 20019 raft_consensus.cc:482] T ca64de16d40f4a44b8b411f5526ba11a P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:43.014698 20019 raft_consensus.cc:504] T ca64de16d40f4a44b8b411f5526ba11a P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:43.015275 20019 leader_election.cc:288] T ca64de16d40f4a44b8b411f5526ba11a P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
W20240430 20:11:43.015839 18417 consensus_peers.cc:485] T ae87f5e92a934e2182164c231bc76df0 P 2efb542aa55e439b99ccb98d1c5c884a -> Peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Couldn't send request to peer 3c51c33d5616482cafb68e26688820f4. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240430 20:11:43.016256 20326 raft_consensus.cc:482] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader b632d6ef4c104a058a6db208024cab33)
I20240430 20:11:43.016512 20326 raft_consensus.cc:504] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b3f98793354c7aaa5c106e8ec31ed8" member_type: VOTER last_known_addr { host: "127.17.177.62" port: 36441 } } peers { permanent_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e" member_type: VOTER last_known_addr { host: "127.17.177.61" port: 34777 } } peers { permanent_uuid: "b632d6ef4c104a058a6db208024cab33" member_type: VOTER last_known_addr { host: "127.17.177.60" port: 37225 } }
I20240430 20:11:43.018851 18750 raft_consensus.cc:3037] T 46251c08277e498da04a41d009310c62 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:43.019253 20327 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b632d6ef4c104a058a6db208024cab33. Latest consensus state: current_term: 5 leader_uuid: "b632d6ef4c104a058a6db208024cab33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b3f98793354c7aaa5c106e8ec31ed8" member_type: VOTER last_known_addr { host: "127.17.177.62" port: 36441 } } peers { permanent_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e" member_type: VOTER last_known_addr { host: "127.17.177.61" port: 34777 } } peers { permanent_uuid: "b632d6ef4c104a058a6db208024cab33" member_type: VOTER last_known_addr { host: "127.17.177.60" port: 37225 } } }
I20240430 20:11:43.019685 20327 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8 [sys.catalog]: This master's current role is: FOLLOWER
I20240430 20:11:43.020187 20406 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 5 leader_uuid: "b632d6ef4c104a058a6db208024cab33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b3f98793354c7aaa5c106e8ec31ed8" member_type: VOTER last_known_addr { host: "127.17.177.62" port: 36441 } } peers { permanent_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e" member_type: VOTER last_known_addr { host: "127.17.177.61" port: 34777 } } peers { permanent_uuid: "b632d6ef4c104a058a6db208024cab33" member_type: VOTER last_known_addr { host: "127.17.177.60" port: 37225 } } }
I20240430 20:11:43.020589 20406 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8 [sys.catalog]: This master's current role is: FOLLOWER
I20240430 20:11:43.021898 18750 raft_consensus.cc:1261] T 46251c08277e498da04a41d009310c62 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Refusing update from remote peer 2efb542aa55e439b99ccb98d1c5c884a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240430 20:11:43.022037 18482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd6083f12c7e4d1e8a2006b3ddf9aa3a" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:43.022454 18474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca64de16d40f4a44b8b411f5526ba11a" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:43.022650 18474 raft_consensus.cc:2454] T ca64de16d40f4a44b8b411f5526ba11a P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:43.027772 18478 raft_consensus.cc:2454] T a59ef6c545dd41a28265a4c1b8ce10e0 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
I20240430 20:11:43.030702 20326 leader_election.cc:288] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 89df84ccf5394c9bb60d4e4ed872fc2e (127.17.177.61:34777), b632d6ef4c104a058a6db208024cab33 (127.17.177.60:37225)
I20240430 20:11:43.022511 18615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca64de16d40f4a44b8b411f5526ba11a" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:43.033049 19471 ts_tablet_manager.cc:1909] T 1555924b5f1444398f12ee82c3cb5b13 P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:43.037356 18446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f3996b7e29941e391ea141dd6cc44b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:43.038841 18461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e28529e1c2ae4c7c9b9b0e4bd709d132. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:43.039355 18457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb7554e115d84e6ca4f05509c5f33e2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:43.047688 20408 consensus_queue.cc:1035] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240430 20:11:43.049067 19495 ts_tablet_manager.cc:1888] T 24dcc20832804229a2664873bf6ddaa7 P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:43.049201 19495 log.cc:1196] T 24dcc20832804229a2664873bf6ddaa7 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/24dcc20832804229a2664873bf6ddaa7
I20240430 20:11:43.050065 18750 raft_consensus.cc:1261] T 7bde1d6934784c9aa5911181e8df9eeb P 3c51c33d5616482cafb68e26688820f4 [term 2 FOLLOWER]: Refusing update from remote peer 2efb542aa55e439b99ccb98d1c5c884a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240430 20:11:43.051815 19980 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33764 (request call id 0) took 4636 ms. Trace:
I20240430 20:11:43.031790 18983 tablet_bootstrap.cc:492] T 30a521916bda46678be20a066bd1d3f3 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:43.052268 18983 ts_tablet_manager.cc:1367] T 30a521916bda46678be20a066bd1d3f3 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.136s user 0.008s sys 0.000s
W20240430 20:11:43.029134 18557 leader_election.cc:341] T 5f02d8862e1f4ccfa9dbda73dbe18615 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
W20240430 20:11:43.052866 18557 leader_election.cc:341] T 5f02d8862e1f4ccfa9dbda73dbe18615 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:43.053002 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 140) took 10446 ms. Trace:
I20240430 20:11:43.053083 19471 rpcz_store.cc:276] 0430 20:11:32.606274 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:32.606335 (+ 61us) service_pool.cc:225] Handling call
0430 20:11:42.920932 (+10314597us) ts_tablet_manager.cc:1120] Deleting tablet 1555924b5f1444398f12ee82c3cb5b13
0430 20:11:42.920945 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:42.933498 (+ 12553us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:43.007825 (+ 74327us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:43.052980 (+ 45155us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":10309387}
W20240430 20:11:43.053244 18557 leader_election.cc:341] T 115c4a12440647a4bb46eb6824f7ea98 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:43.053416 19471 tablet_replica.cc:330] T 20d166756819406ba85044f4fee9c4ce P 5e1c6397f09747f091534faf93542710: stopping tablet replica
W20240430 20:11:43.053519 18557 leader_election.cc:341] T a59ef6c545dd41a28265a4c1b8ce10e0 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:43.053632 19471 raft_consensus.cc:2229] T 20d166756819406ba85044f4fee9c4ce P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Raft consensus shutting down.
W20240430 20:11:43.053809 18557 leader_election.cc:341] T 115c4a12440647a4bb46eb6824f7ea98 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:43.053964 19471 raft_consensus.cc:2258] T 20d166756819406ba85044f4fee9c4ce P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240430 20:11:43.054116 18557 leader_election.cc:341] T a59ef6c545dd41a28265a4c1b8ce10e0 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:43.054077 20408 consensus_queue.cc:1035] T 46251c08277e498da04a41d009310c62 P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:43.054502 19471 ts_tablet_manager.cc:1875] T 20d166756819406ba85044f4fee9c4ce P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:43.054692 20395 consensus_queue.cc:1035] T ca64de16d40f4a44b8b411f5526ba11a P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:43.056161 18983 raft_consensus.cc:348] T 30a521916bda46678be20a066bd1d3f3 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:43.056335 18983 raft_consensus.cc:374] T 30a521916bda46678be20a066bd1d3f3 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:43.056380 18983 raft_consensus.cc:729] T 30a521916bda46678be20a066bd1d3f3 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:43.056586 18983 consensus_queue.cc:260] T 30a521916bda46678be20a066bd1d3f3 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:43.057086 18983 ts_tablet_manager.cc:1398] T 30a521916bda46678be20a066bd1d3f3 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240430 20:11:43.057364 18983 tablet_bootstrap.cc:492] T 50486644ec234933b9a0b45ee39a6685 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
W20240430 20:11:43.034723 18558 consensus_peers.cc:485] T 6e826459c3f4437da81c9e6e09f0c22f P 5e1c6397f09747f091534faf93542710 -> Peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Couldn't send request to peer 2efb542aa55e439b99ccb98d1c5c884a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6e826459c3f4437da81c9e6e09f0c22f. This is attempt 1: this message will repeat every 5th retry.
I20240430 20:11:43.061993 18558 leader_election.cc:302] T a59ef6c545dd41a28265a4c1b8ce10e0 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters: 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:43.062546 18999 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet 1555924b5f1444398f12ee82c3cb5b13 (table default.Table_388d774462294cab8342c6de331f9712 [id=aca48e01301e4f5f9562df8c87829b9d]) successfully deleted
I20240430 20:11:43.063282 20047 raft_consensus.cc:2781] T a59ef6c545dd41a28265a4c1b8ce10e0 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:43.063400 20047 raft_consensus.cc:686] T a59ef6c545dd41a28265a4c1b8ce10e0 P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Becoming Leader. State: Replica: 5e1c6397f09747f091534faf93542710, State: Running, Role: LEADER
I20240430 20:11:43.063668 20047 consensus_queue.cc:237] T a59ef6c545dd41a28265a4c1b8ce10e0 P 5e1c6397f09747f091534faf93542710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:43.067531 19495 ts_tablet_manager.cc:1909] T 24dcc20832804229a2664873bf6ddaa7 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:43.067677 20395 raft_consensus.cc:482] T e6eac92dfa5f4456bc49571878f0f8dd P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:43.067811 20395 raft_consensus.cc:504] T e6eac92dfa5f4456bc49571878f0f8dd P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:43.068413 20395 leader_election.cc:288] T e6eac92dfa5f4456bc49571878f0f8dd P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:43.070130 19051 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: Sending DeleteTablet for 3 replicas of tablet 8901327b3efb4201adce61a9266142ea
I20240430 20:11:43.070117 20395 consensus_queue.cc:1035] T ca64de16d40f4a44b8b411f5526ba11a P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:43.070703 19051 catalog_manager.cc:6554] ReplaceTablet: tablet 8901327b3efb4201adce61a9266142ea deleted and replaced by tablet b1277899f0c040cea7dc2b12f3fd68de
W20240430 20:11:43.071568 18558 leader_election.cc:341] T e6eac92dfa5f4456bc49571878f0f8dd P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Not found: Tablet not found: e6eac92dfa5f4456bc49571878f0f8dd
I20240430 20:11:43.071609 19026 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: Sending DeleteTablet for 3 replicas of tablet aa9998f4e17147aa8f2b35c01980463a
I20240430 20:11:43.075325 18457 tablet_service.cc:1514] Processing DeleteTablet for tablet 8901327b3efb4201adce61a9266142ea with delete_type TABLET_DATA_DELETED (replaced by tablet b1277899f0c040cea7dc2b12f3fd68de) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:43.075562 18461 tablet_service.cc:1514] Processing DeleteTablet for tablet aa9998f4e17147aa8f2b35c01980463a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:42 UTC) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:43.072901 18601 tablet_service.cc:1514] Processing DeleteTablet for tablet 8901327b3efb4201adce61a9266142ea with delete_type TABLET_DATA_DELETED (replaced by tablet b1277899f0c040cea7dc2b12f3fd68de) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:43.079412 18601 tablet_service.cc:1514] Processing DeleteTablet for tablet aa9998f4e17147aa8f2b35c01980463a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:42 UTC) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:43.084594 18699 leader_election.cc:302] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710
I20240430 20:11:43.087002 18699 leader_election.cc:302] T ca64de16d40f4a44b8b411f5526ba11a P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters: 5e1c6397f09747f091534faf93542710
I20240430 20:11:43.087764 20019 raft_consensus.cc:2781] T ca64de16d40f4a44b8b411f5526ba11a P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240430 20:11:43.087853 20019 raft_consensus.cc:482] T ca64de16d40f4a44b8b411f5526ba11a P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:43.087930 20019 raft_consensus.cc:3037] T ca64de16d40f4a44b8b411f5526ba11a P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Advancing to term 2
I20240430 20:11:43.088373 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 206) took 4186 ms. Trace:
I20240430 20:11:43.088474 19495 rpcz_store.cc:276] 0430 20:11:38.902219 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:38.902302 (+ 83us) service_pool.cc:225] Handling call
0430 20:11:42.784784 (+3882482us) ts_tablet_manager.cc:1120] Deleting tablet 24dcc20832804229a2664873bf6ddaa7
0430 20:11:42.784794 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:42.946699 (+161905us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:43.048965 (+102266us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:43.088348 (+ 39383us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3849389}
I20240430 20:11:43.088920 19004 catalog_manager.cc:4830] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): tablet 24dcc20832804229a2664873bf6ddaa7 (table default.Table_be2ebe77f859489f9cd3e912cb1edc8b [id=2d44ec94131148048021b2ee6366ba30]) successfully deleted
I20240430 20:11:43.092063 20198 raft_consensus.cc:2726] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 3c51c33d5616482cafb68e26688820f4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240430 20:11:43.092795 20409 consensus_queue.cc:1035] T 7bde1d6934784c9aa5911181e8df9eeb P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240430 20:11:43.095642 19049 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: Sending DeleteTablet for 3 replicas of tablet 6d1d2193fd424c91b143f5256df5fcd0
I20240430 20:11:43.096213 19049 catalog_manager.cc:6554] ReplaceTablet: tablet 6d1d2193fd424c91b143f5256df5fcd0 deleted and replaced by tablet bbdbb8a3a4b444f58a2761466577a88f
I20240430 20:11:43.097966 18601 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d1d2193fd424c91b143f5256df5fcd0 with delete_type TABLET_DATA_DELETED (replaced by tablet bbdbb8a3a4b444f58a2761466577a88f) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:43.098165 18461 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d1d2193fd424c91b143f5256df5fcd0 with delete_type TABLET_DATA_DELETED (replaced by tablet bbdbb8a3a4b444f58a2761466577a88f) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:43.098644 19026 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_ba82d0fed0264b818de8278e1d142e5d" } modify_external_catalogs: true
I20240430 20:11:43.098910 19026 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_ba82d0fed0264b818de8278e1d142e5d" } modify_external_catalogs: true
I20240430 20:11:43.099623 19048 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_80306c0ee72842209ca8feac52c0f247"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:43.100312 19048 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_80306c0ee72842209ca8feac52c0f247 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:43.101900 19051 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_9c1e6182953140bb9cd594f08458c7d9" } new_table_name: "default.Table_b064d5659f7243f5b0ec167fb6c3f404" modify_external_catalogs: true
I20240430 20:11:43.105903 19049 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_47217f7ee5ef4a50a2e9f146412b198a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:43.106490 19049 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_47217f7ee5ef4a50a2e9f146412b198a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:43.107545 18615 raft_consensus.cc:1261] T 7bde1d6934784c9aa5911181e8df9eeb P 5e1c6397f09747f091534faf93542710 [term 2 FOLLOWER]: Refusing update from remote peer 2efb542aa55e439b99ccb98d1c5c884a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240430 20:11:43.108075 19029 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_be2ebe77f859489f9cd3e912cb1edc8b" } new_table_name: "default.Table_e3160f5ced1b4718b479401ca4df675c" modify_external_catalogs: true
I20240430 20:11:43.108248 20409 consensus_queue.cc:1035] T 7bde1d6934784c9aa5911181e8df9eeb P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.003s
I20240430 20:11:43.111747 19033 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_204dce4e6128482583d8869c83413f25"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240430 20:11:43.107286 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 208) took 4004 ms. Trace:
I20240430 20:11:43.112183 19495 rpcz_store.cc:276] 0430 20:11:39.102649 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:39.102726 (+ 77us) service_pool.cc:225] Handling call
0430 20:11:43.107142 (+4004416us) ts_tablet_manager.cc:1120] Deleting tablet 4e69ca12bd10494fb013614741bc166c
0430 20:11:43.107158 (+ 16us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:43.107259 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4003936}
W20240430 20:11:43.112618 19033 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_204dce4e6128482583d8869c83413f25 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240430 20:11:43.114848 19004 catalog_manager.cc:4797] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): delete failed for tablet 4e69ca12bd10494fb013614741bc166c because the tablet was not found. No further retry: Not found: Tablet not found: 4e69ca12bd10494fb013614741bc166c
I20240430 20:11:43.116649 19023 master_service.cc:946] ReplaceTablet: received request to replace tablet acf3b6fead41483a82c43dc0286fab7c from {username='slave'} at 127.0.0.1:51664
I20240430 20:11:43.117023 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 207) took 4024 ms. Trace:
I20240430 20:11:43.117106 19495 rpcz_store.cc:276] 0430 20:11:39.092479 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:39.093187 (+ 708us) service_pool.cc:225] Handling call
0430 20:11:43.112502 (+4019315us) ts_tablet_manager.cc:1120] Deleting tablet c5fd0730133c472299c93f4b0830e527
0430 20:11:43.116920 (+ 4418us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:43.117004 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3969266}
I20240430 20:11:43.117511 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 216) took 3575 ms. Trace:
W20240430 20:11:43.118418 19004 catalog_manager.cc:4797] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): delete failed for tablet c5fd0730133c472299c93f4b0830e527 because the tablet was not found. No further retry: Not found: Tablet not found: c5fd0730133c472299c93f4b0830e527
I20240430 20:11:43.090574 20019 raft_consensus.cc:504] T ca64de16d40f4a44b8b411f5526ba11a P 3c51c33d5616482cafb68e26688820f4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:43.121381 20019 leader_election.cc:288] T ca64de16d40f4a44b8b411f5526ba11a P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 2 election: Requested vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:43.125808 18616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca64de16d40f4a44b8b411f5526ba11a" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:43.127370 18733 tablet_service.cc:1514] Processing DeleteTablet for tablet aa9998f4e17147aa8f2b35c01980463a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:42 UTC) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:43.127298 18729 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d1d2193fd424c91b143f5256df5fcd0 with delete_type TABLET_DATA_DELETED (replaced by tablet bbdbb8a3a4b444f58a2761466577a88f) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:43.128635 20424 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20240430 20:11:43.128806 18763 raft_consensus.cc:1226] T ca64de16d40f4a44b8b411f5526ba11a P 3c51c33d5616482cafb68e26688820f4 [term 2 FOLLOWER]: Rejecting Update request from peer 5e1c6397f09747f091534faf93542710 for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20240430 20:11:43.128932 18724 tablet_service.cc:1514] Processing DeleteTablet for tablet 8901327b3efb4201adce61a9266142ea with delete_type TABLET_DATA_DELETED (replaced by tablet b1277899f0c040cea7dc2b12f3fd68de) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:43.129576 18756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6eac92dfa5f4456bc49571878f0f8dd" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:43.129781 18756 raft_consensus.cc:2454] T e6eac92dfa5f4456bc49571878f0f8dd P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
I20240430 20:11:43.131083 20396 consensus_queue.cc:1046] T ca64de16d40f4a44b8b411f5526ba11a P 5e1c6397f09747f091534faf93542710 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:43.132321 18557 leader_election.cc:302] T e6eac92dfa5f4456bc49571878f0f8dd P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a
I20240430 20:11:43.132689 20395 raft_consensus.cc:2781] T e6eac92dfa5f4456bc49571878f0f8dd P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240430 20:11:43.132755 20395 raft_consensus.cc:482] T e6eac92dfa5f4456bc49571878f0f8dd P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:43.132827 20395 raft_consensus.cc:3037] T e6eac92dfa5f4456bc49571878f0f8dd P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Advancing to term 2
I20240430 20:11:43.136858 19489 ts_tablet_manager.cc:1888] T fae9a35a2ac844b4bc2ebdf214aa7171 P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240430 20:11:43.138921 20425 tablet.cc:1722] T c9e7660eccc448dbb99d135f82bfd261 P 3c51c33d5616482cafb68e26688820f4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240430 20:11:43.140448 20395 raft_consensus.cc:504] T e6eac92dfa5f4456bc49571878f0f8dd P 5e1c6397f09747f091534faf93542710 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:43.141687 20395 leader_election.cc:288] T e6eac92dfa5f4456bc49571878f0f8dd P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 2 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:43.142326 18763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6eac92dfa5f4456bc49571878f0f8dd" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:43.142459 18983 tablet_bootstrap.cc:654] T 50486644ec234933b9a0b45ee39a6685 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:43.142567 18763 raft_consensus.cc:3037] T e6eac92dfa5f4456bc49571878f0f8dd P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Advancing to term 2
I20240430 20:11:43.145839 18763 raft_consensus.cc:2454] T e6eac92dfa5f4456bc49571878f0f8dd P 3c51c33d5616482cafb68e26688820f4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 2.
I20240430 20:11:43.146313 18557 leader_election.cc:302] T e6eac92dfa5f4456bc49571878f0f8dd P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:43.146765 20395 raft_consensus.cc:2781] T e6eac92dfa5f4456bc49571878f0f8dd P 5e1c6397f09747f091534faf93542710 [term 2 FOLLOWER]: Leader election won for term 2
I20240430 20:11:43.147069 20395 raft_consensus.cc:686] T e6eac92dfa5f4456bc49571878f0f8dd P 5e1c6397f09747f091534faf93542710 [term 2 LEADER]: Becoming Leader. State: Replica: 5e1c6397f09747f091534faf93542710, State: Running, Role: LEADER
I20240430 20:11:43.147431 20395 consensus_queue.cc:237] T e6eac92dfa5f4456bc49571878f0f8dd P 5e1c6397f09747f091534faf93542710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:43.151736 20397 tablet.cc:1722] T 6e754af64cb64fd2a082adae7d5f24da P 5e1c6397f09747f091534faf93542710: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240430 20:11:43.157605 18983 tablet_bootstrap.cc:492] T 50486644ec234933b9a0b45ee39a6685 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:43.157898 18983 ts_tablet_manager.cc:1367] T 50486644ec234933b9a0b45ee39a6685 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.101s user 0.005s sys 0.003s
I20240430 20:11:43.158972 18983 raft_consensus.cc:348] T 50486644ec234933b9a0b45ee39a6685 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:43.159338 18983 raft_consensus.cc:374] T 50486644ec234933b9a0b45ee39a6685 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:43.159463 18983 raft_consensus.cc:729] T 50486644ec234933b9a0b45ee39a6685 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:43.159775 18983 consensus_queue.cc:260] T 50486644ec234933b9a0b45ee39a6685 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:43.160532 18983 ts_tablet_manager.cc:1398] T 50486644ec234933b9a0b45ee39a6685 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240430 20:11:43.160915 18983 tablet_bootstrap.cc:492] T abdaf243e5be4ad58036b23151b14fe4 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:43.162787 18983 tablet_bootstrap.cc:654] T abdaf243e5be4ad58036b23151b14fe4 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:43.167363 19945 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "5e1c6397f09747f091534faf93542710" instance_seqno: 1714507880456452) as {username='slave'} at 127.17.177.2:50593; Asking this server to re-register.
I20240430 20:11:43.171118 18988 tablet_bootstrap.cc:492] T c2b019f529a54228b804437bb025e31d P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:43.171192 18668 heartbeater.cc:464] Registering TS with master...
I20240430 20:11:43.171408 18988 ts_tablet_manager.cc:1367] T c2b019f529a54228b804437bb025e31d P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.181s user 0.007s sys 0.000s
I20240430 20:11:43.172133 18988 raft_consensus.cc:348] T c2b019f529a54228b804437bb025e31d P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:43.172330 18988 raft_consensus.cc:374] T c2b019f529a54228b804437bb025e31d P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:43.172379 18988 raft_consensus.cc:729] T c2b019f529a54228b804437bb025e31d P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:43.183744 18988 consensus_queue.cc:260] T c2b019f529a54228b804437bb025e31d P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:43.184502 18988 ts_tablet_manager.cc:1398] T c2b019f529a54228b804437bb025e31d P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20240430 20:11:43.184903 18988 tablet_bootstrap.cc:492] T 43b5e2c005a34df08cc4335c6e014cb5 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:43.185798 19942 ts_manager.cc:194] Registered new tserver with Master: 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:43.186333 18988 tablet_bootstrap.cc:654] T 43b5e2c005a34df08cc4335c6e014cb5 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:43.188804 20442 tablet.cc:1722] T 6e754af64cb64fd2a082adae7d5f24da P 3c51c33d5616482cafb68e26688820f4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240430 20:11:43.194518 20444 tablet.cc:1722] T 46251c08277e498da04a41d009310c62 P 3c51c33d5616482cafb68e26688820f4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20240430 20:11:43.198283 18417 consensus_peers.cc:485] T 34130f2820064fa0898ef1f9d7de1c6d P 2efb542aa55e439b99ccb98d1c5c884a -> Peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Couldn't send request to peer 3c51c33d5616482cafb68e26688820f4. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240430 20:11:43.199040 20408 raft_consensus.cc:482] T 5bd0362ff41b4faca0205d136d596737 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:43.199177 20408 raft_consensus.cc:504] T 5bd0362ff41b4faca0205d136d596737 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:43.201489 19489 log.cc:1196] T fae9a35a2ac844b4bc2ebdf214aa7171 P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/fae9a35a2ac844b4bc2ebdf214aa7171
I20240430 20:11:43.201930 19489 ts_tablet_manager.cc:1909] T fae9a35a2ac844b4bc2ebdf214aa7171 P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:43.202260 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 119) took 10924 ms. Trace:
I20240430 20:11:43.202349 19489 rpcz_store.cc:276] 0430 20:11:32.277555 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:32.278846 (+ 1291us) service_pool.cc:225] Handling call
0430 20:11:42.773251 (+10494405us) ts_tablet_manager.cc:1120] Deleting tablet fae9a35a2ac844b4bc2ebdf214aa7171
0430 20:11:42.773260 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:43.097700 (+324440us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:43.136749 (+ 39049us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:43.202242 (+ 65493us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":10494182}
I20240430 20:11:43.202845 19489 tablet_replica.cc:330] T 44fa6917a85c46f3aaa363df46ea586b P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:43.203049 19489 raft_consensus.cc:2229] T 44fa6917a85c46f3aaa363df46ea586b P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Raft consensus shutting down.
I20240430 20:11:43.203364 19489 raft_consensus.cc:2258] T 44fa6917a85c46f3aaa363df46ea586b P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:43.209672 20384 raft_consensus.cc:482] T 2e79df8744e74b5991a79ce978ff0202 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:43.209861 20384 raft_consensus.cc:504] T 2e79df8744e74b5991a79ce978ff0202 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:43.210584 20384 leader_election.cc:288] T 2e79df8744e74b5991a79ce978ff0202 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:43.216010 18750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e79df8744e74b5991a79ce978ff0202" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:43.216337 18474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca64de16d40f4a44b8b411f5526ba11a" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:43.217123 18615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bd0362ff41b4faca0205d136d596737" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:43.217433 18699 leader_election.cc:302] T ca64de16d40f4a44b8b411f5526ba11a P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710
W20240430 20:11:43.217718 18557 leader_election.cc:341] T 2e79df8744e74b5991a79ce978ff0202 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
W20240430 20:11:43.217720 18415 leader_election.cc:341] T 5bd0362ff41b4faca0205d136d596737 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:43.218128 20198 raft_consensus.cc:2726] T ca64de16d40f4a44b8b411f5526ba11a P 3c51c33d5616482cafb68e26688820f4 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240430 20:11:43.117581 19495 rpcz_store.cc:276] 0430 20:11:39.542201 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:39.542253 (+ 52us) service_pool.cc:225] Handling call
0430 20:11:43.117375 (+3575122us) ts_tablet_manager.cc:1120] Deleting tablet de1dfe6c82d34a4d8d02e466c55327ff
0430 20:11:43.117385 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:43.117493 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3568244}
I20240430 20:11:43.219100 18750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bd0362ff41b4faca0205d136d596737" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
W20240430 20:11:43.219426 19004 catalog_manager.cc:4810] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): delete failed for tablet de1dfe6c82d34a4d8d02e466c55327ff because tablet deleting was already in progress. No further retry: Already present: State transition of tablet de1dfe6c82d34a4d8d02e466c55327ff already in progress: creating tablet
W20240430 20:11:43.220055 18417 leader_election.cc:341] T 5bd0362ff41b4faca0205d136d596737 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:43.220204 18417 leader_election.cc:302] T 5bd0362ff41b4faca0205d136d596737 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a; no voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710
I20240430 20:11:43.220515 20105 raft_consensus.cc:2726] T 5bd0362ff41b4faca0205d136d596737 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240430 20:11:43.224503 19051 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 66dd767bd3f8494f94061fb89e5e5680 consensus state has no leader
I20240430 20:11:43.225329 19049 master_service.cc:946] ReplaceTablet: received request to replace tablet 32c823d1700949ab8b22adcd83b7bfd1 from {username='slave'} at 127.0.0.1:51664
I20240430 20:11:43.227864 20047 raft_consensus.cc:3032] T ca64de16d40f4a44b8b411f5526ba11a P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Stepping down as leader of term 1
I20240430 20:11:43.228041 20047 raft_consensus.cc:729] T ca64de16d40f4a44b8b411f5526ba11a P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Running, Role: LEADER
I20240430 20:11:43.228425 20047 consensus_queue.cc:260] T ca64de16d40f4a44b8b411f5526ba11a P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:43.228842 18750 raft_consensus.cc:1226] T ca64de16d40f4a44b8b411f5526ba11a P 3c51c33d5616482cafb68e26688820f4 [term 2 FOLLOWER]: Rejecting Update request from peer 5e1c6397f09747f091534faf93542710 for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20240430 20:11:43.228868 20047 raft_consensus.cc:3037] T ca64de16d40f4a44b8b411f5526ba11a P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Advancing to term 2
I20240430 20:11:43.216428 20408 leader_election.cc:288] T 5bd0362ff41b4faca0205d136d596737 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:43.232339 19489 ts_tablet_manager.cc:1875] T 44fa6917a85c46f3aaa363df46ea586b P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:43.232704 18474 raft_consensus.cc:1226] T e3e74e76ab2b4fe492ffcc1838656fd1 P 2efb542aa55e439b99ccb98d1c5c884a [term 3 LEADER]: Rejecting Update request from peer 5e1c6397f09747f091534faf93542710 for earlier term 2. Current term is 3. Ops: []
I20240430 20:11:43.232883 18983 tablet_bootstrap.cc:492] T abdaf243e5be4ad58036b23151b14fe4 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:43.233036 18983 ts_tablet_manager.cc:1367] T abdaf243e5be4ad58036b23151b14fe4 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.072s user 0.008s sys 0.009s
I20240430 20:11:43.234799 18983 raft_consensus.cc:348] T abdaf243e5be4ad58036b23151b14fe4 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:43.236073 18983 raft_consensus.cc:374] T abdaf243e5be4ad58036b23151b14fe4 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:43.236331 18983 raft_consensus.cc:729] T abdaf243e5be4ad58036b23151b14fe4 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:43.236723 18983 consensus_queue.cc:260] T abdaf243e5be4ad58036b23151b14fe4 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:43.237772 18983 ts_tablet_manager.cc:1398] T abdaf243e5be4ad58036b23151b14fe4 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240430 20:11:43.238206 18474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e79df8744e74b5991a79ce978ff0202" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:43.238914 18983 tablet_bootstrap.cc:492] T 7ce09fcbe1ac457a9c05efaf000c3263 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:43.239204 18988 tablet_bootstrap.cc:492] T 43b5e2c005a34df08cc4335c6e014cb5 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:43.227957 19942 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "3c51c33d5616482cafb68e26688820f4" instance_seqno: 1714507880977799) as {username='slave'} at 127.17.177.3:39881; Asking this server to re-register.
W20240430 20:11:43.241818 18558 leader_election.cc:341] T e6eac92dfa5f4456bc49571878f0f8dd P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Not found: Tablet not found: e6eac92dfa5f4456bc49571878f0f8dd
W20240430 20:11:43.242545 18558 consensus_peers.cc:485] T 84568f90b9c54cf292099808d4cbf5dc P 5e1c6397f09747f091534faf93542710 -> Peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Couldn't send request to peer 2efb542aa55e439b99ccb98d1c5c884a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 84568f90b9c54cf292099808d4cbf5dc. This is attempt 1: this message will repeat every 5th retry.
W20240430 20:11:43.242899 18558 leader_election.cc:341] T 2e79df8744e74b5991a79ce978ff0202 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:43.243131 18558 leader_election.cc:302] T 2e79df8744e74b5991a79ce978ff0202 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:43.243692 20384 consensus_queue.cc:1046] T e3e74e76ab2b4fe492ffcc1838656fd1 P 5e1c6397f09747f091534faf93542710 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:43.243958 20384 consensus_queue.cc:1225] T e3e74e76ab2b4fe492ffcc1838656fd1 P 5e1c6397f09747f091534faf93542710 [LEADER]: Peer 2efb542aa55e439b99ccb98d1c5c884a log is divergent from this leader: its last log entry 3.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 0
I20240430 20:11:43.244097 20384 raft_consensus.cc:2726] T 2e79df8744e74b5991a79ce978ff0202 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:43.244289 20384 raft_consensus.cc:3032] T e3e74e76ab2b4fe492ffcc1838656fd1 P 5e1c6397f09747f091534faf93542710 [term 2 LEADER]: Stepping down as leader of term 2
I20240430 20:11:43.244371 20384 raft_consensus.cc:729] T e3e74e76ab2b4fe492ffcc1838656fd1 P 5e1c6397f09747f091534faf93542710 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Running, Role: LEADER
I20240430 20:11:43.244417 19074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88b3f98793354c7aaa5c106e8ec31ed8" candidate_term: 6 candidate_status { last_received { term: 5 index: 499 } } ignore_live_leader: false dest_uuid: "b632d6ef4c104a058a6db208024cab33" is_pre_election: true
I20240430 20:11:43.244657 20384 consensus_queue.cc:260] T e3e74e76ab2b4fe492ffcc1838656fd1 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:43.244997 18983 tablet_bootstrap.cc:654] T 7ce09fcbe1ac457a9c05efaf000c3263 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:43.245056 20384 raft_consensus.cc:3037] T e3e74e76ab2b4fe492ffcc1838656fd1 P 5e1c6397f09747f091534faf93542710 [term 2 FOLLOWER]: Advancing to term 3
I20240430 20:11:43.241147 18809 heartbeater.cc:464] Registering TS with master...
W20240430 20:11:43.248917 18699 consensus_peers.cc:485] T b76a4105a2974fe280e759349037d027 P 3c51c33d5616482cafb68e26688820f4 -> Peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Couldn't send request to peer 2efb542aa55e439b99ccb98d1c5c884a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b76a4105a2974fe280e759349037d027. This is attempt 1: this message will repeat every 5th retry.
I20240430 20:11:43.249797 19942 ts_manager.cc:194] Registered new tserver with Master: 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:43.250700 20398 tablet.cc:1722] T 6e754af64cb64fd2a082adae7d5f24da P 2efb542aa55e439b99ccb98d1c5c884a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20240430 20:11:43.252478 19029 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 62732af95c6a4c749c5b726f0460dc72 consensus state has no leader
I20240430 20:11:43.252763 19029 master_service.cc:946] ReplaceTablet: received request to replace tablet d61c279d20b0471a967942525ceb4fcb from {username='slave'} at 127.0.0.1:51664
I20240430 20:11:43.253358 18988 ts_tablet_manager.cc:1367] T 43b5e2c005a34df08cc4335c6e014cb5 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.069s user 0.003s sys 0.000s
I20240430 20:11:43.254863 19023 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: Sending DeleteTablet for 3 replicas of tablet acf3b6fead41483a82c43dc0286fab7c
I20240430 20:11:43.255043 18237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88b3f98793354c7aaa5c106e8ec31ed8" candidate_term: 6 candidate_status { last_received { term: 5 index: 499 } } ignore_live_leader: false dest_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e" is_pre_election: true
I20240430 20:11:43.256064 19023 catalog_manager.cc:6554] ReplaceTablet: tablet acf3b6fead41483a82c43dc0286fab7c deleted and replaced by tablet 626048d3ee12482a86c9ce860d12d069
I20240430 20:11:43.258556 18461 tablet_service.cc:1514] Processing DeleteTablet for tablet acf3b6fead41483a82c43dc0286fab7c with delete_type TABLET_DATA_DELETED (replaced by tablet 626048d3ee12482a86c9ce860d12d069) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:43.258713 18724 tablet_service.cc:1514] Processing DeleteTablet for tablet acf3b6fead41483a82c43dc0286fab7c with delete_type TABLET_DATA_DELETED (replaced by tablet 626048d3ee12482a86c9ce860d12d069) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:43.259083 18601 tablet_service.cc:1514] Processing DeleteTablet for tablet acf3b6fead41483a82c43dc0286fab7c with delete_type TABLET_DATA_DELETED (replaced by tablet 626048d3ee12482a86c9ce860d12d069) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:43.260025 19029 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_8d32c8a8f05e4b599dd6a713133ed82e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240430 20:11:43.254253 18988 raft_consensus.cc:348] T 43b5e2c005a34df08cc4335c6e014cb5 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:43.260501 18988 raft_consensus.cc:374] T 43b5e2c005a34df08cc4335c6e014cb5 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:43.260573 18988 raft_consensus.cc:729] T 43b5e2c005a34df08cc4335c6e014cb5 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
W20240430 20:11:43.260656 19029 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8d32c8a8f05e4b599dd6a713133ed82e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:43.260813 18988 consensus_queue.cc:260] T 43b5e2c005a34df08cc4335c6e014cb5 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:43.261360 18988 ts_tablet_manager.cc:1398] T 43b5e2c005a34df08cc4335c6e014cb5 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20240430 20:11:43.262502 19023 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_0376ffa6507243ada0b3b67b10264ee0" } new_table_name: "default.Table_1ce204e15fce4c469816cf4e7f80cac7" modify_external_catalogs: true
I20240430 20:11:43.263644 19051 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_276f62c2338b46a0aab8905430e8b61b" } modify_external_catalogs: true
I20240430 20:11:43.263820 19051 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_276f62c2338b46a0aab8905430e8b61b" } modify_external_catalogs: true
I20240430 20:11:43.264258 19048 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_01450a1e661d450a8cbe185e16e8250c" } modify_external_catalogs: true
I20240430 20:11:43.264436 19048 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_01450a1e661d450a8cbe185e16e8250c" } modify_external_catalogs: true
I20240430 20:11:43.265117 19033 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_bab588d4539e4155a9018d0db939e71f" } modify_external_catalogs: true
I20240430 20:11:43.265287 19033 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_bab588d4539e4155a9018d0db939e71f" } modify_external_catalogs: true
I20240430 20:11:43.266011 19049 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_fe491607e9e7480b9d9d2c8eb89ccac9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:43.266539 19049 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fe491607e9e7480b9d9d2c8eb89ccac9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:43.272704 18988 tablet_bootstrap.cc:492] T 84568f90b9c54cf292099808d4cbf5dc P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:43.273630 18461 tablet_service.cc:1467] Processing CreateTablet for tablet 4432d2c97b634a36b44e1d61d8bf2458 (DEFAULT_TABLE table=default.Table_fc68663ab2d74c34bedefcbad2d98a69 [id=d5cfd1c086e543b5a220e614a5241457]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:43.274070 18461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4432d2c97b634a36b44e1d61d8bf2458. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:43.275277 18988 tablet_bootstrap.cc:654] T 84568f90b9c54cf292099808d4cbf5dc P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:43.277535 18724 tablet_service.cc:1467] Processing CreateTablet for tablet 4432d2c97b634a36b44e1d61d8bf2458 (DEFAULT_TABLE table=default.Table_fc68663ab2d74c34bedefcbad2d98a69 [id=d5cfd1c086e543b5a220e614a5241457]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:43.277870 18735 tablet_service.cc:1467] Processing CreateTablet for tablet ffd6e420e6d44c039832de38de001c1b (DEFAULT_TABLE table=default.Table_7390a5615e234bf38ddf41eefe70bdf3 [id=52259eea51074e14b472d8532e08a990]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:43.278149 18724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4432d2c97b634a36b44e1d61d8bf2458. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:43.279225 18601 tablet_service.cc:1467] Processing CreateTablet for tablet 4432d2c97b634a36b44e1d61d8bf2458 (DEFAULT_TABLE table=default.Table_fc68663ab2d74c34bedefcbad2d98a69 [id=d5cfd1c086e543b5a220e614a5241457]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:43.279618 18601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4432d2c97b634a36b44e1d61d8bf2458. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:43.281129 18457 tablet_service.cc:1467] Processing CreateTablet for tablet ffd6e420e6d44c039832de38de001c1b (DEFAULT_TABLE table=default.Table_7390a5615e234bf38ddf41eefe70bdf3 [id=52259eea51074e14b472d8532e08a990]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:43.281507 18457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffd6e420e6d44c039832de38de001c1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:43.282847 18446 tablet_service.cc:1467] Processing CreateTablet for tablet 2fe548ac307d4b159c30659aa83fd76b (DEFAULT_TABLE table=default.Table_1407edea18d440e7adacadcfc8d89054 [id=b74e2a754da140628893bab105373983]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:43.282994 18735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffd6e420e6d44c039832de38de001c1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:43.283246 18446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fe548ac307d4b159c30659aa83fd76b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:43.285966 19899 leader_election.cc:302] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88b3f98793354c7aaa5c106e8ec31ed8; no voters: 89df84ccf5394c9bb60d4e4ed872fc2e, b632d6ef4c104a058a6db208024cab33
I20240430 20:11:43.286423 20326 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240430 20:11:43.286871 18449 tablet_service.cc:1467] Processing CreateTablet for tablet 7eae3a1a8d6c4b438d3424eb909a1fe6 (DEFAULT_TABLE table=default.Table_304521bb46d9434481340ff5ce7ca994 [id=c51917e099fa4676a092373e1ad201f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:43.287225 18449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7eae3a1a8d6c4b438d3424eb909a1fe6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:43.287600 18447 tablet_service.cc:1467] Processing CreateTablet for tablet 72205028d0654e69a6fa20f105604e1c (DEFAULT_TABLE table=default.Table_83b26c6b5e1942da864b8e60e882f5b4 [id=a09a93469a8f450895fa1756e1074c16]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:43.287930 18447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72205028d0654e69a6fa20f105604e1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:43.289570 18985 tablet_bootstrap.cc:492] T 681cdd85ac6847a0a1aaded310456582 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:43.289692 18985 ts_tablet_manager.cc:1367] T 681cdd85ac6847a0a1aaded310456582 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.287s user 0.022s sys 0.022s
I20240430 20:11:43.290452 18985 raft_consensus.cc:348] T 681cdd85ac6847a0a1aaded310456582 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:43.290643 18985 raft_consensus.cc:374] T 681cdd85ac6847a0a1aaded310456582 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:43.290694 18985 raft_consensus.cc:729] T 681cdd85ac6847a0a1aaded310456582 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:43.290859 18459 tablet_service.cc:1467] Processing CreateTablet for tablet b7141e215d6d44bc8406c80508190874 (DEFAULT_TABLE table=default.Table_bab588d4539e4155a9018d0db939e71f [id=ba57e80c90eb4a429ac6311d3ff874a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:43.290889 18448 tablet_service.cc:1467] Processing CreateTablet for tablet e848539446a64a968b37045d367bb3ce (DEFAULT_TABLE table=default.Table_605f23ec0a58457f87356d7f45d3b45c [id=1930f366a0f0456cad9a1c2456eb8fac]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:43.291301 18454 tablet_service.cc:1467] Processing CreateTablet for tablet 5df364628561475391c9a5e10ff5a05f (DEFAULT_TABLE table=default.Table_3c8f9f7b22f64fb1bdc3f1550987233a [id=7fd6f8576f4c4cbc8ffcdb4ab95e15dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:43.291851 18450 tablet_service.cc:1467] Processing CreateTablet for tablet 716c62f14ca74da9af3282128441b44f (DEFAULT_TABLE table=default.Table_2c1b15dc75504997924b1f36ca1f69a8 [id=634df0651c254f1c9e8db138ad908f31]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:43.292410 18445 tablet_service.cc:1467] Processing CreateTablet for tablet 4fa9575d96a84bef876467e21dada803 (DEFAULT_TABLE table=default.Table_27f3d711beeb4387a9088ab291168bc8 [id=434aba8e131342a3a9a1aca3278f9c0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:43.292560 18448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e848539446a64a968b37045d367bb3ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:43.292788 18453 tablet_service.cc:1467] Processing CreateTablet for tablet 1faa2f2c84184b129ba6e4d2857822f0 (DEFAULT_TABLE table=default.Table_2eed103638364cde94cd8af68d3cdf9b [id=1a382ad8009b4522a50d55c307db8f96]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:43.292956 19942 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" instance_seqno: 1714507879927603) as {username='slave'} at 127.17.177.1:47345; Asking this server to re-register.
I20240430 20:11:43.293258 18586 tablet_service.cc:1467] Processing CreateTablet for tablet ffd6e420e6d44c039832de38de001c1b (DEFAULT_TABLE table=default.Table_7390a5615e234bf38ddf41eefe70bdf3 [id=52259eea51074e14b472d8532e08a990]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:43.293707 18586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffd6e420e6d44c039832de38de001c1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:43.277791 19489 ts_tablet_manager.cc:1888] T 44fa6917a85c46f3aaa363df46ea586b P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:43.293972 19489 log.cc:1196] T 44fa6917a85c46f3aaa363df46ea586b P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/44fa6917a85c46f3aaa363df46ea586b
I20240430 20:11:43.294317 19489 ts_tablet_manager.cc:1909] T 44fa6917a85c46f3aaa363df46ea586b P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:43.295063 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 125) took 10947 ms. Trace:
I20240430 20:11:43.294996 18985 consensus_queue.cc:260] T 681cdd85ac6847a0a1aaded310456582 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:43.295142 19489 rpcz_store.cc:276] 0430 20:11:32.348073 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:32.356387 (+ 8314us) service_pool.cc:225] Handling call
0430 20:11:43.202798 (+10846411us) ts_tablet_manager.cc:1120] Deleting tablet 44fa6917a85c46f3aaa363df46ea586b
0430 20:11:43.202808 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:43.273193 (+ 70385us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:43.277712 (+ 4519us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:43.295038 (+ 17326us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":10846103}
I20240430 20:11:43.295521 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 126) took 10886 ms. Trace:
I20240430 20:11:43.295540 18985 ts_tablet_manager.cc:1398] T 681cdd85ac6847a0a1aaded310456582 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240430 20:11:43.295589 19489 rpcz_store.cc:276] 0430 20:11:32.408865 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:32.408928 (+ 63us) service_pool.cc:225] Handling call
0430 20:11:43.295398 (+10886470us) ts_tablet_manager.cc:1120] Deleting tablet 73453a7fa30342e29db552b17b62f0c1
0430 20:11:43.295407 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:43.295503 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":10885162}
I20240430 20:11:43.295862 19489 tablet_replica.cc:330] T b72fe1f0713b4076a81a3fc1c74f7fa7 P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:43.296041 19489 raft_consensus.cc:2229] T b72fe1f0713b4076a81a3fc1c74f7fa7 P 3c51c33d5616482cafb68e26688820f4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:43.296161 19489 pending_rounds.cc:62] T b72fe1f0713b4076a81a3fc1c74f7fa7 P 3c51c33d5616482cafb68e26688820f4: Trying to abort 1 pending ops.
I20240430 20:11:43.296192 19489 pending_rounds.cc:68] T b72fe1f0713b4076a81a3fc1c74f7fa7 P 3c51c33d5616482cafb68e26688820f4: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7022624341382275072 op_type: NO_OP noop_request { }
I20240430 20:11:43.296284 19489 raft_consensus.cc:2866] T b72fe1f0713b4076a81a3fc1c74f7fa7 P 3c51c33d5616482cafb68e26688820f4 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20240430 20:11:43.296350 19489 raft_consensus.cc:2258] T b72fe1f0713b4076a81a3fc1c74f7fa7 P 3c51c33d5616482cafb68e26688820f4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:43.296875 19489 ts_tablet_manager.cc:1875] T b72fe1f0713b4076a81a3fc1c74f7fa7 P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:43.297623 18445 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4fa9575d96a84bef876467e21dada803. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:43.299167 18724 tablet_service.cc:1467] Processing CreateTablet for tablet 2fe548ac307d4b159c30659aa83fd76b (DEFAULT_TABLE table=default.Table_1407edea18d440e7adacadcfc8d89054 [id=b74e2a754da140628893bab105373983]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:43.299479 18729 tablet_service.cc:1467] Processing CreateTablet for tablet 72205028d0654e69a6fa20f105604e1c (DEFAULT_TABLE table=default.Table_83b26c6b5e1942da864b8e60e882f5b4 [id=a09a93469a8f450895fa1756e1074c16]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:43.299727 18744 tablet_service.cc:1467] Processing CreateTablet for tablet 4fa9575d96a84bef876467e21dada803 (DEFAULT_TABLE table=default.Table_27f3d711beeb4387a9088ab291168bc8 [id=434aba8e131342a3a9a1aca3278f9c0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:43.299808 18733 tablet_service.cc:1467] Processing CreateTablet for tablet e848539446a64a968b37045d367bb3ce (DEFAULT_TABLE table=default.Table_605f23ec0a58457f87356d7f45d3b45c [id=1930f366a0f0456cad9a1c2456eb8fac]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:43.300292 18725 tablet_service.cc:1467] Processing CreateTablet for tablet b7141e215d6d44bc8406c80508190874 (DEFAULT_TABLE table=default.Table_bab588d4539e4155a9018d0db939e71f [id=ba57e80c90eb4a429ac6311d3ff874a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:43.300271 18735 tablet_service.cc:1467] Processing CreateTablet for tablet 7eae3a1a8d6c4b438d3424eb909a1fe6 (DEFAULT_TABLE table=default.Table_304521bb46d9434481340ff5ce7ca994 [id=c51917e099fa4676a092373e1ad201f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:43.300717 18734 tablet_service.cc:1467] Processing CreateTablet for tablet 5df364628561475391c9a5e10ff5a05f (DEFAULT_TABLE table=default.Table_3c8f9f7b22f64fb1bdc3f1550987233a [id=7fd6f8576f4c4cbc8ffcdb4ab95e15dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:43.300804 18724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fe548ac307d4b159c30659aa83fd76b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:43.301035 18734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5df364628561475391c9a5e10ff5a05f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:43.301208 18729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72205028d0654e69a6fa20f105604e1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:43.301370 18744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4fa9575d96a84bef876467e21dada803. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:43.301466 18736 tablet_service.cc:1467] Processing CreateTablet for tablet 716c62f14ca74da9af3282128441b44f (DEFAULT_TABLE table=default.Table_2c1b15dc75504997924b1f36ca1f69a8 [id=634df0651c254f1c9e8db138ad908f31]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:43.301766 18736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 716c62f14ca74da9af3282128441b44f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:43.302881 18453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1faa2f2c84184b129ba6e4d2857822f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:43.305405 18725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7141e215d6d44bc8406c80508190874. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:43.307323 18735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7eae3a1a8d6c4b438d3424eb909a1fe6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:43.308188 19001 catalog_manager.cc:4830] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): tablet 44fa6917a85c46f3aaa363df46ea586b (table default.Table_5924e4520c6c49988ccef21ca04d25d4 [id=68c1eb7252974258aabd9873e6654b10]) successfully deleted
W20240430 20:11:43.308441 19001 catalog_manager.cc:4810] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): delete failed for tablet 73453a7fa30342e29db552b17b62f0c1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 73453a7fa30342e29db552b17b62f0c1 already in progress: creating tablet
I20240430 20:11:43.310351 18733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e848539446a64a968b37045d367bb3ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:43.312645 18985 tablet_bootstrap.cc:492] T e87fb2f0361d4aa3ada27bbdfa88e94c P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:43.314860 18454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5df364628561475391c9a5e10ff5a05f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:43.315249 18528 heartbeater.cc:464] Registering TS with master...
I20240430 20:11:43.318526 19945 ts_manager.cc:194] Registered new tserver with Master: 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:43.318827 18450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 716c62f14ca74da9af3282128441b44f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:43.322393 18983 tablet_bootstrap.cc:492] T 7ce09fcbe1ac457a9c05efaf000c3263 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:43.322558 18983 ts_tablet_manager.cc:1367] T 7ce09fcbe1ac457a9c05efaf000c3263 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.084s user 0.003s sys 0.000s
I20240430 20:11:43.322834 18459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7141e215d6d44bc8406c80508190874. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:43.323350 18983 raft_consensus.cc:348] T 7ce09fcbe1ac457a9c05efaf000c3263 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:43.323558 18983 raft_consensus.cc:374] T 7ce09fcbe1ac457a9c05efaf000c3263 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:43.323624 18983 raft_consensus.cc:729] T 7ce09fcbe1ac457a9c05efaf000c3263 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:43.323813 18983 consensus_queue.cc:260] T 7ce09fcbe1ac457a9c05efaf000c3263 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:43.324368 18983 ts_tablet_manager.cc:1398] T 7ce09fcbe1ac457a9c05efaf000c3263 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240430 20:11:43.324656 18983 tablet_bootstrap.cc:492] T 5bd0362ff41b4faca0205d136d596737 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:43.325644 18985 tablet_bootstrap.cc:654] T e87fb2f0361d4aa3ada27bbdfa88e94c P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:43.327813 18983 tablet_bootstrap.cc:654] T 5bd0362ff41b4faca0205d136d596737 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:43.328788 18810 heartbeater.cc:503] Master 127.17.177.60:37225 was elected leader, sending a full tablet report...
I20240430 20:11:43.333125 20408 raft_consensus.cc:482] T f7b0132c17d44a9ba17288e0321a7f11 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:43.333300 20408 raft_consensus.cc:504] T f7b0132c17d44a9ba17288e0321a7f11 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:43.334039 20408 leader_election.cc:288] T f7b0132c17d44a9ba17288e0321a7f11 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:43.334776 18616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7b0132c17d44a9ba17288e0321a7f11" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:43.335604 18726 tablet_service.cc:1467] Processing CreateTablet for tablet cc267ce92ffd415fbe3377afc0503690 (DEFAULT_TABLE table=default.Table_2f6a85489f2e49db8974ff86813c85ee [id=6a249076822e47eda013a4d4f23ab8da]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:43.335774 18737 tablet_service.cc:1467] Processing CreateTablet for tablet 72d316f424214792ac1d5029ffb1b04a (DEFAULT_TABLE table=default.Table_69622a6c03f14409a34ad4d98f24c5a1 [id=43f5d7056f5d4ef98f6861d8f40643f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:43.335860 18732 tablet_service.cc:1467] Processing CreateTablet for tablet bb471d33b0784c9c9de1e5425cdd1fd7 (DEFAULT_TABLE table=default.Table_41ccec912bfa439c888281ac81447a38 [id=da57ee424e3f481f8e0c00399bb21ffa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:43.336002 18730 tablet_service.cc:1467] Processing CreateTablet for tablet 821c55edb3414d72b1489eea97d3f73a (DEFAULT_TABLE table=default.Table_4d68c38d9ba7452a829f8318b2b267b8 [id=d06e3e9045b1429aaeebd1fd67d841de]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:43.336194 18739 tablet_service.cc:1467] Processing CreateTablet for tablet 9e98e20abc924873853d176455f3678e (DEFAULT_TABLE table=default.Table_4839b72b4f2143b2b23e8f3865d3e203 [id=e5126f46127442f1ad6ea61d40f983bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:43.336290 18744 tablet_service.cc:1467] Processing CreateTablet for tablet 1faa2f2c84184b129ba6e4d2857822f0 (DEFAULT_TABLE table=default.Table_2eed103638364cde94cd8af68d3cdf9b [id=1a382ad8009b4522a50d55c307db8f96]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:43.336196 18742 tablet_service.cc:1467] Processing CreateTablet for tablet 024069ab786749028c020abd1c836cf5 (DEFAULT_TABLE table=default.Table_e8c73c89623d4b49b8d7f0a01d729bdd [id=5cd10682ce52414cb5a103e4433a6a0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:43.336416 18740 tablet_service.cc:1467] Processing CreateTablet for tablet bea2ca3d09164afd9941cb7e35f5ae24 (DEFAULT_TABLE table=default.Table_01d62e066e4942d9aa1f582189d60d49 [id=97861b0b1ce94638b62bbd123d774d7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:43.336915 19489 ts_tablet_manager.cc:1888] T b72fe1f0713b4076a81a3fc1c74f7fa7 P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:43.337038 19489 log.cc:1196] T b72fe1f0713b4076a81a3fc1c74f7fa7 P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/b72fe1f0713b4076a81a3fc1c74f7fa7
W20240430 20:11:43.336990 18415 leader_election.cc:341] T f7b0132c17d44a9ba17288e0321a7f11 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:43.337427 19489 ts_tablet_manager.cc:1909] T b72fe1f0713b4076a81a3fc1c74f7fa7 P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:43.337846 20396 raft_consensus.cc:482] T e87fb2f0361d4aa3ada27bbdfa88e94c P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:43.338016 20396 raft_consensus.cc:504] T e87fb2f0361d4aa3ada27bbdfa88e94c P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:43.338692 20396 leader_election.cc:288] T e87fb2f0361d4aa3ada27bbdfa88e94c P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:43.338905 18730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 821c55edb3414d72b1489eea97d3f73a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:43.339264 18732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb471d33b0784c9c9de1e5425cdd1fd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:43.342314 18739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e98e20abc924873853d176455f3678e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:43.346854 18740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bea2ca3d09164afd9941cb7e35f5ae24. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:43.347057 18744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1faa2f2c84184b129ba6e4d2857822f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:43.347244 18737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72d316f424214792ac1d5029ffb1b04a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:43.348274 18742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 024069ab786749028c020abd1c836cf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:43.350487 18726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc267ce92ffd415fbe3377afc0503690. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:43.353652 18453 tablet_service.cc:1467] Processing CreateTablet for tablet 9e98e20abc924873853d176455f3678e (DEFAULT_TABLE table=default.Table_4839b72b4f2143b2b23e8f3865d3e203 [id=e5126f46127442f1ad6ea61d40f983bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:43.354094 18453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e98e20abc924873853d176455f3678e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:43.358225 18585 tablet_service.cc:1467] Processing CreateTablet for tablet e848539446a64a968b37045d367bb3ce (DEFAULT_TABLE table=default.Table_605f23ec0a58457f87356d7f45d3b45c [id=1930f366a0f0456cad9a1c2456eb8fac]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:43.358347 18592 tablet_service.cc:1467] Processing CreateTablet for tablet 5df364628561475391c9a5e10ff5a05f (DEFAULT_TABLE table=default.Table_3c8f9f7b22f64fb1bdc3f1550987233a [id=7fd6f8576f4c4cbc8ffcdb4ab95e15dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:43.358731 18585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e848539446a64a968b37045d367bb3ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:43.358839 18601 tablet_service.cc:1467] Processing CreateTablet for tablet 7eae3a1a8d6c4b438d3424eb909a1fe6 (DEFAULT_TABLE table=default.Table_304521bb46d9434481340ff5ce7ca994 [id=c51917e099fa4676a092373e1ad201f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:43.359144 18601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7eae3a1a8d6c4b438d3424eb909a1fe6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:43.359710 18445 tablet_service.cc:1467] Processing CreateTablet for tablet 024069ab786749028c020abd1c836cf5 (DEFAULT_TABLE table=default.Table_e8c73c89623d4b49b8d7f0a01d729bdd [id=5cd10682ce52414cb5a103e4433a6a0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:43.360036 18445 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 024069ab786749028c020abd1c836cf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:43.362969 18591 tablet_service.cc:1467] Processing CreateTablet for tablet b7141e215d6d44bc8406c80508190874 (DEFAULT_TABLE table=default.Table_bab588d4539e4155a9018d0db939e71f [id=ba57e80c90eb4a429ac6311d3ff874a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:43.363133 18603 tablet_service.cc:1467] Processing CreateTablet for tablet 716c62f14ca74da9af3282128441b44f (DEFAULT_TABLE table=default.Table_2c1b15dc75504997924b1f36ca1f69a8 [id=634df0651c254f1c9e8db138ad908f31]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:43.363358 18591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7141e215d6d44bc8406c80508190874. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:43.363638 18589 tablet_service.cc:1467] Processing CreateTablet for tablet 4fa9575d96a84bef876467e21dada803 (DEFAULT_TABLE table=default.Table_27f3d711beeb4387a9088ab291168bc8 [id=434aba8e131342a3a9a1aca3278f9c0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:43.363667 18586 tablet_service.cc:1467] Processing CreateTablet for tablet 2fe548ac307d4b159c30659aa83fd76b (DEFAULT_TABLE table=default.Table_1407edea18d440e7adacadcfc8d89054 [id=b74e2a754da140628893bab105373983]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:43.364005 18589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4fa9575d96a84bef876467e21dada803. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:43.365028 18586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fe548ac307d4b159c30659aa83fd76b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:43.358268 18588 tablet_service.cc:1467] Processing CreateTablet for tablet 72205028d0654e69a6fa20f105604e1c (DEFAULT_TABLE table=default.Table_83b26c6b5e1942da864b8e60e882f5b4 [id=a09a93469a8f450895fa1756e1074c16]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:43.353672 18458 tablet_service.cc:1467] Processing CreateTablet for tablet bea2ca3d09164afd9941cb7e35f5ae24 (DEFAULT_TABLE table=default.Table_01d62e066e4942d9aa1f582189d60d49 [id=97861b0b1ce94638b62bbd123d774d7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:43.367385 18588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72205028d0654e69a6fa20f105604e1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:43.367432 18458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bea2ca3d09164afd9941cb7e35f5ae24. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:43.371107 18600 tablet_service.cc:1467] Processing CreateTablet for tablet 1faa2f2c84184b129ba6e4d2857822f0 (DEFAULT_TABLE table=default.Table_2eed103638364cde94cd8af68d3cdf9b [id=1a382ad8009b4522a50d55c307db8f96]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:43.371510 18600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1faa2f2c84184b129ba6e4d2857822f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:43.371433 18595 tablet_service.cc:1467] Processing CreateTablet for tablet cc267ce92ffd415fbe3377afc0503690 (DEFAULT_TABLE table=default.Table_2f6a85489f2e49db8974ff86813c85ee [id=6a249076822e47eda013a4d4f23ab8da]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:43.374882 18592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5df364628561475391c9a5e10ff5a05f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:43.375811 18446 tablet_service.cc:1467] Processing CreateTablet for tablet 72d316f424214792ac1d5029ffb1b04a (DEFAULT_TABLE table=default.Table_69622a6c03f14409a34ad4d98f24c5a1 [id=43f5d7056f5d4ef98f6861d8f40643f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:43.376195 18446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72d316f424214792ac1d5029ffb1b04a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:43.378768 18603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 716c62f14ca74da9af3282128441b44f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:43.379688 18447 tablet_service.cc:1467] Processing CreateTablet for tablet cc267ce92ffd415fbe3377afc0503690 (DEFAULT_TABLE table=default.Table_2f6a85489f2e49db8974ff86813c85ee [id=6a249076822e47eda013a4d4f23ab8da]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:43.380059 18447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc267ce92ffd415fbe3377afc0503690. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:43.381980 18459 tablet_service.cc:1467] Processing CreateTablet for tablet 821c55edb3414d72b1489eea97d3f73a (DEFAULT_TABLE table=default.Table_4d68c38d9ba7452a829f8318b2b267b8 [id=d06e3e9045b1429aaeebd1fd67d841de]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:43.382315 18459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 821c55edb3414d72b1489eea97d3f73a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:43.383270 18587 tablet_service.cc:1467] Processing CreateTablet for tablet 72d316f424214792ac1d5029ffb1b04a (DEFAULT_TABLE table=default.Table_69622a6c03f14409a34ad4d98f24c5a1 [id=43f5d7056f5d4ef98f6861d8f40643f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:43.383438 18590 tablet_service.cc:1467] Processing CreateTablet for tablet 9e98e20abc924873853d176455f3678e (DEFAULT_TABLE table=default.Table_4839b72b4f2143b2b23e8f3865d3e203 [id=e5126f46127442f1ad6ea61d40f983bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:43.383512 18602 tablet_service.cc:1467] Processing CreateTablet for tablet 024069ab786749028c020abd1c836cf5 (DEFAULT_TABLE table=default.Table_e8c73c89623d4b49b8d7f0a01d729bdd [id=5cd10682ce52414cb5a103e4433a6a0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:43.383661 18587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72d316f424214792ac1d5029ffb1b04a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:43.383883 18602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 024069ab786749028c020abd1c836cf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:43.384131 18593 tablet_service.cc:1467] Processing CreateTablet for tablet bb471d33b0784c9c9de1e5425cdd1fd7 (DEFAULT_TABLE table=default.Table_41ccec912bfa439c888281ac81447a38 [id=da57ee424e3f481f8e0c00399bb21ffa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:43.384438 18593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb471d33b0784c9c9de1e5425cdd1fd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:43.386693 18590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e98e20abc924873853d176455f3678e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:43.391160 20408 raft_consensus.cc:482] T 2cdccf4be7094ca5b863dd5c3432d55d P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:43.391358 20408 raft_consensus.cc:504] T 2cdccf4be7094ca5b863dd5c3432d55d P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:43.392091 20408 leader_election.cc:288] T 2cdccf4be7094ca5b863dd5c3432d55d P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
W20240430 20:11:43.392854 18415 leader_election.cc:341] T 2cdccf4be7094ca5b863dd5c3432d55d P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Not found: Tablet not found: 2cdccf4be7094ca5b863dd5c3432d55d
I20240430 20:11:43.393266 18602 tablet_service.cc:1467] Processing CreateTablet for tablet bea2ca3d09164afd9941cb7e35f5ae24 (DEFAULT_TABLE table=default.Table_01d62e066e4942d9aa1f582189d60d49 [id=97861b0b1ce94638b62bbd123d774d7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:43.393654 18602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bea2ca3d09164afd9941cb7e35f5ae24. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:43.395617 18588 tablet_service.cc:1467] Processing CreateTablet for tablet 821c55edb3414d72b1489eea97d3f73a (DEFAULT_TABLE table=default.Table_4d68c38d9ba7452a829f8318b2b267b8 [id=d06e3e9045b1429aaeebd1fd67d841de]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:43.395988 18588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 821c55edb3414d72b1489eea97d3f73a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:43.399106 18474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e87fb2f0361d4aa3ada27bbdfa88e94c" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:43.400219 18449 tablet_service.cc:1467] Processing CreateTablet for tablet bb471d33b0784c9c9de1e5425cdd1fd7 (DEFAULT_TABLE table=default.Table_41ccec912bfa439c888281ac81447a38 [id=da57ee424e3f481f8e0c00399bb21ffa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:43.400625 18449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb471d33b0784c9c9de1e5425cdd1fd7. 1 dirs total, 0 dirs full, 0 dirs failed
W20240430 20:11:43.401983 18558 leader_election.cc:341] T e87fb2f0361d4aa3ada27bbdfa88e94c P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:43.401948 20384 consensus_queue.cc:1035] T ebf74809b52f47f3b0ed21b6ac0f541a P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:43.407379 18750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e87fb2f0361d4aa3ada27bbdfa88e94c" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:43.408679 18750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7b0132c17d44a9ba17288e0321a7f11" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:43.409173 18750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cdccf4be7094ca5b863dd5c3432d55d" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
W20240430 20:11:43.409684 18557 leader_election.cc:341] T e87fb2f0361d4aa3ada27bbdfa88e94c P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:43.409799 18557 leader_election.cc:302] T e87fb2f0361d4aa3ada27bbdfa88e94c P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:43.410300 20384 raft_consensus.cc:2726] T e87fb2f0361d4aa3ada27bbdfa88e94c P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240430 20:11:43.410751 18557 consensus_peers.cc:485] T 84568f90b9c54cf292099808d4cbf5dc P 5e1c6397f09747f091534faf93542710 -> Peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Couldn't send request to peer 3c51c33d5616482cafb68e26688820f4. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
W20240430 20:11:43.411170 18417 consensus_peers.cc:485] T 1462ecf1176c4e66be42abddbe36b00c P 2efb542aa55e439b99ccb98d1c5c884a -> Peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Couldn't send request to peer 3c51c33d5616482cafb68e26688820f4. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20240430 20:11:43.411559 18417 leader_election.cc:341] T f7b0132c17d44a9ba17288e0321a7f11 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:43.411617 19026 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: Sending DeleteTablet for 3 replicas of tablet 3ec3993042f5498b818bdf32d5087928
I20240430 20:11:43.411659 18417 leader_election.cc:302] T f7b0132c17d44a9ba17288e0321a7f11 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a; no voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710
I20240430 20:11:43.411885 20408 raft_consensus.cc:2726] T f7b0132c17d44a9ba17288e0321a7f11 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240430 20:11:43.412266 18417 leader_election.cc:341] T 2cdccf4be7094ca5b863dd5c3432d55d P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:43.412365 18417 leader_election.cc:302] T 2cdccf4be7094ca5b863dd5c3432d55d P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a; no voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710
I20240430 20:11:43.412703 20408 raft_consensus.cc:2726] T 2cdccf4be7094ca5b863dd5c3432d55d P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:43.412539 19026 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_75efd32c9420454f901c61882c8c29bc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:43.413199 19026 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_75efd32c9420454f901c61882c8c29bc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240430 20:11:43.413938 19023 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 677481aee5fa4eb4b189322d7a3f8f2e consensus state has no leader
I20240430 20:11:43.414386 19023 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_e311a584368b4233b6e5beb24587ce00"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240430 20:11:43.415030 19033 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: Sending DeleteTablet for 3 replicas of tablet b7141e215d6d44bc8406c80508190874
I20240430 20:11:43.415668 19033 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_ee86b8fcf2064d8b8c3e327e357d3395" } new_table_name: "default.Table_0163ce6f5a4c45299106daeffb3c22b1" modify_external_catalogs: true
I20240430 20:11:43.417965 19024 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 2e79df8744e74b5991a79ce978ff0202 (table default.Table_276bbdc7cfe542d1a2936485a2aad1ec [id=f146a15e6e8a4acbb4c734c1c863da57]). Expected version 1 got 0
I20240430 20:11:43.418659 20408 consensus_queue.cc:1035] T 1462ecf1176c4e66be42abddbe36b00c P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:43.420742 18726 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ec3993042f5498b818bdf32d5087928 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:43 UTC) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:43.422870 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 217) took 3828 ms. Trace:
I20240430 20:11:43.422976 19495 rpcz_store.cc:276] 0430 20:11:39.594193 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:39.594961 (+ 768us) service_pool.cc:225] Handling call
0430 20:11:43.219374 (+3624413us) ts_tablet_manager.cc:1120] Deleting tablet 90a694bd5db649afa50f3c4e752939c1
0430 20:11:43.219385 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:43.422844 (+203459us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3622605}
I20240430 20:11:43.423432 19495 tablet_replica.cc:330] T 53a899fa5a424a0d99f1b47925e8c7ad P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:43.423612 19495 raft_consensus.cc:2229] T 53a899fa5a424a0d99f1b47925e8c7ad P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:43.423772 19495 raft_consensus.cc:2258] T 53a899fa5a424a0d99f1b47925e8c7ad P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus is shut down!
W20240430 20:11:43.424201 19023 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e311a584368b4233b6e5beb24587ce00 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:43.424335 19495 ts_tablet_manager.cc:1875] T 53a899fa5a424a0d99f1b47925e8c7ad P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:43.435077 19024 catalog_manager.cc:5484] T 5f02d8862e1f4ccfa9dbda73dbe18615 P 5e1c6397f09747f091534faf93542710 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e1c6397f09747f091534faf93542710 (127.17.177.2). New cstate: current_term: 1 leader_uuid: "5e1c6397f09747f091534faf93542710" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } health_report { overall_health: UNKNOWN } } }
I20240430 20:11:43.435956 19024 catalog_manager.cc:5484] T ebf74809b52f47f3b0ed21b6ac0f541a P 5e1c6397f09747f091534faf93542710 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e1c6397f09747f091534faf93542710 (127.17.177.2). New cstate: current_term: 1 leader_uuid: "5e1c6397f09747f091534faf93542710" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } health_report { overall_health: UNKNOWN } } }
I20240430 20:11:43.436440 19024 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet ebf74809b52f47f3b0ed21b6ac0f541a (table default.Table_b2d07520b9944a89ad62fca70396174e [id=1c8e0a3a44ec4a798770c0165ed3dffa]). Expected version 1 got 0
I20240430 20:11:43.436705 19024 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 46251c08277e498da04a41d009310c62 (table default.Table_59297b42e8cf4ab3b9e95e2e70d3c1fe [id=b1312a6cd1a54003b1db727a2a9d125d]). Expected version 2 got 0
I20240430 20:11:43.437048 19027 catalog_manager.cc:6029] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: default.Table_685a26737bb046fb85ced15df7ae3423 [id=51d177e8498848b38c43e3f8d11f092c] alter complete (version 1)
I20240430 20:11:43.438792 19051 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: Sending DeleteTablet for 3 replicas of tablet dd6083f12c7e4d1e8a2006b3ddf9aa3a
I20240430 20:11:43.439628 19051 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_4493a13725004bf5a2996bc9369a0892" } modify_external_catalogs: true
I20240430 20:11:43.439860 19051 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_4493a13725004bf5a2996bc9369a0892" } modify_external_catalogs: true
I20240430 20:11:43.441565 19048 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet 626048d3ee12482a86c9ce860d12d069
I20240430 20:11:43.442034 19048 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_e4c7f4d2134445f380a007e589acd65d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:43.442765 19048 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e4c7f4d2134445f380a007e589acd65d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:43.445273 18726 tablet_service.cc:1514] Processing DeleteTablet for tablet dd6083f12c7e4d1e8a2006b3ddf9aa3a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:43 UTC) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:43.448568 19027 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_457e03bc2d524accaa76649790c42fc8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:43.449306 19027 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_457e03bc2d524accaa76649790c42fc8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:43.450670 18453 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ec3993042f5498b818bdf32d5087928 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:43 UTC) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:43.450898 18459 tablet_service.cc:1514] Processing DeleteTablet for tablet 115c4a12440647a4bb46eb6824f7ea98 with delete_type TABLET_DATA_DELETED (Sending initial creation of tablet) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:43.451090 18453 tablet_service.cc:1514] Processing DeleteTablet for tablet dd6083f12c7e4d1e8a2006b3ddf9aa3a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:43 UTC) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:43.451189 18461 tablet_service.cc:1514] Processing DeleteTablet for tablet b7141e215d6d44bc8406c80508190874 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:43 UTC) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:43.451236 18459 tablet_service.cc:1514] Processing DeleteTablet for tablet 50486644ec234933b9a0b45ee39a6685 with delete_type TABLET_DATA_DELETED (replaced by tablet ed0c37a71ede4bfc9eb5ad1a4365f25e) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:43.453157 18590 tablet_service.cc:1514] Processing DeleteTablet for tablet dd6083f12c7e4d1e8a2006b3ddf9aa3a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:43 UTC) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:43.455132 20047 raft_consensus.cc:482] T e2d90ff17e574f88b2061b1c117d4b7d P 5e1c6397f09747f091534faf93542710 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:43.455286 20047 raft_consensus.cc:504] T e2d90ff17e574f88b2061b1c117d4b7d P 5e1c6397f09747f091534faf93542710 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:43.456629 18750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2d90ff17e574f88b2061b1c117d4b7d" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:43.457309 18588 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ec3993042f5498b818bdf32d5087928 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:43 UTC) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:43.465912 20047 leader_election.cc:288] T e2d90ff17e574f88b2061b1c117d4b7d P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:43.474241 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 127) took 11064 ms. Trace:
I20240430 20:11:43.474385 19489 rpcz_store.cc:276] 0430 20:11:32.409964 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:32.410677 (+ 713us) service_pool.cc:225] Handling call
0430 20:11:43.295822 (+10885145us) ts_tablet_manager.cc:1120] Deleting tablet b72fe1f0713b4076a81a3fc1c74f7fa7
0430 20:11:43.295830 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:43.328930 (+ 33100us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:43.331685 (+ 2755us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:43.474208 (+142523us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":10879926}
I20240430 20:11:43.474740 19489 tablet_replica.cc:330] T 87ac32a0ddb84676adc83038ef664591 P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:43.475937 19001 catalog_manager.cc:4830] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): tablet b72fe1f0713b4076a81a3fc1c74f7fa7 (table default.Table_16b082d140234e1c9a1e70654839b4cd [id=c0d8dc87d1f9426a85f6176f161595a1]) successfully deleted
I20240430 20:11:43.479974 19033 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_afeeff199c8b4ba888e8b73423e870a7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:43.480737 19033 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_afeeff199c8b4ba888e8b73423e870a7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240430 20:11:43.484673 19004 catalog_manager.cc:4810] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): delete failed for tablet 90a694bd5db649afa50f3c4e752939c1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 90a694bd5db649afa50f3c4e752939c1 already in progress: creating tablet
I20240430 20:11:43.491616 18985 tablet_bootstrap.cc:492] T e87fb2f0361d4aa3ada27bbdfa88e94c P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:43.491775 18985 ts_tablet_manager.cc:1367] T e87fb2f0361d4aa3ada27bbdfa88e94c P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.179s user 0.028s sys 0.010s
I20240430 20:11:43.492583 18985 raft_consensus.cc:348] T e87fb2f0361d4aa3ada27bbdfa88e94c P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:43.492830 18985 raft_consensus.cc:374] T e87fb2f0361d4aa3ada27bbdfa88e94c P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:43.492880 18985 raft_consensus.cc:729] T e87fb2f0361d4aa3ada27bbdfa88e94c P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:43.493072 18985 consensus_queue.cc:260] T e87fb2f0361d4aa3ada27bbdfa88e94c P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:43.493753 18985 ts_tablet_manager.cc:1398] T e87fb2f0361d4aa3ada27bbdfa88e94c P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240430 20:11:43.494048 18985 tablet_bootstrap.cc:492] T 2e79df8744e74b5991a79ce978ff0202 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:43.495138 19471 ts_tablet_manager.cc:1888] T 20d166756819406ba85044f4fee9c4ce P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:43.495371 18985 tablet_bootstrap.cc:654] T 2e79df8744e74b5991a79ce978ff0202 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:43.496304 19051 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: Sending DeleteTablet for 3 replicas of tablet 489dc838744a49eba542009c86688aa5
I20240430 20:11:43.497248 19051 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_43f360b4669c4bfd8d6849258825fac5" } new_table_name: "default.Table_25ddc3adef6b4ccf8b9c00cc6772031c" modify_external_catalogs: true
I20240430 20:11:43.499147 18726 tablet_service.cc:1514] Processing DeleteTablet for tablet 489dc838744a49eba542009c86688aa5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:43 UTC) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:43.504329 18449 tablet_service.cc:1514] Processing DeleteTablet for tablet 489dc838744a49eba542009c86688aa5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:43 UTC) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:43.506318 19489 raft_consensus.cc:2229] T 87ac32a0ddb84676adc83038ef664591 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:43.507059 19489 raft_consensus.cc:2258] T 87ac32a0ddb84676adc83038ef664591 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:43.507804 19489 ts_tablet_manager.cc:1875] T 87ac32a0ddb84676adc83038ef664591 P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:43.496189 18619 raft_consensus.cc:1261] T e3e74e76ab2b4fe492ffcc1838656fd1 P 5e1c6397f09747f091534faf93542710 [term 3 FOLLOWER]: Refusing update from remote peer 2efb542aa55e439b99ccb98d1c5c884a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20240430 20:11:43.508397 18619 pending_rounds.cc:76] T e3e74e76ab2b4fe492ffcc1838656fd1 P 5e1c6397f09747f091534faf93542710: Aborting all ops after (but not including) 1
I20240430 20:11:43.508464 18619 pending_rounds.cc:97] T e3e74e76ab2b4fe492ffcc1838656fd1 P 5e1c6397f09747f091534faf93542710: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20240430 20:11:43.508515 18619 raft_consensus.cc:2866] T e3e74e76ab2b4fe492ffcc1838656fd1 P 5e1c6397f09747f091534faf93542710 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20240430 20:11:43.506201 18557 consensus_peers.cc:485] T ebf74809b52f47f3b0ed21b6ac0f541a P 5e1c6397f09747f091534faf93542710 -> Peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Couldn't send request to peer 3c51c33d5616482cafb68e26688820f4. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240430 20:11:43.509155 20408 consensus_queue.cc:1035] T e3e74e76ab2b4fe492ffcc1838656fd1 P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:43.509310 19471 log.cc:1196] T 20d166756819406ba85044f4fee9c4ce P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/20d166756819406ba85044f4fee9c4ce
I20240430 20:11:43.511596 19471 ts_tablet_manager.cc:1909] T 20d166756819406ba85044f4fee9c4ce P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:43.516906 18445 tablet_service.cc:1514] Processing DeleteTablet for tablet 5bd0362ff41b4faca0205d136d596737 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:31 UTC) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:43.518147 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 139) took 10911 ms. Trace:
I20240430 20:11:43.518240 20362 consensus_queue.cc:1035] T 34130f2820064fa0898ef1f9d7de1c6d P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:43.518656 19471 rpcz_store.cc:276] 0430 20:11:32.603637 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:32.603691 (+ 54us) service_pool.cc:225] Handling call
0430 20:11:43.053360 (+10449669us) ts_tablet_manager.cc:1120] Deleting tablet 20d166756819406ba85044f4fee9c4ce
0430 20:11:43.053371 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:43.176474 (+123103us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:43.495032 (+318558us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:43.515355 (+ 20323us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":10441713}
I20240430 20:11:43.521167 18999 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet 20d166756819406ba85044f4fee9c4ce (table default.Table_b331bf8aeaae4911ac01ed3fbd2627bd [id=1bf5c29b08134670984b6569db298129]) successfully deleted
I20240430 20:11:43.523826 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 141) took 10907 ms. Trace:
I20240430 20:11:43.524987 18750 raft_consensus.cc:1261] T e3e74e76ab2b4fe492ffcc1838656fd1 P 3c51c33d5616482cafb68e26688820f4 [term 3 FOLLOWER]: Refusing update from remote peer 2efb542aa55e439b99ccb98d1c5c884a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240430 20:11:43.525691 20362 consensus_queue.cc:1035] T e3e74e76ab2b4fe492ffcc1838656fd1 P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240430 20:11:43.534427 18474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2d90ff17e574f88b2061b1c117d4b7d" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:43.534794 18474 raft_consensus.cc:2454] T e2d90ff17e574f88b2061b1c117d4b7d P 2efb542aa55e439b99ccb98d1c5c884a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 2.
I20240430 20:11:43.524075 18586 tablet_service.cc:1514] Processing DeleteTablet for tablet b7141e215d6d44bc8406c80508190874 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:43 UTC) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:43.538898 18558 leader_election.cc:302] T e2d90ff17e574f88b2061b1c117d4b7d P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters: 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:43.539228 18601 tablet_service.cc:1514] Processing DeleteTablet for tablet 489dc838744a49eba542009c86688aa5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:43 UTC) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:43.539372 18737 tablet_service.cc:1514] Processing DeleteTablet for tablet b7141e215d6d44bc8406c80508190874 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:43 UTC) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:43.539579 20395 raft_consensus.cc:2781] T e2d90ff17e574f88b2061b1c117d4b7d P 5e1c6397f09747f091534faf93542710 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240430 20:11:43.539683 20395 raft_consensus.cc:482] T e2d90ff17e574f88b2061b1c117d4b7d P 5e1c6397f09747f091534faf93542710 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:43.539764 20395 raft_consensus.cc:3037] T e2d90ff17e574f88b2061b1c117d4b7d P 5e1c6397f09747f091534faf93542710 [term 2 FOLLOWER]: Advancing to term 3
I20240430 20:11:43.542937 19495 ts_tablet_manager.cc:1888] T 53a899fa5a424a0d99f1b47925e8c7ad P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:43.543099 19495 log.cc:1196] T 53a899fa5a424a0d99f1b47925e8c7ad P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/53a899fa5a424a0d99f1b47925e8c7ad
I20240430 20:11:43.543263 20395 raft_consensus.cc:504] T e2d90ff17e574f88b2061b1c117d4b7d P 5e1c6397f09747f091534faf93542710 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:43.543498 19495 ts_tablet_manager.cc:1909] T 53a899fa5a424a0d99f1b47925e8c7ad P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:43.544804 18750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2d90ff17e574f88b2061b1c117d4b7d" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:43.545481 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 218) took 3854 ms. Trace:
I20240430 20:11:43.545588 19495 rpcz_store.cc:276] 0430 20:11:39.698362 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:39.707812 (+ 9450us) service_pool.cc:225] Handling call
0430 20:11:43.423380 (+3715568us) ts_tablet_manager.cc:1120] Deleting tablet 53a899fa5a424a0d99f1b47925e8c7ad
0430 20:11:43.423392 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:43.427482 (+ 4090us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:43.542727 (+115245us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:43.545455 (+ 2728us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3698854}
I20240430 20:11:43.545964 19495 tablet_replica.cc:330] T aec9c09cc0454acab6006bb3e10982b7 P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:43.545760 20395 leader_election.cc:288] T e2d90ff17e574f88b2061b1c117d4b7d P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 3 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:43.546435 18474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2d90ff17e574f88b2061b1c117d4b7d" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:43.546784 18474 raft_consensus.cc:3037] T e2d90ff17e574f88b2061b1c117d4b7d P 2efb542aa55e439b99ccb98d1c5c884a [term 2 FOLLOWER]: Advancing to term 3
I20240430 20:11:43.531059 18983 tablet_bootstrap.cc:492] T 5bd0362ff41b4faca0205d136d596737 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:43.547814 18983 ts_tablet_manager.cc:1367] T 5bd0362ff41b4faca0205d136d596737 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.223s user 0.027s sys 0.030s
I20240430 20:11:43.548678 18983 raft_consensus.cc:348] T 5bd0362ff41b4faca0205d136d596737 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:43.548903 18983 raft_consensus.cc:374] T 5bd0362ff41b4faca0205d136d596737 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:43.548970 18983 raft_consensus.cc:729] T 5bd0362ff41b4faca0205d136d596737 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:43.538894 19471 rpcz_store.cc:276] 0430 20:11:32.616648 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:32.616715 (+ 67us) service_pool.cc:225] Handling call
0430 20:11:43.519731 (+10903016us) ts_tablet_manager.cc:1120] Deleting tablet eb76f60448774fe1967fffe7c6873cfc
0430 20:11:43.519746 (+ 15us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:43.523792 (+ 4046us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":10902114}
I20240430 20:11:43.549573 19471 tablet_replica.cc:330] T 6e826459c3f4437da81c9e6e09f0c22f P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:43.549839 19471 raft_consensus.cc:2229] T 6e826459c3f4437da81c9e6e09f0c22f P 5e1c6397f09747f091534faf93542710 [term 2 LEADER]: Raft consensus shutting down.
I20240430 20:11:43.550561 19471 raft_consensus.cc:2258] T 6e826459c3f4437da81c9e6e09f0c22f P 5e1c6397f09747f091534faf93542710 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:43.554960 19495 raft_consensus.cc:2229] T aec9c09cc0454acab6006bb3e10982b7 P 2efb542aa55e439b99ccb98d1c5c884a [term 3 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:43.556181 19495 raft_consensus.cc:2258] T aec9c09cc0454acab6006bb3e10982b7 P 2efb542aa55e439b99ccb98d1c5c884a [term 3 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:43.556766 19495 ts_tablet_manager.cc:1875] T aec9c09cc0454acab6006bb3e10982b7 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:43.556941 19489 ts_tablet_manager.cc:1888] T 87ac32a0ddb84676adc83038ef664591 P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:43.557052 19489 log.cc:1196] T 87ac32a0ddb84676adc83038ef664591 P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/87ac32a0ddb84676adc83038ef664591
I20240430 20:11:43.557360 19489 ts_tablet_manager.cc:1909] T 87ac32a0ddb84676adc83038ef664591 P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:43.559651 20019 raft_consensus.cc:482] T c2b019f529a54228b804437bb025e31d P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:43.559857 20019 raft_consensus.cc:504] T c2b019f529a54228b804437bb025e31d P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:43.560626 20019 leader_election.cc:288] T c2b019f529a54228b804437bb025e31d P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
W20240430 20:11:43.561184 18999 catalog_manager.cc:4810] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): delete failed for tablet eb76f60448774fe1967fffe7c6873cfc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet eb76f60448774fe1967fffe7c6873cfc already in progress: creating tablet
I20240430 20:11:43.565518 19024 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_f45eee4d4b6a4baf83111f12dfd25a9f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:43.566505 19024 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f45eee4d4b6a4baf83111f12dfd25a9f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:43.572544 20362 consensus_queue.cc:1035] T 43b5e2c005a34df08cc4335c6e014cb5 P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240430 20:11:43.575636 18699 leader_election.cc:341] T c2b019f529a54228b804437bb025e31d P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Not found: Tablet not found: c2b019f529a54228b804437bb025e31d
I20240430 20:11:43.583588 18988 tablet_bootstrap.cc:492] T 84568f90b9c54cf292099808d4cbf5dc P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:43.583752 18988 ts_tablet_manager.cc:1367] T 84568f90b9c54cf292099808d4cbf5dc P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.311s user 0.008s sys 0.007s
I20240430 20:11:43.584712 18988 raft_consensus.cc:348] T 84568f90b9c54cf292099808d4cbf5dc P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:43.584944 18988 raft_consensus.cc:374] T 84568f90b9c54cf292099808d4cbf5dc P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:43.584997 18988 raft_consensus.cc:729] T 84568f90b9c54cf292099808d4cbf5dc P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:43.585209 18988 consensus_queue.cc:260] T 84568f90b9c54cf292099808d4cbf5dc P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:43.585861 18988 ts_tablet_manager.cc:1398] T 84568f90b9c54cf292099808d4cbf5dc P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240430 20:11:43.586171 18988 tablet_bootstrap.cc:492] T 1462ecf1176c4e66be42abddbe36b00c P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:43.587643 18988 tablet_bootstrap.cc:654] T 1462ecf1176c4e66be42abddbe36b00c P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:43.583084 19495 ts_tablet_manager.cc:1888] T aec9c09cc0454acab6006bb3e10982b7 P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240430 20:11:43.599593 19495 log.cc:1196] T aec9c09cc0454acab6006bb3e10982b7 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/aec9c09cc0454acab6006bb3e10982b7
I20240430 20:11:43.600028 19495 ts_tablet_manager.cc:1909] T aec9c09cc0454acab6006bb3e10982b7 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:43.600842 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 219) took 3852 ms. Trace:
I20240430 20:11:43.600950 19495 rpcz_store.cc:276] 0430 20:11:39.748231 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:39.748292 (+ 61us) service_pool.cc:225] Handling call
0430 20:11:43.545901 (+3797609us) ts_tablet_manager.cc:1120] Deleting tablet aec9c09cc0454acab6006bb3e10982b7
0430 20:11:43.545915 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:43.574916 (+ 29001us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:43.582945 (+ 8029us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:43.600813 (+ 17868us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3797159}
I20240430 20:11:43.601567 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 220) took 3801 ms. Trace:
I20240430 20:11:43.601642 19495 rpcz_store.cc:276] 0430 20:11:39.800584 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:39.800846 (+ 262us) service_pool.cc:225] Handling call
0430 20:11:43.601419 (+3800573us) ts_tablet_manager.cc:1120] Deleting tablet 04b3dff86d5c48a09d03da58bd48f360
0430 20:11:43.601430 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:43.601551 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3798143}
I20240430 20:11:43.602013 19495 tablet_replica.cc:330] T 294644d7a16b4ca2bbb6d106bcb8c7ff P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:43.603197 19495 raft_consensus.cc:2229] T 294644d7a16b4ca2bbb6d106bcb8c7ff P 2efb542aa55e439b99ccb98d1c5c884a [term 2 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:43.603649 19495 raft_consensus.cc:2258] T 294644d7a16b4ca2bbb6d106bcb8c7ff P 2efb542aa55e439b99ccb98d1c5c884a [term 2 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:43.604259 19495 ts_tablet_manager.cc:1875] T 294644d7a16b4ca2bbb6d106bcb8c7ff P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:43.553369 18983 consensus_queue.cc:260] T 5bd0362ff41b4faca0205d136d596737 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:43.536543 18595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc267ce92ffd415fbe3377afc0503690. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:43.620425 19495 ts_tablet_manager.cc:1888] T 294644d7a16b4ca2bbb6d106bcb8c7ff P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.3
I20240430 20:11:43.561338 18617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2b019f529a54228b804437bb025e31d" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:43.620574 19495 log.cc:1196] T 294644d7a16b4ca2bbb6d106bcb8c7ff P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/294644d7a16b4ca2bbb6d106bcb8c7ff
I20240430 20:11:43.620617 18983 ts_tablet_manager.cc:1398] T 5bd0362ff41b4faca0205d136d596737 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.073s user 0.002s sys 0.000s
I20240430 20:11:43.620688 18617 raft_consensus.cc:2454] T c2b019f529a54228b804437bb025e31d P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:43.575629 18623 raft_consensus.cc:1261] T 724e51ff05ec4baebc2bb55e05c8eddc P 5e1c6397f09747f091534faf93542710 [term 3 FOLLOWER]: Refusing update from remote peer 2efb542aa55e439b99ccb98d1c5c884a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240430 20:11:43.620947 18983 tablet_bootstrap.cc:492] T d61c279d20b0471a967942525ceb4fcb P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:43.620975 19495 ts_tablet_manager.cc:1909] T 294644d7a16b4ca2bbb6d106bcb8c7ff P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:43.584841 18601 tablet_service.cc:1514] Processing DeleteTablet for tablet 115c4a12440647a4bb46eb6824f7ea98 with delete_type TABLET_DATA_DELETED (Sending initial creation of tablet) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:43.621279 18620 raft_consensus.cc:1226] T aec9c09cc0454acab6006bb3e10982b7 P 5e1c6397f09747f091534faf93542710 [term 3 LEADER]: Rejecting Update request from peer 3c51c33d5616482cafb68e26688820f4 for earlier term 2. Current term is 3. Ops: []
I20240430 20:11:43.621800 18696 leader_election.cc:302] T c2b019f529a54228b804437bb025e31d P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a
I20240430 20:11:43.622802 20362 consensus_queue.cc:1035] T 724e51ff05ec4baebc2bb55e05c8eddc P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240430 20:11:43.622972 18983 tablet_bootstrap.cc:654] T d61c279d20b0471a967942525ceb4fcb P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:43.626044 20019 raft_consensus.cc:2781] T c2b019f529a54228b804437bb025e31d P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:43.626179 20019 raft_consensus.cc:482] T c2b019f529a54228b804437bb025e31d P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:43.626261 20019 raft_consensus.cc:3037] T c2b019f529a54228b804437bb025e31d P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:43.627645 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 221) took 3823 ms. Trace:
I20240430 20:11:43.627858 19471 ts_tablet_manager.cc:1875] T 6e826459c3f4437da81c9e6e09f0c22f P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:43.627741 19495 rpcz_store.cc:276] 0430 20:11:39.803875 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:39.803926 (+ 51us) service_pool.cc:225] Handling call
0430 20:11:43.601969 (+3798043us) ts_tablet_manager.cc:1120] Deleting tablet 294644d7a16b4ca2bbb6d106bcb8c7ff
0430 20:11:43.601977 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:43.610048 (+ 8071us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:43.620279 (+ 10231us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:43.627616 (+ 7337us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":973,"tablet-delete.queue_time_us":3793186}
I20240430 20:11:43.628363 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 222) took 3823 ms. Trace:
I20240430 20:11:43.628435 19495 rpcz_store.cc:276] 0430 20:11:39.805273 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:39.805558 (+ 285us) service_pool.cc:225] Handling call
0430 20:11:43.628223 (+3822665us) ts_tablet_manager.cc:1120] Deleting tablet 1f4e860484784aa9a2d28af2e9fb611c
0430 20:11:43.628236 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:43.628346 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3818788}
I20240430 20:11:43.628875 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 227) took 3797 ms. Trace:
I20240430 20:11:43.628943 19495 rpcz_store.cc:276] 0430 20:11:39.831689 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:39.831736 (+ 47us) service_pool.cc:225] Handling call
0430 20:11:43.628756 (+3797020us) ts_tablet_manager.cc:1120] Deleting tablet d61c279d20b0471a967942525ceb4fcb
0430 20:11:43.628766 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:43.628849 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3796104}
I20240430 20:11:43.628968 20019 raft_consensus.cc:504] T c2b019f529a54228b804437bb025e31d P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:43.630664 18623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2b019f529a54228b804437bb025e31d" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:43.630869 18623 raft_consensus.cc:3037] T c2b019f529a54228b804437bb025e31d P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:43.630828 20205 consensus_queue.cc:1046] T aec9c09cc0454acab6006bb3e10982b7 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240430 20:11:43.631044 20205 consensus_queue.cc:1225] T aec9c09cc0454acab6006bb3e10982b7 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Peer 5e1c6397f09747f091534faf93542710 log is divergent from this leader: its last log entry 3.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20240430 20:11:43.631168 20205 raft_consensus.cc:3032] T aec9c09cc0454acab6006bb3e10982b7 P 3c51c33d5616482cafb68e26688820f4 [term 2 LEADER]: Stepping down as leader of term 2
I20240430 20:11:43.631235 20205 raft_consensus.cc:729] T aec9c09cc0454acab6006bb3e10982b7 P 3c51c33d5616482cafb68e26688820f4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Running, Role: LEADER
I20240430 20:11:43.631283 19495 tablet_replica.cc:330] T e2d90ff17e574f88b2061b1c117d4b7d P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:43.631521 20205 consensus_queue.cc:260] T aec9c09cc0454acab6006bb3e10982b7 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:43.631884 20205 raft_consensus.cc:3037] T aec9c09cc0454acab6006bb3e10982b7 P 3c51c33d5616482cafb68e26688820f4 [term 2 FOLLOWER]: Advancing to term 3
I20240430 20:11:43.633430 18623 raft_consensus.cc:2454] T c2b019f529a54228b804437bb025e31d P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:43.635414 18696 leader_election.cc:302] T c2b019f529a54228b804437bb025e31d P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:43.635772 20198 raft_consensus.cc:2781] T c2b019f529a54228b804437bb025e31d P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:43.636906 18474 raft_consensus.cc:2454] T e2d90ff17e574f88b2061b1c117d4b7d P 2efb542aa55e439b99ccb98d1c5c884a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 3.
I20240430 20:11:43.637151 19495 raft_consensus.cc:2229] T e2d90ff17e574f88b2061b1c117d4b7d P 2efb542aa55e439b99ccb98d1c5c884a [term 3 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:43.637804 19495 raft_consensus.cc:2258] T e2d90ff17e574f88b2061b1c117d4b7d P 2efb542aa55e439b99ccb98d1c5c884a [term 3 FOLLOWER]: Raft consensus is shut down!
W20240430 20:11:43.638177 18699 consensus_peers.cc:485] T e2d90ff17e574f88b2061b1c117d4b7d P 3c51c33d5616482cafb68e26688820f4 -> Peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Couldn't send request to peer 2efb542aa55e439b99ccb98d1c5c884a. Error code: UNKNOWN_ERROR (1). Status: Illegal state: RaftConsensus is not running: State = Stopping. This is attempt 1: this message will repeat every 5th retry.
I20240430 20:11:43.638396 19495 ts_tablet_manager.cc:1875] T e2d90ff17e574f88b2061b1c117d4b7d P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:43.644869 19495 ts_tablet_manager.cc:1888] T e2d90ff17e574f88b2061b1c117d4b7d P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240430 20:11:43.644997 19495 log.cc:1196] T e2d90ff17e574f88b2061b1c117d4b7d P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/e2d90ff17e574f88b2061b1c117d4b7d
I20240430 20:11:43.644932 20398 tablet.cc:1722] T 372b135dd01f44f98cd713ca8857ba61 P 2efb542aa55e439b99ccb98d1c5c884a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240430 20:11:43.648658 20442 tablet.cc:1722] T 372b135dd01f44f98cd713ca8857ba61 P 3c51c33d5616482cafb68e26688820f4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240430 20:11:43.649089 18756 raft_consensus.cc:3037] T 43b5e2c005a34df08cc4335c6e014cb5 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
W20240430 20:11:43.649387 18558 consensus_peers.cc:485] T 53a899fa5a424a0d99f1b47925e8c7ad P 5e1c6397f09747f091534faf93542710 -> Peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Couldn't send request to peer 2efb542aa55e439b99ccb98d1c5c884a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 53a899fa5a424a0d99f1b47925e8c7ad. This is attempt 1: this message will repeat every 5th retry.
I20240430 20:11:43.650971 18558 leader_election.cc:302] T e2d90ff17e574f88b2061b1c117d4b7d P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters: 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:43.651458 20384 raft_consensus.cc:2781] T e2d90ff17e574f88b2061b1c117d4b7d P 5e1c6397f09747f091534faf93542710 [term 3 FOLLOWER]: Leader election won for term 3
I20240430 20:11:43.651583 20384 raft_consensus.cc:686] T e2d90ff17e574f88b2061b1c117d4b7d P 5e1c6397f09747f091534faf93542710 [term 3 LEADER]: Becoming Leader. State: Replica: 5e1c6397f09747f091534faf93542710, State: Running, Role: LEADER
I20240430 20:11:43.651821 20384 consensus_queue.cc:237] T e2d90ff17e574f88b2061b1c117d4b7d P 5e1c6397f09747f091534faf93542710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:43.653718 20362 consensus_queue.cc:1035] T 43b5e2c005a34df08cc4335c6e014cb5 P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:43.658798 19495 ts_tablet_manager.cc:1909] T e2d90ff17e574f88b2061b1c117d4b7d P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:43.659076 18988 tablet_bootstrap.cc:492] T 1462ecf1176c4e66be42abddbe36b00c P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:43.659394 18988 ts_tablet_manager.cc:1367] T 1462ecf1176c4e66be42abddbe36b00c P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.073s user 0.003s sys 0.000s
I20240430 20:11:43.660337 18988 raft_consensus.cc:348] T 1462ecf1176c4e66be42abddbe36b00c P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:43.660524 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 228) took 3585 ms. Trace:
I20240430 20:11:43.660616 19495 rpcz_store.cc:276] 0430 20:11:40.074782 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:40.074846 (+ 64us) service_pool.cc:225] Handling call
0430 20:11:43.631233 (+3556387us) ts_tablet_manager.cc:1120] Deleting tablet e2d90ff17e574f88b2061b1c117d4b7d
0430 20:11:43.631244 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:43.637702 (+ 6458us) spinlock_profiling.cc:243] Waited 5.68 ms on lock 0x616000106f54. stack: 00007f9c64594756 00007f9c64594510 00007f9c635c748d 0000000000955804 00000000009557cd 000000000095519a 00007f9c6fe02a00 00007f9c710dc853 00007f9c73b6f4bc 00007f9c73ba1814 00007f9c73b82b00 00007f9c73b8288d 0000000000a60bcc 00007f9c645dab74 00007f9c645de19b 00007f9c645de018
0430 20:11:43.641717 (+ 4015us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:43.644728 (+ 3011us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:43.660499 (+ 15771us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":13059328,"tablet-delete.queue_time_us":3556076}
I20240430 20:11:43.661702 18988 raft_consensus.cc:374] T 1462ecf1176c4e66be42abddbe36b00c P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:43.661931 18988 raft_consensus.cc:729] T 1462ecf1176c4e66be42abddbe36b00c P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:43.663452 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 233) took 3328 ms. Trace:
I20240430 20:11:43.663882 18988 consensus_queue.cc:260] T 1462ecf1176c4e66be42abddbe36b00c P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:43.663640 19495 rpcz_store.cc:276] 0430 20:11:40.335488 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:40.335616 (+ 128us) service_pool.cc:225] Handling call
0430 20:11:43.663260 (+3327644us) ts_tablet_manager.cc:1120] Deleting tablet 898b2c91adf34545808df9c786e218c9
0430 20:11:43.663277 (+ 17us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:43.663426 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3323512}
I20240430 20:11:43.664728 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 234) took 3232 ms. Trace:
I20240430 20:11:43.664846 19495 rpcz_store.cc:276] 0430 20:11:40.431916 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:40.432031 (+ 115us) service_pool.cc:225] Handling call
0430 20:11:43.664506 (+3232475us) ts_tablet_manager.cc:1120] Deleting tablet d4d6c37b578c4fe1aa666b0a6b0bc7c9
0430 20:11:43.664515 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:43.664710 (+ 195us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3232198}
I20240430 20:11:43.665035 18983 tablet_bootstrap.cc:492] T d61c279d20b0471a967942525ceb4fcb P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:43.665171 18983 ts_tablet_manager.cc:1367] T d61c279d20b0471a967942525ceb4fcb P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.044s user 0.002s sys 0.009s
I20240430 20:11:43.665964 18983 raft_consensus.cc:348] T d61c279d20b0471a967942525ceb4fcb P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:43.666177 18983 raft_consensus.cc:374] T d61c279d20b0471a967942525ceb4fcb P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:43.666229 18983 raft_consensus.cc:729] T d61c279d20b0471a967942525ceb4fcb P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:43.666488 18983 consensus_queue.cc:260] T d61c279d20b0471a967942525ceb4fcb P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:43.667001 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 130) took 11227 ms. Trace:
I20240430 20:11:43.667013 18983 ts_tablet_manager.cc:1398] T d61c279d20b0471a967942525ceb4fcb P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240430 20:11:43.667402 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 236) took 3193 ms. Trace:
I20240430 20:11:43.667492 19495 rpcz_store.cc:276] 0430 20:11:40.474090 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:40.474147 (+ 57us) service_pool.cc:225] Handling call
0430 20:11:43.667278 (+3193131us) ts_tablet_manager.cc:1120] Deleting tablet f3711610610542128c4a84e17bce4a3e
0430 20:11:43.667289 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:43.667384 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3192886}
I20240430 20:11:43.667945 19495 tablet_replica.cc:330] T 5809e7c9c2204ba293426c24129fb16e P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:43.668159 19495 raft_consensus.cc:2229] T 5809e7c9c2204ba293426c24129fb16e P 2efb542aa55e439b99ccb98d1c5c884a [term 2 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:43.668332 19495 raft_consensus.cc:2258] T 5809e7c9c2204ba293426c24129fb16e P 2efb542aa55e439b99ccb98d1c5c884a [term 2 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:43.668895 19495 ts_tablet_manager.cc:1875] T 5809e7c9c2204ba293426c24129fb16e P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:43.669070 18983 tablet_bootstrap.cc:492] T f7b0132c17d44a9ba17288e0321a7f11 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:43.669202 19489 rpcz_store.cc:276] 0430 20:11:32.439087 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:32.439140 (+ 53us) service_pool.cc:225] Handling call
0430 20:11:43.474689 (+11035549us) ts_tablet_manager.cc:1120] Deleting tablet 87ac32a0ddb84676adc83038ef664591
0430 20:11:43.474701 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:43.553926 (+ 79225us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:43.556835 (+ 2909us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:43.666980 (+110145us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":31295,"tablet-delete.queue_time_us":11035295}
I20240430 20:11:43.670640 18983 tablet_bootstrap.cc:654] T f7b0132c17d44a9ba17288e0321a7f11 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:43.666648 18988 ts_tablet_manager.cc:1398] T 1462ecf1176c4e66be42abddbe36b00c P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240430 20:11:43.669674 18763 raft_consensus.cc:1261] T 724e51ff05ec4baebc2bb55e05c8eddc P 3c51c33d5616482cafb68e26688820f4 [term 3 FOLLOWER]: Refusing update from remote peer 2efb542aa55e439b99ccb98d1c5c884a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240430 20:11:43.672601 18988 tablet_bootstrap.cc:492] T 34130f2820064fa0898ef1f9d7de1c6d P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:43.672981 20408 consensus_queue.cc:1035] T 724e51ff05ec4baebc2bb55e05c8eddc P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240430 20:11:43.673938 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 131) took 11214 ms. Trace:
I20240430 20:11:43.674022 19489 rpcz_store.cc:276] 0430 20:11:32.459393 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:32.459453 (+ 60us) service_pool.cc:225] Handling call
0430 20:11:43.673798 (+11214345us) ts_tablet_manager.cc:1120] Deleting tablet f940d17b02b541289121463e05b0bb5f
0430 20:11:43.673809 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:43.673914 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":11166890}
I20240430 20:11:43.675956 19471 ts_tablet_manager.cc:1888] T 6e826459c3f4437da81c9e6e09f0c22f P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.3
I20240430 20:11:43.676733 19495 ts_tablet_manager.cc:1888] T 5809e7c9c2204ba293426c24129fb16e P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.2
I20240430 20:11:43.676867 19495 log.cc:1196] T 5809e7c9c2204ba293426c24129fb16e P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/5809e7c9c2204ba293426c24129fb16e
I20240430 20:11:43.677197 19495 ts_tablet_manager.cc:1909] T 5809e7c9c2204ba293426c24129fb16e P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:43.677954 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 235) took 3205 ms. Trace:
I20240430 20:11:43.678066 19495 rpcz_store.cc:276] 0430 20:11:40.472501 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:40.475136 (+ 2635us) service_pool.cc:225] Handling call
0430 20:11:43.667900 (+3192764us) ts_tablet_manager.cc:1120] Deleting tablet 5809e7c9c2204ba293426c24129fb16e
0430 20:11:43.667909 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:43.673738 (+ 5829us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:43.676643 (+ 2905us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:43.677929 (+ 1286us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3192574}
I20240430 20:11:43.678478 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 237) took 3187 ms. Trace:
I20240430 20:11:43.678556 19495 rpcz_store.cc:276] 0430 20:11:40.491072 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:40.491125 (+ 53us) service_pool.cc:225] Handling call
0430 20:11:43.678352 (+3187227us) ts_tablet_manager.cc:1120] Deleting tablet d7c1fc7e81364fb890bbc13d7fb6848b
0430 20:11:43.678363 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:43.678464 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3187022}
I20240430 20:11:43.678930 19495 tablet_replica.cc:330] T fd40967ab4a04001819e16c06a6c4300 P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:43.679191 19495 raft_consensus.cc:2229] T fd40967ab4a04001819e16c06a6c4300 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:43.679389 19495 raft_consensus.cc:2258] T fd40967ab4a04001819e16c06a6c4300 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:43.680096 19495 ts_tablet_manager.cc:1875] T fd40967ab4a04001819e16c06a6c4300 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:43.685609 19489 tablet_replica.cc:330] T 6e826459c3f4437da81c9e6e09f0c22f P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:43.685840 19489 raft_consensus.cc:2229] T 6e826459c3f4437da81c9e6e09f0c22f P 3c51c33d5616482cafb68e26688820f4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:43.686010 19489 raft_consensus.cc:2258] T 6e826459c3f4437da81c9e6e09f0c22f P 3c51c33d5616482cafb68e26688820f4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:43.686074 20019 leader_election.cc:288] T c2b019f529a54228b804437bb025e31d P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Requested vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:43.686551 19489 ts_tablet_manager.cc:1875] T 6e826459c3f4437da81c9e6e09f0c22f P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240430 20:11:43.687304 18699 leader_election.cc:341] T c2b019f529a54228b804437bb025e31d P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Not found: Tablet not found: c2b019f529a54228b804437bb025e31d
I20240430 20:11:43.688803 19471 log.cc:1196] T 6e826459c3f4437da81c9e6e09f0c22f P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/6e826459c3f4437da81c9e6e09f0c22f
I20240430 20:11:43.690456 19001 catalog_manager.cc:4830] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): tablet 87ac32a0ddb84676adc83038ef664591 (table default.Table_763f0800e8bb4fa69976886416250f42 [id=61c8a5b99ef34a0baeeb343fef7cfd05]) successfully deleted
I20240430 20:11:43.691464 19471 ts_tablet_manager.cc:1909] T 6e826459c3f4437da81c9e6e09f0c22f P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
W20240430 20:11:43.691984 19001 catalog_manager.cc:4810] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): delete failed for tablet f940d17b02b541289121463e05b0bb5f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f940d17b02b541289121463e05b0bb5f already in progress: creating tablet
I20240430 20:11:43.692426 19004 catalog_manager.cc:4830] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): tablet aec9c09cc0454acab6006bb3e10982b7 (table default.Table_ff91abfadbc746d6b01fdbfecd16fac5 [id=26d4860fb33047b7b812ebd4309f2e28]) successfully deleted
W20240430 20:11:43.692660 19004 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.17.177.1:41217, user_credentials={real_user=slave}} blocked reactor thread for 91236.9us
I20240430 20:11:43.693516 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 142) took 11065 ms. Trace:
I20240430 20:11:43.694957 19471 rpcz_store.cc:276] 0430 20:11:32.628341 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:32.628784 (+ 443us) service_pool.cc:225] Handling call
0430 20:11:43.549502 (+10920718us) ts_tablet_manager.cc:1120] Deleting tablet 6e826459c3f4437da81c9e6e09f0c22f
0430 20:11:43.549526 (+ 24us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:43.672325 (+122799us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:43.675823 (+ 3498us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:43.693488 (+ 17665us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":10913833}
W20240430 20:11:43.695317 19004 catalog_manager.cc:4810] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): delete failed for tablet 04b3dff86d5c48a09d03da58bd48f360 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 04b3dff86d5c48a09d03da58bd48f360 already in progress: creating tablet
I20240430 20:11:43.695664 19004 catalog_manager.cc:4830] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): tablet 294644d7a16b4ca2bbb6d106bcb8c7ff (table default.Table_190b2d86515a44898fa38a49517b9df1 [id=97d8a36ddfae4bb5b99c04a7a08544af]) successfully deleted
W20240430 20:11:43.696184 19004 catalog_manager.cc:4810] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): delete failed for tablet 1f4e860484784aa9a2d28af2e9fb611c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1f4e860484784aa9a2d28af2e9fb611c already in progress: creating tablet
I20240430 20:11:43.696707 18999 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet 6e826459c3f4437da81c9e6e09f0c22f (table default.Table_873c21f6dffd4d47a742b3699ab85268 [id=d7e2c4300ff94fa2acc65d6d6c932c65]) successfully deleted
I20240430 20:11:43.697373 18988 tablet_bootstrap.cc:654] T 34130f2820064fa0898ef1f9d7de1c6d P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
W20240430 20:11:43.699520 19004 catalog_manager.cc:4810] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): delete failed for tablet d61c279d20b0471a967942525ceb4fcb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d61c279d20b0471a967942525ceb4fcb already in progress: creating tablet
W20240430 20:11:43.703275 19004 catalog_manager.cc:4810] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): delete failed for tablet 898b2c91adf34545808df9c786e218c9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 898b2c91adf34545808df9c786e218c9 already in progress: creating tablet
W20240430 20:11:43.703661 19004 catalog_manager.cc:4810] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): delete failed for tablet d4d6c37b578c4fe1aa666b0a6b0bc7c9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d4d6c37b578c4fe1aa666b0a6b0bc7c9 already in progress: creating tablet
W20240430 20:11:43.704317 19004 catalog_manager.cc:4810] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): delete failed for tablet f3711610610542128c4a84e17bce4a3e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f3711610610542128c4a84e17bce4a3e already in progress: creating tablet
I20240430 20:11:43.704680 19004 catalog_manager.cc:4830] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): tablet 5809e7c9c2204ba293426c24129fb16e (table default.Table_d82824c897a94dd0ba45955f9473b626 [id=d794640d21a843a185038673f7eb2b00]) successfully deleted
W20240430 20:11:43.705327 19004 catalog_manager.cc:4810] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): delete failed for tablet d7c1fc7e81364fb890bbc13d7fb6848b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d7c1fc7e81364fb890bbc13d7fb6848b already in progress: creating tablet
I20240430 20:11:43.707139 19495 ts_tablet_manager.cc:1888] T fd40967ab4a04001819e16c06a6c4300 P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:43.707263 19495 log.cc:1196] T fd40967ab4a04001819e16c06a6c4300 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/fd40967ab4a04001819e16c06a6c4300
I20240430 20:11:43.707631 19495 ts_tablet_manager.cc:1909] T fd40967ab4a04001819e16c06a6c4300 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:43.707873 20019 raft_consensus.cc:482] T 1462ecf1176c4e66be42abddbe36b00c P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:43.708071 20019 raft_consensus.cc:504] T 1462ecf1176c4e66be42abddbe36b00c P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
W20240430 20:11:43.708595 18557 consensus_peers.cc:485] T 5f02d8862e1f4ccfa9dbda73dbe18615 P 5e1c6397f09747f091534faf93542710 -> Peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Couldn't send request to peer 3c51c33d5616482cafb68e26688820f4. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240430 20:11:43.708796 20019 leader_election.cc:288] T 1462ecf1176c4e66be42abddbe36b00c P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:43.709712 18474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1462ecf1176c4e66be42abddbe36b00c" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:43.710863 18620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1462ecf1176c4e66be42abddbe36b00c" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
W20240430 20:11:43.711661 18699 consensus_peers.cc:485] T 24dcc20832804229a2664873bf6ddaa7 P 3c51c33d5616482cafb68e26688820f4 -> Peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Couldn't send request to peer 2efb542aa55e439b99ccb98d1c5c884a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 24dcc20832804229a2664873bf6ddaa7. This is attempt 1: this message will repeat every 5th retry.
I20240430 20:11:43.712219 18699 leader_election.cc:302] T 1462ecf1176c4e66be42abddbe36b00c P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710
I20240430 20:11:43.713608 19023 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_6a30a0abb67d4a298f87216515a92153" } new_table_name: "default.Table_5ed3ea587cf843db9a1ff1ffab40b678" modify_external_catalogs: true
I20240430 20:11:43.714959 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 143) took 11064 ms. Trace:
I20240430 20:11:43.715174 19471 rpcz_store.cc:276] 0430 20:11:32.650340 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:32.650576 (+ 236us) service_pool.cc:225] Handling call
0430 20:11:43.714806 (+11064230us) ts_tablet_manager.cc:1120] Deleting tablet 2873af6da928463f9809864b8eaf0c4a
0430 20:11:43.714819 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:43.714936 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":11056349}
W20240430 20:11:43.715770 18999 catalog_manager.cc:4810] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): delete failed for tablet 2873af6da928463f9809864b8eaf0c4a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2873af6da928463f9809864b8eaf0c4a already in progress: creating tablet
I20240430 20:11:43.716230 19049 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_b869ed2a85d64fcf83a0eb41c1a86446"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240430 20:11:43.716372 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 148) took 10627 ms. Trace:
I20240430 20:11:43.716456 19471 rpcz_store.cc:276] 0430 20:11:33.088866 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:33.092175 (+ 3309us) service_pool.cc:225] Handling call
0430 20:11:43.716243 (+10624068us) ts_tablet_manager.cc:1120] Deleting tablet 246fb63a5e9842df8263800e28a5935c
0430 20:11:43.716255 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:43.716354 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":10617062}
I20240430 20:11:43.717114 19471 tablet_replica.cc:330] T c2b019f529a54228b804437bb025e31d P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:43.717352 19471 raft_consensus.cc:2229] T c2b019f529a54228b804437bb025e31d P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:43.717597 19471 raft_consensus.cc:2258] T c2b019f529a54228b804437bb025e31d P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240430 20:11:43.717552 19049 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b869ed2a85d64fcf83a0eb41c1a86446 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:43.718245 19471 ts_tablet_manager.cc:1875] T c2b019f529a54228b804437bb025e31d P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240430 20:11:43.716908 18999 catalog_manager.cc:4810] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): delete failed for tablet 246fb63a5e9842df8263800e28a5935c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 246fb63a5e9842df8263800e28a5935c already in progress: creating tablet
I20240430 20:11:43.719271 19027 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_f5a3451cd2284390be4357d771c03390" } new_table_name: "default.Table_1cf914acf7fe4e98b8dbc297b6372f21" modify_external_catalogs: true
I20240430 20:11:43.721554 19026 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_d770977de70d4cac8511d8d125c1b160"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240430 20:11:43.723779 19033 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_d95c70be474e415cba6a345438fad1ba" } modify_external_catalogs: true
I20240430 20:11:43.724088 19033 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_d95c70be474e415cba6a345438fad1ba" } modify_external_catalogs: true
W20240430 20:11:43.722564 19026 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d770977de70d4cac8511d8d125c1b160 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:43.726894 20019 raft_consensus.cc:2726] T 1462ecf1176c4e66be42abddbe36b00c P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:43.733686 19024 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_8c841af5208742f9b56a4591c02c573e" } new_table_name: "default.Table_c1f36f7df5ae4e7c8eb8b4e99b89748b" modify_external_catalogs: true
I20240430 20:11:43.738956 18983 tablet_bootstrap.cc:492] T f7b0132c17d44a9ba17288e0321a7f11 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:43.739126 18983 ts_tablet_manager.cc:1367] T f7b0132c17d44a9ba17288e0321a7f11 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.072s user 0.000s sys 0.006s
I20240430 20:11:43.739945 18983 raft_consensus.cc:348] T f7b0132c17d44a9ba17288e0321a7f11 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:43.740209 18983 raft_consensus.cc:374] T f7b0132c17d44a9ba17288e0321a7f11 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:43.740269 18983 raft_consensus.cc:729] T f7b0132c17d44a9ba17288e0321a7f11 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:43.740521 18983 consensus_queue.cc:260] T f7b0132c17d44a9ba17288e0321a7f11 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:43.741667 19489 ts_tablet_manager.cc:1888] T 6e826459c3f4437da81c9e6e09f0c22f P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.3
I20240430 20:11:43.741793 19489 log.cc:1196] T 6e826459c3f4437da81c9e6e09f0c22f P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/6e826459c3f4437da81c9e6e09f0c22f
I20240430 20:11:43.751026 18726 tablet_service.cc:1514] Processing DeleteTablet for tablet fae9a35a2ac844b4bc2ebdf214aa7171 with delete_type TABLET_DATA_DELETED (replaced by tablet b22aaedb0a6c4fd9ad0ea8fa019001e1) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:43.751103 18740 tablet_service.cc:1514] Processing DeleteTablet for tablet 73453a7fa30342e29db552b17b62f0c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:32 UTC) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:43.751026 18737 tablet_service.cc:1514] Processing DeleteTablet for tablet b72fe1f0713b4076a81a3fc1c74f7fa7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:32 UTC) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:43.752008 18983 ts_tablet_manager.cc:1398] T f7b0132c17d44a9ba17288e0321a7f11 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.013s user 0.003s sys 0.000s
I20240430 20:11:43.752094 18988 tablet_bootstrap.cc:492] T 34130f2820064fa0898ef1f9d7de1c6d P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:43.752238 18988 ts_tablet_manager.cc:1367] T 34130f2820064fa0898ef1f9d7de1c6d P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.080s user 0.003s sys 0.000s
I20240430 20:11:43.752391 18983 tablet_bootstrap.cc:492] T 681cdd85ac6847a0a1aaded310456582 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:43.752774 19489 ts_tablet_manager.cc:1909] T 6e826459c3f4437da81c9e6e09f0c22f P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:43.753585 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 148) took 11078 ms. Trace:
I20240430 20:11:43.753669 19489 rpcz_store.cc:276] 0430 20:11:32.674781 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:32.676482 (+ 1701us) service_pool.cc:225] Handling call
0430 20:11:43.685547 (+11009065us) ts_tablet_manager.cc:1120] Deleting tablet 6e826459c3f4437da81c9e6e09f0c22f
0430 20:11:43.685559 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:43.716319 (+ 30760us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:43.741563 (+ 25244us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:43.753563 (+ 12000us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":10984671}
I20240430 20:11:43.754204 19001 catalog_manager.cc:4830] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): tablet 6e826459c3f4437da81c9e6e09f0c22f (table default.Table_873c21f6dffd4d47a742b3699ab85268 [id=d7e2c4300ff94fa2acc65d6d6c932c65]) successfully deleted
I20240430 20:11:43.757233 18742 tablet_service.cc:1514] Processing DeleteTablet for tablet f940d17b02b541289121463e05b0bb5f with delete_type TABLET_DATA_DELETED (replaced by tablet 53da4b3855344b6d92211af767195dc8) from {username='slave'} at 127.0.0.1:59830
W20240430 20:11:43.761881 19001 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 1701a15b115741aea13f8327e3d26b34 (table default.Table_3776a4b9d4f24aa6b7f882bb4d0116ef [id=7127dbd368864f7ba80a755549fe7ede], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 1701a15b115741aea13f8327e3d26b34 consensus state has no leader
I20240430 20:11:43.763373 18988 raft_consensus.cc:348] T 34130f2820064fa0898ef1f9d7de1c6d P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:43.763602 18988 raft_consensus.cc:374] T 34130f2820064fa0898ef1f9d7de1c6d P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:43.763657 18988 raft_consensus.cc:729] T 34130f2820064fa0898ef1f9d7de1c6d P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:43.763862 18988 consensus_queue.cc:260] T 34130f2820064fa0898ef1f9d7de1c6d P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:43.764963 18988 ts_tablet_manager.cc:1398] T 34130f2820064fa0898ef1f9d7de1c6d P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20240430 20:11:43.765398 18988 tablet_bootstrap.cc:492] T ebf74809b52f47f3b0ed21b6ac0f541a P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:43.766361 19471 ts_tablet_manager.cc:1888] T c2b019f529a54228b804437bb025e31d P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:43.766492 19471 log.cc:1196] T c2b019f529a54228b804437bb025e31d P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/c2b019f529a54228b804437bb025e31d
I20240430 20:11:43.766832 19471 ts_tablet_manager.cc:1909] T c2b019f529a54228b804437bb025e31d P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:43.766829 18988 tablet_bootstrap.cc:654] T ebf74809b52f47f3b0ed21b6ac0f541a P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:43.767149 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 147) took 10678 ms. Trace:
I20240430 20:11:43.767227 19471 rpcz_store.cc:276] 0430 20:11:33.088308 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:33.163362 (+ 75054us) service_pool.cc:225] Handling call
0430 20:11:43.717070 (+10553708us) ts_tablet_manager.cc:1120] Deleting tablet c2b019f529a54228b804437bb025e31d
0430 20:11:43.717078 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:43.721387 (+ 4309us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:43.766246 (+ 44859us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:43.767126 (+ 880us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":10523270}
I20240430 20:11:43.767692 18999 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet c2b019f529a54228b804437bb025e31d (table default.Table_8028554bc993419e80dbf7d526b63bf4 [id=27ef5b3429fb4353afe80dc3089e9914]) successfully deleted
I20240430 20:11:43.768656 19471 tablet_replica.cc:330] T a745334c12c545c1ba825c7552c60359 P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:43.768853 19471 raft_consensus.cc:2229] T a745334c12c545c1ba825c7552c60359 P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Raft consensus shutting down.
I20240430 20:11:43.769141 19471 raft_consensus.cc:2258] T a745334c12c545c1ba825c7552c60359 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:43.769723 19471 ts_tablet_manager.cc:1875] T a745334c12c545c1ba825c7552c60359 P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240430 20:11:43.771454 18558 consensus_peers.cc:485] T fd40967ab4a04001819e16c06a6c4300 P 5e1c6397f09747f091534faf93542710 -> Peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Couldn't send request to peer 2efb542aa55e439b99ccb98d1c5c884a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fd40967ab4a04001819e16c06a6c4300. This is attempt 1: this message will repeat every 5th retry.
W20240430 20:11:43.772053 19028 catalog_manager.cc:5676] Failed to send AlterTable RPC for tablet dbb427aefa6b47b0805d1751fea4efcb (table default.Table_4130caf534004edfb413056d7f6cfd52 [id=da9ad991953a40c4b77b31317597ad1d], current schema version=4): Not found: failed to reset TS proxy: No leader found: Tablet dbb427aefa6b47b0805d1751fea4efcb consensus state has no leader
I20240430 20:11:43.772498 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 134) took 11099 ms. Trace:
I20240430 20:11:43.772588 19489 rpcz_store.cc:276] 0430 20:11:32.673415 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:32.675941 (+ 2526us) service_pool.cc:225] Handling call
0430 20:11:43.772278 (+11096337us) ts_tablet_manager.cc:1120] Deleting tablet 2cdccf4be7094ca5b863dd5c3432d55d
0430 20:11:43.772291 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:43.772476 (+ 185us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":11069385}
W20240430 20:11:43.773068 19001 catalog_manager.cc:4810] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): delete failed for tablet 2cdccf4be7094ca5b863dd5c3432d55d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2cdccf4be7094ca5b863dd5c3432d55d already in progress: creating tablet
I20240430 20:11:43.773509 19489 tablet_replica.cc:330] T 20d166756819406ba85044f4fee9c4ce P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:43.773694 19489 raft_consensus.cc:2229] T 20d166756819406ba85044f4fee9c4ce P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:43.773856 19489 raft_consensus.cc:2258] T 20d166756819406ba85044f4fee9c4ce P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:43.775004 18742 tablet_service.cc:1467] Processing CreateTablet for tablet fd423a49fb694695a8de7ecd54330127 (DEFAULT_TABLE table=default.Table_204dce4e6128482583d8869c83413f25 [id=618fcf2891c84a238fcde6f9b62bba3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:43.775427 18742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd423a49fb694695a8de7ecd54330127. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:43.775521 19029 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet ebf74809b52f47f3b0ed21b6ac0f541a (table default.Table_b2d07520b9944a89ad62fca70396174e [id=1c8e0a3a44ec4a798770c0165ed3dffa]). Expected version 1 got 0
I20240430 20:11:43.776382 18740 tablet_service.cc:1467] Processing CreateTablet for tablet 0a558d18cdfa4b33b4f72dc9041bfd53 (DEFAULT_TABLE table=default.Table_50a4836df84543a888fcdb06ccc5ec41 [id=4cc347c303a04ae2864738e6b2cdc519]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:43.776700 18740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a558d18cdfa4b33b4f72dc9041bfd53. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:43.778024 18454 tablet_service.cc:1467] Processing CreateTablet for tablet fd423a49fb694695a8de7ecd54330127 (DEFAULT_TABLE table=default.Table_204dce4e6128482583d8869c83413f25 [id=618fcf2891c84a238fcde6f9b62bba3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:43.779629 18726 tablet_service.cc:1467] Processing CreateTablet for tablet ad01c979b2f7439bab83137f87f1405c (DEFAULT_TABLE table=default.Table_e3165b7ce52a44bf9d2bed43f9c5d6c7 [id=3546514024eb4b8c9ad58c7af8d63aff]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:43.779611 18449 tablet_service.cc:1467] Processing CreateTablet for tablet 0a558d18cdfa4b33b4f72dc9041bfd53 (DEFAULT_TABLE table=default.Table_50a4836df84543a888fcdb06ccc5ec41 [id=4cc347c303a04ae2864738e6b2cdc519]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:43.779762 18739 tablet_service.cc:1467] Processing CreateTablet for tablet f1073fc792954c5795a05aa7d0ec24df (DEFAULT_TABLE table=default.Table_6e886b67708e4dc1b9e1753b1dd00800 [id=aedf11fda8914a3b843a57f8a7b69a53]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:43.780005 18726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad01c979b2f7439bab83137f87f1405c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:43.779968 18446 tablet_service.cc:1467] Processing CreateTablet for tablet ad01c979b2f7439bab83137f87f1405c (DEFAULT_TABLE table=default.Table_e3165b7ce52a44bf9d2bed43f9c5d6c7 [id=3546514024eb4b8c9ad58c7af8d63aff]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:43.780052 18447 tablet_service.cc:1467] Processing CreateTablet for tablet f1073fc792954c5795a05aa7d0ec24df (DEFAULT_TABLE table=default.Table_6e886b67708e4dc1b9e1753b1dd00800 [id=aedf11fda8914a3b843a57f8a7b69a53]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:43.780395 18739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1073fc792954c5795a05aa7d0ec24df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:43.780295 18733 tablet_service.cc:1467] Processing CreateTablet for tablet bbdbb8a3a4b444f58a2761466577a88f (DEFAULT_TABLE table=default.Table_dd819f96762949c786839dda00487cc2 [id=e1e2e2a5b0b740cba09110991dd28193]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:43.780618 18733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbdbb8a3a4b444f58a2761466577a88f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:43.780598 18459 tablet_service.cc:1467] Processing CreateTablet for tablet 9b95098d3af04dd5a9ae3972e44b3a6c (DEFAULT_TABLE table=default.Table_80306c0ee72842209ca8feac52c0f247 [id=ac6acb1d504243e292723b12e0ad2b33]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:43.780292 18732 tablet_service.cc:1467] Processing CreateTablet for tablet 9b95098d3af04dd5a9ae3972e44b3a6c (DEFAULT_TABLE table=default.Table_80306c0ee72842209ca8feac52c0f247 [id=ac6acb1d504243e292723b12e0ad2b33]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:43.780953 18732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b95098d3af04dd5a9ae3972e44b3a6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:43.782913 18453 tablet_service.cc:1467] Processing CreateTablet for tablet c4953578de5f45f3a6ba092a13357460 (DEFAULT_TABLE table=default.Table_9110c0cb4db441dbb8bdce422eadc189 [id=a826c08c7eef44ec83feae3a6cb384e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:43.783181 18445 tablet_service.cc:1467] Processing CreateTablet for tablet c85a6d11d8af486492d8b843c75632e6 (DEFAULT_TABLE table=default.Table_dfb59f35018441fd9a72ea860cdd01f6 [id=880609dd541c4ce5af4d27b5a2e0c9a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:43.783713 18461 tablet_service.cc:1467] Processing CreateTablet for tablet bbdbb8a3a4b444f58a2761466577a88f (DEFAULT_TABLE table=default.Table_dd819f96762949c786839dda00487cc2 [id=e1e2e2a5b0b740cba09110991dd28193]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:43.792665 18737 tablet_service.cc:1467] Processing CreateTablet for tablet c85a6d11d8af486492d8b843c75632e6 (DEFAULT_TABLE table=default.Table_dfb59f35018441fd9a72ea860cdd01f6 [id=880609dd541c4ce5af4d27b5a2e0c9a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:43.793133 18737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c85a6d11d8af486492d8b843c75632e6. 1 dirs total, 0 dirs full, 0 dirs failed
W20240430 20:11:43.793422 18699 consensus_peers.cc:485] T 5809e7c9c2204ba293426c24129fb16e P 3c51c33d5616482cafb68e26688820f4 -> Peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Couldn't send request to peer 2efb542aa55e439b99ccb98d1c5c884a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5809e7c9c2204ba293426c24129fb16e. This is attempt 1: this message will repeat every 5th retry.
W20240430 20:11:43.794052 18558 consensus_peers.cc:485] T aec9c09cc0454acab6006bb3e10982b7 P 5e1c6397f09747f091534faf93542710 -> Peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Couldn't send request to peer 2efb542aa55e439b99ccb98d1c5c884a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: aec9c09cc0454acab6006bb3e10982b7. This is attempt 1: this message will repeat every 5th retry.
I20240430 20:11:43.799584 18588 tablet_service.cc:1467] Processing CreateTablet for tablet f1073fc792954c5795a05aa7d0ec24df (DEFAULT_TABLE table=default.Table_6e886b67708e4dc1b9e1753b1dd00800 [id=aedf11fda8914a3b843a57f8a7b69a53]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:43.799780 18586 tablet_service.cc:1467] Processing CreateTablet for tablet 0a558d18cdfa4b33b4f72dc9041bfd53 (DEFAULT_TABLE table=default.Table_50a4836df84543a888fcdb06ccc5ec41 [id=4cc347c303a04ae2864738e6b2cdc519]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:43.800151 18602 tablet_service.cc:1467] Processing CreateTablet for tablet ad01c979b2f7439bab83137f87f1405c (DEFAULT_TABLE table=default.Table_e3165b7ce52a44bf9d2bed43f9c5d6c7 [id=3546514024eb4b8c9ad58c7af8d63aff]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:43.800531 18590 tablet_service.cc:1467] Processing CreateTablet for tablet 9b95098d3af04dd5a9ae3972e44b3a6c (DEFAULT_TABLE table=default.Table_80306c0ee72842209ca8feac52c0f247 [id=ac6acb1d504243e292723b12e0ad2b33]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:43.800834 18603 tablet_service.cc:1467] Processing CreateTablet for tablet bbdbb8a3a4b444f58a2761466577a88f (DEFAULT_TABLE table=default.Table_dd819f96762949c786839dda00487cc2 [id=e1e2e2a5b0b740cba09110991dd28193]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:43.799613 18601 tablet_service.cc:1467] Processing CreateTablet for tablet c85a6d11d8af486492d8b843c75632e6 (DEFAULT_TABLE table=default.Table_dfb59f35018441fd9a72ea860cdd01f6 [id=880609dd541c4ce5af4d27b5a2e0c9a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:43.801388 18595 tablet_service.cc:1467] Processing CreateTablet for tablet fd423a49fb694695a8de7ecd54330127 (DEFAULT_TABLE table=default.Table_204dce4e6128482583d8869c83413f25 [id=618fcf2891c84a238fcde6f9b62bba3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:43.805168 18742 tablet_service.cc:1467] Processing CreateTablet for tablet c4953578de5f45f3a6ba092a13357460 (DEFAULT_TABLE table=default.Table_9110c0cb4db441dbb8bdce422eadc189 [id=a826c08c7eef44ec83feae3a6cb384e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:43.806619 18592 tablet_service.cc:1467] Processing CreateTablet for tablet c4953578de5f45f3a6ba092a13357460 (DEFAULT_TABLE table=default.Table_9110c0cb4db441dbb8bdce422eadc189 [id=a826c08c7eef44ec83feae3a6cb384e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:43.806717 20384 raft_consensus.cc:482] T 30a521916bda46678be20a066bd1d3f3 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:43.806854 20384 raft_consensus.cc:504] T 30a521916bda46678be20a066bd1d3f3 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:43.807552 20384 leader_election.cc:288] T 30a521916bda46678be20a066bd1d3f3 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:43.807603 18458 tablet_service.cc:1467] Processing CreateTablet for tablet 975d145de1794f05b1dd844b4b86ce40 (DEFAULT_TABLE table=default.Table_2e5bcbca76d345fea6d7b7dac08a7afb [id=e2122b91429d4acc939e280d8261c452]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:43.808539 18750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30a521916bda46678be20a066bd1d3f3" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
W20240430 20:11:43.809063 18557 leader_election.cc:341] T 30a521916bda46678be20a066bd1d3f3 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:43.809113 18732 tablet_service.cc:1467] Processing CreateTablet for tablet 975d145de1794f05b1dd844b4b86ce40 (DEFAULT_TABLE table=default.Table_2e5bcbca76d345fea6d7b7dac08a7afb [id=e2122b91429d4acc939e280d8261c452]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:43.809427 18482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30a521916bda46678be20a066bd1d3f3" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
W20240430 20:11:43.809970 18558 leader_election.cc:341] T 30a521916bda46678be20a066bd1d3f3 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:43.810081 18558 leader_election.cc:302] T 30a521916bda46678be20a066bd1d3f3 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:43.810379 18732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 975d145de1794f05b1dd844b4b86ce40. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:43.810523 20384 raft_consensus.cc:2726] T 30a521916bda46678be20a066bd1d3f3 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:43.811090 19489 ts_tablet_manager.cc:1875] T 20d166756819406ba85044f4fee9c4ce P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:43.815454 18600 tablet_service.cc:1467] Processing CreateTablet for tablet 975d145de1794f05b1dd844b4b86ce40 (DEFAULT_TABLE table=default.Table_2e5bcbca76d345fea6d7b7dac08a7afb [id=e2122b91429d4acc939e280d8261c452]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:43.817340 18457 tablet_service.cc:1467] Processing CreateTablet for tablet 107638df57ed43adb6e13d47997df541 (DEFAULT_TABLE table=default.Table_650b1b52ac7a453a95434b798c408f6e [id=68e19fd08f164090a84319c51b938b03]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:43.819773 18585 tablet_service.cc:1467] Processing CreateTablet for tablet 107638df57ed43adb6e13d47997df541 (DEFAULT_TABLE table=default.Table_650b1b52ac7a453a95434b798c408f6e [id=68e19fd08f164090a84319c51b938b03]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:43.820711 18448 tablet_service.cc:1467] Processing CreateTablet for tablet cefce2f2a392495bb71399ed75db938c (DEFAULT_TABLE table=default.Table_47217f7ee5ef4a50a2e9f146412b198a [id=3b5dcd10bb0d4f2493dcfaae7a2b9d19]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:43.821704 18726 tablet_service.cc:1467] Processing CreateTablet for tablet 107638df57ed43adb6e13d47997df541 (DEFAULT_TABLE table=default.Table_650b1b52ac7a453a95434b798c408f6e [id=68e19fd08f164090a84319c51b938b03]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:43.829324 18619 raft_consensus.cc:1226] T e2d90ff17e574f88b2061b1c117d4b7d P 5e1c6397f09747f091534faf93542710 [term 3 LEADER]: Rejecting Update request from peer 3c51c33d5616482cafb68e26688820f4 for earlier term 2. Current term is 3. Ops: []
I20240430 20:11:43.830121 20019 consensus_queue.cc:1046] T e2d90ff17e574f88b2061b1c117d4b7d P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240430 20:11:43.830685 20019 consensus_queue.cc:1225] T e2d90ff17e574f88b2061b1c117d4b7d P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Peer 5e1c6397f09747f091534faf93542710 log is divergent from this leader: its last log entry 3.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20240430 20:11:43.831106 20205 raft_consensus.cc:3032] T e2d90ff17e574f88b2061b1c117d4b7d P 3c51c33d5616482cafb68e26688820f4 [term 2 LEADER]: Stepping down as leader of term 2
I20240430 20:11:43.831189 20205 raft_consensus.cc:729] T e2d90ff17e574f88b2061b1c117d4b7d P 3c51c33d5616482cafb68e26688820f4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Running, Role: LEADER
I20240430 20:11:43.831470 20205 consensus_queue.cc:260] T e2d90ff17e574f88b2061b1c117d4b7d P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.3, Last appended by leader: 3, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:43.831777 20205 raft_consensus.cc:3037] T e2d90ff17e574f88b2061b1c117d4b7d P 3c51c33d5616482cafb68e26688820f4 [term 2 FOLLOWER]: Advancing to term 3
I20240430 20:11:43.836885 18742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4953578de5f45f3a6ba092a13357460. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:43.839277 18726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 107638df57ed43adb6e13d47997df541. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:43.841869 18983 tablet_bootstrap.cc:654] T 681cdd85ac6847a0a1aaded310456582 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:43.842211 19489 ts_tablet_manager.cc:1888] T 20d166756819406ba85044f4fee9c4ce P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:43.842344 19489 log.cc:1196] T 20d166756819406ba85044f4fee9c4ce P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/20d166756819406ba85044f4fee9c4ce
I20240430 20:11:43.842774 19489 ts_tablet_manager.cc:1909] T 20d166756819406ba85044f4fee9c4ce P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
W20240430 20:11:43.847962 19028 catalog_manager.cc:5676] Failed to send AlterTable RPC for tablet aa9998f4e17147aa8f2b35c01980463a (table default.Table_ffb3192f443d437ab63f56f628e596ad [id=608890834da843fa9af0cfc0d548029e], current schema version=2): Not found: failed to reset TS proxy: No leader found: Tablet aa9998f4e17147aa8f2b35c01980463a consensus state has no leader
I20240430 20:11:43.849668 18739 tablet_service.cc:1514] Processing DeleteTablet for tablet 44fa6917a85c46f3aaa363df46ea586b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:32 UTC) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:43.852224 18740 tablet_service.cc:1514] Processing DeleteTablet for tablet 87ac32a0ddb84676adc83038ef664591 with delete_type TABLET_DATA_DELETED (replaced by tablet 331c031d44854d679a27d10d9151d0ea) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:43.852471 18733 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e826459c3f4437da81c9e6e09f0c22f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:32 UTC) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:43.859279 18589 tablet_service.cc:1467] Processing CreateTablet for tablet cefce2f2a392495bb71399ed75db938c (DEFAULT_TABLE table=default.Table_47217f7ee5ef4a50a2e9f146412b198a [id=3b5dcd10bb0d4f2493dcfaae7a2b9d19]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:43.859537 18587 tablet_service.cc:1467] Processing CreateTablet for tablet b1277899f0c040cea7dc2b12f3fd68de (DEFAULT_TABLE table=default.Table_2582fc0f97c1470d8b6b21cd1240697e [id=fc059935f9c943389180fab316c33a05]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:43.862653 20476 tablet.cc:1722] T 372b135dd01f44f98cd713ca8857ba61 P 5e1c6397f09747f091534faf93542710: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240430 20:11:43.861111 18450 tablet_service.cc:1467] Processing CreateTablet for tablet b1277899f0c040cea7dc2b12f3fd68de (DEFAULT_TABLE table=default.Table_2582fc0f97c1470d8b6b21cd1240697e [id=fc059935f9c943389180fab316c33a05]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:43.860697 18985 tablet_bootstrap.cc:492] T 2e79df8744e74b5991a79ce978ff0202 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:43.863802 18985 ts_tablet_manager.cc:1367] T 2e79df8744e74b5991a79ce978ff0202 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.370s user 0.001s sys 0.041s
I20240430 20:11:43.864781 18985 raft_consensus.cc:348] T 2e79df8744e74b5991a79ce978ff0202 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
W20240430 20:11:43.868146 18557 consensus_peers.cc:485] T a59ef6c545dd41a28265a4c1b8ce10e0 P 5e1c6397f09747f091534faf93542710 -> Peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Couldn't send request to peer 3c51c33d5616482cafb68e26688820f4. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240430 20:11:43.868218 18740 tablet_service.cc:1467] Processing CreateTablet for tablet b1277899f0c040cea7dc2b12f3fd68de (DEFAULT_TABLE table=default.Table_2582fc0f97c1470d8b6b21cd1240697e [id=fc059935f9c943389180fab316c33a05]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:43.868634 18740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1277899f0c040cea7dc2b12f3fd68de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:43.869081 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 145) took 11194 ms. Trace:
I20240430 20:11:43.869225 19489 rpcz_store.cc:276] 0430 20:11:32.674509 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:32.675942 (+ 1433us) service_pool.cc:225] Handling call
0430 20:11:43.773456 (+11097514us) ts_tablet_manager.cc:1120] Deleting tablet 20d166756819406ba85044f4fee9c4ce
0430 20:11:43.773467 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:43.836893 (+ 63426us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:43.842089 (+ 5196us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:43.869058 (+ 26969us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":11066653}
I20240430 20:11:43.869473 18733 tablet_service.cc:1467] Processing CreateTablet for tablet cefce2f2a392495bb71399ed75db938c (DEFAULT_TABLE table=default.Table_47217f7ee5ef4a50a2e9f146412b198a [id=3b5dcd10bb0d4f2493dcfaae7a2b9d19]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:43.869887 18733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cefce2f2a392495bb71399ed75db938c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:43.869859 19001 catalog_manager.cc:4830] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): tablet 20d166756819406ba85044f4fee9c4ce (table default.Table_b331bf8aeaae4911ac01ed3fbd2627bd [id=1bf5c29b08134670984b6569db298129]) successfully deleted
I20240430 20:11:43.870270 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 147) took 11195 ms. Trace:
I20240430 20:11:43.870343 19489 rpcz_store.cc:276] 0430 20:11:32.674702 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:32.676459 (+ 1757us) service_pool.cc:225] Handling call
0430 20:11:43.870154 (+11193695us) ts_tablet_manager.cc:1120] Deleting tablet eb76f60448774fe1967fffe7c6873cfc
0430 20:11:43.870163 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:43.870255 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":11160137}
W20240430 20:11:43.870932 19001 catalog_manager.cc:4810] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): delete failed for tablet eb76f60448774fe1967fffe7c6873cfc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet eb76f60448774fe1967fffe7c6873cfc already in progress: creating tablet
I20240430 20:11:43.873566 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 133) took 11200 ms. Trace:
I20240430 20:11:43.873659 19489 rpcz_store.cc:276] 0430 20:11:32.673281 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:32.675912 (+ 2631us) service_pool.cc:225] Handling call
0430 20:11:43.871351 (+11195439us) ts_tablet_manager.cc:1120] Deleting tablet 115c4a12440647a4bb46eb6824f7ea98
0430 20:11:43.871362 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:43.873548 (+ 2186us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":11159594}
W20240430 20:11:43.874146 19001 catalog_manager.cc:4810] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): delete failed for tablet 115c4a12440647a4bb46eb6824f7ea98 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 115c4a12440647a4bb46eb6824f7ea98 already in progress: creating tablet
I20240430 20:11:43.882858 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 135) took 11209 ms. Trace:
I20240430 20:11:43.883080 19489 rpcz_store.cc:276] 0430 20:11:32.673507 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:32.675965 (+ 2458us) service_pool.cc:225] Handling call
0430 20:11:43.877292 (+11201327us) ts_tablet_manager.cc:1120] Deleting tablet d9e578c99058424a9b86390641f9c34f
0430 20:11:43.877304 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:43.882823 (+ 5519us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":11164335}
W20240430 20:11:43.883733 19001 catalog_manager.cc:4797] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): delete failed for tablet d9e578c99058424a9b86390641f9c34f because the tablet was not found. No further retry: Not found: Tablet not found: d9e578c99058424a9b86390641f9c34f
I20240430 20:11:43.884341 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 149) took 11209 ms. Trace:
I20240430 20:11:43.884505 19489 rpcz_store.cc:276] 0430 20:11:32.674879 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:32.676506 (+ 1627us) service_pool.cc:225] Handling call
0430 20:11:43.884200 (+11207694us) ts_tablet_manager.cc:1120] Deleting tablet 2873af6da928463f9809864b8eaf0c4a
0430 20:11:43.884213 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:43.884319 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":11169430}
W20240430 20:11:43.885166 19001 catalog_manager.cc:4810] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): delete failed for tablet 2873af6da928463f9809864b8eaf0c4a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2873af6da928463f9809864b8eaf0c4a already in progress: creating tablet
I20240430 20:11:43.885613 19489 tablet_replica.cc:330] T 1555924b5f1444398f12ee82c3cb5b13 P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:43.886911 18985 raft_consensus.cc:374] T 2e79df8744e74b5991a79ce978ff0202 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:43.887096 18985 raft_consensus.cc:729] T 2e79df8744e74b5991a79ce978ff0202 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:43.888092 18985 consensus_queue.cc:260] T 2e79df8744e74b5991a79ce978ff0202 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:43.889359 18988 tablet_bootstrap.cc:492] T ebf74809b52f47f3b0ed21b6ac0f541a P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:43.889571 18988 ts_tablet_manager.cc:1367] T ebf74809b52f47f3b0ed21b6ac0f541a P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.124s user 0.011s sys 0.005s
I20240430 20:11:43.890259 19489 raft_consensus.cc:2229] T 1555924b5f1444398f12ee82c3cb5b13 P 3c51c33d5616482cafb68e26688820f4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:43.891084 19489 raft_consensus.cc:2258] T 1555924b5f1444398f12ee82c3cb5b13 P 3c51c33d5616482cafb68e26688820f4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:43.892488 20384 raft_consensus.cc:482] T 50486644ec234933b9a0b45ee39a6685 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:43.892693 20384 raft_consensus.cc:504] T 50486644ec234933b9a0b45ee39a6685 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:43.893424 20384 leader_election.cc:288] T 50486644ec234933b9a0b45ee39a6685 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:43.894260 18737 tablet_service.cc:1514] Processing DeleteTablet for tablet 2873af6da928463f9809864b8eaf0c4a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:32 UTC) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:43.895563 18600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 975d145de1794f05b1dd844b4b86ce40. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:43.898732 18592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4953578de5f45f3a6ba092a13357460. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:43.894884 18482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50486644ec234933b9a0b45ee39a6685" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:43.900701 18482 raft_consensus.cc:2454] T 50486644ec234933b9a0b45ee39a6685 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 0.
I20240430 20:11:43.901207 18558 leader_election.cc:302] T 50486644ec234933b9a0b45ee39a6685 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:43.901477 20384 raft_consensus.cc:2781] T 50486644ec234933b9a0b45ee39a6685 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:43.901551 20384 raft_consensus.cc:482] T 50486644ec234933b9a0b45ee39a6685 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:43.901619 20384 raft_consensus.cc:3037] T 50486644ec234933b9a0b45ee39a6685 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:43.902467 18585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 107638df57ed43adb6e13d47997df541. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:43.904311 20384 raft_consensus.cc:504] T 50486644ec234933b9a0b45ee39a6685 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:43.905627 18586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a558d18cdfa4b33b4f72dc9041bfd53. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:43.908819 18587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1277899f0c040cea7dc2b12f3fd68de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:43.910722 18603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbdbb8a3a4b444f58a2761466577a88f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:43.910995 18589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cefce2f2a392495bb71399ed75db938c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:43.911808 18482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50486644ec234933b9a0b45ee39a6685" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:43.912027 18482 raft_consensus.cc:3037] T 50486644ec234933b9a0b45ee39a6685 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:43.914721 18602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad01c979b2f7439bab83137f87f1405c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:43.918970 18590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b95098d3af04dd5a9ae3972e44b3a6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:43.920378 18482 raft_consensus.cc:2454] T 50486644ec234933b9a0b45ee39a6685 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
I20240430 20:11:43.921025 18558 leader_election.cc:302] T 50486644ec234933b9a0b45ee39a6685 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:43.922210 20395 raft_consensus.cc:2781] T 50486644ec234933b9a0b45ee39a6685 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:43.922336 20395 raft_consensus.cc:686] T 50486644ec234933b9a0b45ee39a6685 P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Becoming Leader. State: Replica: 5e1c6397f09747f091534faf93542710, State: Running, Role: LEADER
I20240430 20:11:43.922470 18595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd423a49fb694695a8de7ecd54330127. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:43.922566 20395 consensus_queue.cc:237] T 50486644ec234933b9a0b45ee39a6685 P 5e1c6397f09747f091534faf93542710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:43.067013 19980 rpcz_store.cc:276] 0430 20:11:38.415699 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:38.416626 (+ 927us) service_pool.cc:225] Handling call
0430 20:11:41.329990 (+2913364us) raft_consensus.cc:1457] Updating replica for 191 ops
0430 20:11:41.896335 (+566345us) raft_consensus.cc:1520] Early marking committed up to index 0
0430 20:11:41.896343 (+ 8us) raft_consensus.cc:1525] Triggering prepare for 191 ops
0430 20:11:41.898121 (+ 1778us) write_op.cc:270] Start()
0430 20:11:41.899051 (+ 930us) write_op.cc:276] Timestamp: P: 1714507889107665 usec, L: 0
0430 20:11:41.904376 (+ 5325us) write_op.cc:270] Start()
0430 20:11:41.904433 (+ 57us) write_op.cc:276] Timestamp: P: 1714507889109205 usec, L: 0
0430 20:11:41.904640 (+ 207us) write_op.cc:270] Start()
0430 20:11:41.904679 (+ 39us) write_op.cc:276] Timestamp: P: 1714507889110794 usec, L: 0
0430 20:11:41.904862 (+ 183us) write_op.cc:270] Start()
0430 20:11:41.904882 (+ 20us) write_op.cc:276] Timestamp: P: 1714507889111323 usec, L: 0
0430 20:11:41.905021 (+ 139us) write_op.cc:270] Start()
0430 20:11:41.905046 (+ 25us) write_op.cc:276] Timestamp: P: 1714507889112257 usec, L: 0
0430 20:11:41.905178 (+ 132us) write_op.cc:270] Start()
0430 20:11:41.905196 (+ 18us) write_op.cc:276] Timestamp: P: 1714507889112829 usec, L: 0
0430 20:11:41.908267 (+ 3071us) write_op.cc:270] Start()
0430 20:11:41.908326 (+ 59us) write_op.cc:276] Timestamp: P: 1714507889115208 usec, L: 0
0430 20:11:41.908495 (+ 169us) write_op.cc:270] Start()
0430 20:11:41.908516 (+ 21us) write_op.cc:276] Timestamp: P: 1714507889125201 usec, L: 0
0430 20:11:41.908640 (+ 124us) write_op.cc:270] Start()
0430 20:11:41.908659 (+ 19us) write_op.cc:276] Timestamp: P: 1714507889304200 usec, L: 0
0430 20:11:41.908812 (+ 153us) write_op.cc:270] Start()
0430 20:11:41.908832 (+ 20us) write_op.cc:276] Timestamp: P: 1714507889305207 usec, L: 0
0430 20:11:41.908975 (+ 143us) write_op.cc:270] Start()
0430 20:11:41.908995 (+ 20us) write_op.cc:276] Timestamp: P: 1714507889305862 usec, L: 0
0430 20:11:41.917533 (+ 8538us) write_op.cc:270] Start()
0430 20:11:41.918109 (+ 576us) write_op.cc:276] Timestamp: P: 1714507889306653 usec, L: 0
0430 20:11:41.918388 (+ 279us) write_op.cc:270] Start()
0430 20:11:41.918428 (+ 40us) write_op.cc:276] Timestamp: P: 1714507889322291 usec, L: 0
0430 20:11:41.918613 (+ 185us) write_op.cc:270] Start()
0430 20:11:41.918639 (+ 26us) write_op.cc:276] Timestamp: P: 1714507889325349 usec, L: 0
0430 20:11:41.918777 (+ 138us) write_op.cc:270] Start()
0430 20:11:41.918804 (+ 27us) write_op.cc:276] Timestamp: P: 1714507889326483 usec, L: 0
0430 20:11:41.918962 (+ 158us) write_op.cc:270] Start()
0430 20:11:41.918986 (+ 24us) write_op.cc:276] Timestamp: P: 1714507889327550 usec, L: 0
0430 20:11:41.919136 (+ 150us) write_op.cc:270] Start()
0430 20:11:41.919174 (+ 38us) write_op.cc:276] Timestamp: P: 1714507889328556 usec, L: 0
0430 20:11:41.919346 (+ 172us) write_op.cc:270] Start()
0430 20:11:41.919369 (+ 23us) write_op.cc:276] Timestamp: P: 1714507889434212 usec, L: 0
0430 20:11:41.919516 (+ 147us) write_op.cc:270] Start()
0430 20:11:41.919536 (+ 20us) write_op.cc:276] Timestamp: P: 1714507889552254 usec, L: 0
0430 20:11:41.919645 (+ 109us) write_op.cc:270] Start()
0430 20:11:41.919660 (+ 15us) write_op.cc:276] Timestamp: P: 1714507889552779 usec, L: 0
0430 20:11:41.919771 (+ 111us) write_op.cc:270] Start()
0430 20:11:41.919786 (+ 15us) write_op.cc:276] Timestamp: P: 1714507889553266 usec, L: 0
0430 20:11:41.919906 (+ 120us) write_op.cc:270] Start()
0430 20:11:41.919927 (+ 21us) write_op.cc:276] Timestamp: P: 1714507889595525 usec, L: 0
0430 20:11:41.920089 (+ 162us) write_op.cc:270] Start()
0430 20:11:41.920112 (+ 23us) write_op.cc:276] Timestamp: P: 1714507889649132 usec, L: 0
0430 20:11:41.920248 (+ 136us) write_op.cc:270] Start()
0430 20:11:41.920272 (+ 24us) write_op.cc:276] Timestamp: P: 1714507889654324 usec, L: 0
0430 20:11:41.920471 (+ 199us) write_op.cc:270] Start()
0430 20:11:41.920498 (+ 27us) write_op.cc:276] Timestamp: P: 1714507889707775 usec, L: 0
0430 20:11:41.920651 (+ 153us) write_op.cc:270] Start()
0430 20:11:41.920675 (+ 24us) write_op.cc:276] Timestamp: P: 1714507889731960 usec, L: 0
0430 20:11:41.920810 (+ 135us) write_op.cc:270] Start()
0430 20:11:41.920833 (+ 23us) write_op.cc:276] Timestamp: P: 1714507889752398 usec, L: 0
0430 20:11:41.920969 (+ 136us) write_op.cc:270] Start()
0430 20:11:41.920985 (+ 16us) write_op.cc:276] Timestamp: P: 1714507889777263 usec, L: 0
0430 20:11:41.921097 (+ 112us) write_op.cc:270] Start()
0430 20:11:41.921118 (+ 21us) write_op.cc:276] Timestamp: P: 1714507889796813 usec, L: 0
0430 20:11:41.921233 (+ 115us) write_op.cc:270] Start()
0430 20:11:41.921252 (+ 19us) write_op.cc:276] Timestamp: P: 1714507889802117 usec, L: 0
0430 20:11:41.921371 (+ 119us) write_op.cc:270] Start()
0430 20:11:41.921389 (+ 18us) write_op.cc:276] Timestamp: P: 1714507889817862 usec, L: 0
0430 20:11:41.921517 (+ 128us) write_op.cc:270] Start()
0430 20:11:41.921537 (+ 20us) write_op.cc:276] Timestamp: P: 1714507889897225 usec, L: 0
0430 20:11:41.921707 (+ 170us) write_op.cc:270] Start()
0430 20:11:41.921728 (+ 21us) write_op.cc:276] Timestamp: P: 1714507889921187 usec, L: 0
0430 20:11:41.921878 (+ 150us) write_op.cc:270] Start()
0430 20:11:41.921905 (+ 27us) write_op.cc:276] Timestamp: P: 1714507889990898 usec, L: 0
0430 20:11:41.922095 (+ 190us) write_op.cc:270] Start()
0430 20:11:41.922144 (+ 49us) write_op.cc:276] Timestamp: P: 1714507889993344 usec, L: 0
0430 20:11:41.922312 (+ 168us) write_op.cc:270] Start()
0430 20:11:41.922338 (+ 26us) write_op.cc:276] Timestamp: P: 1714507890243865 usec, L: 0
0430 20:11:41.922497 (+ 159us) write_op.cc:270] Start()
0430 20:11:41.922543 (+ 46us) write_op.cc:276] Timestamp: P: 1714507890307733 usec, L: 0
0430 20:11:42.040062 (+117519us) write_op.cc:270] Start()
0430 20:11:42.040102 (+ 40us) write_op.cc:276] Timestamp: P: 1714507890362883 usec, L: 0
0430 20:11:42.040257 (+ 155us) write_op.cc:270] Start()
0430 20:11:42.040277 (+ 20us) write_op.cc:276] Timestamp: P: 1714507890407379 usec, L: 0
0430 20:11:42.040433 (+ 156us) write_op.cc:270] Start()
0430 20:11:42.040462 (+ 29us) write_op.cc:276] Timestamp: P: 1714507890410920 usec, L: 0
0430 20:11:42.040585 (+ 123us) write_op.cc:270] Start()
0430 20:11:42.040602 (+ 17us) write_op.cc:276] Timestamp: P: 1714507890551822 usec, L: 0
0430 20:11:42.040714 (+ 112us) write_op.cc:270] Start()
0430 20:11:42.040730 (+ 16us) write_op.cc:276] Timestamp: P: 1714507890763437 usec, L: 0
0430 20:11:42.040830 (+ 100us) write_op.cc:270] Start()
0430 20:11:42.040847 (+ 17us) write_op.cc:276] Timestamp: P: 1714507890830989 usec, L: 0
0430 20:11:42.040953 (+ 106us) write_op.cc:270] Start()
0430 20:11:42.040970 (+ 17us) write_op.cc:276] Timestamp: P: 1714507890831770 usec, L: 0
0430 20:11:42.041080 (+ 110us) write_op.cc:270] Start()
0430 20:11:42.041097 (+ 17us) write_op.cc:276] Timestamp: P: 1714507890840144 usec, L: 0
0430 20:11:42.041208 (+ 111us) write_op.cc:270] Start()
0430 20:11:42.041225 (+ 17us) write_op.cc:276] Timestamp: P: 1714507890841302 usec, L: 0
0430 20:11:42.041326 (+ 101us) write_op.cc:270] Start()
0430 20:11:42.041341 (+ 15us) write_op.cc:276] Timestamp: P: 1714507890879967 usec, L: 0
0430 20:11:42.041446 (+ 105us) write_op.cc:270] Start()
0430 20:11:42.041461 (+ 15us) write_op.cc:276] Timestamp: P: 1714507891146668 usec, L: 0
0430 20:11:42.041611 (+ 150us) write_op.cc:270] Start()
0430 20:11:42.041629 (+ 18us) write_op.cc:276] Timestamp: P: 1714507891147239 usec, L: 0
0430 20:11:42.041807 (+ 178us) write_op.cc:270] Start()
0430 20:11:42.041824 (+ 17us) write_op.cc:276] Timestamp: P: 1714507891147841 usec, L: 0
0430 20:11:42.041932 (+ 108us) write_op.cc:270] Start()
0430 20:11:42.041950 (+ 18us) write_op.cc:276] Timestamp: P: 1714507891159741 usec, L: 0
0430 20:11:42.042051 (+ 101us) write_op.cc:270] Start()
0430 20:11:42.042072 (+ 21us) write_op.cc:276] Timestamp: P: 1714507891166971 usec, L: 0
0430 20:11:42.042198 (+ 126us) write_op.cc:270] Start()
0430 20:11:42.042218 (+ 20us) write_op.cc:276] Timestamp: P: 1714507891167635 usec, L: 0
0430 20:11:42.071937 (+ 29719us) write_op.cc:270] Start()
0430 20:11:42.071984 (+ 47us) write_op.cc:276] Timestamp: P: 1714507891200720 usec, L: 0
0430 20:11:42.072187 (+ 203us) write_op.cc:270] Start()
0430 20:11:42.072209 (+ 22us) write_op.cc:276] Timestamp: P: 1714507891203534 usec, L: 0
0430 20:11:42.072359 (+ 150us) write_op.cc:270] Start()
0430 20:11:42.072382 (+ 23us) write_op.cc:276] Timestamp: P: 1714507891212192 usec, L: 0
0430 20:11:42.072549 (+ 167us) write_op.cc:270] Start()
0430 20:11:42.072569 (+ 20us) write_op.cc:276] Timestamp: P: 1714507891221460 usec, L: 0
0430 20:11:42.072677 (+ 108us) write_op.cc:270] Start()
0430 20:11:42.072694 (+ 17us) write_op.cc:276] Timestamp: P: 1714507891274679 usec, L: 0
0430 20:11:42.072797 (+ 103us) write_op.cc:270] Start()
0430 20:11:42.072813 (+ 16us) write_op.cc:276] Timestamp: P: 1714507891277910 usec, L: 0
0430 20:11:42.072920 (+ 107us) write_op.cc:270] Start()
0430 20:11:42.072958 (+ 38us) write_op.cc:276] Timestamp: P: 1714507891284724 usec, L: 0
0430 20:11:42.073081 (+ 123us) write_op.cc:270] Start()
0430 20:11:42.073098 (+ 17us) write_op.cc:276] Timestamp: P: 1714507891296163 usec, L: 0
0430 20:11:42.073206 (+ 108us) write_op.cc:270] Start()
0430 20:11:42.073221 (+ 15us) write_op.cc:276] Timestamp: P: 1714507891312308 usec, L: 0
0430 20:11:42.073308 (+ 87us) write_op.cc:270] Start()
0430 20:11:42.073324 (+ 16us) write_op.cc:276] Timestamp: P: 1714507891316218 usec, L: 0
0430 20:11:42.073477 (+ 153us) write_op.cc:270] Start()
0430 20:11:42.073495 (+ 18us) write_op.cc:276] Timestamp: P: 1714507891317209 usec, L: 0
0430 20:11:42.073661 (+ 166us) write_op.cc:270] Start()
0430 20:11:42.073681 (+ 20us) write_op.cc:276] Timestamp: P: 1714507891339480 usec, L: 0
0430 20:11:42.073804 (+ 123us) write_op.cc:270] Start()
0430 20:11:42.073822 (+ 18us) write_op.cc:276] Timestamp: P: 1714507891347888 usec, L: 0
0430 20:11:42.073940 (+ 118us) write_op.cc:270] Start()
0430 20:11:42.073960 (+ 20us) write_op.cc:276] Timestamp: P: 1714507891386662 usec, L: 0
0430 20:11:42.074080 (+ 120us) write_op.cc:270] Start()
0430 20:11:42.074103 (+ 23us) write_op.cc:276] Timestamp: P: 1714507891399603 usec, L: 0
0430 20:11:42.074290 (+ 187us) write_op.cc:270] Start()
0430 20:11:42.074319 (+ 29us) write_op.cc:276] Timestamp: P: 1714507891473959 usec, L: 0
0430 20:11:42.074450 (+ 131us) write_op.cc:270] Start()
0430 20:11:42.074468 (+ 18us) write_op.cc:276] Timestamp: P: 1714507891599781 usec, L: 0
0430 20:11:42.117168 (+ 42700us) write_op.cc:270] Start()
0430 20:11:42.117231 (+ 63us) write_op.cc:276] Timestamp: P: 1714507891605794 usec, L: 0
0430 20:11:42.117549 (+ 318us) write_op.cc:270] Start()
0430 20:11:42.117579 (+ 30us) write_op.cc:276] Timestamp: P: 1714507891624255 usec, L: 0
0430 20:11:42.117730 (+ 151us) write_op.cc:270] Start()
0430 20:11:42.117754 (+ 24us) write_op.cc:276] Timestamp: P: 1714507891636684 usec, L: 0
0430 20:11:42.117879 (+ 125us) write_op.cc:270] Start()
0430 20:11:42.117903 (+ 24us) write_op.cc:276] Timestamp: P: 1714507891638990 usec, L: 0
0430 20:11:42.118072 (+ 169us) write_op.cc:270] Start()
0430 20:11:42.118101 (+ 29us) write_op.cc:276] Timestamp: P: 1714507891659744 usec, L: 0
0430 20:11:42.118231 (+ 130us) write_op.cc:270] Start()
0430 20:11:42.118252 (+ 21us) write_op.cc:276] Timestamp: P: 1714507891668300 usec, L: 0
0430 20:11:42.118385 (+ 133us) write_op.cc:270] Start()
0430 20:11:42.118410 (+ 25us) write_op.cc:276] Timestamp: P: 1714507891740799 usec, L: 0
0430 20:11:42.118563 (+ 153us) write_op.cc:270] Start()
0430 20:11:42.118581 (+ 18us) write_op.cc:276] Timestamp: P: 1714507891912142 usec, L: 0
0430 20:11:42.138420 (+ 19839us) write_op.cc:270] Start()
0430 20:11:42.138474 (+ 54us) write_op.cc:276] Timestamp: P: 1714507891925010 usec, L: 0
0430 20:11:42.142512 (+ 4038us) write_op.cc:270] Start()
0430 20:11:42.143034 (+ 522us) write_op.cc:276] Timestamp: P: 1714507891956771 usec, L: 0
0430 20:11:42.144154 (+ 1120us) write_op.cc:270] Start()
0430 20:11:42.144192 (+ 38us) write_op.cc:276] Timestamp: P: 1714507891991382 usec, L: 0
0430 20:11:42.144352 (+ 160us) write_op.cc:270] Start()
0430 20:11:42.144382 (+ 30us) write_op.cc:276] Timestamp: P: 1714507892046709 usec, L: 0
0430 20:11:42.144522 (+ 140us) write_op.cc:270] Start()
0430 20:11:42.144542 (+ 20us) write_op.cc:276] Timestamp: P: 1714507892058410 usec, L: 0
0430 20:11:42.144692 (+ 150us) write_op.cc:270] Start()
0430 20:11:42.144712 (+ 20us) write_op.cc:276] Timestamp: P: 1714507892073804 usec, L: 0
0430 20:11:42.144876 (+ 164us) write_op.cc:270] Start()
0430 20:11:42.144896 (+ 20us) write_op.cc:276] Timestamp: P: 1714507892105599 usec, L: 0
0430 20:11:42.145031 (+ 135us) write_op.cc:270] Start()
0430 20:11:42.145050 (+ 19us) write_op.cc:276] Timestamp: P: 1714507892106275 usec, L: 0
0430 20:11:42.145167 (+ 117us) write_op.cc:270] Start()
0430 20:11:42.145186 (+ 19us) write_op.cc:276] Timestamp: P: 1714507892181418 usec, L: 0
0430 20:11:42.145311 (+ 125us) write_op.cc:270] Start()
0430 20:11:42.145330 (+ 19us) write_op.cc:276] Timestamp: P: 1714507892188041 usec, L: 0
0430 20:11:42.145470 (+ 140us) write_op.cc:270] Start()
0430 20:11:42.145489 (+ 19us) write_op.cc:276] Timestamp: P: 1714507892190588 usec, L: 0
0430 20:11:42.156921 (+ 11432us) write_op.cc:270] Start()
0430 20:11:42.156959 (+ 38us) write_op.cc:276] Timestamp: P: 1714507892191986 usec, L: 0
0430 20:11:42.157141 (+ 182us) write_op.cc:270] Start()
0430 20:11:42.157163 (+ 22us) write_op.cc:276] Timestamp: P: 1714507892193125 usec, L: 0
0430 20:11:42.157282 (+ 119us) write_op.cc:270] Start()
0430 20:11:42.157301 (+ 19us) write_op.cc:276] Timestamp: P: 1714507892233202 usec, L: 0
0430 20:11:42.157449 (+ 148us) write_op.cc:270] Start()
0430 20:11:42.157468 (+ 19us) write_op.cc:276] Timestamp: P: 1714507892251617 usec, L: 0
0430 20:11:42.161216 (+ 3748us) write_op.cc:270] Start()
0430 20:11:42.161242 (+ 26us) write_op.cc:276] Timestamp: P: 1714507892252301 usec, L: 0
0430 20:11:42.161408 (+ 166us) write_op.cc:270] Start()
0430 20:11:42.161429 (+ 21us) write_op.cc:276] Timestamp: P: 1714507892253027 usec, L: 0
0430 20:11:42.161631 (+ 202us) write_op.cc:270] Start()
0430 20:11:42.161656 (+ 25us) write_op.cc:276] Timestamp: P: 1714507892253659 usec, L: 0
0430 20:11:42.161809 (+ 153us) write_op.cc:270] Start()
0430 20:11:42.161830 (+ 21us) write_op.cc:276] Timestamp: P: 1714507892258075 usec, L: 0
0430 20:11:42.161965 (+ 135us) write_op.cc:270] Start()
0430 20:11:42.161987 (+ 22us) write_op.cc:276] Timestamp: P: 1714507892289729 usec, L: 0
0430 20:11:42.162111 (+ 124us) write_op.cc:270] Start()
0430 20:11:42.162130 (+ 19us) write_op.cc:276] Timestamp: P: 1714507892326718 usec, L: 0
0430 20:11:42.162330 (+ 200us) write_op.cc:270] Start()
0430 20:11:42.162352 (+ 22us) write_op.cc:276] Timestamp: P: 1714507892327436 usec, L: 0
0430 20:11:42.162497 (+ 145us) write_op.cc:270] Start()
0430 20:11:42.162516 (+ 19us) write_op.cc:276] Timestamp: P: 1714507892338973 usec, L: 0
0430 20:11:42.193208 (+ 30692us) write_op.cc:270] Start()
0430 20:11:42.298667 (+105459us) write_op.cc:276] Timestamp: P: 1714507892339592 usec, L: 0
0430 20:11:42.298971 (+ 304us) write_op.cc:270] Start()
0430 20:11:42.299011 (+ 40us) write_op.cc:276] Timestamp: P: 1714507892340118 usec, L: 0
0430 20:11:42.299219 (+ 208us) write_op.cc:270] Start()
0430 20:11:42.299244 (+ 25us) write_op.cc:276] Timestamp: P: 1714507892344026 usec, L: 0
0430 20:11:42.299562 (+ 318us) write_op.cc:270] Start()
0430 20:11:42.299585 (+ 23us) write_op.cc:276] Timestamp: P: 1714507892372295 usec, L: 0
0430 20:11:42.299711 (+ 126us) write_op.cc:270] Start()
0430 20:11:42.299727 (+ 16us) write_op.cc:276] Timestamp: P: 1714507892372897 usec, L: 0
0430 20:11:42.299857 (+ 130us) write_op.cc:270] Start()
0430 20:11:42.299878 (+ 21us) write_op.cc:276] Timestamp: P: 1714507892404394 usec, L: 0
0430 20:11:42.299994 (+ 116us) write_op.cc:270] Start()
0430 20:11:42.300018 (+ 24us) write_op.cc:276] Timestamp: P: 1714507892448455 usec, L: 0
0430 20:11:42.300126 (+ 108us) write_op.cc:270] Start()
0430 20:11:42.300142 (+ 16us) write_op.cc:276] Timestamp: P: 1714507892449382 usec, L: 0
0430 20:11:42.300237 (+ 95us) write_op.cc:270] Start()
0430 20:11:42.300251 (+ 14us) write_op.cc:276] Timestamp: P: 1714507892464141 usec, L: 0
0430 20:11:42.300339 (+ 88us) write_op.cc:270] Start()
0430 20:11:42.300353 (+ 14us) write_op.cc:276] Timestamp: P: 1714507892464885 usec, L: 0
0430 20:11:42.300454 (+ 101us) write_op.cc:270] Start()
0430 20:11:42.300528 (+ 74us) write_op.cc:276] Timestamp: P: 1714507892465455 usec, L: 0
0430 20:11:42.300648 (+ 120us) write_op.cc:270] Start()
0430 20:11:42.300662 (+ 14us) write_op.cc:276] Timestamp: P: 1714507892497099 usec, L: 0
0430 20:11:42.300802 (+ 140us) write_op.cc:270] Start()
0430 20:11:42.300823 (+ 21us) write_op.cc:276] Timestamp: P: 1714507892502081 usec, L: 0
0430 20:11:42.300964 (+ 141us) write_op.cc:270] Start()
0430 20:11:42.300986 (+ 22us) write_op.cc:276] Timestamp: P: 1714507892526910 usec, L: 0
0430 20:11:42.301927 (+ 941us) write_op.cc:270] Start()
0430 20:11:42.301955 (+ 28us) write_op.cc:276] Timestamp: P: 1714507892552427 usec, L: 0
0430 20:11:42.302100 (+ 145us) write_op.cc:270] Start()
0430 20:11:42.302117 (+ 17us) write_op.cc:276] Timestamp: P: 1714507892563765 usec, L: 0
0430 20:11:42.302234 (+ 117us) write_op.cc:270] Start()
0430 20:11:42.302248 (+ 14us) write_op.cc:276] Timestamp: P: 1714507892572034 usec, L: 0
0430 20:11:42.302347 (+ 99us) write_op.cc:270] Start()
0430 20:11:42.302361 (+ 14us) write_op.cc:276] Timestamp: P: 1714507892582537 usec, L: 0
0430 20:11:42.302463 (+ 102us) write_op.cc:270] Start()
0430 20:11:42.302478 (+ 15us) write_op.cc:276] Timestamp: P: 1714507892592987 usec, L: 0
0430 20:11:42.302611 (+ 133us) write_op.cc:270] Start()
0430 20:11:42.302629 (+ 18us) write_op.cc:276] Timestamp: P: 1714507892602685 usec, L: 0
0430 20:11:42.358357 (+ 55728us) write_op.cc:270] Start()
0430 20:11:42.358392 (+ 35us) write_op.cc:276] Timestamp: P: 1714507892610459 usec, L: 0
0430 20:11:42.358686 (+ 294us) write_op.cc:270] Start()
0430 20:11:42.358715 (+ 29us) write_op.cc:276] Timestamp: P: 1714507892617541 usec, L: 0
0430 20:11:42.358889 (+ 174us) write_op.cc:270] Start()
0430 20:11:42.358910 (+ 21us) write_op.cc:276] Timestamp: P: 1714507892629113 usec, L: 0
0430 20:11:42.359047 (+ 137us) write_op.cc:270] Start()
0430 20:11:42.359068 (+ 21us) write_op.cc:276] Timestamp: P: 1714507892647575 usec, L: 0
0430 20:11:42.359211 (+ 143us) write_op.cc:270] Start()
0430 20:11:42.359230 (+ 19us) write_op.cc:276] Timestamp: P: 1714507892649401 usec, L: 0
0430 20:11:42.359351 (+ 121us) write_op.cc:270] Start()
0430 20:11:42.359368 (+ 17us) write_op.cc:276] Timestamp: P: 1714507892653209 usec, L: 0
0430 20:11:42.359503 (+ 135us) write_op.cc:270] Start()
0430 20:11:42.359527 (+ 24us) write_op.cc:276] Timestamp: P: 1714507892672953 usec, L: 0
0430 20:11:42.359719 (+ 192us) write_op.cc:270] Start()
0430 20:11:42.359743 (+ 24us) write_op.cc:276] Timestamp: P: 1714507892680629 usec, L: 0
0430 20:11:42.359900 (+ 157us) write_op.cc:270] Start()
0430 20:11:42.359919 (+ 19us) write_op.cc:276] Timestamp: P: 1714507892681368 usec, L: 0
0430 20:11:42.360057 (+ 138us) write_op.cc:270] Start()
0430 20:11:42.360098 (+ 41us) write_op.cc:276] Timestamp: P: 1714507892685531 usec, L: 0
0430 20:11:42.360230 (+ 132us) write_op.cc:270] Start()
0430 20:11:42.360248 (+ 18us) write_op.cc:276] Timestamp: P: 1714507892686493 usec, L: 0
0430 20:11:42.360368 (+ 120us) write_op.cc:270] Start()
0430 20:11:42.360389 (+ 21us) write_op.cc:276] Timestamp: P: 1714507892700243 usec, L: 0
0430 20:11:42.360524 (+ 135us) write_op.cc:270] Start()
0430 20:11:42.360544 (+ 20us) write_op.cc:276] Timestamp: P: 1714507892702798 usec, L: 0
0430 20:11:42.360680 (+ 136us) write_op.cc:270] Start()
0430 20:11:42.360701 (+ 21us) write_op.cc:276] Timestamp: P: 1714507892748303 usec, L: 0
0430 20:11:42.360837 (+ 136us) write_op.cc:270] Start()
0430 20:11:42.360856 (+ 19us) write_op.cc:276] Timestamp: P: 1714507892823227 usec, L: 0
0430 20:11:42.360992 (+ 136us) write_op.cc:270] Start()
0430 20:11:42.361014 (+ 22us) write_op.cc:276] Timestamp: P: 1714507892851808 usec, L: 0
0430 20:11:42.361143 (+ 129us) write_op.cc:270] Start()
0430 20:11:42.361164 (+ 21us) write_op.cc:276] Timestamp: P: 1714507892999385 usec, L: 0
0430 20:11:42.361299 (+ 135us) write_op.cc:270] Start()
0430 20:11:42.361323 (+ 24us) write_op.cc:276] Timestamp: P: 1714507893000299 usec, L: 0
0430 20:11:42.361463 (+ 140us) write_op.cc:270] Start()
0430 20:11:42.361484 (+ 21us) write_op.cc:276] Timestamp: P: 1714507893003920 usec, L: 0
0430 20:11:42.361606 (+ 122us) write_op.cc:270] Start()
0430 20:11:42.361627 (+ 21us) write_op.cc:276] Timestamp: P: 1714507893005326 usec, L: 0
0430 20:11:42.361743 (+ 116us) write_op.cc:270] Start()
0430 20:11:42.361761 (+ 18us) write_op.cc:276] Timestamp: P: 1714507893007753 usec, L: 0
0430 20:11:42.361929 (+ 168us) write_op.cc:270] Start()
0430 20:11:42.361950 (+ 21us) write_op.cc:276] Timestamp: P: 1714507893009512 usec, L: 0
0430 20:11:42.362083 (+ 133us) write_op.cc:270] Start()
0430 20:11:42.362132 (+ 49us) write_op.cc:276] Timestamp: P: 1714507893011705 usec, L: 0
0430 20:11:42.362309 (+ 177us) write_op.cc:270] Start()
0430 20:11:42.362329 (+ 20us) write_op.cc:276] Timestamp: P: 1714507893034247 usec, L: 0
0430 20:11:42.362467 (+ 138us) write_op.cc:270] Start()
0430 20:11:42.362486 (+ 19us) write_op.cc:276] Timestamp: P: 1714507893037500 usec, L: 0
0430 20:11:42.426432 (+ 63946us) write_op.cc:270] Start()
0430 20:11:42.426479 (+ 47us) write_op.cc:276] Timestamp: P: 1714507893043535 usec, L: 0
0430 20:11:42.426751 (+ 272us) write_op.cc:270] Start()
0430 20:11:42.426781 (+ 30us) write_op.cc:276] Timestamp: P: 1714507893048744 usec, L: 0
0430 20:11:42.426926 (+ 145us) write_op.cc:270] Start()
0430 20:11:42.426948 (+ 22us) write_op.cc:276] Timestamp: P: 1714507893063203 usec, L: 0
0430 20:11:42.427091 (+ 143us) write_op.cc:270] Start()
0430 20:11:42.427116 (+ 25us) write_op.cc:276] Timestamp: P: 1714507893197503 usec, L: 0
0430 20:11:42.427265 (+ 149us) write_op.cc:270] Start()
0430 20:11:42.427287 (+ 22us) write_op.cc:276] Timestamp: P: 1714507893242404 usec, L: 0
0430 20:11:42.427624 (+ 337us) write_op.cc:270] Start()
0430 20:11:42.427649 (+ 25us) write_op.cc:276] Timestamp: P: 1714507893285583 usec, L: 0
0430 20:11:42.427782 (+ 133us) write_op.cc:270] Start()
0430 20:11:42.427806 (+ 24us) write_op.cc:276] Timestamp: P: 1714507893295675 usec, L: 0
0430 20:11:42.427932 (+ 126us) write_op.cc:270] Start()
0430 20:11:42.427956 (+ 24us) write_op.cc:276] Timestamp: P: 1714507893386143 usec, L: 0
0430 20:11:42.428115 (+ 159us) write_op.cc:270] Start()
0430 20:11:42.428150 (+ 35us) write_op.cc:276] Timestamp: P: 1714507893811658 usec, L: 0
0430 20:11:42.428291 (+ 141us) write_op.cc:270] Start()
0430 20:11:42.428310 (+ 19us) write_op.cc:276] Timestamp: P: 1714507893831053 usec, L: 0
0430 20:11:42.428413 (+ 103us) write_op.cc:270] Start()
0430 20:11:42.428438 (+ 25us) write_op.cc:276] Timestamp: P: 1714507893838430 usec, L: 0
0430 20:11:42.428566 (+ 128us) write_op.cc:270] Start()
0430 20:11:42.428586 (+ 20us) write_op.cc:276] Timestamp: P: 1714507893842010 usec, L: 0
0430 20:11:42.428717 (+ 131us) write_op.cc:270] Start()
0430 20:11:42.428737 (+ 20us) write_op.cc:276] Timestamp: P: 1714507893865151 usec, L: 0
0430 20:11:42.428878 (+ 141us) write_op.cc:270] Start()
0430 20:11:42.428904 (+ 26us) write_op.cc:276] Timestamp: P: 1714507894069495 usec, L: 0
0430 20:11:42.429032 (+ 128us) write_op.cc:270] Start()
0430 20:11:42.429051 (+ 19us) write_op.cc:276] Timestamp: P: 1714507894083041 usec, L: 0
0430 20:11:42.429164 (+ 113us) write_op.cc:270] Start()
0430 20:11:42.429181 (+ 17us) write_op.cc:276] Timestamp: P: 1714507894083674 usec, L: 0
0430 20:11:42.429289 (+ 108us) write_op.cc:270] Start()
0430 20:11:42.429306 (+ 17us) write_op.cc:276] Timestamp: P: 1714507894084238 usec, L: 0
0430 20:11:42.429421 (+ 115us) write_op.cc:270] Start()
0430 20:11:42.429453 (+ 32us) write_op.cc:276] Timestamp: P: 1714507894084755 usec, L: 0
0430 20:11:42.429558 (+ 105us) write_op.cc:270] Start()
0430 20:11:42.429577 (+ 19us) write_op.cc:276] Timestamp: P: 1714507894104114 usec, L: 0
0430 20:11:42.429734 (+ 157us) write_op.cc:270] Start()
0430 20:11:42.429759 (+ 25us) write_op.cc:276] Timestamp: P: 1714507894109151 usec, L: 0
0430 20:11:42.429927 (+ 168us) write_op.cc:270] Start()
0430 20:11:42.429987 (+ 60us) write_op.cc:276] Timestamp: P: 1714507894111494 usec, L: 0
0430 20:11:42.430160 (+ 173us) write_op.cc:270] Start()
0430 20:11:42.430184 (+ 24us) write_op.cc:276] Timestamp: P: 1714507894112584 usec, L: 0
0430 20:11:42.430329 (+ 145us) write_op.cc:270] Start()
0430 20:11:42.430350 (+ 21us) write_op.cc:276] Timestamp: P: 1714507894191668 usec, L: 0
0430 20:11:42.430487 (+ 137us) write_op.cc:270] Start()
0430 20:11:42.430510 (+ 23us) write_op.cc:276] Timestamp: P: 1714507894268500 usec, L: 0
0430 20:11:42.493284 (+ 62774us) write_op.cc:270] Start()
0430 20:11:42.493332 (+ 48us) write_op.cc:276] Timestamp: P: 1714507894444649 usec, L: 0
0430 20:11:42.493579 (+ 247us) write_op.cc:270] Start()
0430 20:11:42.493603 (+ 24us) write_op.cc:276] Timestamp: P: 1714507894458313 usec, L: 0
0430 20:11:42.493740 (+ 137us) write_op.cc:270] Start()
0430 20:11:42.493757 (+ 17us) write_op.cc:276] Timestamp: P: 1714507894459047 usec, L: 0
0430 20:11:42.493876 (+ 119us) write_op.cc:270] Start()
0430 20:11:42.493897 (+ 21us) write_op.cc:276] Timestamp: P: 1714507894470056 usec, L: 0
0430 20:11:42.494024 (+ 127us) write_op.cc:270] Start()
0430 20:11:42.494049 (+ 25us) write_op.cc:276] Timestamp: P: 1714507894480332 usec, L: 0
0430 20:11:42.494161 (+ 112us) write_op.cc:270] Start()
0430 20:11:42.494188 (+ 27us) write_op.cc:276] Timestamp: P: 1714507894782302 usec, L: 0
0430 20:11:42.526889 (+ 32701us) write_op.cc:270] Start()
0430 20:11:42.526928 (+ 39us) write_op.cc:276] Timestamp: P: 1714507894924888 usec, L: 0
0430 20:11:42.527190 (+ 262us) write_op.cc:270] Start()
0430 20:11:42.527217 (+ 27us) write_op.cc:276] Timestamp: P: 1714507895517233 usec, L: 0
0430 20:11:42.527374 (+ 157us) write_op.cc:270] Start()
0430 20:11:42.527399 (+ 25us) write_op.cc:276] Timestamp: P: 1714507895654553 usec, L: 0
0430 20:11:42.527548 (+ 149us) write_op.cc:270] Start()
0430 20:11:42.527568 (+ 20us) write_op.cc:276] Timestamp: P: 1714507895830757 usec, L: 0
0430 20:11:42.527700 (+ 132us) write_op.cc:270] Start()
0430 20:11:42.527719 (+ 19us) write_op.cc:276] Timestamp: P: 1714507895844699 usec, L: 0
0430 20:11:42.527891 (+ 172us) write_op.cc:270] Start()
0430 20:11:42.527911 (+ 20us) write_op.cc:276] Timestamp: P: 1714507895845367 usec, L: 0
0430 20:11:42.528036 (+ 125us) write_op.cc:270] Start()
0430 20:11:42.528054 (+ 18us) write_op.cc:276] Timestamp: P: 1714507895864259 usec, L: 0
0430 20:11:42.528196 (+ 142us) write_op.cc:270] Start()
0430 20:11:42.528215 (+ 19us) write_op.cc:276] Timestamp: P: 1714507895866100 usec, L: 0
0430 20:11:42.528333 (+ 118us) write_op.cc:270] Start()
0430 20:11:42.528350 (+ 17us) write_op.cc:276] Timestamp: P: 1714507895871699 usec, L: 0
0430 20:11:42.528629 (+ 279us) write_op.cc:270] Start()
0430 20:11:42.528647 (+ 18us) write_op.cc:276] Timestamp: P: 1714507896295007 usec, L: 0
0430 20:11:42.528774 (+ 127us) write_op.cc:270] Start()
0430 20:11:42.528793 (+ 19us) write_op.cc:276] Timestamp: P: 1714507896983063 usec, L: 0
0430 20:11:42.528929 (+ 136us) write_op.cc:270] Start()
0430 20:11:42.528950 (+ 21us) write_op.cc:276] Timestamp: P: 1714507897036203 usec, L: 0
0430 20:11:42.529073 (+ 123us) write_op.cc:270] Start()
0430 20:11:42.529095 (+ 22us) write_op.cc:276] Timestamp: P: 1714507897462944 usec, L: 0
0430 20:11:42.529227 (+ 132us) write_op.cc:270] Start()
0430 20:11:42.529246 (+ 19us) write_op.cc:276] Timestamp: P: 1714507897463794 usec, L: 0
0430 20:11:42.735615 (+206369us) log.cc:842] Serialized 128305 byte log entry
0430 20:11:42.785889 (+ 50274us) raft_consensus.cc:1631] Marking committed up to 497
0430 20:11:42.786080 (+ 191us) log.cc:842] Serialized 15 byte log entry
0430 20:11:43.015667 (+229587us) raft_consensus.cc:1683] Filling consensus response to leader.
0430 20:11:43.019185 (+ 3518us) spinlock_profiling.cc:243] Waited 1.12 s on lock 0x616000070654. stack: 00007f6d23ac8756 00007f6d23ac8510 00007f6d22afb48d 0000000000955804 00000000009557cd 00007f6d3260715d 00007f6d32607253 00007f6d2f339d41 00007f6d3710d5f5 00007f6d37120755 00007f6d37120501 00007f6d30640a8d 00007f6d2f34a27e 00007f6d2f349fcd 0000000000a60bcc 00007f6d23b0eb74
0430 20:11:43.020126 (+ 941us) spinlock_profiling.cc:243] Waited 131 ms on lock 0x616000070654. stack: 00007f6d23ac8756 00007f6d23ac8510 00007f6d22afb48d 0000000000955804 00000000009557cd 00007f6d3260715d 00007f6d32607253 00007f6d2f339d41 00007f6d3710d5f5 00007f6d37120755 00007f6d37120501 00007f6d30640a8d 00007f6d2f3
I20240430 20:11:43.931831 18983 tablet_bootstrap.cc:492] T 681cdd85ac6847a0a1aaded310456582 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:43.932061 18983 ts_tablet_manager.cc:1367] T 681cdd85ac6847a0a1aaded310456582 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.180s user 0.004s sys 0.001s
I20240430 20:11:43.932915 18983 raft_consensus.cc:348] T 681cdd85ac6847a0a1aaded310456582 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:43.933233 18983 raft_consensus.cc:374] T 681cdd85ac6847a0a1aaded310456582 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:43.933475 18983 raft_consensus.cc:729] T 681cdd85ac6847a0a1aaded310456582 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:43.933949 18983 consensus_queue.cc:260] T 681cdd85ac6847a0a1aaded310456582 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:43.935222 18983 ts_tablet_manager.cc:1398] T 681cdd85ac6847a0a1aaded310456582 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240430 20:11:43.935757 18983 tablet_bootstrap.cc:492] T f940d17b02b541289121463e05b0bb5f P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:43.894202 18750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50486644ec234933b9a0b45ee39a6685" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:43.938136 19489 ts_tablet_manager.cc:1875] T 1555924b5f1444398f12ee82c3cb5b13 P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:43.939782 18763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50486644ec234933b9a0b45ee39a6685" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
W20240430 20:11:43.943205 18557 leader_election.cc:341] T 50486644ec234933b9a0b45ee39a6685 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
W20240430 20:11:43.943709 18557 leader_election.cc:341] T 50486644ec234933b9a0b45ee39a6685 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:43.944572 20384 leader_election.cc:288] T 50486644ec234933b9a0b45ee39a6685 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:43.922703 18588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1073fc792954c5795a05aa7d0ec24df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:43.946677 18601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c85a6d11d8af486492d8b843c75632e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:43.946946 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 238) took 3450 ms. Trace:
I20240430 20:11:43.947070 19495 rpcz_store.cc:276] 0430 20:11:40.496952 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:40.497061 (+ 109us) service_pool.cc:225] Handling call
0430 20:11:43.678880 (+3181819us) ts_tablet_manager.cc:1120] Deleting tablet fd40967ab4a04001819e16c06a6c4300
0430 20:11:43.678890 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:43.695186 (+ 16296us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:43.706994 (+ 11808us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:43.946916 (+239922us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3181594}
I20240430 20:11:43.947283 19489 ts_tablet_manager.cc:1888] T 1555924b5f1444398f12ee82c3cb5b13 P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.2
I20240430 20:11:43.947400 19489 log.cc:1196] T 1555924b5f1444398f12ee82c3cb5b13 P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/1555924b5f1444398f12ee82c3cb5b13
I20240430 20:11:43.947481 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 240) took 3449 ms. Trace:
I20240430 20:11:43.947576 19495 rpcz_store.cc:276] 0430 20:11:40.497785 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:40.497834 (+ 49us) service_pool.cc:225] Handling call
0430 20:11:43.947368 (+3449534us) ts_tablet_manager.cc:1120] Deleting tablet e6eac92dfa5f4456bc49571878f0f8dd
0430 20:11:43.947380 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:43.947461 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3449362}
I20240430 20:11:43.947932 19489 ts_tablet_manager.cc:1909] T 1555924b5f1444398f12ee82c3cb5b13 P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:43.947926 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 239) took 3450 ms. Trace:
I20240430 20:11:43.947997 19495 rpcz_store.cc:276] 0430 20:11:40.497643 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:40.498086 (+ 443us) service_pool.cc:225] Handling call
0430 20:11:43.947834 (+3449748us) ts_tablet_manager.cc:1120] Deleting tablet a9e0659564894b0e8cb7f600bb07a085
0430 20:11:43.947852 (+ 18us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:43.947908 (+ 56us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3449617}
I20240430 20:11:43.948316 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 241) took 3377 ms. Trace:
I20240430 20:11:43.948395 19495 rpcz_store.cc:276] 0430 20:11:40.571125 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:40.571190 (+ 65us) service_pool.cc:225] Handling call
0430 20:11:43.948223 (+3377033us) ts_tablet_manager.cc:1120] Deleting tablet e4cffa295d8c401aa8415ce7e8f56526
0430 20:11:43.948230 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:43.948301 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3365254}
I20240430 20:11:43.948671 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 252) took 3235 ms. Trace:
I20240430 20:11:43.948745 19495 rpcz_store.cc:276] 0430 20:11:40.713634 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:40.713700 (+ 66us) service_pool.cc:225] Handling call
0430 20:11:43.948584 (+3234884us) ts_tablet_manager.cc:1120] Deleting tablet a9f37f0f7ce8425f9b24699513d41d61
0430 20:11:43.948592 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:43.948659 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3234642}
I20240430 20:11:43.948838 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 146) took 11274 ms. Trace:
I20240430 20:11:43.948910 19489 rpcz_store.cc:276] 0430 20:11:32.674612 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:32.676437 (+ 1825us) service_pool.cc:225] Handling call
0430 20:11:43.885556 (+11209119us) ts_tablet_manager.cc:1120] Deleting tablet 1555924b5f1444398f12ee82c3cb5b13
0430 20:11:43.885569 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:43.941182 (+ 55613us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:43.947177 (+ 5995us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:43.948822 (+ 1645us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":11170186}
I20240430 20:11:43.949031 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 253) took 3234 ms. Trace:
I20240430 20:11:43.949091 19495 rpcz_store.cc:276] 0430 20:11:40.714138 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:40.714187 (+ 49us) service_pool.cc:225] Handling call
0430 20:11:43.948945 (+3234758us) ts_tablet_manager.cc:1120] Deleting tablet 4dac8b3838954d839094021f1c0503da
0430 20:11:43.948953 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:43.949019 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3234584}
I20240430 20:11:43.949334 19495 tablet_replica.cc:330] T 0a9066d895744c5bb7d9532b2940603c P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:43.949395 19001 catalog_manager.cc:4830] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): tablet 1555924b5f1444398f12ee82c3cb5b13 (table default.Table_388d774462294cab8342c6de331f9712 [id=aca48e01301e4f5f9562df8c87829b9d]) successfully deleted
I20240430 20:11:43.949529 19495 raft_consensus.cc:2229] T 0a9066d895744c5bb7d9532b2940603c P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:43.949698 19495 raft_consensus.cc:2258] T 0a9066d895744c5bb7d9532b2940603c P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:43.949826 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 154) took 10890 ms. Trace:
I20240430 20:11:43.949908 19489 rpcz_store.cc:276] 0430 20:11:33.059046 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:33.059102 (+ 56us) service_pool.cc:225] Handling call
0430 20:11:43.949689 (+10890587us) ts_tablet_manager.cc:1120] Deleting tablet 246fb63a5e9842df8263800e28a5935c
0430 20:11:43.949700 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:43.949807 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":10890283}
I20240430 20:11:43.950301 19495 ts_tablet_manager.cc:1875] T 0a9066d895744c5bb7d9532b2940603c P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240430 20:11:43.950450 19001 catalog_manager.cc:4810] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): delete failed for tablet 246fb63a5e9842df8263800e28a5935c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 246fb63a5e9842df8263800e28a5935c already in progress: creating tablet
I20240430 20:11:43.951105 20105 raft_consensus.cc:482] T e87fb2f0361d4aa3ada27bbdfa88e94c P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:43.951197 19004 catalog_manager.cc:4830] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): tablet fd40967ab4a04001819e16c06a6c4300 (table default.Table_4130caf534004edfb413056d7f6cfd52 [id=da9ad991953a40c4b77b31317597ad1d]) successfully deleted
I20240430 20:11:43.951236 20105 raft_consensus.cc:504] T e87fb2f0361d4aa3ada27bbdfa88e94c P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
W20240430 20:11:43.951455 19004 catalog_manager.cc:4797] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): delete failed for tablet e6eac92dfa5f4456bc49571878f0f8dd because the tablet was not found. No further retry: Not found: Tablet not found: e6eac92dfa5f4456bc49571878f0f8dd
W20240430 20:11:43.951648 19004 catalog_manager.cc:4797] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): delete failed for tablet a9e0659564894b0e8cb7f600bb07a085 because the tablet was not found. No further retry: Not found: Tablet not found: a9e0659564894b0e8cb7f600bb07a085
W20240430 20:11:43.951841 19004 catalog_manager.cc:4810] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): delete failed for tablet e4cffa295d8c401aa8415ce7e8f56526 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e4cffa295d8c401aa8415ce7e8f56526 already in progress: creating tablet
W20240430 20:11:43.952105 19004 catalog_manager.cc:4810] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): delete failed for tablet a9f37f0f7ce8425f9b24699513d41d61 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a9f37f0f7ce8425f9b24699513d41d61 already in progress: creating tablet
W20240430 20:11:43.952311 19004 catalog_manager.cc:4810] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): delete failed for tablet 4dac8b3838954d839094021f1c0503da because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4dac8b3838954d839094021f1c0503da already in progress: creating tablet
I20240430 20:11:43.952332 18763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e87fb2f0361d4aa3ada27bbdfa88e94c" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
W20240430 20:11:43.952909 18417 leader_election.cc:341] T e87fb2f0361d4aa3ada27bbdfa88e94c P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:43.953178 18454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd423a49fb694695a8de7ecd54330127. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:43.953648 18619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e87fb2f0361d4aa3ada27bbdfa88e94c" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:43.953860 18619 raft_consensus.cc:2454] T e87fb2f0361d4aa3ada27bbdfa88e94c P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 0.
I20240430 20:11:43.954236 18415 leader_election.cc:302] T e87fb2f0361d4aa3ada27bbdfa88e94c P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters: 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:43.954494 20235 raft_consensus.cc:2781] T e87fb2f0361d4aa3ada27bbdfa88e94c P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:43.954550 20235 raft_consensus.cc:482] T e87fb2f0361d4aa3ada27bbdfa88e94c P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:43.954632 20235 raft_consensus.cc:3037] T e87fb2f0361d4aa3ada27bbdfa88e94c P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
W20240430 20:11:43.955157 19028 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.17.177.3:42157 (request call id 85) took 1628 ms (client timeout 1000 ms). Trace:
W20240430 20:11:43.955289 19028 rpcz_store.cc:269] 0430 20:11:42.326526 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:42.326623 (+ 97us) service_pool.cc:225] Handling call
0430 20:11:43.955113 (+1628490us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:42.758893 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:42.758992 (+ 99us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:42.758997 (+ 5us) write_op.cc:427] Acquired schema lock
0430 20:11:42.758999 (+ 2us) tablet.cc:660] Decoding operations
0430 20:11:42.759082 (+ 83us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:42.759090 (+ 8us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:42.759094 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0430 20:11:42.759139 (+ 45us) tablet.cc:699] Row locks acquired
0430 20:11:42.759141 (+ 2us) write_op.cc:260] PREPARE: finished
0430 20:11:42.759166 (+ 25us) write_op.cc:270] Start()
0430 20:11:42.759188 (+ 22us) write_op.cc:276] Timestamp: P: 1714507902759118 usec, L: 0
0430 20:11:42.759192 (+ 4us) op_driver.cc:331] REPLICATION: starting
0430 20:11:42.759345 (+ 153us) log.cc:842] Serialized 530 byte log entry
0430 20:11:42.766671 (+ 7326us) op_driver.cc:444] REPLICATION: finished
0430 20:11:42.767361 (+ 690us) write_op.cc:301] APPLY: starting
0430 20:11:42.767371 (+ 10us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:42.767401 (+ 30us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:42.767421 (+ 20us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:42.767473 (+ 52us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:42.767498 (+ 25us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0430 20:11:42.767504 (+ 6us) write_op.cc:312] APPLY: finished
0430 20:11:42.767573 (+ 69us) log.cc:842] Serialized 23 byte log entry
0430 20:11:42.767618 (+ 45us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:42.767634 (+ 16us) write_op.cc:446] Released schema lock
0430 20:11:42.767640 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:42.767691 (+ 51us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:42.767695 (+ 4us) write_op.cc:446] Released schema lock
Metrics: {"reported_tablets":184,"spinlock_wait_cycles":547072,"child_traces":[["op",{"apply.queue_time_us":658,"apply.run_cpu_time_us":376,"apply.run_wall_time_us":376,"num_ops":1,"prepare.queue_time_us":7390,"prepare.run_cpu_time_us":496,"prepare.run_wall_time_us":496,"replication_time_us":7464}]]}
I20240430 20:11:43.956089 19028 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_4bce1dae3391497ebea5f1b536700d62" } modify_external_catalogs: true
I20240430 20:11:43.957194 20235 raft_consensus.cc:504] T e87fb2f0361d4aa3ada27bbdfa88e94c P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:43.957826 20235 leader_election.cc:288] T e87fb2f0361d4aa3ada27bbdfa88e94c P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:43.958993 18449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a558d18cdfa4b33b4f72dc9041bfd53. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:43.959450 18763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e87fb2f0361d4aa3ada27bbdfa88e94c" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
W20240430 20:11:43.959971 18417 leader_election.cc:341] T e87fb2f0361d4aa3ada27bbdfa88e94c P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:43.960608 18619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e87fb2f0361d4aa3ada27bbdfa88e94c" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:43.960800 18619 raft_consensus.cc:3037] T e87fb2f0361d4aa3ada27bbdfa88e94c P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:43.962827 18446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad01c979b2f7439bab83137f87f1405c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:43.963582 18619 raft_consensus.cc:2454] T e87fb2f0361d4aa3ada27bbdfa88e94c P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 1.
I20240430 20:11:43.964136 18415 leader_election.cc:302] T e87fb2f0361d4aa3ada27bbdfa88e94c P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters: 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:43.964597 20235 raft_consensus.cc:2781] T e87fb2f0361d4aa3ada27bbdfa88e94c P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:43.965245 20235 raft_consensus.cc:686] T e87fb2f0361d4aa3ada27bbdfa88e94c P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Becoming Leader. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Running, Role: LEADER
I20240430 20:11:43.965507 20235 consensus_queue.cc:237] T e87fb2f0361d4aa3ada27bbdfa88e94c P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:43.966555 20105 leader_election.cc:288] T e87fb2f0361d4aa3ada27bbdfa88e94c P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:43.966972 18447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1073fc792954c5795a05aa7d0ec24df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:43.968165 18459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b95098d3af04dd5a9ae3972e44b3a6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:43.968315 18453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4953578de5f45f3a6ba092a13357460. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:43.969676 19489 tablet_replica.cc:330] T c2b019f529a54228b804437bb025e31d P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:43.973639 19495 ts_tablet_manager.cc:1888] T 0a9066d895744c5bb7d9532b2940603c P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:43.973757 19495 log.cc:1196] T 0a9066d895744c5bb7d9532b2940603c P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/0a9066d895744c5bb7d9532b2940603c
I20240430 20:11:43.973928 18983 tablet_bootstrap.cc:654] T f940d17b02b541289121463e05b0bb5f P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:43.974099 19495 ts_tablet_manager.cc:1909] T 0a9066d895744c5bb7d9532b2940603c P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:43.975054 18445 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c85a6d11d8af486492d8b843c75632e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:43.975227 18458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 975d145de1794f05b1dd844b4b86ce40. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:43.978824 18457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 107638df57ed43adb6e13d47997df541. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:43.979024 18448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cefce2f2a392495bb71399ed75db938c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:43.979236 18450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1277899f0c040cea7dc2b12f3fd68de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:43.982739 18461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbdbb8a3a4b444f58a2761466577a88f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:43.993193 20324 consensus_queue.cc:1035] T 5f02d8862e1f4ccfa9dbda73dbe18615 P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:43.997485 19471 ts_tablet_manager.cc:1888] T a745334c12c545c1ba825c7552c60359 P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:43.997785 19471 log.cc:1196] T a745334c12c545c1ba825c7552c60359 P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/a745334c12c545c1ba825c7552c60359
I20240430 20:11:43.998265 19471 ts_tablet_manager.cc:1909] T a745334c12c545c1ba825c7552c60359 P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:43.999264 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 157) took 10177 ms. Trace:
I20240430 20:11:44.009934 18758 raft_consensus.cc:1261] T aec9c09cc0454acab6006bb3e10982b7 P 3c51c33d5616482cafb68e26688820f4 [term 3 FOLLOWER]: Refusing update from remote peer 5e1c6397f09747f091534faf93542710: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240430 20:11:44.010041 18758 pending_rounds.cc:76] T aec9c09cc0454acab6006bb3e10982b7 P 3c51c33d5616482cafb68e26688820f4: Aborting all ops after (but not including) 3
I20240430 20:11:44.010077 18758 pending_rounds.cc:97] T aec9c09cc0454acab6006bb3e10982b7 P 3c51c33d5616482cafb68e26688820f4: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240430 20:11:44.010115 18758 raft_consensus.cc:2866] T aec9c09cc0454acab6006bb3e10982b7 P 3c51c33d5616482cafb68e26688820f4 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240430 20:11:44.012377 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 254) took 3264 ms. Trace:
I20240430 20:11:44.012356 20047 consensus_queue.cc:1035] T aec9c09cc0454acab6006bb3e10982b7 P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240430 20:11:44.012488 19495 rpcz_store.cc:276] 0430 20:11:40.748304 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:40.748357 (+ 53us) service_pool.cc:225] Handling call
0430 20:11:43.949295 (+3200938us) ts_tablet_manager.cc:1120] Deleting tablet 0a9066d895744c5bb7d9532b2940603c
0430 20:11:43.949303 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:43.970721 (+ 21418us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:43.973525 (+ 2804us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:44.012353 (+ 38828us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3189196}
I20240430 20:11:44.012809 19495 tablet_replica.cc:330] T 34130f2820064fa0898ef1f9d7de1c6d P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:44.013003 19495 raft_consensus.cc:2229] T 34130f2820064fa0898ef1f9d7de1c6d P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Raft consensus shutting down.
I20240430 20:11:44.013314 19495 raft_consensus.cc:2258] T 34130f2820064fa0898ef1f9d7de1c6d P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:44.013979 19004 catalog_manager.cc:4830] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): tablet 0a9066d895744c5bb7d9532b2940603c (table default.Table_7b72c19b4765456dba543151abaf1d44 [id=ec7831570fea4ff2944bc6304fc17bc9]) successfully deleted
I20240430 20:11:44.018899 19495 ts_tablet_manager.cc:1875] T 34130f2820064fa0898ef1f9d7de1c6d P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:44.022248 19028 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_4bce1dae3391497ebea5f1b536700d62" } modify_external_catalogs: true
I20240430 20:11:44.026846 19049 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_e8c73c89623d4b49b8d7f0a01d729bdd" } modify_external_catalogs: true
I20240430 20:11:44.027117 19049 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_e8c73c89623d4b49b8d7f0a01d729bdd" } modify_external_catalogs: true
I20240430 20:11:44.033633 19495 ts_tablet_manager.cc:1888] T 34130f2820064fa0898ef1f9d7de1c6d P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:44.035784 20395 consensus_queue.cc:1035] T 115c4a12440647a4bb46eb6824f7ea98 P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:44.038388 19495 log.cc:1196] T 34130f2820064fa0898ef1f9d7de1c6d P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/34130f2820064fa0898ef1f9d7de1c6d
I20240430 20:11:44.039475 19495 ts_tablet_manager.cc:1909] T 34130f2820064fa0898ef1f9d7de1c6d P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:44.045734 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 255) took 3260 ms. Trace:
I20240430 20:11:44.050798 19495 rpcz_store.cc:276] 0430 20:11:40.785057 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:40.785122 (+ 65us) service_pool.cc:225] Handling call
0430 20:11:44.012759 (+3227637us) ts_tablet_manager.cc:1120] Deleting tablet 34130f2820064fa0898ef1f9d7de1c6d
0430 20:11:44.012769 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:44.022279 (+ 9510us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:44.033500 (+ 11221us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:44.045706 (+ 12206us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3227445}
I20240430 20:11:44.052528 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 256) took 3262 ms. Trace:
I20240430 20:11:44.052623 19004 catalog_manager.cc:4830] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): tablet 34130f2820064fa0898ef1f9d7de1c6d (table default.Table_69892a8264a4477db48ee6248758e046 [id=5a1c4a7be2fc465fac89a50854eb3322]) successfully deleted
I20240430 20:11:44.056761 19495 rpcz_store.cc:276] 0430 20:11:40.790540 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:40.802681 (+ 12141us) service_pool.cc:225] Handling call
0430 20:11:44.052366 (+3249685us) ts_tablet_manager.cc:1120] Deleting tablet 436579f473844c6ca1038b496201a550
0430 20:11:44.052385 (+ 19us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:44.052509 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3249493}
W20240430 20:11:44.058238 19004 catalog_manager.cc:4810] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): delete failed for tablet 436579f473844c6ca1038b496201a550 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 436579f473844c6ca1038b496201a550 already in progress: creating tablet
I20240430 20:11:44.058408 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 257) took 3192 ms. Trace:
I20240430 20:11:44.058883 20489 raft_consensus.cc:482] T d61c279d20b0471a967942525ceb4fcb P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:44.058970 19495 rpcz_store.cc:276] 0430 20:11:40.866284 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:40.866358 (+ 74us) service_pool.cc:225] Handling call
0430 20:11:44.058230 (+3191872us) ts_tablet_manager.cc:1120] Deleting tablet f7d3dea34cf54fe2b949d1b7ca68577a
0430 20:11:44.058246 (+ 16us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:44.058386 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3190959}
I20240430 20:11:44.059469 20489 raft_consensus.cc:504] T d61c279d20b0471a967942525ceb4fcb P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.059852 19495 tablet_replica.cc:330] T e6c0c808bfc94be3837ab835292b3c1b P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
W20240430 20:11:44.059929 19004 catalog_manager.cc:4810] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): delete failed for tablet f7d3dea34cf54fe2b949d1b7ca68577a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f7d3dea34cf54fe2b949d1b7ca68577a already in progress: creating tablet
I20240430 20:11:44.060333 19495 raft_consensus.cc:2229] T e6c0c808bfc94be3837ab835292b3c1b P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:44.060832 19495 raft_consensus.cc:2258] T e6c0c808bfc94be3837ab835292b3c1b P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:44.061053 18750 raft_consensus.cc:1261] T e6eac92dfa5f4456bc49571878f0f8dd P 3c51c33d5616482cafb68e26688820f4 [term 2 FOLLOWER]: Refusing update from remote peer 5e1c6397f09747f091534faf93542710: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20240430 20:11:44.062031 19495 ts_tablet_manager.cc:1875] T e6c0c808bfc94be3837ab835292b3c1b P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:44.062358 20395 consensus_queue.cc:1035] T e6eac92dfa5f4456bc49571878f0f8dd P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:44.064134 18475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d61c279d20b0471a967942525ceb4fcb" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:44.065706 18750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d61c279d20b0471a967942525ceb4fcb" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
W20240430 20:11:44.066749 18558 leader_election.cc:341] T d61c279d20b0471a967942525ceb4fcb P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
W20240430 20:11:44.076303 18557 leader_election.cc:341] T d61c279d20b0471a967942525ceb4fcb P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:44.076584 18557 leader_election.cc:302] T d61c279d20b0471a967942525ceb4fcb P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:44.077301 20395 raft_consensus.cc:2726] T d61c279d20b0471a967942525ceb4fcb P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:44.077729 20489 leader_election.cc:288] T d61c279d20b0471a967942525ceb4fcb P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:44.087616 20395 raft_consensus.cc:482] T 681cdd85ac6847a0a1aaded310456582 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:44.087857 20395 raft_consensus.cc:504] T 681cdd85ac6847a0a1aaded310456582 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.088624 20395 leader_election.cc:288] T 681cdd85ac6847a0a1aaded310456582 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:44.089634 18478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "681cdd85ac6847a0a1aaded310456582" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:44.089919 18478 raft_consensus.cc:2454] T 681cdd85ac6847a0a1aaded310456582 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 0.
I20240430 20:11:44.090584 18558 leader_election.cc:302] T 681cdd85ac6847a0a1aaded310456582 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:44.091527 20384 raft_consensus.cc:2781] T 681cdd85ac6847a0a1aaded310456582 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:44.091693 20384 raft_consensus.cc:482] T 681cdd85ac6847a0a1aaded310456582 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:44.091845 20384 raft_consensus.cc:3037] T 681cdd85ac6847a0a1aaded310456582 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:44.092671 19495 ts_tablet_manager.cc:1888] T e6c0c808bfc94be3837ab835292b3c1b P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240430 20:11:44.094345 18756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "681cdd85ac6847a0a1aaded310456582" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:44.095872 20476 tablet.cc:1722] T a05e32a33aa7488eb287cc6314c09c44 P 5e1c6397f09747f091534faf93542710: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240430 20:11:44.096104 20384 raft_consensus.cc:504] T 681cdd85ac6847a0a1aaded310456582 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
W20240430 20:11:44.096371 18557 leader_election.cc:341] T 681cdd85ac6847a0a1aaded310456582 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:44.096701 20384 leader_election.cc:288] T 681cdd85ac6847a0a1aaded310456582 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:44.097801 18750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "681cdd85ac6847a0a1aaded310456582" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:44.098871 19495 log.cc:1196] T e6c0c808bfc94be3837ab835292b3c1b P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/e6c0c808bfc94be3837ab835292b3c1b
I20240430 20:11:44.099514 19495 ts_tablet_manager.cc:1909] T e6c0c808bfc94be3837ab835292b3c1b P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:44.099967 18475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "681cdd85ac6847a0a1aaded310456582" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:44.100175 18475 raft_consensus.cc:3037] T 681cdd85ac6847a0a1aaded310456582 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:44.100790 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 258) took 2948 ms. Trace:
I20240430 20:11:44.100869 19495 rpcz_store.cc:276] 0430 20:11:41.152367 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:41.152424 (+ 57us) service_pool.cc:225] Handling call
0430 20:11:44.059786 (+2907362us) ts_tablet_manager.cc:1120] Deleting tablet e6c0c808bfc94be3837ab835292b3c1b
0430 20:11:44.059800 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:44.076134 (+ 16334us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:44.092532 (+ 16398us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:44.100768 (+ 8236us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2907184}
I20240430 20:11:44.101671 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 262) took 2707 ms. Trace:
I20240430 20:11:44.101735 19495 rpcz_store.cc:276] 0430 20:11:41.394057 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:41.394114 (+ 57us) service_pool.cc:225] Handling call
0430 20:11:44.101548 (+2707434us) ts_tablet_manager.cc:1120] Deleting tablet 3b7fdc6ec7e046c5af894f9a6eda17b0
0430 20:11:44.101558 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:44.101656 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2707247}
I20240430 20:11:44.102165 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 269) took 2579 ms. Trace:
I20240430 20:11:44.102231 19495 rpcz_store.cc:276] 0430 20:11:41.523201 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:41.523267 (+ 66us) service_pool.cc:225] Handling call
0430 20:11:44.102058 (+2578791us) ts_tablet_manager.cc:1120] Deleting tablet 95383d9d5a234a4d9d841a8b114428b0
0430 20:11:44.102066 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:44.102149 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2578576}
I20240430 20:11:44.102658 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 275) took 1953 ms. Trace:
I20240430 20:11:44.105325 19495 rpcz_store.cc:276] 0430 20:11:42.149575 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:42.149883 (+ 308us) service_pool.cc:225] Handling call
0430 20:11:44.102540 (+1952657us) ts_tablet_manager.cc:1120] Deleting tablet 283e5e73cbc04daba0de19f4541f9fd0
0430 20:11:44.102551 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:44.102644 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1952480}
I20240430 20:11:44.070569 20102 sys_catalog.cc:455] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 5 leader_uuid: "b632d6ef4c104a058a6db208024cab33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b3f98793354c7aaa5c106e8ec31ed8" member_type: VOTER last_known_addr { host: "127.17.177.62" port: 36441 } } peers { permanent_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e" member_type: VOTER last_known_addr { host: "127.17.177.61" port: 34777 } } peers { permanent_uuid: "b632d6ef4c104a058a6db208024cab33" member_type: VOTER last_known_addr { host: "127.17.177.60" port: 37225 } } }
I20240430 20:11:44.101374 19004 catalog_manager.cc:4830] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): tablet e6c0c808bfc94be3837ab835292b3c1b (table default.Table_5fc77d5f60ae44f4bb4ca9f7f183672c [id=6ca223ca6354449c959d85ed2ac6a9c8]) successfully deleted
W20240430 20:11:44.107905 19004 catalog_manager.cc:4810] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): delete failed for tablet 3b7fdc6ec7e046c5af894f9a6eda17b0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3b7fdc6ec7e046c5af894f9a6eda17b0 already in progress: creating tablet
W20240430 20:11:44.108278 19004 catalog_manager.cc:4810] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): delete failed for tablet 95383d9d5a234a4d9d841a8b114428b0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 95383d9d5a234a4d9d841a8b114428b0 already in progress: creating tablet
I20240430 20:11:44.108942 20102 sys_catalog.cc:458] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33 [sys.catalog]: This master's current role is: LEADER
I20240430 20:11:44.103219 18475 raft_consensus.cc:2454] T 681cdd85ac6847a0a1aaded310456582 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
W20240430 20:11:44.109488 19004 catalog_manager.cc:4810] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): delete failed for tablet 283e5e73cbc04daba0de19f4541f9fd0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 283e5e73cbc04daba0de19f4541f9fd0 already in progress: creating tablet
I20240430 20:11:44.109954 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 277) took 1960 ms. Trace:
I20240430 20:11:44.110085 19495 rpcz_store.cc:276] 0430 20:11:42.149825 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:42.149877 (+ 52us) service_pool.cc:225] Handling call
0430 20:11:44.109824 (+1959947us) ts_tablet_manager.cc:1120] Deleting tablet 9c73d0a6bc414d3b9900570848d19a51
0430 20:11:44.109838 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:44.109935 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1959623}
I20240430 20:11:44.110168 20198 raft_consensus.cc:686] T c2b019f529a54228b804437bb025e31d P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Becoming Leader. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Running, Role: LEADER
W20240430 20:11:44.110611 19004 catalog_manager.cc:4810] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): delete failed for tablet 9c73d0a6bc414d3b9900570848d19a51 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9c73d0a6bc414d3b9900570848d19a51 already in progress: creating tablet
I20240430 20:11:44.110960 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 276) took 1961 ms. Trace:
I20240430 20:11:44.110563 20198 consensus_queue.cc:237] T c2b019f529a54228b804437bb025e31d P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.112793 19495 rpcz_store.cc:276] 0430 20:11:42.149720 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:42.149897 (+ 177us) service_pool.cc:225] Handling call
0430 20:11:44.110845 (+1960948us) ts_tablet_manager.cc:1120] Deleting tablet 2aa2a489116940dea3173cad16d94fa5
0430 20:11:44.110855 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:44.110943 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1960603}
W20240430 20:11:44.113363 19004 catalog_manager.cc:4810] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): delete failed for tablet 2aa2a489116940dea3173cad16d94fa5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2aa2a489116940dea3173cad16d94fa5 already in progress: creating tablet
I20240430 20:11:44.113708 19495 tablet_replica.cc:330] T f7b0132c17d44a9ba17288e0321a7f11 P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:44.113938 19495 raft_consensus.cc:2229] T f7b0132c17d44a9ba17288e0321a7f11 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:44.114151 19495 raft_consensus.cc:2258] T f7b0132c17d44a9ba17288e0321a7f11 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:44.117676 19495 ts_tablet_manager.cc:1875] T f7b0132c17d44a9ba17288e0321a7f11 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:44.119187 20442 tablet.cc:1722] T a05e32a33aa7488eb287cc6314c09c44 P 3c51c33d5616482cafb68e26688820f4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20240430 20:11:44.098752 18558 consensus_peers.cc:485] T e6eac92dfa5f4456bc49571878f0f8dd P 5e1c6397f09747f091534faf93542710 -> Peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Couldn't send request to peer 2efb542aa55e439b99ccb98d1c5c884a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e6eac92dfa5f4456bc49571878f0f8dd. This is attempt 1: this message will repeat every 5th retry.
I20240430 20:11:44.108732 20395 raft_consensus.cc:482] T f7b0132c17d44a9ba17288e0321a7f11 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:44.124568 20395 raft_consensus.cc:504] T f7b0132c17d44a9ba17288e0321a7f11 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.125586 20395 leader_election.cc:288] T f7b0132c17d44a9ba17288e0321a7f11 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
W20240430 20:11:44.111369 18557 leader_election.cc:341] T 681cdd85ac6847a0a1aaded310456582 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:44.128865 18558 leader_election.cc:302] T 681cdd85ac6847a0a1aaded310456582 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters: 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:44.129268 20324 raft_consensus.cc:2781] T 681cdd85ac6847a0a1aaded310456582 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:44.129482 20324 raft_consensus.cc:686] T 681cdd85ac6847a0a1aaded310456582 P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Becoming Leader. State: Replica: 5e1c6397f09747f091534faf93542710, State: Running, Role: LEADER
I20240430 20:11:44.129822 20324 consensus_queue.cc:237] T 681cdd85ac6847a0a1aaded310456582 P 5e1c6397f09747f091534faf93542710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.130241 18988 raft_consensus.cc:348] T ebf74809b52f47f3b0ed21b6ac0f541a P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.130445 18988 raft_consensus.cc:374] T ebf74809b52f47f3b0ed21b6ac0f541a P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:44.130496 18988 raft_consensus.cc:729] T ebf74809b52f47f3b0ed21b6ac0f541a P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:44.130719 18988 consensus_queue.cc:260] T ebf74809b52f47f3b0ed21b6ac0f541a P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.131208 18988 ts_tablet_manager.cc:1398] T ebf74809b52f47f3b0ed21b6ac0f541a P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.242s user 0.002s sys 0.000s
I20240430 20:11:44.131479 18988 tablet_bootstrap.cc:492] T 3fd0e8c360734dcbb75dc2958b70c2b8 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:44.131758 18750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7b0132c17d44a9ba17288e0321a7f11" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:44.132874 18988 tablet_bootstrap.cc:654] T 3fd0e8c360734dcbb75dc2958b70c2b8 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:44.134212 18474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7b0132c17d44a9ba17288e0321a7f11" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:44.134510 20398 tablet.cc:1722] T a05e32a33aa7488eb287cc6314c09c44 P 2efb542aa55e439b99ccb98d1c5c884a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20240430 20:11:44.137240 18558 leader_election.cc:341] T f7b0132c17d44a9ba17288e0321a7f11 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: Tablet not RUNNING: STOPPED
I20240430 20:11:44.139791 19495 ts_tablet_manager.cc:1888] T f7b0132c17d44a9ba17288e0321a7f11 P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20240430 20:11:44.140914 18557 leader_election.cc:341] T f7b0132c17d44a9ba17288e0321a7f11 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:44.141064 18557 leader_election.cc:302] T f7b0132c17d44a9ba17288e0321a7f11 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4
W20240430 20:11:44.141611 18557 consensus_peers.cc:485] T 115c4a12440647a4bb46eb6824f7ea98 P 5e1c6397f09747f091534faf93542710 -> Peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Couldn't send request to peer 3c51c33d5616482cafb68e26688820f4. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240430 20:11:44.142498 20469 raft_consensus.cc:482] T 4e69ca12bd10494fb013614741bc166c P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:44.142695 20469 raft_consensus.cc:504] T 4e69ca12bd10494fb013614741bc166c P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.143517 20469 leader_election.cc:288] T 4e69ca12bd10494fb013614741bc166c P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
W20240430 20:11:44.147504 18699 leader_election.cc:341] T 4e69ca12bd10494fb013614741bc166c P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Not found: Tablet not found: 4e69ca12bd10494fb013614741bc166c
I20240430 20:11:44.149139 20384 raft_consensus.cc:2726] T f7b0132c17d44a9ba17288e0321a7f11 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:44.150689 19495 log.cc:1196] T f7b0132c17d44a9ba17288e0321a7f11 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/f7b0132c17d44a9ba17288e0321a7f11
I20240430 20:11:44.151235 19495 ts_tablet_manager.cc:1909] T f7b0132c17d44a9ba17288e0321a7f11 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:44.156926 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 278) took 1974 ms. Trace:
I20240430 20:11:44.157171 19495 rpcz_store.cc:276] 0430 20:11:42.182557 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:42.182624 (+ 67us) service_pool.cc:225] Handling call
0430 20:11:44.113660 (+1931036us) ts_tablet_manager.cc:1120] Deleting tablet f7b0132c17d44a9ba17288e0321a7f11
0430 20:11:44.113670 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:44.136356 (+ 22686us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:44.139659 (+ 3303us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:44.156901 (+ 17242us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1930839}
I20240430 20:11:44.157765 19004 catalog_manager.cc:4830] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): tablet f7b0132c17d44a9ba17288e0321a7f11 (table default.Table_01d62e066e4942d9aa1f582189d60d49 [id=97861b0b1ce94638b62bbd123d774d7f]) successfully deleted
I20240430 20:11:44.158326 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 279) took 1975 ms. Trace:
I20240430 20:11:44.158504 19495 rpcz_store.cc:276] 0430 20:11:42.182692 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:42.182865 (+ 173us) service_pool.cc:225] Handling call
0430 20:11:44.158161 (+1975296us) ts_tablet_manager.cc:1120] Deleting tablet 4f6f9ae722da49e58c9b72107f6da2fa
0430 20:11:44.158183 (+ 22us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:44.158305 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1975140}
W20240430 20:11:44.159361 19004 catalog_manager.cc:4810] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): delete failed for tablet 4f6f9ae722da49e58c9b72107f6da2fa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4f6f9ae722da49e58c9b72107f6da2fa already in progress: creating tablet
I20240430 20:11:44.160303 19495 tablet_replica.cc:330] T 32c823d1700949ab8b22adcd83b7bfd1 P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:44.160584 19495 raft_consensus.cc:2229] T 32c823d1700949ab8b22adcd83b7bfd1 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Raft consensus shutting down.
I20240430 20:11:44.160971 19495 raft_consensus.cc:2258] T 32c823d1700949ab8b22adcd83b7bfd1 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:44.162883 20047 raft_consensus.cc:482] T abdaf243e5be4ad58036b23151b14fe4 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:44.163107 19495 ts_tablet_manager.cc:1875] T 32c823d1700949ab8b22adcd83b7bfd1 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:44.163087 20047 raft_consensus.cc:504] T abdaf243e5be4ad58036b23151b14fe4 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.164024 20047 leader_election.cc:288] T abdaf243e5be4ad58036b23151b14fe4 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:44.164541 18482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abdaf243e5be4ad58036b23151b14fe4" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:44.164855 18482 raft_consensus.cc:2454] T abdaf243e5be4ad58036b23151b14fe4 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 0.
I20240430 20:11:44.165808 18558 leader_election.cc:302] T abdaf243e5be4ad58036b23151b14fe4 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:44.166630 19489 raft_consensus.cc:2229] T c2b019f529a54228b804437bb025e31d P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Raft consensus shutting down.
I20240430 20:11:44.167299 19489 pending_rounds.cc:62] T c2b019f529a54228b804437bb025e31d P 3c51c33d5616482cafb68e26688820f4: Trying to abort 1 pending ops.
I20240430 20:11:44.167359 19489 pending_rounds.cc:68] T c2b019f529a54228b804437bb025e31d P 3c51c33d5616482cafb68e26688820f4: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7022624375268708352 op_type: NO_OP noop_request { }
I20240430 20:11:44.167500 19489 raft_consensus.cc:2866] T c2b019f529a54228b804437bb025e31d P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240430 20:11:44.167608 19489 raft_consensus.cc:2258] T c2b019f529a54228b804437bb025e31d P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:44.167865 20198 raft_consensus.cc:482] T f5546f9edeea44d7a6c5cb954f5157ba P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:44.168095 20198 raft_consensus.cc:504] T f5546f9edeea44d7a6c5cb954f5157ba P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.168388 20047 raft_consensus.cc:2781] T abdaf243e5be4ad58036b23151b14fe4 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:44.168506 20047 raft_consensus.cc:482] T abdaf243e5be4ad58036b23151b14fe4 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:44.168598 20047 raft_consensus.cc:3037] T abdaf243e5be4ad58036b23151b14fe4 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:44.169505 20198 leader_election.cc:288] T f5546f9edeea44d7a6c5cb954f5157ba P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
W20240430 20:11:44.170082 18699 leader_election.cc:341] T f5546f9edeea44d7a6c5cb954f5157ba P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Not found: Tablet not found: f5546f9edeea44d7a6c5cb954f5157ba
W20240430 20:11:44.170862 18696 leader_election.cc:341] T f5546f9edeea44d7a6c5cb954f5157ba P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Not found: Tablet not found: f5546f9edeea44d7a6c5cb954f5157ba
I20240430 20:11:44.170982 18696 leader_election.cc:302] T f5546f9edeea44d7a6c5cb954f5157ba P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710
I20240430 20:11:44.171590 20198 raft_consensus.cc:2726] T f5546f9edeea44d7a6c5cb954f5157ba P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240430 20:11:44.172847 20047 raft_consensus.cc:504] T abdaf243e5be4ad58036b23151b14fe4 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.173857 20047 leader_election.cc:288] T abdaf243e5be4ad58036b23151b14fe4 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:44.174609 18482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abdaf243e5be4ad58036b23151b14fe4" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:44.174870 18482 raft_consensus.cc:3037] T abdaf243e5be4ad58036b23151b14fe4 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:44.175288 19495 ts_tablet_manager.cc:1888] T 32c823d1700949ab8b22adcd83b7bfd1 P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:44.179023 18988 tablet_bootstrap.cc:492] T 3fd0e8c360734dcbb75dc2958b70c2b8 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:44.179242 18988 ts_tablet_manager.cc:1367] T 3fd0e8c360734dcbb75dc2958b70c2b8 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.048s user 0.002s sys 0.005s
I20240430 20:11:44.180001 18988 raft_consensus.cc:348] T 3fd0e8c360734dcbb75dc2958b70c2b8 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.180285 18988 raft_consensus.cc:374] T 3fd0e8c360734dcbb75dc2958b70c2b8 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:44.180374 18988 raft_consensus.cc:729] T 3fd0e8c360734dcbb75dc2958b70c2b8 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:44.180632 18988 consensus_queue.cc:260] T 3fd0e8c360734dcbb75dc2958b70c2b8 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.181181 18988 ts_tablet_manager.cc:1398] T 3fd0e8c360734dcbb75dc2958b70c2b8 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240430 20:11:44.181490 18988 tablet_bootstrap.cc:492] T a59ef6c545dd41a28265a4c1b8ce10e0 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:44.183085 18988 tablet_bootstrap.cc:654] T a59ef6c545dd41a28265a4c1b8ce10e0 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:44.184149 19495 log.cc:1196] T 32c823d1700949ab8b22adcd83b7bfd1 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/32c823d1700949ab8b22adcd83b7bfd1
W20240430 20:11:44.184319 18696 leader_election.cc:341] T 4e69ca12bd10494fb013614741bc166c P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Not found: Tablet not found: 4e69ca12bd10494fb013614741bc166c
I20240430 20:11:44.184445 18696 leader_election.cc:302] T 4e69ca12bd10494fb013614741bc166c P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710
I20240430 20:11:44.184783 19495 ts_tablet_manager.cc:1909] T 32c823d1700949ab8b22adcd83b7bfd1 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:44.185194 18756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abdaf243e5be4ad58036b23151b14fe4" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:44.185287 18758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abdaf243e5be4ad58036b23151b14fe4" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:44.185725 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 280) took 2002 ms. Trace:
I20240430 20:11:44.185603 20198 raft_consensus.cc:2726] T 4e69ca12bd10494fb013614741bc166c P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:44.190266 20198 raft_consensus.cc:482] T 3fd0e8c360734dcbb75dc2958b70c2b8 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:44.190486 20198 raft_consensus.cc:504] T 3fd0e8c360734dcbb75dc2958b70c2b8 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.190644 19495 rpcz_store.cc:276] 0430 20:11:42.182814 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:42.182894 (+ 80us) service_pool.cc:225] Handling call
0430 20:11:44.160252 (+1977358us) ts_tablet_manager.cc:1120] Deleting tablet 32c823d1700949ab8b22adcd83b7bfd1
0430 20:11:44.160264 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:44.172114 (+ 11850us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:44.175176 (+ 3062us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:44.185701 (+ 10525us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1977142}
I20240430 20:11:44.177614 18482 raft_consensus.cc:2454] T abdaf243e5be4ad58036b23151b14fe4 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
I20240430 20:11:44.191514 19004 catalog_manager.cc:4830] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): tablet 32c823d1700949ab8b22adcd83b7bfd1 (table default.Table_e8c73c89623d4b49b8d7f0a01d729bdd [id=5cd10682ce52414cb5a103e4433a6a0d]) successfully deleted
I20240430 20:11:44.191552 18558 leader_election.cc:302] T abdaf243e5be4ad58036b23151b14fe4 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:44.191991 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 281) took 2004 ms. Trace:
I20240430 20:11:44.192116 19495 rpcz_store.cc:276] 0430 20:11:42.187091 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:42.187148 (+ 57us) service_pool.cc:225] Handling call
0430 20:11:44.191851 (+2004703us) ts_tablet_manager.cc:1120] Deleting tablet b22aaedb0a6c4fd9ad0ea8fa019001e1
0430 20:11:44.191864 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:44.191971 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2004505}
I20240430 20:11:44.192620 20198 leader_election.cc:288] T 3fd0e8c360734dcbb75dc2958b70c2b8 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
W20240430 20:11:44.192623 19004 catalog_manager.cc:4810] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): delete failed for tablet b22aaedb0a6c4fd9ad0ea8fa019001e1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b22aaedb0a6c4fd9ad0ea8fa019001e1 already in progress: creating tablet
I20240430 20:11:44.192926 19495 tablet_replica.cc:330] T ae87f5e92a934e2182164c231bc76df0 P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:44.192996 20489 raft_consensus.cc:2781] T abdaf243e5be4ad58036b23151b14fe4 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:44.193154 20489 raft_consensus.cc:686] T abdaf243e5be4ad58036b23151b14fe4 P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Becoming Leader. State: Replica: 5e1c6397f09747f091534faf93542710, State: Running, Role: LEADER
I20240430 20:11:44.193171 19495 raft_consensus.cc:2229] T ae87f5e92a934e2182164c231bc76df0 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Raft consensus shutting down.
I20240430 20:11:44.193493 19495 raft_consensus.cc:2258] T ae87f5e92a934e2182164c231bc76df0 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:44.193446 20489 consensus_queue.cc:237] T abdaf243e5be4ad58036b23151b14fe4 P 5e1c6397f09747f091534faf93542710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.194480 18482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd0e8c360734dcbb75dc2958b70c2b8" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:44.195078 19495 ts_tablet_manager.cc:1875] T ae87f5e92a934e2182164c231bc76df0 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:44.196657 20332 mvcc.cc:205] Tried to move back new op lower bound from 7022624348011700224 to 7022624310852739072. Current Snapshot: MvccSnapshot[applied={T|T < 7022624348011700224 or (T in {7022624348011700224})}]
W20240430 20:11:44.197412 18557 leader_election.cc:341] T abdaf243e5be4ad58036b23151b14fe4 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
W20240430 20:11:44.198022 18557 leader_election.cc:341] T abdaf243e5be4ad58036b23151b14fe4 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:44.199127 18615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd0e8c360734dcbb75dc2958b70c2b8" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:44.201597 20384 consensus_queue.cc:1035] T a59ef6c545dd41a28265a4c1b8ce10e0 P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:44.196748 18985 ts_tablet_manager.cc:1398] T 2e79df8744e74b5991a79ce978ff0202 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.333s user 0.001s sys 0.001s
I20240430 20:11:44.204550 18985 tablet_bootstrap.cc:492] T d61c279d20b0471a967942525ceb4fcb P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:44.209697 18696 leader_election.cc:302] T 3fd0e8c360734dcbb75dc2958b70c2b8 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710
I20240430 20:11:44.209837 18983 tablet_bootstrap.cc:492] T f940d17b02b541289121463e05b0bb5f P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:44.209990 18983 ts_tablet_manager.cc:1367] T f940d17b02b541289121463e05b0bb5f P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.274s user 0.045s sys 0.060s
I20240430 20:11:44.210325 20198 raft_consensus.cc:2726] T 3fd0e8c360734dcbb75dc2958b70c2b8 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:44.210811 18983 raft_consensus.cc:348] T f940d17b02b541289121463e05b0bb5f P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.211112 18983 raft_consensus.cc:374] T f940d17b02b541289121463e05b0bb5f P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:44.211212 18983 raft_consensus.cc:729] T f940d17b02b541289121463e05b0bb5f P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:44.211293 18985 tablet_bootstrap.cc:654] T d61c279d20b0471a967942525ceb4fcb P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:44.211477 18983 consensus_queue.cc:260] T f940d17b02b541289121463e05b0bb5f P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.212059 18983 ts_tablet_manager.cc:1398] T f940d17b02b541289121463e05b0bb5f P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240430 20:11:44.212661 18983 tablet_bootstrap.cc:492] T 2da900b6bbfa4fa99fe8494fd2e68da8 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:44.214812 18983 tablet_bootstrap.cc:654] T 2da900b6bbfa4fa99fe8494fd2e68da8 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:44.219729 19495 ts_tablet_manager.cc:1888] T ae87f5e92a934e2182164c231bc76df0 P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:44.220666 20198 raft_consensus.cc:482] T ebf74809b52f47f3b0ed21b6ac0f541a P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:44.220960 20198 raft_consensus.cc:504] T ebf74809b52f47f3b0ed21b6ac0f541a P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.222582 20198 leader_election.cc:288] T ebf74809b52f47f3b0ed21b6ac0f541a P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:44.222815 19495 log.cc:1196] T ae87f5e92a934e2182164c231bc76df0 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/ae87f5e92a934e2182164c231bc76df0
I20240430 20:11:44.222726 18478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebf74809b52f47f3b0ed21b6ac0f541a" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:44.223181 18619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebf74809b52f47f3b0ed21b6ac0f541a" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:44.223414 18983 tablet_bootstrap.cc:492] T 2da900b6bbfa4fa99fe8494fd2e68da8 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:44.223639 18983 ts_tablet_manager.cc:1367] T 2da900b6bbfa4fa99fe8494fd2e68da8 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20240430 20:11:44.223678 18699 leader_election.cc:302] T ebf74809b52f47f3b0ed21b6ac0f541a P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710
I20240430 20:11:44.223953 20198 raft_consensus.cc:2726] T ebf74809b52f47f3b0ed21b6ac0f541a P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:44.224261 18988 tablet_bootstrap.cc:492] T a59ef6c545dd41a28265a4c1b8ce10e0 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:44.224407 19495 ts_tablet_manager.cc:1909] T ae87f5e92a934e2182164c231bc76df0 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:44.224599 18983 raft_consensus.cc:348] T 2da900b6bbfa4fa99fe8494fd2e68da8 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.224972 18983 raft_consensus.cc:374] T 2da900b6bbfa4fa99fe8494fd2e68da8 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:44.225106 18983 raft_consensus.cc:729] T 2da900b6bbfa4fa99fe8494fd2e68da8 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:44.225366 18983 consensus_queue.cc:260] T 2da900b6bbfa4fa99fe8494fd2e68da8 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.225981 18988 ts_tablet_manager.cc:1367] T a59ef6c545dd41a28265a4c1b8ce10e0 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.045s user 0.004s sys 0.001s
I20240430 20:11:44.226155 18983 ts_tablet_manager.cc:1398] T 2da900b6bbfa4fa99fe8494fd2e68da8 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240430 20:11:44.226889 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 282) took 2020 ms. Trace:
I20240430 20:11:44.226863 18988 raft_consensus.cc:348] T a59ef6c545dd41a28265a4c1b8ce10e0 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.227816 18985 tablet_bootstrap.cc:492] T d61c279d20b0471a967942525ceb4fcb P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:44.228668 18985 ts_tablet_manager.cc:1367] T d61c279d20b0471a967942525ceb4fcb P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.002s
I20240430 20:11:44.229341 18983 tablet_bootstrap.cc:492] T 7683caba80594627900a0185f772c131 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:44.229348 18985 raft_consensus.cc:348] T d61c279d20b0471a967942525ceb4fcb P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.229547 18985 raft_consensus.cc:374] T d61c279d20b0471a967942525ceb4fcb P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:44.229682 18985 raft_consensus.cc:729] T d61c279d20b0471a967942525ceb4fcb P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:44.229943 18985 consensus_queue.cc:260] T d61c279d20b0471a967942525ceb4fcb P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.230634 18985 ts_tablet_manager.cc:1398] T d61c279d20b0471a967942525ceb4fcb P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240430 20:11:44.230973 18985 tablet_bootstrap.cc:492] T 30a521916bda46678be20a066bd1d3f3 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:44.011936 19471 rpcz_store.cc:276] 0430 20:11:33.822054 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:33.822203 (+ 149us) service_pool.cc:225] Handling call
0430 20:11:43.768598 (+9946395us) ts_tablet_manager.cc:1120] Deleting tablet a745334c12c545c1ba825c7552c60359
0430 20:11:43.768612 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:43.989827 (+221215us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:43.997329 (+ 7502us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:43.999230 (+ 1901us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":9681904}
I20240430 20:11:44.232791 18985 tablet_bootstrap.cc:654] T 30a521916bda46678be20a066bd1d3f3 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:44.232798 18988 raft_consensus.cc:374] T a59ef6c545dd41a28265a4c1b8ce10e0 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:44.232875 18988 raft_consensus.cc:729] T a59ef6c545dd41a28265a4c1b8ce10e0 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:44.233747 18988 consensus_queue.cc:260] T a59ef6c545dd41a28265a4c1b8ce10e0 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.234279 19495 rpcz_store.cc:276] 0430 20:11:42.206235 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:42.206318 (+ 83us) service_pool.cc:225] Handling call
0430 20:11:44.192869 (+1986551us) ts_tablet_manager.cc:1120] Deleting tablet ae87f5e92a934e2182164c231bc76df0
0430 20:11:44.192880 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:44.209023 (+ 16143us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:44.219613 (+ 10590us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:44.226864 (+ 7251us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1974786}
I20240430 20:11:44.234853 19004 catalog_manager.cc:4830] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): tablet ae87f5e92a934e2182164c231bc76df0 (table default.Table_27f3d711beeb4387a9088ab291168bc8 [id=434aba8e131342a3a9a1aca3278f9c0d]) successfully deleted
I20240430 20:11:44.235215 18988 ts_tablet_manager.cc:1398] T a59ef6c545dd41a28265a4c1b8ce10e0 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240430 20:11:44.235204 18983 tablet_bootstrap.cc:654] T 7683caba80594627900a0185f772c131 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:44.236111 19471 tablet_replica.cc:330] T 50486644ec234933b9a0b45ee39a6685 P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:44.236466 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 283) took 2028 ms. Trace:
I20240430 20:11:44.236517 18999 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet a745334c12c545c1ba825c7552c60359 (table default.Table_36adeb442f1245aba00549559bb329f3 [id=4ab2e4b038744db8b2cc94c26dbcc35f]) successfully deleted
I20240430 20:11:44.236618 19495 rpcz_store.cc:276] 0430 20:11:42.207710 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:42.207762 (+ 52us) service_pool.cc:225] Handling call
0430 20:11:44.236308 (+2028546us) ts_tablet_manager.cc:1120] Deleting tablet fdf53061efd74fa29a9d8208d1fd210e
0430 20:11:44.236320 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:44.236445 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2018196}
W20240430 20:11:44.237216 19004 catalog_manager.cc:4810] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): delete failed for tablet fdf53061efd74fa29a9d8208d1fd210e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fdf53061efd74fa29a9d8208d1fd210e already in progress: creating tablet
I20240430 20:11:44.237896 19471 raft_consensus.cc:2229] T 50486644ec234933b9a0b45ee39a6685 P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Raft consensus shutting down.
I20240430 20:11:44.238833 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 286) took 1798 ms. Trace:
I20240430 20:11:44.239015 19495 rpcz_store.cc:276] 0430 20:11:42.440754 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:42.440813 (+ 59us) service_pool.cc:225] Handling call
0430 20:11:44.238691 (+1797878us) ts_tablet_manager.cc:1120] Deleting tablet 73453a7fa30342e29db552b17b62f0c1
0430 20:11:44.238706 (+ 15us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:44.238809 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1797661}
I20240430 20:11:44.239372 19471 pending_rounds.cc:62] T 50486644ec234933b9a0b45ee39a6685 P 5e1c6397f09747f091534faf93542710: Trying to abort 1 pending ops.
I20240430 20:11:44.235378 19489 ts_tablet_manager.cc:1875] T c2b019f529a54228b804437bb025e31d P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240430 20:11:44.240083 19004 catalog_manager.cc:4797] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): delete failed for tablet 73453a7fa30342e29db552b17b62f0c1 because the tablet was not found. No further retry: Not found: Tablet not found: 73453a7fa30342e29db552b17b62f0c1
I20240430 20:11:44.240466 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 288) took 1799 ms. Trace:
I20240430 20:11:44.243693 19495 rpcz_store.cc:276] 0430 20:11:42.441247 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:42.441283 (+ 36us) service_pool.cc:225] Handling call
0430 20:11:44.240363 (+1799080us) ts_tablet_manager.cc:1120] Deleting tablet fcc9698abda4491b98e1e879589913a8
0430 20:11:44.240373 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:44.240451 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1798935}
I20240430 20:11:44.244429 19471 pending_rounds.cc:68] T 50486644ec234933b9a0b45ee39a6685 P 5e1c6397f09747f091534faf93542710: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7022624374473142272 op_type: NO_OP noop_request { }
I20240430 20:11:44.244921 19471 raft_consensus.cc:2866] T 50486644ec234933b9a0b45ee39a6685 P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
W20240430 20:11:44.245178 19004 catalog_manager.cc:4797] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): delete failed for tablet fcc9698abda4491b98e1e879589913a8 because the tablet was not found. No further retry: Not found: Tablet not found: fcc9698abda4491b98e1e879589913a8
I20240430 20:11:44.245653 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 287) took 1804 ms. Trace:
I20240430 20:11:44.245775 19489 ts_tablet_manager.cc:1888] T c2b019f529a54228b804437bb025e31d P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:44.245795 19495 rpcz_store.cc:276] 0430 20:11:42.441162 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:42.441457 (+ 295us) service_pool.cc:225] Handling call
0430 20:11:44.245527 (+1804070us) ts_tablet_manager.cc:1120] Deleting tablet 6e826459c3f4437da81c9e6e09f0c22f
0430 20:11:44.245542 (+ 15us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:44.245631 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1803936}
I20240430 20:11:44.245954 19489 log.cc:1196] T c2b019f529a54228b804437bb025e31d P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/c2b019f529a54228b804437bb025e31d
I20240430 20:11:44.246436 19489 ts_tablet_manager.cc:1909] T c2b019f529a54228b804437bb025e31d P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:44.247294 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 153) took 11195 ms. Trace:
I20240430 20:11:44.247434 19489 rpcz_store.cc:276] 0430 20:11:33.052027 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:33.186685 (+134658us) service_pool.cc:225] Handling call
0430 20:11:43.969614 (+10782929us) ts_tablet_manager.cc:1120] Deleting tablet c2b019f529a54228b804437bb025e31d
0430 20:11:43.969627 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:44.167051 (+197424us) spinlock_profiling.cc:243] Waited 197 ms on lock 0x616000183a54. stack: 00007f30e7d63756 00007f30e7d63510 00007f30e6d9648d 0000000000955804 00000000009557cd 000000000095519a 00007f30f35d1a00 00007f30f48ab853 00007f30f733e4bc 00007f30f7370814 00007f30f7351b00 00007f30f735188d 0000000000a60bcc 00007f30e7da9b74 00007f30e7dad19b 00007f30e7dad018
0430 20:11:44.242556 (+ 75505us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:44.245654 (+ 3098us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:44.247270 (+ 1616us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":452496128,"tablet-delete.queue_time_us":10782619}
I20240430 20:11:44.248142 19001 catalog_manager.cc:4830] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): tablet c2b019f529a54228b804437bb025e31d (table default.Table_8028554bc993419e80dbf7d526b63bf4 [id=27ef5b3429fb4353afe80dc3089e9914]) successfully deleted
I20240430 20:11:44.248524 19489 tablet_replica.cc:330] T a745334c12c545c1ba825c7552c60359 P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:44.248756 19489 raft_consensus.cc:2229] T a745334c12c545c1ba825c7552c60359 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:44.248971 19489 raft_consensus.cc:2258] T a745334c12c545c1ba825c7552c60359 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240430 20:11:44.249405 19004 catalog_manager.cc:4797] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): delete failed for tablet 6e826459c3f4437da81c9e6e09f0c22f because the tablet was not found. No further retry: Not found: Tablet not found: 6e826459c3f4437da81c9e6e09f0c22f
I20240430 20:11:44.249792 18988 tablet_bootstrap.cc:492] T 73453a7fa30342e29db552b17b62f0c1 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:44.249825 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 290) took 1807 ms. Trace:
I20240430 20:11:44.249951 19495 rpcz_store.cc:276] 0430 20:11:42.441854 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:42.441903 (+ 49us) service_pool.cc:225] Handling call
0430 20:11:44.249713 (+1807810us) ts_tablet_manager.cc:1120] Deleting tablet 75ebf051383e42529dcb5f7dd0770a4b
0430 20:11:44.249725 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:44.249805 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1807673}
W20240430 20:11:44.250528 19004 catalog_manager.cc:4797] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): delete failed for tablet 75ebf051383e42529dcb5f7dd0770a4b because the tablet was not found. No further retry: Not found: Tablet not found: 75ebf051383e42529dcb5f7dd0770a4b
I20240430 20:11:44.250928 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 289) took 1809 ms. Trace:
I20240430 20:11:44.251053 19495 rpcz_store.cc:276] 0430 20:11:42.441748 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:42.442056 (+ 308us) service_pool.cc:225] Handling call
0430 20:11:44.250843 (+1808787us) ts_tablet_manager.cc:1120] Deleting tablet 2f61e3f88a6448dfa323204bd8d14cc5
0430 20:11:44.250853 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:44.250912 (+ 59us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1808688}
I20240430 20:11:44.251492 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 291) took 1809 ms. Trace:
I20240430 20:11:44.251627 19495 rpcz_store.cc:276] 0430 20:11:42.442323 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:42.442361 (+ 38us) service_pool.cc:225] Handling call
0430 20:11:44.251407 (+1809046us) ts_tablet_manager.cc:1120] Deleting tablet 7542c7fe6cad452ab718f9ef54dbbbbe
0430 20:11:44.251415 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:44.251475 (+ 60us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1808930}
W20240430 20:11:44.251708 19004 catalog_manager.cc:4797] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): delete failed for tablet 2f61e3f88a6448dfa323204bd8d14cc5 because the tablet was not found. No further retry: Not found: Tablet not found: 2f61e3f88a6448dfa323204bd8d14cc5
I20240430 20:11:44.252055 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 292) took 1809 ms. Trace:
I20240430 20:11:44.252166 19495 rpcz_store.cc:276] 0430 20:11:42.442617 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:42.442651 (+ 34us) service_pool.cc:225] Handling call
0430 20:11:44.251975 (+1809324us) ts_tablet_manager.cc:1120] Deleting tablet 87ac32a0ddb84676adc83038ef664591
0430 20:11:44.251983 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:44.252041 (+ 58us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1809199}
W20240430 20:11:44.252271 19004 catalog_manager.cc:4797] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): delete failed for tablet 7542c7fe6cad452ab718f9ef54dbbbbe because the tablet was not found. No further retry: Not found: Tablet not found: 7542c7fe6cad452ab718f9ef54dbbbbe
W20240430 20:11:44.252601 19004 catalog_manager.cc:4797] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): delete failed for tablet 87ac32a0ddb84676adc83038ef664591 because the tablet was not found. No further retry: Not found: Tablet not found: 87ac32a0ddb84676adc83038ef664591
I20240430 20:11:44.252617 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 293) took 1809 ms. Trace:
I20240430 20:11:44.252696 19495 rpcz_store.cc:276] 0430 20:11:42.443002 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:42.443048 (+ 46us) service_pool.cc:225] Handling call
0430 20:11:44.252539 (+1809491us) ts_tablet_manager.cc:1120] Deleting tablet c2b019f529a54228b804437bb025e31d
0430 20:11:44.252548 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:44.252602 (+ 54us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1809362}
W20240430 20:11:44.253082 19004 catalog_manager.cc:4797] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): delete failed for tablet c2b019f529a54228b804437bb025e31d because the tablet was not found. No further retry: Not found: Tablet not found: c2b019f529a54228b804437bb025e31d
I20240430 20:11:44.246271 18983 tablet_bootstrap.cc:492] T 7683caba80594627900a0185f772c131 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:44.253574 18988 tablet_bootstrap.cc:654] T 73453a7fa30342e29db552b17b62f0c1 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:44.253605 19471 raft_consensus.cc:2258] T 50486644ec234933b9a0b45ee39a6685 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:44.253682 18983 ts_tablet_manager.cc:1367] T 7683caba80594627900a0185f772c131 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.027s user 0.008s sys 0.004s
I20240430 20:11:44.253908 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 305) took 1574 ms. Trace:
I20240430 20:11:44.253973 19495 rpcz_store.cc:276] 0430 20:11:42.679404 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:42.679577 (+ 173us) service_pool.cc:225] Handling call
0430 20:11:44.253793 (+1574216us) ts_tablet_manager.cc:1120] Deleting tablet dad16ae02b8b4e29a724d8eec67bb4b8
0430 20:11:44.253802 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:44.253894 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1574028}
W20240430 20:11:44.254357 19004 catalog_manager.cc:4810] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): delete failed for tablet dad16ae02b8b4e29a724d8eec67bb4b8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dad16ae02b8b4e29a724d8eec67bb4b8 already in progress: creating tablet
I20240430 20:11:44.255336 18985 tablet_bootstrap.cc:492] T 30a521916bda46678be20a066bd1d3f3 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:44.255467 18985 ts_tablet_manager.cc:1367] T 30a521916bda46678be20a066bd1d3f3 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.000s
I20240430 20:11:44.255525 19471 ts_tablet_manager.cc:1875] T 50486644ec234933b9a0b45ee39a6685 P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:44.256139 18985 raft_consensus.cc:348] T 30a521916bda46678be20a066bd1d3f3 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.256323 18985 raft_consensus.cc:374] T 30a521916bda46678be20a066bd1d3f3 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:44.256376 18985 raft_consensus.cc:729] T 30a521916bda46678be20a066bd1d3f3 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:44.256610 18985 consensus_queue.cc:260] T 30a521916bda46678be20a066bd1d3f3 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.257366 18985 ts_tablet_manager.cc:1398] T 30a521916bda46678be20a066bd1d3f3 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240430 20:11:44.257652 18985 tablet_bootstrap.cc:492] T f940d17b02b541289121463e05b0bb5f P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:44.251919 19489 ts_tablet_manager.cc:1875] T a745334c12c545c1ba825c7552c60359 P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:44.254459 18983 raft_consensus.cc:348] T 7683caba80594627900a0185f772c131 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.259269 18983 raft_consensus.cc:374] T 7683caba80594627900a0185f772c131 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:44.259338 18983 raft_consensus.cc:729] T 7683caba80594627900a0185f772c131 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:44.259557 18983 consensus_queue.cc:260] T 7683caba80594627900a0185f772c131 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.260066 18983 ts_tablet_manager.cc:1398] T 7683caba80594627900a0185f772c131 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240430 20:11:44.260310 18983 tablet_bootstrap.cc:492] T 99203cc522de424184c6f97453f166fd P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:44.263036 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 311) took 1517 ms. Trace:
I20240430 20:11:44.266811 20489 raft_consensus.cc:482] T 7ce09fcbe1ac457a9c05efaf000c3263 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:44.266978 20489 raft_consensus.cc:504] T 7ce09fcbe1ac457a9c05efaf000c3263 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.267542 19495 rpcz_store.cc:276] 0430 20:11:42.745180 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:42.745242 (+ 62us) service_pool.cc:225] Handling call
0430 20:11:44.262879 (+1517637us) ts_tablet_manager.cc:1120] Deleting tablet 7683caba80594627900a0185f772c131
0430 20:11:44.262891 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:44.263010 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1517417}
I20240430 20:11:44.267711 18983 tablet_bootstrap.cc:654] T 99203cc522de424184c6f97453f166fd P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
W20240430 20:11:44.268134 19004 catalog_manager.cc:4810] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): delete failed for tablet 7683caba80594627900a0185f772c131 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7683caba80594627900a0185f772c131 already in progress: creating tablet
I20240430 20:11:44.268185 18758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ce09fcbe1ac457a9c05efaf000c3263" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:44.268136 20487 raft_consensus.cc:482] T 5bd0362ff41b4faca0205d136d596737 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:44.268448 20487 raft_consensus.cc:504] T 5bd0362ff41b4faca0205d136d596737 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
W20240430 20:11:44.268735 18557 leader_election.cc:341] T 7ce09fcbe1ac457a9c05efaf000c3263 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:44.268867 18985 tablet_bootstrap.cc:654] T f940d17b02b541289121463e05b0bb5f P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:44.269516 20489 leader_election.cc:288] T 7ce09fcbe1ac457a9c05efaf000c3263 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:44.269619 18758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bd0362ff41b4faca0205d136d596737" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:44.269799 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 312) took 1356 ms. Trace:
I20240430 20:11:44.269876 19495 rpcz_store.cc:276] 0430 20:11:42.912915 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:42.913197 (+ 282us) service_pool.cc:225] Handling call
0430 20:11:44.269659 (+1356462us) ts_tablet_manager.cc:1120] Deleting tablet 2797eed593fb4b06a3c4d6a28e8075a2
0430 20:11:44.269669 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:44.269779 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1356260}
I20240430 20:11:44.269958 18482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ce09fcbe1ac457a9c05efaf000c3263" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:44.270160 18482 raft_consensus.cc:2454] T 7ce09fcbe1ac457a9c05efaf000c3263 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 0.
W20240430 20:11:44.270210 18557 leader_election.cc:341] T 5bd0362ff41b4faca0205d136d596737 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
W20240430 20:11:44.270313 19004 catalog_manager.cc:4810] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): delete failed for tablet 2797eed593fb4b06a3c4d6a28e8075a2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2797eed593fb4b06a3c4d6a28e8075a2 already in progress: creating tablet
I20240430 20:11:44.270474 20487 leader_election.cc:288] T 5bd0362ff41b4faca0205d136d596737 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:44.270689 19495 tablet_replica.cc:330] T a05e32a33aa7488eb287cc6314c09c44 P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:44.270933 19495 raft_consensus.cc:2229] T a05e32a33aa7488eb287cc6314c09c44 P 2efb542aa55e439b99ccb98d1c5c884a [term 2 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:44.270882 18482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bd0362ff41b4faca0205d136d596737" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:44.271134 18482 raft_consensus.cc:2454] T 5bd0362ff41b4faca0205d136d596737 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 0.
I20240430 20:11:44.271126 18558 leader_election.cc:302] T 7ce09fcbe1ac457a9c05efaf000c3263 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters: 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:44.271615 19495 raft_consensus.cc:2258] T a05e32a33aa7488eb287cc6314c09c44 P 2efb542aa55e439b99ccb98d1c5c884a [term 2 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:44.271598 18558 leader_election.cc:302] T 5bd0362ff41b4faca0205d136d596737 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters: 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:44.271891 20487 raft_consensus.cc:2781] T 7ce09fcbe1ac457a9c05efaf000c3263 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:44.272212 20489 raft_consensus.cc:2781] T 5bd0362ff41b4faca0205d136d596737 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:44.272284 20487 raft_consensus.cc:482] T 7ce09fcbe1ac457a9c05efaf000c3263 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:44.272342 20489 raft_consensus.cc:482] T 5bd0362ff41b4faca0205d136d596737 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:44.272629 20489 raft_consensus.cc:3037] T 5bd0362ff41b4faca0205d136d596737 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:44.275100 19495 ts_tablet_manager.cc:1875] T a05e32a33aa7488eb287cc6314c09c44 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:44.272382 20487 raft_consensus.cc:3037] T 7ce09fcbe1ac457a9c05efaf000c3263 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:44.277765 20487 raft_consensus.cc:504] T 7ce09fcbe1ac457a9c05efaf000c3263 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.278779 20487 leader_election.cc:288] T 7ce09fcbe1ac457a9c05efaf000c3263 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:44.279341 18758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ce09fcbe1ac457a9c05efaf000c3263" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:44.279433 18482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ce09fcbe1ac457a9c05efaf000c3263" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:44.279598 18482 raft_consensus.cc:3037] T 7ce09fcbe1ac457a9c05efaf000c3263 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
W20240430 20:11:44.280092 18557 leader_election.cc:341] T 7ce09fcbe1ac457a9c05efaf000c3263 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:44.282307 18482 raft_consensus.cc:2454] T 7ce09fcbe1ac457a9c05efaf000c3263 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
I20240430 20:11:44.282783 18983 tablet_bootstrap.cc:492] T 99203cc522de424184c6f97453f166fd P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:44.282999 18558 leader_election.cc:302] T 7ce09fcbe1ac457a9c05efaf000c3263 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters: 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:44.283187 18983 ts_tablet_manager.cc:1367] T 99203cc522de424184c6f97453f166fd P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.023s user 0.005s sys 0.002s
I20240430 20:11:44.278061 20489 raft_consensus.cc:504] T 5bd0362ff41b4faca0205d136d596737 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.284127 20489 leader_election.cc:288] T 5bd0362ff41b4faca0205d136d596737 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:44.284701 18482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bd0362ff41b4faca0205d136d596737" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:44.284978 18482 raft_consensus.cc:3037] T 5bd0362ff41b4faca0205d136d596737 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:44.285188 18985 tablet_bootstrap.cc:492] T f940d17b02b541289121463e05b0bb5f P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:44.285363 18985 ts_tablet_manager.cc:1367] T f940d17b02b541289121463e05b0bb5f P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.028s user 0.000s sys 0.003s
I20240430 20:11:44.285696 18758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bd0362ff41b4faca0205d136d596737" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:44.286198 18985 raft_consensus.cc:348] T f940d17b02b541289121463e05b0bb5f P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.286381 18985 raft_consensus.cc:374] T f940d17b02b541289121463e05b0bb5f P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:44.286425 18985 raft_consensus.cc:729] T f940d17b02b541289121463e05b0bb5f P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
W20240430 20:11:44.286414 18557 leader_election.cc:341] T 5bd0362ff41b4faca0205d136d596737 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:44.286568 19048 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 3a03b6997921408cb5a65f4cae223e9c (table default.Table_744b5c96ea014bdb8feae740356a790e [id=83b1ac2e9c3d4a10b02f96ebb17faf44]). Expected version 1 got 0
I20240430 20:11:44.286639 18985 consensus_queue.cc:260] T f940d17b02b541289121463e05b0bb5f P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.286810 19048 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet ae20f44df50343289b0a512a4ea53692 (table default.Table_1844f6b1bda84812825384e14a2fd94e [id=71da9acc3cff4d95bbac4a686b708fd5]). Expected version 1 got 0
I20240430 20:11:44.287161 19048 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet eb6c96047b114ebba73db7a371f2da8e (table default.Table_7d5f6f18cbd64c869739872a7dff9be2 [id=e67d619e36c0485f8f22b4a81cb4cb6c]). Expected version 1 got 0
I20240430 20:11:44.287328 19048 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 62732af95c6a4c749c5b726f0460dc72 (table default.Table_e3160f5ced1b4718b479401ca4df675c [id=2d44ec94131148048021b2ee6366ba30]). Expected version 3 got 0
I20240430 20:11:44.287475 18985 ts_tablet_manager.cc:1398] T f940d17b02b541289121463e05b0bb5f P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240430 20:11:44.287573 18482 raft_consensus.cc:2454] T 5bd0362ff41b4faca0205d136d596737 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
I20240430 20:11:44.287652 19048 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet a0baee91f5b44945b595d0e234af018e (table default.Table_6a30a0abb67d4a298f87216515a92153 [id=b3d1c2f97c4648518dbea83a46cadd85]). Expected version 1 got 0
I20240430 20:11:44.287742 18985 tablet_bootstrap.cc:492] T 7683caba80594627900a0185f772c131 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:44.287788 19048 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 266fd23786bc435db5083370f3330ed0 (table default.Table_cf60b652e9dc49e7a948ce66fd1cbffe [id=9734bb8dd05a400ca4244b3829f01dee]). Expected version 1 got 0
I20240430 20:11:44.287947 19048 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet ed0c37a71ede4bfc9eb5ad1a4365f25e (table default.Table_a2e85efc6b454524b0b122102ec07002 [id=86d795a82e674d859104741ebd04f4e8]). Expected version 1 got 0
I20240430 20:11:44.287963 18558 leader_election.cc:302] T 5bd0362ff41b4faca0205d136d596737 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters: 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:44.288101 19048 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 3e5225f1f5f544ecbf8dfa470af3daca (table default.Table_c7c87a8e019944cab3ec99e9563ac01a [id=abf3765074eb4d7d9d2bcc4ab721e6bd]). Expected version 1 got 0
I20240430 20:11:44.288388 20488 raft_consensus.cc:2781] T 7ce09fcbe1ac457a9c05efaf000c3263 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:44.288503 20488 raft_consensus.cc:686] T 7ce09fcbe1ac457a9c05efaf000c3263 P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Becoming Leader. State: Replica: 5e1c6397f09747f091534faf93542710, State: Running, Role: LEADER
I20240430 20:11:44.288525 19048 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 1701a15b115741aea13f8327e3d26b34 (table default.Table_3776a4b9d4f24aa6b7f882bb4d0116ef [id=7127dbd368864f7ba80a755549fe7ede]). Expected version 1 got 0
I20240430 20:11:44.288738 20488 consensus_queue.cc:237] T 7ce09fcbe1ac457a9c05efaf000c3263 P 5e1c6397f09747f091534faf93542710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.289031 19048 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 12795670f21447a8964fe1efc6e92398 (table default.Table_896bbae1f6944005b0272ca8ebc97218 [id=d5cea90cbc2c48df9d8b80da1eccc949]). Expected version 3 got 0
I20240430 20:11:44.289100 18985 tablet_bootstrap.cc:654] T 7683caba80594627900a0185f772c131 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:44.289386 19048 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet a59ef6c545dd41a28265a4c1b8ce10e0 (table default.Table_07e40af8f1c546048975c8edf88170d7 [id=86df5d161d8641c89070e7eed9a39360]). Expected version 2 got 0
I20240430 20:11:44.289525 20047 raft_consensus.cc:2781] T 5bd0362ff41b4faca0205d136d596737 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:44.289816 20047 raft_consensus.cc:686] T 5bd0362ff41b4faca0205d136d596737 P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Becoming Leader. State: Replica: 5e1c6397f09747f091534faf93542710, State: Running, Role: LEADER
I20240430 20:11:44.289956 19048 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet f5eaf2413e1d4e69b56fae61f768f0ce (table default.Table_1280cfe32d5b41abb4d3eb1bd70641bf [id=8489b872074a42b6abb713e420968141]). Expected version 1 got 0
I20240430 20:11:44.290033 20047 consensus_queue.cc:237] T 5bd0362ff41b4faca0205d136d596737 P 5e1c6397f09747f091534faf93542710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.291643 19048 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 66dd767bd3f8494f94061fb89e5e5680 (table default.Table_b064d5659f7243f5b0ec167fb6c3f404 [id=5227206f44d04fa89b4842b899f33afb]). Expected version 2 got 0
I20240430 20:11:44.292207 19048 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 22fef8bb6b034db098c3372ae992f7a7 (table default.Table_4440df38610340758047115845b42b95 [id=7d186ca20c094cc19aef792fe1c094e9]). Expected version 1 got 0
I20240430 20:11:44.292483 19048 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 086b3dfa84c045a18c8c4f177dce606a (table default.Table_df19cb8a6f4d46699350ec5b69c14e5a [id=197923633fc54a4582207942b51530e2]). Expected version 2 got 0
I20240430 20:11:44.292644 19048 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet dbb427aefa6b47b0805d1751fea4efcb (table default.Table_4130caf534004edfb413056d7f6cfd52 [id=da9ad991953a40c4b77b31317597ad1d]). Expected version 4 got 0
I20240430 20:11:44.292912 19048 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet e041756c962b4a9f97c2cb314e7d8c05 (table default.Table_de88cafd92b149f9b545369bff71033c [id=c4bf4806f3d84579b16a3038d8d982c3]). Expected version 1 got 0
I20240430 20:11:44.293509 19048 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 488d0e609b224674932f890939664df2 (table default.Table_58f35a2120874472978ca0e5f816e86c [id=ff18001c6d814afc8c1768d0c84a6bac]). Expected version 1 got 0
I20240430 20:11:44.294126 19048 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 69f88532952a4591bf1577541c3430e6 (table default.Table_1177828cad1f4a24bd397619dad96170 [id=a95fed1bc3fa4865bf9a35dfc51b659c]). Expected version 2 got 0
I20240430 20:11:44.295898 19048 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet acd23f0a63d5448aaa7ed78422e67bad (table default.Table_972f13be4dca4cd19b9a04534dc59f51 [id=69f37db4de634a3185613e40013e0ed7]). Expected version 1 got 0
I20240430 20:11:44.284590 18983 raft_consensus.cc:348] T 99203cc522de424184c6f97453f166fd P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:44.296223 18983 raft_consensus.cc:374] T 99203cc522de424184c6f97453f166fd P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:44.296342 18983 raft_consensus.cc:729] T 99203cc522de424184c6f97453f166fd P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:44.296684 18983 consensus_queue.cc:260] T 99203cc522de424184c6f97453f166fd P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:44.297319 18983 ts_tablet_manager.cc:1398] T 99203cc522de424184c6f97453f166fd P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20240430 20:11:44.297591 19048 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 2e79df8744e74b5991a79ce978ff0202 (table default.Table_276bbdc7cfe542d1a2936485a2aad1ec [id=f146a15e6e8a4acbb4c734c1c863da57]). Expected version 1 got 0
I20240430 20:11:44.297964 18983 tablet_bootstrap.cc:492] T ae20f44df50343289b0a512a4ea53692 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:44.298308 19048 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 1050ab528c7d41559831652a1437d358 (table default.Table_acddc4cbf6334546b5fb3e4752fef2a5 [id=274cb15fd751410e83327ac4e8dc4629]). Expected version 2 got 0
I20240430 20:11:44.300206 18983 tablet_bootstrap.cc:654] T ae20f44df50343289b0a512a4ea53692 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:44.300560 19048 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 372b135dd01f44f98cd713ca8857ba61 (table default.Table_0163ce6f5a4c45299106daeffb3c22b1 [id=7f3a15a442984827a7521d5d1c98dc3d]). Expected version 3 got 2
I20240430 20:11:44.300859 19495 ts_tablet_manager.cc:1888] T a05e32a33aa7488eb287cc6314c09c44 P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.3
I20240430 20:11:44.301029 19048 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 76d6e9d547ca4a848eae852944b1d117 (table default.Table_104ae015924a48fd8d27220792f07470 [id=ff86c0357c164dd0991098c416087773]). Expected version 2 got 0
I20240430 20:11:44.302052 19048 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet fbdeb4714b8c4b93ab72822c43329aae (table default.Table_c12ca6938d424fbfa2e1df5cdb00fdbc [id=0fb93f36bc17408ea56bc4cafcfd8580]). Expected version 1 got 0
I20240430 20:11:44.303957 19495 log.cc:1196] T a05e32a33aa7488eb287cc6314c09c44 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/a05e32a33aa7488eb287cc6314c09c44
I20240430 20:11:44.307750 19048 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 772eba5f24c549c791cad75565858285 (table default.Table_5fc77d5f60ae44f4bb4ca9f7f183672c [id=6ca223ca6354449c959d85ed2ac6a9c8]). Expected version 1 got 0
I20240430 20:11:44.308276 19048 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet c6e3d09402574f1e8fe9e4160c70a3ee (table default.Table_656db97355f64d7c8aa29b1352dfe9c8 [id=450f5d1c7e084a1bb88d035c0ccfcf82]). Expected version 1 got 0
I20240430 20:11:44.308598 19048 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 1462ecf1176c4e66be42abddbe36b00c (table default.Table_b49a7942aa8c4f58acb94c35d1cdbb70 [id=6257559b0e9545c58848a76af0c5a5f7]). Expected version 1 got 0
I20240430 20:11:44.308637 19489 ts_tablet_manager.cc:1888] T a745334c12c545c1ba825c7552c60359 P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:44.308847 19489 log.cc:1196] T a745334c12c545c1ba825c7552c60359 P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/a745334c12c545c1ba825c7552c60359
I20240430 20:11:44.308971 19048 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 78ba48dadef548708153878810cc46f8 (table default.Table_b9983e101a6547649eb44ee04b5e1147 [id=29769be2b9ec4ecba9de5c0a65692f85]). Expected version 1 got 0
I20240430 20:11:44.309202 19048 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 47bf64ecc41b445da4ff697e141243dd (table default.Table_f1ad29d5986d4b799cfc67afb2b0dfdf [id=838fc8b61a864dea8877a6f144e4cb4f]). Expected version 2 got 0
I20240430 20:11:44.309466 19489 ts_tablet_manager.cc:1909] T a745334c12c545c1ba825c7552c60359 P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:44.309633 19048 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 331c031d44854d679a27d10d9151d0ea (table default.Table_763f0800e8bb4fa69976886416250f42 [id=61c8a5b99ef34a0baeeb343fef7cfd05]). Expected version 1 got 0
I20240430 20:11:44.309883 19048 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 7ce09fcbe1ac457a9c05efaf000c3263 (table default.Table_edc4c27ed846454288f73f6a0d88cb69 [id=78fd3f516083489ab3b42d90dcef7e4d]). Expected version 1 got 0
I20240430 20:11:44.309935 19495 ts_tablet_manager.cc:1909] T a05e32a33aa7488eb287cc6314c09c44 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:44.310144 19048 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet ebf74809b52f47f3b0ed21b6ac0f541a (table default.Table_b2d07520b9944a89ad62fca70396174e [id=1c8e0a3a44ec4a798770c0165ed3dffa]). Expected version 1 got 0
I20240430 20:11:44.310393 19048 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 0368d37dd9e2452982fb81f539c6805b (table default.Table_404b186c544e4b30bb36cb77d1a14120 [id=4330626e342a48699ea939add5c91b11]). Expected version 1 got 0
I20240430 20:11:44.310406 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 162) took 10497 ms. Trace:
I20240430 20:11:44.310488 19489 rpcz_store.cc:276] 0430 20:11:33.813219 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:33.817795 (+ 4576us) service_pool.cc:225] Handling call
0430 20:11:44.248470 (+10430675us) ts_tablet_manager.cc:1120] Deleting tablet a745334c12c545c1ba825c7552c60359
0430 20:11:44.248481 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:44.270930 (+ 22449us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:44.308121 (+ 37191us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:44.310377 (+ 2256us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":9794393}
I20240430 20:11:44.310992 19048 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 2da900b6bbfa4fa99fe8494fd2e68da8 (table default.Table_19a91a7f17ff456ebb145e8dd89c0940 [id=67283dd3995a44c6a7a7bc8722e4f955]). Expected version 1 got 0
I20240430 20:11:44.311753 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 316) took 1364 ms. Trace:
I20240430 20:11:44.310928 18985 tablet_bootstrap.cc:492] T 7683caba80594627900a0185f772c131 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:44.312032 18985 ts_tablet_manager.cc:1367] T 7683caba80594627900a0185f772c131 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.001s
I20240430 20:11:44.312705 18985 raft_consensus.cc:348] T 7683caba80594627900a0185f772c131 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.312950 18985 raft_consensus.cc:374] T 7683caba80594627900a0185f772c131 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:44.313046 18985 raft_consensus.cc:729] T 7683caba80594627900a0185f772c131 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:44.313336 18985 consensus_queue.cc:260] T 7683caba80594627900a0185f772c131 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.314761 19048 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 8e91920e7e36418d8c27d1f0bfd9b981 (table default.Table_e860191f401c434e8cf0085ef41952b9 [id=f393bcc9669c4ea985a2d40deb161f0c]). Expected version 1 got 0
I20240430 20:11:44.315480 18985 ts_tablet_manager.cc:1398] T 7683caba80594627900a0185f772c131 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240430 20:11:44.316782 18983 tablet_bootstrap.cc:492] T ae20f44df50343289b0a512a4ea53692 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:44.316993 18983 ts_tablet_manager.cc:1367] T ae20f44df50343289b0a512a4ea53692 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.000s
I20240430 20:11:44.317394 18985 tablet_bootstrap.cc:492] T 2da900b6bbfa4fa99fe8494fd2e68da8 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:44.311640 18988 tablet_bootstrap.cc:492] T 73453a7fa30342e29db552b17b62f0c1 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:44.317967 18983 raft_consensus.cc:348] T ae20f44df50343289b0a512a4ea53692 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.318225 18988 ts_tablet_manager.cc:1367] T 73453a7fa30342e29db552b17b62f0c1 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.068s user 0.004s sys 0.002s
I20240430 20:11:44.318238 18983 raft_consensus.cc:374] T ae20f44df50343289b0a512a4ea53692 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:44.318301 18983 raft_consensus.cc:729] T ae20f44df50343289b0a512a4ea53692 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:44.318604 18983 consensus_queue.cc:260] T ae20f44df50343289b0a512a4ea53692 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.318974 18985 tablet_bootstrap.cc:654] T 2da900b6bbfa4fa99fe8494fd2e68da8 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:44.319103 18983 ts_tablet_manager.cc:1398] T ae20f44df50343289b0a512a4ea53692 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240430 20:11:44.319420 18983 tablet_bootstrap.cc:492] T a9f37f0f7ce8425f9b24699513d41d61 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:44.319311 18988 raft_consensus.cc:348] T 73453a7fa30342e29db552b17b62f0c1 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.319485 18988 raft_consensus.cc:374] T 73453a7fa30342e29db552b17b62f0c1 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:44.319595 18988 raft_consensus.cc:729] T 73453a7fa30342e29db552b17b62f0c1 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:44.319888 18988 consensus_queue.cc:260] T 73453a7fa30342e29db552b17b62f0c1 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.320644 18988 ts_tablet_manager.cc:1398] T 73453a7fa30342e29db552b17b62f0c1 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240430 20:11:44.321017 18988 tablet_bootstrap.cc:492] T ae87f5e92a934e2182164c231bc76df0 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:44.321023 18983 tablet_bootstrap.cc:654] T a9f37f0f7ce8425f9b24699513d41d61 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:44.320902 19495 rpcz_store.cc:276] 0430 20:11:42.946958 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:42.947171 (+ 213us) service_pool.cc:225] Handling call
0430 20:11:44.270615 (+1323444us) ts_tablet_manager.cc:1120] Deleting tablet a05e32a33aa7488eb287cc6314c09c44
0430 20:11:44.270624 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:44.278068 (+ 7444us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:44.300730 (+ 22662us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:44.311731 (+ 11001us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1323211}
I20240430 20:11:44.322038 19048 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 99203cc522de424184c6f97453f166fd (table default.Table_cb2a4b9ceaf4431facfff2c9bcf94c5d [id=3f8690a437ae49a3beb009fa7899d5d0]). Expected version 1 got 0
I20240430 20:11:44.322365 19048 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 5b26319488644d6a8b6be60460a5e7af (table default.Table_69892a8264a4477db48ee6248758e046 [id=5a1c4a7be2fc465fac89a50854eb3322]). Expected version 3 got 0
I20240430 20:11:44.322685 19048 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 677481aee5fa4eb4b189322d7a3f8f2e (table default.Table_1ce204e15fce4c469816cf4e7f80cac7 [id=d268c31d82df4871bc1a78aea5871b33]). Expected version 1 got 0
I20240430 20:11:44.322736 18988 tablet_bootstrap.cc:654] T ae87f5e92a934e2182164c231bc76df0 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:44.323155 19048 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet a482f4622f774d60b4efd6827b8be7d7 (table default.Table_e686cdfcbcc046749fa5b51578a64794 [id=c3dc9f05316d46d6a3ef4d03f40877ec]). Expected version 1 got 0
I20240430 20:11:44.323525 19048 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 130007acc8ba4e709c3eb85fabe75903 (table default.Table_78987875650940c293247bc43db345ab [id=36f478a20a1f4a53a52fc29ade62b7f6]). Expected version 1 got 0
I20240430 20:11:44.323917 19001 catalog_manager.cc:4830] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): tablet a745334c12c545c1ba825c7552c60359 (table default.Table_36adeb442f1245aba00549559bb329f3 [id=4ab2e4b038744db8b2cc94c26dbcc35f]) successfully deleted
I20240430 20:11:44.324390 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 179) took 10184 ms. Trace:
I20240430 20:11:44.324525 19489 rpcz_store.cc:276] 0430 20:11:34.140137 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:34.140533 (+ 396us) service_pool.cc:225] Handling call
0430 20:11:44.324237 (+10183704us) ts_tablet_manager.cc:1120] Deleting tablet 06d28b6f48a34b17b0d7fb3102df2f4c
0430 20:11:44.324248 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:44.324373 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":9531768}
W20240430 20:11:44.325138 19001 catalog_manager.cc:4810] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): delete failed for tablet 06d28b6f48a34b17b0d7fb3102df2f4c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 06d28b6f48a34b17b0d7fb3102df2f4c already in progress: creating tablet
I20240430 20:11:44.326412 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 318) took 1377 ms. Trace:
I20240430 20:11:44.326510 19495 rpcz_store.cc:276] 0430 20:11:42.948621 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:42.948829 (+ 208us) service_pool.cc:225] Handling call
0430 20:11:44.326283 (+1377454us) ts_tablet_manager.cc:1120] Deleting tablet 84568f90b9c54cf292099808d4cbf5dc
0430 20:11:44.326295 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:44.326390 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1377214}
I20240430 20:11:44.326933 18988 tablet_bootstrap.cc:492] T ae87f5e92a934e2182164c231bc76df0 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:44.327121 18988 ts_tablet_manager.cc:1367] T ae87f5e92a934e2182164c231bc76df0 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20240430 20:11:44.327271 19495 tablet_replica.cc:330] T 2cdccf4be7094ca5b863dd5c3432d55d P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:44.327495 19495 raft_consensus.cc:2229] T 2cdccf4be7094ca5b863dd5c3432d55d P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:44.327656 19495 raft_consensus.cc:2258] T 2cdccf4be7094ca5b863dd5c3432d55d P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:44.328068 18988 raft_consensus.cc:348] T ae87f5e92a934e2182164c231bc76df0 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.328186 19495 ts_tablet_manager.cc:1875] T 2cdccf4be7094ca5b863dd5c3432d55d P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:44.328336 18988 raft_consensus.cc:374] T ae87f5e92a934e2182164c231bc76df0 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:44.328447 18988 raft_consensus.cc:729] T ae87f5e92a934e2182164c231bc76df0 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:44.328794 18758 raft_consensus.cc:3037] T 84568f90b9c54cf292099808d4cbf5dc P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:44.328728 18988 consensus_queue.cc:260] T ae87f5e92a934e2182164c231bc76df0 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.329303 18988 ts_tablet_manager.cc:1398] T ae87f5e92a934e2182164c231bc76df0 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240430 20:11:44.334247 19495 ts_tablet_manager.cc:1888] T 2cdccf4be7094ca5b863dd5c3432d55d P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:44.334379 19495 log.cc:1196] T 2cdccf4be7094ca5b863dd5c3432d55d P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/2cdccf4be7094ca5b863dd5c3432d55d
I20240430 20:11:44.338785 19489 tablet_replica.cc:330] T 0b4d9e860d7c45fd91d7a566605c9283 P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:44.339149 19489 raft_consensus.cc:2229] T 0b4d9e860d7c45fd91d7a566605c9283 P 3c51c33d5616482cafb68e26688820f4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:44.339372 19489 raft_consensus.cc:2258] T 0b4d9e860d7c45fd91d7a566605c9283 P 3c51c33d5616482cafb68e26688820f4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:44.339774 19004 catalog_manager.cc:4830] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): tablet a05e32a33aa7488eb287cc6314c09c44 (table default.Table_2d1cfc78363a47aa8dba7ef7d80f8c9a [id=955d833ff8f14bf6957abadf9ef669e2]) successfully deleted
I20240430 20:11:44.340039 19489 ts_tablet_manager.cc:1875] T 0b4d9e860d7c45fd91d7a566605c9283 P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240430 20:11:44.340286 19004 catalog_manager.cc:4797] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): delete failed for tablet 84568f90b9c54cf292099808d4cbf5dc because the tablet was not found. No further retry: Not found: Tablet not found: 84568f90b9c54cf292099808d4cbf5dc
I20240430 20:11:44.339767 19048 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet b285e3eb16d148779c74e057458d8461 (table default.Table_7fb60e4b56f444009ab5da219238eec7 [id=d1655c507b434f129ad7ae0b04365e51]). Expected version 1 got 0
I20240430 20:11:44.342898 18988 tablet_bootstrap.cc:492] T 5f02d8862e1f4ccfa9dbda73dbe18615 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:44.343079 19033 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: Sending DeleteTablet for 3 replicas of tablet c1017a50d1d646c486c0b9a75cb88ce4
I20240430 20:11:44.344090 19033 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_94da8b9b6d284bd28c2310417f57e095" } modify_external_catalogs: true
I20240430 20:11:44.344324 19033 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_94da8b9b6d284bd28c2310417f57e095" } modify_external_catalogs: true
I20240430 20:11:44.344444 18988 tablet_bootstrap.cc:654] T 5f02d8862e1f4ccfa9dbda73dbe18615 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:44.345860 18742 tablet_service.cc:1514] Processing DeleteTablet for tablet c1017a50d1d646c486c0b9a75cb88ce4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:43 UTC) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:44.346072 18588 tablet_service.cc:1514] Processing DeleteTablet for tablet c1017a50d1d646c486c0b9a75cb88ce4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:43 UTC) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:44.346472 18445 tablet_service.cc:1514] Processing DeleteTablet for tablet c1017a50d1d646c486c0b9a75cb88ce4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:43 UTC) from {username='slave'} at 127.0.0.1:41680
W20240430 20:11:44.346695 19051 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 94fbd4a57a1e438891fcb456f6b40be3 consensus state has no leader
I20240430 20:11:44.346894 19051 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:51664 (request call id 342) took 1019 ms. Trace:
I20240430 20:11:44.346976 19051 rpcz_store.cc:276] 0430 20:11:43.327448 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:43.497003 (+169555us) service_pool.cc:225] Handling call
0430 20:11:43.497558 (+ 555us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0430 20:11:43.497819 (+ 261us) catalog_manager.cc:3613] Apply alter schema
0430 20:11:43.498064 (+ 245us) catalog_manager.cc:3636] Acquired catalog manager lock
0430 20:11:43.498162 (+ 98us) catalog_manager.cc:3783] Updating metadata on disk
0430 20:11:43.613792 (+115630us) catalog_manager.cc:3814] Committing alterations to in-memory state
0430 20:11:44.346839 (+733047us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:43.509801 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:43.509908 (+ 107us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:43.509915 (+ 7us) write_op.cc:427] Acquired schema lock
0430 20:11:43.509919 (+ 4us) tablet.cc:660] Decoding operations
0430 20:11:43.510031 (+ 112us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:43.510040 (+ 9us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:43.510046 (+ 6us) tablet.cc:683] Acquiring locks for 1 operations
0430 20:11:43.510100 (+ 54us) tablet.cc:699] Row locks acquired
0430 20:11:43.510104 (+ 4us) write_op.cc:260] PREPARE: finished
0430 20:11:43.510132 (+ 28us) write_op.cc:270] Start()
0430 20:11:43.510160 (+ 28us) write_op.cc:276] Timestamp: P: 1714507903510110 usec, L: 0
0430 20:11:43.510166 (+ 6us) op_driver.cc:331] REPLICATION: starting
0430 20:11:43.541289 (+ 31123us) log.cc:842] Serialized 405 byte log entry
0430 20:11:43.603907 (+ 62618us) op_driver.cc:444] REPLICATION: finished
0430 20:11:43.604950 (+ 1043us) write_op.cc:301] APPLY: starting
0430 20:11:43.604963 (+ 13us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:43.604999 (+ 36us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:43.610853 (+ 5854us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:43.612311 (+ 1458us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:43.612349 (+ 38us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0430 20:11:43.612373 (+ 24us) write_op.cc:312] APPLY: finished
0430 20:11:43.612494 (+ 121us) log.cc:842] Serialized 23 byte log entry
0430 20:11:43.612549 (+ 55us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:43.612565 (+ 16us) write_op.cc:446] Released schema lock
0430 20:11:43.612572 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:43.612622 (+ 50us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:43.612625 (+ 3us) write_op.cc:446] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1008,"apply.run_cpu_time_us":538,"apply.run_wall_time_us":7721,"num_ops":1,"prepare.queue_time_us":10493,"prepare.run_cpu_time_us":704,"prepare.run_wall_time_us":62265,"replication_time_us":93729}]]}
I20240430 20:11:44.347895 19051 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_e4c7f4d2134445f380a007e589acd65d" } new_table_name: "default.Table_5f6e0fce999b46de927f9271e5de5071" modify_external_catalogs: true
I20240430 20:11:44.349161 19495 ts_tablet_manager.cc:1909] T 2cdccf4be7094ca5b863dd5c3432d55d P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:44.350154 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 319) took 1400 ms. Trace:
I20240430 20:11:44.350579 19471 ts_tablet_manager.cc:1888] T 50486644ec234933b9a0b45ee39a6685 P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:44.351037 19471 log.cc:1196] T 50486644ec234933b9a0b45ee39a6685 P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/50486644ec234933b9a0b45ee39a6685
I20240430 20:11:44.351395 19471 ts_tablet_manager.cc:1909] T 50486644ec234933b9a0b45ee39a6685 P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:44.352326 19489 ts_tablet_manager.cc:1888] T 0b4d9e860d7c45fd91d7a566605c9283 P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.3
I20240430 20:11:44.352666 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 175) took 9563 ms. Trace:
I20240430 20:11:44.352758 19471 rpcz_store.cc:276] 0430 20:11:34.789239 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:34.789333 (+ 94us) service_pool.cc:225] Handling call
0430 20:11:44.236052 (+9446719us) ts_tablet_manager.cc:1120] Deleting tablet 50486644ec234933b9a0b45ee39a6685
0430 20:11:44.236066 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:44.321822 (+ 85756us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:44.350461 (+ 28639us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:44.352640 (+ 2179us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":9446438}
I20240430 20:11:44.353116 19471 tablet_replica.cc:330] T c66c142419ca4e1399ecd1c8d1e62a59 P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:44.353294 19471 raft_consensus.cc:2229] T c66c142419ca4e1399ecd1c8d1e62a59 P 5e1c6397f09747f091534faf93542710 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:44.353493 19471 raft_consensus.cc:2258] T c66c142419ca4e1399ecd1c8d1e62a59 P 5e1c6397f09747f091534faf93542710 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:44.354472 18999 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet 50486644ec234933b9a0b45ee39a6685 (table default.Table_a2e85efc6b454524b0b122102ec07002 [id=86d795a82e674d859104741ebd04f4e8]) successfully deleted
I20240430 20:11:44.355365 19489 log.cc:1196] T 0b4d9e860d7c45fd91d7a566605c9283 P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/0b4d9e860d7c45fd91d7a566605c9283
I20240430 20:11:44.356405 19471 ts_tablet_manager.cc:1875] T c66c142419ca4e1399ecd1c8d1e62a59 P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:44.359601 18983 tablet_bootstrap.cc:492] T a9f37f0f7ce8425f9b24699513d41d61 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:44.359768 18983 ts_tablet_manager.cc:1367] T a9f37f0f7ce8425f9b24699513d41d61 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.040s user 0.003s sys 0.000s
I20240430 20:11:44.360535 18983 raft_consensus.cc:348] T a9f37f0f7ce8425f9b24699513d41d61 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
W20240430 20:11:44.360718 19051 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 37b2ea4be7494f0985808ade788dbe2b has no consensus state
I20240430 20:11:44.360738 18983 raft_consensus.cc:374] T a9f37f0f7ce8425f9b24699513d41d61 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:44.360791 18983 raft_consensus.cc:729] T a9f37f0f7ce8425f9b24699513d41d61 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:44.360992 18983 consensus_queue.cc:260] T a9f37f0f7ce8425f9b24699513d41d61 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.361485 18983 ts_tablet_manager.cc:1398] T a9f37f0f7ce8425f9b24699513d41d61 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240430 20:11:44.361773 18983 tablet_bootstrap.cc:492] T 06d28b6f48a34b17b0d7fb3102df2f4c P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:44.363204 19489 ts_tablet_manager.cc:1909] T 0b4d9e860d7c45fd91d7a566605c9283 P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:44.363143 19051 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_83d417131d35409582550b7b8ce4a141" } modify_external_catalogs: true
I20240430 20:11:44.363358 19051 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_83d417131d35409582550b7b8ce4a141" } modify_external_catalogs: true
I20240430 20:11:44.363986 18983 tablet_bootstrap.cc:654] T 06d28b6f48a34b17b0d7fb3102df2f4c P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:44.364128 19049 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: Sending DeleteTablet for 3 replicas of tablet 024069ab786749028c020abd1c836cf5
I20240430 20:11:44.364212 19028 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: Sending DeleteTablet for 3 replicas of tablet f6cfd089f08b47409fb2a6cf4f6b0385
I20240430 20:11:44.365036 19049 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_b99234d57c5c42d58e4a6516fc1482d4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240430 20:11:44.365089 18742 tablet_service.cc:1514] Processing DeleteTablet for tablet f6cfd089f08b47409fb2a6cf4f6b0385 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:44 UTC) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:44.365200 18733 tablet_service.cc:1514] Processing DeleteTablet for tablet 024069ab786749028c020abd1c836cf5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:44 UTC) from {username='slave'} at 127.0.0.1:59830
W20240430 20:11:44.365648 19049 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b99234d57c5c42d58e4a6516fc1482d4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:44.364998 19028 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_70ba80736dca466e9ca9c231a40e7bd4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240430 20:11:44.366039 19495 rpcz_store.cc:276] 0430 20:11:42.949286 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:42.949500 (+ 214us) service_pool.cc:225] Handling call
0430 20:11:44.327223 (+1377723us) ts_tablet_manager.cc:1120] Deleting tablet 2cdccf4be7094ca5b863dd5c3432d55d
0430 20:11:44.327233 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:44.331269 (+ 4036us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:44.334146 (+ 2877us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:44.350128 (+ 15982us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1377545}
W20240430 20:11:44.366287 19028 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_70ba80736dca466e9ca9c231a40e7bd4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:44.366927 18445 tablet_service.cc:1514] Processing DeleteTablet for tablet 024069ab786749028c020abd1c836cf5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:44 UTC) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:44.366508 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 320) took 1294 ms. Trace:
I20240430 20:11:44.365872 18450 tablet_service.cc:1514] Processing DeleteTablet for tablet f6cfd089f08b47409fb2a6cf4f6b0385 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:44 UTC) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:44.367591 19004 catalog_manager.cc:4830] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): tablet 2cdccf4be7094ca5b863dd5c3432d55d (table default.Table_f33a9e03642c44efbf1c27f1c7e70fb9 [id=c64cf189816b46cdb25574e5fddd9684]) successfully deleted
I20240430 20:11:44.368353 18985 tablet_bootstrap.cc:492] T 2da900b6bbfa4fa99fe8494fd2e68da8 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:44.368860 18588 tablet_service.cc:1514] Processing DeleteTablet for tablet f6cfd089f08b47409fb2a6cf4f6b0385 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:44 UTC) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:44.368947 18601 tablet_service.cc:1514] Processing DeleteTablet for tablet 024069ab786749028c020abd1c836cf5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:44 UTC) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:44.369144 18985 ts_tablet_manager.cc:1367] T 2da900b6bbfa4fa99fe8494fd2e68da8 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.053s user 0.002s sys 0.006s
I20240430 20:11:44.369339 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 180) took 10229 ms. Trace:
I20240430 20:11:44.367218 19495 rpcz_store.cc:276] 0430 20:11:43.072494 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:43.075266 (+ 2772us) service_pool.cc:225] Handling call
0430 20:11:44.366351 (+1291085us) ts_tablet_manager.cc:1120] Deleting tablet 8901327b3efb4201adce61a9266142ea
0430 20:11:44.366374 (+ 23us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:44.366490 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1290848}
I20240430 20:11:44.370121 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 321) took 1295 ms. Trace:
W20240430 20:11:44.370159 19004 catalog_manager.cc:4810] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): delete failed for tablet 8901327b3efb4201adce61a9266142ea because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8901327b3efb4201adce61a9266142ea already in progress: creating tablet
I20240430 20:11:44.370211 19495 rpcz_store.cc:276] 0430 20:11:43.074250 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:43.075529 (+ 1279us) service_pool.cc:225] Handling call
0430 20:11:44.369953 (+1294424us) ts_tablet_manager.cc:1120] Deleting tablet aa9998f4e17147aa8f2b35c01980463a
0430 20:11:44.369968 (+ 15us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:44.370099 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1294303}
W20240430 20:11:44.370695 19004 catalog_manager.cc:4810] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): delete failed for tablet aa9998f4e17147aa8f2b35c01980463a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aa9998f4e17147aa8f2b35c01980463a already in progress: creating tablet
I20240430 20:11:44.370153 19489 rpcz_store.cc:276] 0430 20:11:34.140289 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:34.140822 (+ 533us) service_pool.cc:225] Handling call
0430 20:11:44.325444 (+10184622us) ts_tablet_manager.cc:1120] Deleting tablet 0b4d9e860d7c45fd91d7a566605c9283
0430 20:11:44.338684 (+ 13240us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:44.347461 (+ 8777us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:44.352216 (+ 4755us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:44.369314 (+ 17098us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":9529589}
I20240430 20:11:44.371981 18985 raft_consensus.cc:348] T 2da900b6bbfa4fa99fe8494fd2e68da8 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.373270 19001 catalog_manager.cc:4830] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): tablet 0b4d9e860d7c45fd91d7a566605c9283 (table default.Table_896bbae1f6944005b0272ca8ebc97218 [id=d5cea90cbc2c48df9d8b80da1eccc949]) successfully deleted
I20240430 20:11:44.373762 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 181) took 10233 ms. Trace:
I20240430 20:11:44.374245 19489 rpcz_store.cc:276] 0430 20:11:34.140455 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:34.140995 (+ 540us) service_pool.cc:225] Handling call
0430 20:11:44.373607 (+10232612us) ts_tablet_manager.cc:1120] Deleting tablet 50486644ec234933b9a0b45ee39a6685
0430 20:11:44.373619 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:44.373742 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":8930877}
W20240430 20:11:44.375380 19001 catalog_manager.cc:4810] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): delete failed for tablet 50486644ec234933b9a0b45ee39a6685 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 50486644ec234933b9a0b45ee39a6685 already in progress: creating tablet
I20240430 20:11:44.375928 19489 tablet_replica.cc:330] T c66c142419ca4e1399ecd1c8d1e62a59 P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:44.377028 19489 raft_consensus.cc:2229] T c66c142419ca4e1399ecd1c8d1e62a59 P 3c51c33d5616482cafb68e26688820f4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:44.378679 18985 raft_consensus.cc:374] T 2da900b6bbfa4fa99fe8494fd2e68da8 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:44.380087 19028 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_326878a52ab041609f3c339cd65b7013"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240430 20:11:44.380301 18985 raft_consensus.cc:729] T 2da900b6bbfa4fa99fe8494fd2e68da8 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:44.380623 19489 raft_consensus.cc:2258] T c66c142419ca4e1399ecd1c8d1e62a59 P 3c51c33d5616482cafb68e26688820f4 [term 2 FOLLOWER]: Raft consensus is shut down!
W20240430 20:11:44.380832 19028 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_326878a52ab041609f3c339cd65b7013 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:44.380805 18985 consensus_queue.cc:260] T 2da900b6bbfa4fa99fe8494fd2e68da8 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.381834 18985 ts_tablet_manager.cc:1398] T 2da900b6bbfa4fa99fe8494fd2e68da8 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.013s user 0.000s sys 0.002s
I20240430 20:11:44.383780 19489 ts_tablet_manager.cc:1875] T c66c142419ca4e1399ecd1c8d1e62a59 P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:44.380380 18988 tablet_bootstrap.cc:492] T 5f02d8862e1f4ccfa9dbda73dbe18615 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:44.384093 18988 ts_tablet_manager.cc:1367] T 5f02d8862e1f4ccfa9dbda73dbe18615 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.041s user 0.006s sys 0.003s
I20240430 20:11:44.384178 18985 tablet_bootstrap.cc:492] T 99203cc522de424184c6f97453f166fd P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:44.385066 18988 raft_consensus.cc:348] T 5f02d8862e1f4ccfa9dbda73dbe18615 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.385349 18988 raft_consensus.cc:374] T 5f02d8862e1f4ccfa9dbda73dbe18615 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:44.385461 18988 raft_consensus.cc:729] T 5f02d8862e1f4ccfa9dbda73dbe18615 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:44.385522 18983 tablet_bootstrap.cc:492] T 06d28b6f48a34b17b0d7fb3102df2f4c P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:44.385921 18988 consensus_queue.cc:260] T 5f02d8862e1f4ccfa9dbda73dbe18615 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.386415 18983 ts_tablet_manager.cc:1367] T 06d28b6f48a34b17b0d7fb3102df2f4c P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.025s user 0.002s sys 0.006s
I20240430 20:11:44.386451 18988 ts_tablet_manager.cc:1398] T 5f02d8862e1f4ccfa9dbda73dbe18615 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240430 20:11:44.375136 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 322) took 1277 ms. Trace:
I20240430 20:11:44.386758 18988 tablet_bootstrap.cc:492] T e87fb2f0361d4aa3ada27bbdfa88e94c P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:44.386809 19495 rpcz_store.cc:276] 0430 20:11:43.098028 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:43.098101 (+ 73us) service_pool.cc:225] Handling call
0430 20:11:44.374985 (+1276884us) ts_tablet_manager.cc:1120] Deleting tablet 6d1d2193fd424c91b143f5256df5fcd0
0430 20:11:44.374997 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:44.375113 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1276683}
W20240430 20:11:44.387527 19004 catalog_manager.cc:4810] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): delete failed for tablet 6d1d2193fd424c91b143f5256df5fcd0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6d1d2193fd424c91b143f5256df5fcd0 already in progress: creating tablet
I20240430 20:11:44.387972 19495 tablet_replica.cc:330] T acf3b6fead41483a82c43dc0286fab7c P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:44.388168 19495 raft_consensus.cc:2229] T acf3b6fead41483a82c43dc0286fab7c P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:44.388329 19495 raft_consensus.cc:2258] T acf3b6fead41483a82c43dc0286fab7c P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:44.389153 18988 tablet_bootstrap.cc:654] T e87fb2f0361d4aa3ada27bbdfa88e94c P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:44.390166 18983 raft_consensus.cc:348] T 06d28b6f48a34b17b0d7fb3102df2f4c P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.390447 18983 raft_consensus.cc:374] T 06d28b6f48a34b17b0d7fb3102df2f4c P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:44.390543 18983 raft_consensus.cc:729] T 06d28b6f48a34b17b0d7fb3102df2f4c P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:44.390846 18983 consensus_queue.cc:260] T 06d28b6f48a34b17b0d7fb3102df2f4c P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.391477 18983 ts_tablet_manager.cc:1398] T 06d28b6f48a34b17b0d7fb3102df2f4c P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240430 20:11:44.391803 18983 tablet_bootstrap.cc:492] T 06e82c706b9c4f35bf53faddf57b3fd7 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:44.393956 19049 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_059bef7a615f4ba8a42ed1ffa8cd3c72"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:44.394549 19049 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_059bef7a615f4ba8a42ed1ffa8cd3c72 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:44.393513 18983 tablet_bootstrap.cc:654] T 06e82c706b9c4f35bf53faddf57b3fd7 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:44.396108 19495 ts_tablet_manager.cc:1875] T acf3b6fead41483a82c43dc0286fab7c P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:44.397598 18985 tablet_bootstrap.cc:654] T 99203cc522de424184c6f97453f166fd P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:44.403975 19489 ts_tablet_manager.cc:1888] T c66c142419ca4e1399ecd1c8d1e62a59 P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.3
I20240430 20:11:44.406617 19489 log.cc:1196] T c66c142419ca4e1399ecd1c8d1e62a59 P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/c66c142419ca4e1399ecd1c8d1e62a59
W20240430 20:11:44.407148 18699 consensus_peers.cc:485] T 0a9066d895744c5bb7d9532b2940603c P 3c51c33d5616482cafb68e26688820f4 -> Peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Couldn't send request to peer 2efb542aa55e439b99ccb98d1c5c884a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0a9066d895744c5bb7d9532b2940603c. This is attempt 1: this message will repeat every 5th retry.
I20240430 20:11:44.407790 19489 ts_tablet_manager.cc:1909] T c66c142419ca4e1399ecd1c8d1e62a59 P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:44.408651 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 191) took 9940 ms. Trace:
I20240430 20:11:44.408761 19489 rpcz_store.cc:276] 0430 20:11:34.468636 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:35.859155 (+1390519us) service_pool.cc:225] Handling call
0430 20:11:44.375842 (+8516687us) ts_tablet_manager.cc:1120] Deleting tablet c66c142419ca4e1399ecd1c8d1e62a59
0430 20:11:44.375858 (+ 16us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:44.395705 (+ 19847us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:44.403838 (+ 8133us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:44.408629 (+ 4791us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":8516478}
I20240430 20:11:44.409267 19001 catalog_manager.cc:4830] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): tablet c66c142419ca4e1399ecd1c8d1e62a59 (table default.Table_0443359dbeff437fb6cffb0db33cbb30 [id=93c44abdf6b147228224388398cd7aeb]) successfully deleted
I20240430 20:11:44.410737 19049 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_9c3d369e0c4f4bb393b7278a5fdcb2e8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:44.411685 19049 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9c3d369e0c4f4bb393b7278a5fdcb2e8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:44.415375 19028 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_9c955414a012419eb8f904a0878a6336"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240430 20:11:44.415817 19489 tablet_replica.cc:330] T 84568f90b9c54cf292099808d4cbf5dc P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
W20240430 20:11:44.416251 19028 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9c955414a012419eb8f904a0878a6336 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:44.419270 19471 ts_tablet_manager.cc:1888] T c66c142419ca4e1399ecd1c8d1e62a59 P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.3
I20240430 20:11:44.419466 19471 log.cc:1196] T c66c142419ca4e1399ecd1c8d1e62a59 P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/c66c142419ca4e1399ecd1c8d1e62a59
I20240430 20:11:44.419468 18988 tablet_bootstrap.cc:492] T e87fb2f0361d4aa3ada27bbdfa88e94c P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:44.419610 18988 ts_tablet_manager.cc:1367] T e87fb2f0361d4aa3ada27bbdfa88e94c P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.033s user 0.007s sys 0.003s
W20240430 20:11:44.420147 19024 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 330453ddeea6456e8ef93fa92608b0d3 consensus state has no leader
I20240430 20:11:44.420501 18988 raft_consensus.cc:348] T e87fb2f0361d4aa3ada27bbdfa88e94c P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.420679 18988 raft_consensus.cc:374] T e87fb2f0361d4aa3ada27bbdfa88e94c P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:44.420727 18988 raft_consensus.cc:729] T e87fb2f0361d4aa3ada27bbdfa88e94c P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:44.420650 19024 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_43043172400c49598042d752412c3a28"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240430 20:11:44.420959 18988 consensus_queue.cc:260] T e87fb2f0361d4aa3ada27bbdfa88e94c P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
W20240430 20:11:44.421240 19024 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_43043172400c49598042d752412c3a28 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:44.421473 18988 ts_tablet_manager.cc:1398] T e87fb2f0361d4aa3ada27bbdfa88e94c P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240430 20:11:44.421739 18988 tablet_bootstrap.cc:492] T abdaf243e5be4ad58036b23151b14fe4 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
W20240430 20:11:44.422520 19023 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a0baee91f5b44945b595d0e234af018e consensus state has no leader
I20240430 20:11:44.423239 18988 tablet_bootstrap.cc:654] T abdaf243e5be4ad58036b23151b14fe4 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:44.419656 18983 tablet_bootstrap.cc:492] T 06e82c706b9c4f35bf53faddf57b3fd7 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:44.424005 18983 ts_tablet_manager.cc:1367] T 06e82c706b9c4f35bf53faddf57b3fd7 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.032s user 0.003s sys 0.000s
I20240430 20:11:44.424731 18983 raft_consensus.cc:348] T 06e82c706b9c4f35bf53faddf57b3fd7 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.424963 18983 raft_consensus.cc:374] T 06e82c706b9c4f35bf53faddf57b3fd7 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:44.425019 18983 raft_consensus.cc:729] T 06e82c706b9c4f35bf53faddf57b3fd7 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:44.425235 18983 consensus_queue.cc:260] T 06e82c706b9c4f35bf53faddf57b3fd7 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.425743 18983 ts_tablet_manager.cc:1398] T 06e82c706b9c4f35bf53faddf57b3fd7 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240430 20:11:44.425041 19023 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_8236b26459a348f99005b9ef3033a30f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240430 20:11:44.426057 18983 tablet_bootstrap.cc:492] T 677481aee5fa4eb4b189322d7a3f8f2e P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
W20240430 20:11:44.426275 19023 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8236b26459a348f99005b9ef3033a30f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:44.427603 18983 tablet_bootstrap.cc:654] T 677481aee5fa4eb4b189322d7a3f8f2e P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:44.427723 19471 ts_tablet_manager.cc:1909] T c66c142419ca4e1399ecd1c8d1e62a59 P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:44.431733 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 176) took 9642 ms. Trace:
I20240430 20:11:44.433681 19049 master_service.cc:946] ReplaceTablet: received request to replace tablet 5809e7c9c2204ba293426c24129fb16e from {username='slave'} at 127.0.0.1:51664
I20240430 20:11:44.434545 18988 tablet_bootstrap.cc:492] T abdaf243e5be4ad58036b23151b14fe4 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:44.434737 18988 ts_tablet_manager.cc:1367] T abdaf243e5be4ad58036b23151b14fe4 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.000s
I20240430 20:11:44.435441 18988 raft_consensus.cc:348] T abdaf243e5be4ad58036b23151b14fe4 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.435668 18988 raft_consensus.cc:374] T abdaf243e5be4ad58036b23151b14fe4 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:44.435747 18988 raft_consensus.cc:729] T abdaf243e5be4ad58036b23151b14fe4 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:44.436040 18988 consensus_queue.cc:260] T abdaf243e5be4ad58036b23151b14fe4 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.436581 18988 ts_tablet_manager.cc:1398] T abdaf243e5be4ad58036b23151b14fe4 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240430 20:11:44.437116 18988 tablet_bootstrap.cc:492] T 7ce09fcbe1ac457a9c05efaf000c3263 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:44.437142 19049 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_9ba2556565fc49dfb771b92645ec3fb7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:44.437973 19049 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9ba2556565fc49dfb771b92645ec3fb7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:44.439262 18985 tablet_bootstrap.cc:492] T 99203cc522de424184c6f97453f166fd P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:44.439425 18985 ts_tablet_manager.cc:1367] T 99203cc522de424184c6f97453f166fd P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.055s user 0.000s sys 0.003s
I20240430 20:11:44.440191 18985 raft_consensus.cc:348] T 99203cc522de424184c6f97453f166fd P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:44.440382 18985 raft_consensus.cc:374] T 99203cc522de424184c6f97453f166fd P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:44.440434 18985 raft_consensus.cc:729] T 99203cc522de424184c6f97453f166fd P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:44.440665 18985 consensus_queue.cc:260] T 99203cc522de424184c6f97453f166fd P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:44.441962 18985 ts_tablet_manager.cc:1398] T 99203cc522de424184c6f97453f166fd P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240430 20:11:44.441987 18988 tablet_bootstrap.cc:654] T 7ce09fcbe1ac457a9c05efaf000c3263 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:44.442232 18985 tablet_bootstrap.cc:492] T 06d28b6f48a34b17b0d7fb3102df2f4c P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:44.445868 19026 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 1462ecf1176c4e66be42abddbe36b00c (table default.Table_b49a7942aa8c4f58acb94c35d1cdbb70 [id=6257559b0e9545c58848a76af0c5a5f7]). Expected version 1 got 0
I20240430 20:11:44.446209 19026 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet ebf74809b52f47f3b0ed21b6ac0f541a (table default.Table_b2d07520b9944a89ad62fca70396174e [id=1c8e0a3a44ec4a798770c0165ed3dffa]). Expected version 1 got 0
I20240430 20:11:44.446527 19026 catalog_manager.cc:5484] T a59ef6c545dd41a28265a4c1b8ce10e0 P 5e1c6397f09747f091534faf93542710 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e1c6397f09747f091534faf93542710 (127.17.177.2). New cstate: current_term: 1 leader_uuid: "5e1c6397f09747f091534faf93542710" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } health_report { overall_health: UNKNOWN } } }
I20240430 20:11:44.447010 19026 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet a59ef6c545dd41a28265a4c1b8ce10e0 (table default.Table_07e40af8f1c546048975c8edf88170d7 [id=86df5d161d8641c89070e7eed9a39360]). Expected version 2 got 0
I20240430 20:11:44.447470 19026 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 7ce09fcbe1ac457a9c05efaf000c3263 (table default.Table_edc4c27ed846454288f73f6a0d88cb69 [id=78fd3f516083489ab3b42d90dcef7e4d]). Expected version 1 got 0
I20240430 20:11:44.449131 19029 catalog_manager.cc:6029] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: default.Table_59297b42e8cf4ab3b9e95e2e70d3c1fe [id=b1312a6cd1a54003b1db727a2a9d125d] alter complete (version 2)
I20240430 20:11:44.450485 19027 master_service.cc:946] ReplaceTablet: received request to replace tablet b22aaedb0a6c4fd9ad0ea8fa019001e1 from {username='slave'} at 127.0.0.1:51664
W20240430 20:11:44.441507 18558 consensus_peers.cc:485] T e2d90ff17e574f88b2061b1c117d4b7d P 5e1c6397f09747f091534faf93542710 -> Peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Couldn't send request to peer 2efb542aa55e439b99ccb98d1c5c884a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e2d90ff17e574f88b2061b1c117d4b7d. This is attempt 1: this message will repeat every 5th retry.
W20240430 20:11:44.452335 18601 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40218: Illegal state: replica 5e1c6397f09747f091534faf93542710 is not leader of this config: current role FOLLOWER
I20240430 20:11:44.453994 18983 tablet_bootstrap.cc:492] T 677481aee5fa4eb4b189322d7a3f8f2e P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:44.454144 18983 ts_tablet_manager.cc:1367] T 677481aee5fa4eb4b189322d7a3f8f2e P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.028s user 0.006s sys 0.001s
I20240430 20:11:44.455137 18983 raft_consensus.cc:348] T 677481aee5fa4eb4b189322d7a3f8f2e P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.455430 18983 raft_consensus.cc:374] T 677481aee5fa4eb4b189322d7a3f8f2e P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:44.456058 18983 raft_consensus.cc:729] T 677481aee5fa4eb4b189322d7a3f8f2e P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
W20240430 20:11:44.456127 18999 catalog_manager.cc:4909] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): alter failed for tablet ca64de16d40f4a44b8b411f5526ba11a (table default.Table_1cf914acf7fe4e98b8dbc297b6372f21 [id=f4ba9536e9c141deb68f4f62b7e8fa70]): Illegal state: replica 5e1c6397f09747f091534faf93542710 is not leader of this config: current role FOLLOWER
I20240430 20:11:44.456384 18983 consensus_queue.cc:260] T 677481aee5fa4eb4b189322d7a3f8f2e P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.457504 18756 raft_consensus.cc:3037] T ebf74809b52f47f3b0ed21b6ac0f541a P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:44.461928 19028 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_4adefb6ee63f4301a479e1d042ff1c8b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240430 20:11:44.462410 19027 master_service.cc:946] ReplaceTablet: received request to replace tablet 24dcc20832804229a2664873bf6ddaa7 from {username='slave'} at 127.0.0.1:51664
W20240430 20:11:44.462651 19028 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4adefb6ee63f4301a479e1d042ff1c8b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:44.462934 19027 master_service.cc:946] ReplaceTablet: received request to replace tablet e3e74e76ab2b4fe492ffcc1838656fd1 from {username='slave'} at 127.0.0.1:51664
I20240430 20:11:44.466080 18988 tablet_bootstrap.cc:492] T 7ce09fcbe1ac457a9c05efaf000c3263 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:44.466238 18988 ts_tablet_manager.cc:1367] T 7ce09fcbe1ac457a9c05efaf000c3263 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.029s user 0.002s sys 0.006s
I20240430 20:11:44.466632 19471 rpcz_store.cc:276] 0430 20:11:34.789367 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:34.789658 (+ 291us) service_pool.cc:225] Handling call
0430 20:11:44.353065 (+9563407us) ts_tablet_manager.cc:1120] Deleting tablet c66c142419ca4e1399ecd1c8d1e62a59
0430 20:11:44.353077 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:44.387427 (+ 34350us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:44.419143 (+ 31716us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:44.431706 (+ 12563us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":9563212}
I20240430 20:11:44.467028 18988 raft_consensus.cc:348] T 7ce09fcbe1ac457a9c05efaf000c3263 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.467231 18988 raft_consensus.cc:374] T 7ce09fcbe1ac457a9c05efaf000c3263 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:44.467284 18988 raft_consensus.cc:729] T 7ce09fcbe1ac457a9c05efaf000c3263 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:44.467548 18988 consensus_queue.cc:260] T 7ce09fcbe1ac457a9c05efaf000c3263 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.468027 18988 ts_tablet_manager.cc:1398] T 7ce09fcbe1ac457a9c05efaf000c3263 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240430 20:11:44.468279 18988 tablet_bootstrap.cc:492] T 50486644ec234933b9a0b45ee39a6685 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:44.469274 19024 master_service.cc:946] ReplaceTablet: received request to replace tablet 2aa2a489116940dea3173cad16d94fa5 from {username='slave'} at 127.0.0.1:51664
I20240430 20:11:44.469906 18988 tablet_bootstrap.cc:654] T 50486644ec234933b9a0b45ee39a6685 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:44.471827 18999 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet c66c142419ca4e1399ecd1c8d1e62a59 (table default.Table_0443359dbeff437fb6cffb0db33cbb30 [id=93c44abdf6b147228224388398cd7aeb]) successfully deleted
I20240430 20:11:44.473093 18985 tablet_bootstrap.cc:654] T 06d28b6f48a34b17b0d7fb3102df2f4c P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:44.477070 19471 tablet_replica.cc:330] T 0b4d9e860d7c45fd91d7a566605c9283 P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:44.477999 18983 ts_tablet_manager.cc:1398] T 677481aee5fa4eb4b189322d7a3f8f2e P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.024s user 0.013s sys 0.001s
I20240430 20:11:44.478430 18983 tablet_bootstrap.cc:492] T ae24e75a0d42441ebc97400ffa94145e P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:44.480484 18983 tablet_bootstrap.cc:654] T ae24e75a0d42441ebc97400ffa94145e P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:44.482970 19471 raft_consensus.cc:2229] T 0b4d9e860d7c45fd91d7a566605c9283 P 5e1c6397f09747f091534faf93542710 [term 2 LEADER]: Raft consensus shutting down.
W20240430 20:11:44.483024 18601 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40218: Illegal state: replica 5e1c6397f09747f091534faf93542710 is not leader of this config: current role FOLLOWER
I20240430 20:11:44.484287 19471 raft_consensus.cc:2258] T 0b4d9e860d7c45fd91d7a566605c9283 P 5e1c6397f09747f091534faf93542710 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:44.484583 18988 tablet_bootstrap.cc:492] T 50486644ec234933b9a0b45ee39a6685 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:44.484642 20235 raft_consensus.cc:482] T d61c279d20b0471a967942525ceb4fcb P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:44.484737 18988 ts_tablet_manager.cc:1367] T 50486644ec234933b9a0b45ee39a6685 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.003s
I20240430 20:11:44.484788 20235 raft_consensus.cc:504] T d61c279d20b0471a967942525ceb4fcb P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.485502 20235 leader_election.cc:288] T d61c279d20b0471a967942525ceb4fcb P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:44.485579 18988 raft_consensus.cc:348] T 50486644ec234933b9a0b45ee39a6685 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.485752 18988 raft_consensus.cc:374] T 50486644ec234933b9a0b45ee39a6685 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:44.485797 18988 raft_consensus.cc:729] T 50486644ec234933b9a0b45ee39a6685 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:44.486019 18988 consensus_queue.cc:260] T 50486644ec234933b9a0b45ee39a6685 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.486227 18750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d61c279d20b0471a967942525ceb4fcb" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:44.486671 18988 ts_tablet_manager.cc:1398] T 50486644ec234933b9a0b45ee39a6685 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240430 20:11:44.486924 18988 tablet_bootstrap.cc:492] T 5bd0362ff41b4faca0205d136d596737 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:44.487071 18615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d61c279d20b0471a967942525ceb4fcb" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:44.487273 18615 raft_consensus.cc:2454] T d61c279d20b0471a967942525ceb4fcb P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 0.
W20240430 20:11:44.488349 18417 leader_election.cc:341] T d61c279d20b0471a967942525ceb4fcb P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:44.488565 19471 ts_tablet_manager.cc:1875] T 0b4d9e860d7c45fd91d7a566605c9283 P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:44.488678 18988 tablet_bootstrap.cc:654] T 5bd0362ff41b4faca0205d136d596737 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:44.488996 18415 leader_election.cc:302] T d61c279d20b0471a967942525ceb4fcb P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters: 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:44.489755 20235 raft_consensus.cc:2781] T d61c279d20b0471a967942525ceb4fcb P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:44.489833 20235 raft_consensus.cc:482] T d61c279d20b0471a967942525ceb4fcb P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:44.489888 20235 raft_consensus.cc:3037] T d61c279d20b0471a967942525ceb4fcb P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:44.492517 20235 raft_consensus.cc:504] T d61c279d20b0471a967942525ceb4fcb P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.493932 18750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d61c279d20b0471a967942525ceb4fcb" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:44.494179 18988 tablet_bootstrap.cc:492] T 5bd0362ff41b4faca0205d136d596737 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:44.494305 18988 ts_tablet_manager.cc:1367] T 5bd0362ff41b4faca0205d136d596737 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
W20240430 20:11:44.494474 18417 leader_election.cc:341] T d61c279d20b0471a967942525ceb4fcb P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:44.495185 19023 rpcz_store.cc:275] Call kudu.master.MasterService.GetMasterRegistration from 127.0.0.1:39680 (request call id 1) took 1187 ms. Trace:
I20240430 20:11:44.495307 19023 rpcz_store.cc:276] 0430 20:11:43.307830 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:44.487884 (+1180054us) service_pool.cc:225] Handling call
0430 20:11:44.494998 (+ 7114us) spinlock_profiling.cc:243] Waited 6.75 ms on lock 0x616000040654. stack: 00007feb16c74756 00007feb16c74510 00007feb15ca748d 0000000000955804 00000000009557cd 00007feb257b315d 00007feb257b3253 00007feb224e5d41 00007feb2a0d134d 00007feb2a2661bf 00007feb21058227 00007feb21057f55 00007feb1d97c04e 00007feb1d97af2e 00007feb1d981405 00007feb1d981d2f
0430 20:11:44.495133 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":15536384}
I20240430 20:11:44.495416 18988 raft_consensus.cc:348] T 5bd0362ff41b4faca0205d136d596737 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.495581 18988 raft_consensus.cc:374] T 5bd0362ff41b4faca0205d136d596737 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:44.495621 18988 raft_consensus.cc:729] T 5bd0362ff41b4faca0205d136d596737 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:44.496464 18988 consensus_queue.cc:260] T 5bd0362ff41b4faca0205d136d596737 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.496979 18988 ts_tablet_manager.cc:1398] T 5bd0362ff41b4faca0205d136d596737 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240430 20:11:44.497252 18988 tablet_bootstrap.cc:492] T d61c279d20b0471a967942525ceb4fcb P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:44.497740 18750 raft_consensus.cc:3037] T 1462ecf1176c4e66be42abddbe36b00c P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:44.499238 18988 tablet_bootstrap.cc:654] T d61c279d20b0471a967942525ceb4fcb P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:44.501901 18983 tablet_bootstrap.cc:492] T ae24e75a0d42441ebc97400ffa94145e P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:44.502045 18983 ts_tablet_manager.cc:1367] T ae24e75a0d42441ebc97400ffa94145e P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.024s user 0.004s sys 0.000s
I20240430 20:11:44.502290 18615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d61c279d20b0471a967942525ceb4fcb" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:44.502460 18615 raft_consensus.cc:3037] T d61c279d20b0471a967942525ceb4fcb P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:44.502758 18983 raft_consensus.cc:348] T ae24e75a0d42441ebc97400ffa94145e P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.502954 18983 raft_consensus.cc:374] T ae24e75a0d42441ebc97400ffa94145e P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:44.503007 18983 raft_consensus.cc:729] T ae24e75a0d42441ebc97400ffa94145e P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:44.503218 18983 consensus_queue.cc:260] T ae24e75a0d42441ebc97400ffa94145e P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.503723 18983 ts_tablet_manager.cc:1398] T ae24e75a0d42441ebc97400ffa94145e P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240430 20:11:44.504053 18983 tablet_bootstrap.cc:492] T eb76f60448774fe1967fffe7c6873cfc P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:44.505091 18615 raft_consensus.cc:2454] T d61c279d20b0471a967942525ceb4fcb P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 1.
I20240430 20:11:44.505457 19489 raft_consensus.cc:2229] T 84568f90b9c54cf292099808d4cbf5dc P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:44.505766 19489 pending_rounds.cc:62] T 84568f90b9c54cf292099808d4cbf5dc P 3c51c33d5616482cafb68e26688820f4: Trying to abort 1 pending ops.
I20240430 20:11:44.505806 19489 pending_rounds.cc:68] T 84568f90b9c54cf292099808d4cbf5dc P 3c51c33d5616482cafb68e26688820f4: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7022624367617343488 op_type: NO_OP noop_request { }
I20240430 20:11:44.505901 19489 raft_consensus.cc:2866] T 84568f90b9c54cf292099808d4cbf5dc P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20240430 20:11:44.505970 19489 raft_consensus.cc:2258] T 84568f90b9c54cf292099808d4cbf5dc P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:44.505978 18415 leader_election.cc:302] T d61c279d20b0471a967942525ceb4fcb P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters: 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:44.506289 20105 raft_consensus.cc:2781] T d61c279d20b0471a967942525ceb4fcb P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:44.506486 19471 ts_tablet_manager.cc:1888] T 0b4d9e860d7c45fd91d7a566605c9283 P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.3
I20240430 20:11:44.506577 19471 log.cc:1196] T 0b4d9e860d7c45fd91d7a566605c9283 P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/0b4d9e860d7c45fd91d7a566605c9283
I20240430 20:11:44.507422 19471 ts_tablet_manager.cc:1909] T 0b4d9e860d7c45fd91d7a566605c9283 P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:44.508263 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 174) took 9719 ms. Trace:
I20240430 20:11:44.508410 19471 rpcz_store.cc:276] 0430 20:11:34.789123 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:34.790818 (+ 1695us) service_pool.cc:225] Handling call
0430 20:11:44.476987 (+9686169us) ts_tablet_manager.cc:1120] Deleting tablet 0b4d9e860d7c45fd91d7a566605c9283
0430 20:11:44.477000 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:44.491835 (+ 14835us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:44.506383 (+ 14548us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:44.508237 (+ 1854us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5126,"tablet-delete.queue_time_us":9685936}
I20240430 20:11:44.508971 18999 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet 0b4d9e860d7c45fd91d7a566605c9283 (table default.Table_896bbae1f6944005b0272ca8ebc97218 [id=d5cea90cbc2c48df9d8b80da1eccc949]) successfully deleted
I20240430 20:11:44.507071 18983 tablet_bootstrap.cc:654] T eb76f60448774fe1967fffe7c6873cfc P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:44.510006 19471 tablet_replica.cc:330] T 06d28b6f48a34b17b0d7fb3102df2f4c P 5e1c6397f09747f091534faf93542710: stopping tablet replica
W20240430 20:11:44.510191 18557 consensus_peers.cc:485] T 84568f90b9c54cf292099808d4cbf5dc P 5e1c6397f09747f091534faf93542710 -> Peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Couldn't send request to peer 3c51c33d5616482cafb68e26688820f4. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: STOPPING. This is attempt 1: this message will repeat every 5th retry.
I20240430 20:11:44.511152 20235 leader_election.cc:288] T d61c279d20b0471a967942525ceb4fcb P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:44.516686 19489 ts_tablet_manager.cc:1875] T 84568f90b9c54cf292099808d4cbf5dc P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:44.520478 18983 tablet_bootstrap.cc:492] T eb76f60448774fe1967fffe7c6873cfc P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:44.520748 18983 ts_tablet_manager.cc:1367] T eb76f60448774fe1967fffe7c6873cfc P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.000s
I20240430 20:11:44.521486 18983 raft_consensus.cc:348] T eb76f60448774fe1967fffe7c6873cfc P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.521761 18983 raft_consensus.cc:374] T eb76f60448774fe1967fffe7c6873cfc P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:44.521874 18983 raft_consensus.cc:729] T eb76f60448774fe1967fffe7c6873cfc P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:44.522179 18983 consensus_queue.cc:260] T eb76f60448774fe1967fffe7c6873cfc P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.522816 18983 ts_tablet_manager.cc:1398] T eb76f60448774fe1967fffe7c6873cfc P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240430 20:11:44.523190 18983 tablet_bootstrap.cc:492] T f6cfd089f08b47409fb2a6cf4f6b0385 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:44.510215 19471 raft_consensus.cc:2229] T 06d28b6f48a34b17b0d7fb3102df2f4c P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:44.524205 19471 raft_consensus.cc:2258] T 06d28b6f48a34b17b0d7fb3102df2f4c P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:44.524497 19495 ts_tablet_manager.cc:1888] T acf3b6fead41483a82c43dc0286fab7c P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:44.524974 19495 log.cc:1196] T acf3b6fead41483a82c43dc0286fab7c P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/acf3b6fead41483a82c43dc0286fab7c
I20240430 20:11:44.526954 19489 ts_tablet_manager.cc:1888] T 84568f90b9c54cf292099808d4cbf5dc P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:44.527086 19489 log.cc:1196] T 84568f90b9c54cf292099808d4cbf5dc P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/84568f90b9c54cf292099808d4cbf5dc
I20240430 20:11:44.527493 19489 ts_tablet_manager.cc:1909] T 84568f90b9c54cf292099808d4cbf5dc P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:44.524682 18983 tablet_bootstrap.cc:654] T f6cfd089f08b47409fb2a6cf4f6b0385 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:44.528399 20047 raft_consensus.cc:482] T a9f37f0f7ce8425f9b24699513d41d61 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:44.528548 20047 raft_consensus.cc:504] T a9f37f0f7ce8425f9b24699513d41d61 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.529172 20047 leader_election.cc:288] T a9f37f0f7ce8425f9b24699513d41d61 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:44.530315 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 192) took 10061 ms. Trace:
I20240430 20:11:44.530763 18756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9f37f0f7ce8425f9b24699513d41d61" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:44.530885 20235 raft_consensus.cc:482] T 2da900b6bbfa4fa99fe8494fd2e68da8 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:44.531538 20235 raft_consensus.cc:504] T 2da900b6bbfa4fa99fe8494fd2e68da8 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.533123 19495 ts_tablet_manager.cc:1909] T acf3b6fead41483a82c43dc0286fab7c P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:44.533928 18810 heartbeater.cc:503] Master 127.17.177.60:37225 was elected leader, sending a full tablet report...
W20240430 20:11:44.535125 18557 leader_election.cc:341] T a9f37f0f7ce8425f9b24699513d41d61 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:44.535163 18482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9f37f0f7ce8425f9b24699513d41d61" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:44.535682 18756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2da900b6bbfa4fa99fe8494fd2e68da8" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
W20240430 20:11:44.535988 18558 leader_election.cc:341] T a9f37f0f7ce8425f9b24699513d41d61 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:44.536109 18558 leader_election.cc:302] T a9f37f0f7ce8425f9b24699513d41d61 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:44.536536 20047 raft_consensus.cc:2726] T a9f37f0f7ce8425f9b24699513d41d61 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:44.536805 20047 raft_consensus.cc:482] T ae24e75a0d42441ebc97400ffa94145e P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:44.536926 20047 raft_consensus.cc:504] T ae24e75a0d42441ebc97400ffa94145e P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.537510 20047 leader_election.cc:288] T ae24e75a0d42441ebc97400ffa94145e P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
W20240430 20:11:44.537701 18601 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40218: Illegal state: replica 5e1c6397f09747f091534faf93542710 is not leader of this config: current role FOLLOWER
I20240430 20:11:44.538307 18482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae24e75a0d42441ebc97400ffa94145e" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
W20240430 20:11:44.538959 18558 leader_election.cc:341] T ae24e75a0d42441ebc97400ffa94145e P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:44.539013 20205 raft_consensus.cc:482] T a59ef6c545dd41a28265a4c1b8ce10e0 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240430 20:11:44.539249 18417 leader_election.cc:341] T 2da900b6bbfa4fa99fe8494fd2e68da8 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:44.539157 20205 raft_consensus.cc:504] T a59ef6c545dd41a28265a4c1b8ce10e0 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.539593 18758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae24e75a0d42441ebc97400ffa94145e" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:44.539896 20205 leader_election.cc:288] T a59ef6c545dd41a28265a4c1b8ce10e0 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:44.540237 18615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a59ef6c545dd41a28265a4c1b8ce10e0" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:44.540397 18482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a59ef6c545dd41a28265a4c1b8ce10e0" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:44.528383 19471 ts_tablet_manager.cc:1875] T 06d28b6f48a34b17b0d7fb3102df2f4c P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240430 20:11:44.540895 18557 leader_election.cc:341] T ae24e75a0d42441ebc97400ffa94145e P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:44.541028 18557 leader_election.cc:302] T ae24e75a0d42441ebc97400ffa94145e P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:44.541097 18699 leader_election.cc:302] T a59ef6c545dd41a28265a4c1b8ce10e0 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710
I20240430 20:11:44.541522 20047 raft_consensus.cc:2726] T ae24e75a0d42441ebc97400ffa94145e P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:44.542150 19033 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: Sending DeleteTablet for 3 replicas of tablet a37d917bcc45468f9e3ee805640fba98
I20240430 20:11:44.530429 19489 rpcz_store.cc:276] 0430 20:11:34.468767 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:35.859371 (+1390604us) service_pool.cc:225] Handling call
0430 20:11:44.415755 (+8556384us) ts_tablet_manager.cc:1120] Deleting tablet 84568f90b9c54cf292099808d4cbf5dc
0430 20:11:44.415768 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:44.505690 (+ 89922us) spinlock_profiling.cc:243] Waited 89.4 ms on lock 0x6160000e4a54. stack: 00007f30e7d63756 00007f30e7d63510 00007f30e6d9648d 0000000000955804 00000000009557cd 000000000095519a 00007f30f35d1a00 00007f30f48ab853 00007f30f733e4bc 00007f30f7370814 00007f30f7351b00 00007f30f735188d 0000000000a60bcc 00007f30e7da9b74 00007f30e7dad19b 00007f30e7dad018
0430 20:11:44.521950 (+ 16260us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:44.526827 (+ 4877us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:44.530294 (+ 3467us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":205608192,"tablet-delete.queue_time_us":8556209}
I20240430 20:11:44.542754 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 193) took 9744 ms. Trace:
I20240430 20:11:44.542896 19489 rpcz_store.cc:276] 0430 20:11:34.798522 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:35.859558 (+1061036us) service_pool.cc:225] Handling call
0430 20:11:44.542629 (+8683071us) ts_tablet_manager.cc:1120] Deleting tablet 29974dc2c7dd45b99b6cc0712378fc30
0430 20:11:44.542644 (+ 15us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:44.542734 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":8682868}
I20240430 20:11:44.543259 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 194) took 9744 ms. Trace:
I20240430 20:11:44.543326 19489 rpcz_store.cc:276] 0430 20:11:34.798646 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:35.859733 (+1061087us) service_pool.cc:225] Handling call
0430 20:11:44.543135 (+8683402us) ts_tablet_manager.cc:1120] Deleting tablet 6db55bb510b24b0cae897936211e15f0
0430 20:11:44.543144 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:44.543242 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":8683268}
I20240430 20:11:44.543352 18450 tablet_service.cc:1514] Processing DeleteTablet for tablet a37d917bcc45468f9e3ee805640fba98 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:44 UTC) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:44.543571 19489 tablet_replica.cc:330] T 680553d4d0bc4fcb839235c1832757e1 P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:44.543680 18742 tablet_service.cc:1514] Processing DeleteTablet for tablet a37d917bcc45468f9e3ee805640fba98 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:44 UTC) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:44.543776 19489 raft_consensus.cc:2229] T 680553d4d0bc4fcb839235c1832757e1 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:44.543901 19001 catalog_manager.cc:4830] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): tablet 84568f90b9c54cf292099808d4cbf5dc (table default.Table_0034641cfa4d42a68ff4341e87fcb396 [id=53c496365b43473fa7e61ca706cdd984]) successfully deleted
I20240430 20:11:44.544034 19489 raft_consensus.cc:2258] T 680553d4d0bc4fcb839235c1832757e1 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:44.544299 18615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2da900b6bbfa4fa99fe8494fd2e68da8" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:44.544529 18615 raft_consensus.cc:2454] T 2da900b6bbfa4fa99fe8494fd2e68da8 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 0.
I20240430 20:11:44.544628 19489 ts_tablet_manager.cc:1875] T 680553d4d0bc4fcb839235c1832757e1 P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240430 20:11:44.544755 19001 catalog_manager.cc:4797] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): delete failed for tablet 29974dc2c7dd45b99b6cc0712378fc30 because the tablet was not found. No further retry: Not found: Tablet not found: 29974dc2c7dd45b99b6cc0712378fc30
I20240430 20:11:44.544960 18415 leader_election.cc:302] T 2da900b6bbfa4fa99fe8494fd2e68da8 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters: 3c51c33d5616482cafb68e26688820f4
W20240430 20:11:44.545056 19001 catalog_manager.cc:4810] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): delete failed for tablet 6db55bb510b24b0cae897936211e15f0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6db55bb510b24b0cae897936211e15f0 already in progress: creating tablet
I20240430 20:11:44.545286 20235 leader_election.cc:288] T 2da900b6bbfa4fa99fe8494fd2e68da8 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:44.545548 20235 raft_consensus.cc:2781] T 2da900b6bbfa4fa99fe8494fd2e68da8 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:44.545648 20235 raft_consensus.cc:482] T 2da900b6bbfa4fa99fe8494fd2e68da8 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:44.545723 20235 raft_consensus.cc:3037] T 2da900b6bbfa4fa99fe8494fd2e68da8 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:44.546710 20205 raft_consensus.cc:2726] T a59ef6c545dd41a28265a4c1b8ce10e0 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:44.547259 18601 tablet_service.cc:1514] Processing DeleteTablet for tablet a37d917bcc45468f9e3ee805640fba98 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:44 UTC) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:44.548862 18756 raft_consensus.cc:3037] T 3fd0e8c360734dcbb75dc2958b70c2b8 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:44.551839 18988 tablet_bootstrap.cc:492] T d61c279d20b0471a967942525ceb4fcb P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:44.552014 18988 ts_tablet_manager.cc:1367] T d61c279d20b0471a967942525ceb4fcb P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.055s user 0.006s sys 0.000s
I20240430 20:11:44.552781 18988 raft_consensus.cc:348] T d61c279d20b0471a967942525ceb4fcb P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.553020 18988 raft_consensus.cc:374] T d61c279d20b0471a967942525ceb4fcb P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:44.553083 18988 raft_consensus.cc:729] T d61c279d20b0471a967942525ceb4fcb P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:44.553460 18988 consensus_queue.cc:260] T d61c279d20b0471a967942525ceb4fcb P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.553795 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 323) took 1295 ms. Trace:
I20240430 20:11:44.553908 19495 rpcz_store.cc:276] 0430 20:11:43.258461 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:43.258509 (+ 48us) service_pool.cc:225] Handling call
0430 20:11:44.387151 (+1128642us) ts_tablet_manager.cc:1120] Deleting tablet acf3b6fead41483a82c43dc0286fab7c
0430 20:11:44.387163 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:44.415826 (+ 28663us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:44.524358 (+108532us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:44.553765 (+ 29407us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1128436}
I20240430 20:11:44.554080 18988 ts_tablet_manager.cc:1398] T d61c279d20b0471a967942525ceb4fcb P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240430 20:11:44.554355 18988 tablet_bootstrap.cc:492] T 2e79df8744e74b5991a79ce978ff0202 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:44.553601 20235 raft_consensus.cc:504] T 2da900b6bbfa4fa99fe8494fd2e68da8 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.556010 18988 tablet_bootstrap.cc:654] T 2e79df8744e74b5991a79ce978ff0202 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:44.557273 18758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2da900b6bbfa4fa99fe8494fd2e68da8" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
W20240430 20:11:44.557879 18417 leader_election.cc:341] T 2da900b6bbfa4fa99fe8494fd2e68da8 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:44.558017 19033 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_059bef7a615f4ba8a42ed1ffa8cd3c72" } modify_external_catalogs: true
I20240430 20:11:44.558240 19033 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_059bef7a615f4ba8a42ed1ffa8cd3c72" } modify_external_catalogs: true
I20240430 20:11:44.559908 19049 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_9ba2556565fc49dfb771b92645ec3fb7" } new_table_name: "default.Table_94e5f2a77765434e8a76d9a558b23ae2" modify_external_catalogs: true
W20240430 20:11:44.560910 18558 consensus_peers.cc:485] T 294644d7a16b4ca2bbb6d106bcb8c7ff P 5e1c6397f09747f091534faf93542710 -> Peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Couldn't send request to peer 2efb542aa55e439b99ccb98d1c5c884a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 294644d7a16b4ca2bbb6d106bcb8c7ff. This is attempt 1: this message will repeat every 5th retry.
I20240430 20:11:44.561038 18615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2da900b6bbfa4fa99fe8494fd2e68da8" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:44.561226 18615 raft_consensus.cc:3037] T 2da900b6bbfa4fa99fe8494fd2e68da8 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:44.561429 20105 raft_consensus.cc:686] T d61c279d20b0471a967942525ceb4fcb P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Becoming Leader. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Running, Role: LEADER
I20240430 20:11:44.561694 20105 consensus_queue.cc:237] T d61c279d20b0471a967942525ceb4fcb P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.561971 19489 ts_tablet_manager.cc:1888] T 680553d4d0bc4fcb839235c1832757e1 P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240430 20:11:44.562069 19489 log.cc:1196] T 680553d4d0bc4fcb839235c1832757e1 P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/680553d4d0bc4fcb839235c1832757e1
I20240430 20:11:44.562431 19489 ts_tablet_manager.cc:1909] T 680553d4d0bc4fcb839235c1832757e1 P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:44.562784 19023 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_afeeff199c8b4ba888e8b73423e870a7" } new_table_name: "default.Table_3da5b4c8668c470483ae2a0c7773eaaf" modify_external_catalogs: true
I20240430 20:11:44.564469 18615 raft_consensus.cc:2454] T 2da900b6bbfa4fa99fe8494fd2e68da8 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 1.
I20240430 20:11:44.565047 18415 leader_election.cc:302] T 2da900b6bbfa4fa99fe8494fd2e68da8 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters: 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:44.565676 20538 raft_consensus.cc:2781] T 2da900b6bbfa4fa99fe8494fd2e68da8 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:44.565786 20538 raft_consensus.cc:686] T 2da900b6bbfa4fa99fe8494fd2e68da8 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Becoming Leader. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Running, Role: LEADER
I20240430 20:11:44.566001 20538 consensus_queue.cc:237] T 2da900b6bbfa4fa99fe8494fd2e68da8 P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.566967 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 201) took 9101 ms. Trace:
I20240430 20:11:44.567060 19489 rpcz_store.cc:276] 0430 20:11:35.465344 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:38.413065 (+2947721us) service_pool.cc:225] Handling call
0430 20:11:44.543526 (+6130461us) ts_tablet_manager.cc:1120] Deleting tablet 680553d4d0bc4fcb839235c1832757e1
0430 20:11:44.543536 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:44.549745 (+ 6209us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:44.561843 (+ 12098us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:44.566941 (+ 5098us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5942961}
I20240430 20:11:44.554791 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 342) took 1104 ms. Trace:
I20240430 20:11:44.567912 19495 rpcz_store.cc:276] 0430 20:11:43.450497 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:43.450574 (+ 77us) service_pool.cc:225] Handling call
0430 20:11:44.554644 (+1104070us) ts_tablet_manager.cc:1120] Deleting tablet 3ec3993042f5498b818bdf32d5087928
0430 20:11:44.554656 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:44.554770 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1103824}
W20240430 20:11:44.568435 19004 catalog_manager.cc:4810] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): delete failed for tablet 3ec3993042f5498b818bdf32d5087928 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3ec3993042f5498b818bdf32d5087928 already in progress: creating tablet
I20240430 20:11:44.569031 19051 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: Sending DeleteTablet for 3 replicas of tablet a445fdfde01c4371a5e6b3daea5b00ab
I20240430 20:11:44.569909 18988 tablet_bootstrap.cc:492] T 2e79df8744e74b5991a79ce978ff0202 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:44.570086 18988 ts_tablet_manager.cc:1367] T 2e79df8744e74b5991a79ce978ff0202 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.000s
I20240430 20:11:44.570436 18450 tablet_service.cc:1514] Processing DeleteTablet for tablet a445fdfde01c4371a5e6b3daea5b00ab with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:44 UTC) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:44.570504 19489 tablet_replica.cc:330] T e87fb2f0361d4aa3ada27bbdfa88e94c P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:44.570931 19489 raft_consensus.cc:2229] T e87fb2f0361d4aa3ada27bbdfa88e94c P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:44.571125 19489 raft_consensus.cc:2258] T e87fb2f0361d4aa3ada27bbdfa88e94c P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:44.572104 18601 tablet_service.cc:1514] Processing DeleteTablet for tablet a445fdfde01c4371a5e6b3daea5b00ab with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:44 UTC) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:44.572499 18985 tablet_bootstrap.cc:492] T 06d28b6f48a34b17b0d7fb3102df2f4c P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:44.572644 18985 ts_tablet_manager.cc:1367] T 06d28b6f48a34b17b0d7fb3102df2f4c P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.130s user 0.005s sys 0.001s
I20240430 20:11:44.572845 20047 raft_consensus.cc:482] T f940d17b02b541289121463e05b0bb5f P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:44.573020 20047 raft_consensus.cc:504] T f940d17b02b541289121463e05b0bb5f P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.573386 18985 raft_consensus.cc:348] T 06d28b6f48a34b17b0d7fb3102df2f4c P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.573565 18985 raft_consensus.cc:374] T 06d28b6f48a34b17b0d7fb3102df2f4c P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:44.573614 18985 raft_consensus.cc:729] T 06d28b6f48a34b17b0d7fb3102df2f4c P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:44.573673 20047 leader_election.cc:288] T f940d17b02b541289121463e05b0bb5f P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:44.573837 18985 consensus_queue.cc:260] T 06d28b6f48a34b17b0d7fb3102df2f4c P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.574460 18482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f940d17b02b541289121463e05b0bb5f" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:44.574697 18482 raft_consensus.cc:2454] T f940d17b02b541289121463e05b0bb5f P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 0.
I20240430 20:11:44.575088 18558 leader_election.cc:302] T f940d17b02b541289121463e05b0bb5f P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:44.575358 20047 raft_consensus.cc:2781] T f940d17b02b541289121463e05b0bb5f P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:44.575423 20047 raft_consensus.cc:482] T f940d17b02b541289121463e05b0bb5f P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:44.575436 19495 tablet_replica.cc:330] T 115c4a12440647a4bb46eb6824f7ea98 P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:44.575475 20047 raft_consensus.cc:3037] T f940d17b02b541289121463e05b0bb5f P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:44.575659 19495 raft_consensus.cc:2229] T 115c4a12440647a4bb46eb6824f7ea98 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:44.575817 19495 raft_consensus.cc:2258] T 115c4a12440647a4bb46eb6824f7ea98 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:44.576763 19495 ts_tablet_manager.cc:1875] T 115c4a12440647a4bb46eb6824f7ea98 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:44.579828 20047 raft_consensus.cc:504] T f940d17b02b541289121463e05b0bb5f P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.581431 20047 leader_election.cc:288] T f940d17b02b541289121463e05b0bb5f P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:44.581697 18482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f940d17b02b541289121463e05b0bb5f" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:44.581888 18482 raft_consensus.cc:3037] T f940d17b02b541289121463e05b0bb5f P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:44.583807 20235 leader_election.cc:288] T 2da900b6bbfa4fa99fe8494fd2e68da8 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:44.584583 19051 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_5b6a4bb3832e497888af8e8db7b96c12" } modify_external_catalogs: true
I20240430 20:11:44.584928 19051 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_5b6a4bb3832e497888af8e8db7b96c12" } modify_external_catalogs: true
I20240430 20:11:44.585137 18742 tablet_service.cc:1514] Processing DeleteTablet for tablet a445fdfde01c4371a5e6b3daea5b00ab with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:44 UTC) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:44.585597 18750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f940d17b02b541289121463e05b0bb5f" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:44.586198 20047 raft_consensus.cc:482] T 06e82c706b9c4f35bf53faddf57b3fd7 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:44.586391 20047 raft_consensus.cc:504] T 06e82c706b9c4f35bf53faddf57b3fd7 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.587493 20047 leader_election.cc:288] T 06e82c706b9c4f35bf53faddf57b3fd7 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:44.584643 18482 raft_consensus.cc:2454] T f940d17b02b541289121463e05b0bb5f P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
I20240430 20:11:44.587880 18478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06e82c706b9c4f35bf53faddf57b3fd7" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:44.588261 18558 leader_election.cc:302] T f940d17b02b541289121463e05b0bb5f P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:44.588534 20047 raft_consensus.cc:2781] T f940d17b02b541289121463e05b0bb5f P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election won for term 1
W20240430 20:11:44.588600 18558 leader_election.cc:341] T 06e82c706b9c4f35bf53faddf57b3fd7 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:44.588665 20047 raft_consensus.cc:686] T f940d17b02b541289121463e05b0bb5f P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Becoming Leader. State: Replica: 5e1c6397f09747f091534faf93542710, State: Running, Role: LEADER
I20240430 20:11:44.588918 20047 consensus_queue.cc:237] T f940d17b02b541289121463e05b0bb5f P 5e1c6397f09747f091534faf93542710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.589247 19489 ts_tablet_manager.cc:1875] T e87fb2f0361d4aa3ada27bbdfa88e94c P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:44.594396 18763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f940d17b02b541289121463e05b0bb5f" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:44.595259 18763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06e82c706b9c4f35bf53faddf57b3fd7" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
W20240430 20:11:44.595703 18557 leader_election.cc:341] T f940d17b02b541289121463e05b0bb5f P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
W20240430 20:11:44.596459 18557 leader_election.cc:341] T 06e82c706b9c4f35bf53faddf57b3fd7 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:44.596576 18557 leader_election.cc:302] T 06e82c706b9c4f35bf53faddf57b3fd7 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4
W20240430 20:11:44.597047 18558 consensus_peers.cc:485] T e6c0c808bfc94be3837ab835292b3c1b P 5e1c6397f09747f091534faf93542710 -> Peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Couldn't send request to peer 2efb542aa55e439b99ccb98d1c5c884a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e6c0c808bfc94be3837ab835292b3c1b. This is attempt 1: this message will repeat every 5th retry.
I20240430 20:11:44.597280 20489 raft_consensus.cc:2726] T 06e82c706b9c4f35bf53faddf57b3fd7 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:44.598165 19489 ts_tablet_manager.cc:1888] T e87fb2f0361d4aa3ada27bbdfa88e94c P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:44.598584 18988 raft_consensus.cc:348] T 2e79df8744e74b5991a79ce978ff0202 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.599846 18988 raft_consensus.cc:374] T 2e79df8744e74b5991a79ce978ff0202 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:44.599958 18988 raft_consensus.cc:729] T 2e79df8744e74b5991a79ce978ff0202 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:44.600268 18988 consensus_queue.cc:260] T 2e79df8744e74b5991a79ce978ff0202 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.601004 18988 ts_tablet_manager.cc:1398] T 2e79df8744e74b5991a79ce978ff0202 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.031s user 0.003s sys 0.000s
I20240430 20:11:44.601320 18988 tablet_bootstrap.cc:492] T 30a521916bda46678be20a066bd1d3f3 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:44.602965 18988 tablet_bootstrap.cc:654] T 30a521916bda46678be20a066bd1d3f3 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:44.603616 20047 raft_consensus.cc:482] T 7683caba80594627900a0185f772c131 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:44.603814 20047 raft_consensus.cc:504] T 7683caba80594627900a0185f772c131 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.605069 20047 leader_election.cc:288] T 7683caba80594627900a0185f772c131 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:44.605593 18478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7683caba80594627900a0185f772c131" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:44.605820 18478 raft_consensus.cc:2454] T 7683caba80594627900a0185f772c131 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 0.
I20240430 20:11:44.606269 18558 leader_election.cc:302] T 7683caba80594627900a0185f772c131 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:44.606515 20047 raft_consensus.cc:2781] T 7683caba80594627900a0185f772c131 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:44.606631 20047 raft_consensus.cc:482] T 7683caba80594627900a0185f772c131 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:44.606722 20047 raft_consensus.cc:3037] T 7683caba80594627900a0185f772c131 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:44.607053 18763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7683caba80594627900a0185f772c131" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
W20240430 20:11:44.607826 18557 leader_election.cc:341] T 7683caba80594627900a0185f772c131 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:44.608418 19489 log.cc:1196] T e87fb2f0361d4aa3ada27bbdfa88e94c P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/e87fb2f0361d4aa3ada27bbdfa88e94c
I20240430 20:11:44.611317 20047 raft_consensus.cc:504] T 7683caba80594627900a0185f772c131 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.612015 19489 ts_tablet_manager.cc:1909] T e87fb2f0361d4aa3ada27bbdfa88e94c P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:44.612339 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 208) took 7100 ms. Trace:
I20240430 20:11:44.612447 19489 rpcz_store.cc:276] 0430 20:11:37.511737 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:38.607707 (+1095970us) service_pool.cc:225] Handling call
0430 20:11:44.570437 (+5962730us) ts_tablet_manager.cc:1120] Deleting tablet e87fb2f0361d4aa3ada27bbdfa88e94c
0430 20:11:44.570450 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:44.592605 (+ 22155us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:44.598034 (+ 5429us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:44.612316 (+ 14282us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5962527}
I20240430 20:11:44.612901 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 210) took 6204 ms. Trace:
I20240430 20:11:44.612967 19001 catalog_manager.cc:4830] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): tablet e87fb2f0361d4aa3ada27bbdfa88e94c (table default.Table_2f122c3c1c5949108231e55f6326e934 [id=02a19d9d05264236b35244d4e9998d36]) successfully deleted
I20240430 20:11:44.613188 20047 leader_election.cc:288] T 7683caba80594627900a0185f772c131 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:44.613444 18763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7683caba80594627900a0185f772c131" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:44.613950 18983 tablet_bootstrap.cc:492] T f6cfd089f08b47409fb2a6cf4f6b0385 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:44.614084 18983 ts_tablet_manager.cc:1367] T f6cfd089f08b47409fb2a6cf4f6b0385 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.091s user 0.005s sys 0.006s
I20240430 20:11:44.614246 18478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7683caba80594627900a0185f772c131" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:44.614431 18478 raft_consensus.cc:3037] T 7683caba80594627900a0185f772c131 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:44.615351 18985 ts_tablet_manager.cc:1398] T 06d28b6f48a34b17b0d7fb3102df2f4c P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.043s user 0.001s sys 0.001s
I20240430 20:11:44.615942 18983 raft_consensus.cc:348] T f6cfd089f08b47409fb2a6cf4f6b0385 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.616124 18983 raft_consensus.cc:374] T f6cfd089f08b47409fb2a6cf4f6b0385 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:44.616180 18983 raft_consensus.cc:729] T f6cfd089f08b47409fb2a6cf4f6b0385 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:44.617223 18478 raft_consensus.cc:2454] T 7683caba80594627900a0185f772c131 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
I20240430 20:11:44.617136 18983 consensus_queue.cc:260] T f6cfd089f08b47409fb2a6cf4f6b0385 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.617676 18983 ts_tablet_manager.cc:1398] T f6cfd089f08b47409fb2a6cf4f6b0385 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240430 20:11:44.618299 18558 leader_election.cc:302] T 7683caba80594627900a0185f772c131 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:44.618779 20488 raft_consensus.cc:2781] T 7683caba80594627900a0185f772c131 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:44.618880 20488 raft_consensus.cc:686] T 7683caba80594627900a0185f772c131 P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Becoming Leader. State: Replica: 5e1c6397f09747f091534faf93542710, State: Running, Role: LEADER
I20240430 20:11:44.619186 18983 tablet_bootstrap.cc:492] T 66dd767bd3f8494f94061fb89e5e5680 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:44.619088 20488 consensus_queue.cc:237] T 7683caba80594627900a0185f772c131 P 5e1c6397f09747f091534faf93542710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.620213 18985 tablet_bootstrap.cc:492] T ae20f44df50343289b0a512a4ea53692 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
W20240430 20:11:44.620337 18557 leader_election.cc:341] T 7683caba80594627900a0185f772c131 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:44.621191 18763 raft_consensus.cc:1261] T e2d90ff17e574f88b2061b1c117d4b7d P 3c51c33d5616482cafb68e26688820f4 [term 3 FOLLOWER]: Refusing update from remote peer 5e1c6397f09747f091534faf93542710: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 3. (term mismatch)
I20240430 20:11:44.621285 18763 pending_rounds.cc:76] T e2d90ff17e574f88b2061b1c117d4b7d P 3c51c33d5616482cafb68e26688820f4: Aborting all ops after (but not including) 2
I20240430 20:11:44.621316 18763 pending_rounds.cc:97] T e2d90ff17e574f88b2061b1c117d4b7d P 3c51c33d5616482cafb68e26688820f4: Aborting uncommitted NO_OP operation due to leader change: 2.3
I20240430 20:11:44.621351 18763 raft_consensus.cc:2866] T e2d90ff17e574f88b2061b1c117d4b7d P 3c51c33d5616482cafb68e26688820f4 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240430 20:11:44.622054 20047 consensus_queue.cc:1035] T e2d90ff17e574f88b2061b1c117d4b7d P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240430 20:11:44.623953 18985 tablet_bootstrap.cc:654] T ae20f44df50343289b0a512a4ea53692 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:44.628159 19471 ts_tablet_manager.cc:1888] T 06d28b6f48a34b17b0d7fb3102df2f4c P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:44.628297 19471 log.cc:1196] T 06d28b6f48a34b17b0d7fb3102df2f4c P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/06d28b6f48a34b17b0d7fb3102df2f4c
I20240430 20:11:44.628615 19471 ts_tablet_manager.cc:1909] T 06d28b6f48a34b17b0d7fb3102df2f4c P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:44.632026 18983 tablet_bootstrap.cc:654] T 66dd767bd3f8494f94061fb89e5e5680 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:44.640056 19489 rpcz_store.cc:276] 0430 20:11:38.407972 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:38.608301 (+200329us) service_pool.cc:225] Handling call
0430 20:11:44.612744 (+6004443us) ts_tablet_manager.cc:1120] Deleting tablet 558e71ec370d46ca8fbdcec5b57eb2ec
0430 20:11:44.612754 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:44.612884 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6004239}
W20240430 20:11:44.640637 19001 catalog_manager.cc:4810] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): delete failed for tablet 558e71ec370d46ca8fbdcec5b57eb2ec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 558e71ec370d46ca8fbdcec5b57eb2ec already in progress: creating tablet
I20240430 20:11:44.641413 19489 tablet_replica.cc:330] T b76a4105a2974fe280e759349037d027 P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:44.641611 19489 raft_consensus.cc:2229] T b76a4105a2974fe280e759349037d027 P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Raft consensus shutting down.
I20240430 20:11:44.641928 19489 raft_consensus.cc:2258] T b76a4105a2974fe280e759349037d027 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:44.643945 19489 ts_tablet_manager.cc:1875] T b76a4105a2974fe280e759349037d027 P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:44.647634 18985 tablet_bootstrap.cc:492] T ae20f44df50343289b0a512a4ea53692 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:44.647786 18985 ts_tablet_manager.cc:1367] T ae20f44df50343289b0a512a4ea53692 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.028s user 0.004s sys 0.003s
I20240430 20:11:44.648759 18985 raft_consensus.cc:348] T ae20f44df50343289b0a512a4ea53692 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.648996 18985 raft_consensus.cc:374] T ae20f44df50343289b0a512a4ea53692 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:44.649055 18985 raft_consensus.cc:729] T ae20f44df50343289b0a512a4ea53692 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:44.649278 18985 consensus_queue.cc:260] T ae20f44df50343289b0a512a4ea53692 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.649925 18985 ts_tablet_manager.cc:1398] T ae20f44df50343289b0a512a4ea53692 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240430 20:11:44.650202 19028 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_104ae015924a48fd8d27220792f07470" } modify_external_catalogs: true
I20240430 20:11:44.650413 18985 tablet_bootstrap.cc:492] T a9f37f0f7ce8425f9b24699513d41d61 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:44.650410 19028 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_104ae015924a48fd8d27220792f07470" } modify_external_catalogs: true
I20240430 20:11:44.659471 18983 tablet_bootstrap.cc:492] T 66dd767bd3f8494f94061fb89e5e5680 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:44.664237 19033 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet aa7acba830044a0d80084ca2ab12422d
I20240430 20:11:44.665084 18988 tablet_bootstrap.cc:492] T 30a521916bda46678be20a066bd1d3f3 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:44.665231 18988 ts_tablet_manager.cc:1367] T 30a521916bda46678be20a066bd1d3f3 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.064s user 0.004s sys 0.000s
I20240430 20:11:44.665958 18988 raft_consensus.cc:348] T 30a521916bda46678be20a066bd1d3f3 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.666146 18988 raft_consensus.cc:374] T 30a521916bda46678be20a066bd1d3f3 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:44.666188 18988 raft_consensus.cc:729] T 30a521916bda46678be20a066bd1d3f3 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:44.666374 18988 consensus_queue.cc:260] T 30a521916bda46678be20a066bd1d3f3 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.666911 18988 ts_tablet_manager.cc:1398] T 30a521916bda46678be20a066bd1d3f3 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240430 20:11:44.667191 18988 tablet_bootstrap.cc:492] T 115c4a12440647a4bb46eb6824f7ea98 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:44.668332 20205 raft_consensus.cc:482] T 73453a7fa30342e29db552b17b62f0c1 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:44.668483 20205 raft_consensus.cc:504] T 73453a7fa30342e29db552b17b62f0c1 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.659641 18983 ts_tablet_manager.cc:1367] T 66dd767bd3f8494f94061fb89e5e5680 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.041s user 0.003s sys 0.000s
I20240430 20:11:44.669570 19033 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_71775acb989f4c2f87d670dff67a93b6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240430 20:11:44.669975 20205 leader_election.cc:288] T 73453a7fa30342e29db552b17b62f0c1 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
W20240430 20:11:44.670183 19033 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_71775acb989f4c2f87d670dff67a93b6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:44.671289 19489 ts_tablet_manager.cc:1888] T b76a4105a2974fe280e759349037d027 P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:44.671397 19489 log.cc:1196] T b76a4105a2974fe280e759349037d027 P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/b76a4105a2974fe280e759349037d027
I20240430 20:11:44.671650 18983 raft_consensus.cc:348] T 66dd767bd3f8494f94061fb89e5e5680 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.671859 18983 raft_consensus.cc:374] T 66dd767bd3f8494f94061fb89e5e5680 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:44.671980 18983 raft_consensus.cc:729] T 66dd767bd3f8494f94061fb89e5e5680 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:44.672225 18983 consensus_queue.cc:260] T 66dd767bd3f8494f94061fb89e5e5680 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.672899 18983 ts_tablet_manager.cc:1398] T 66dd767bd3f8494f94061fb89e5e5680 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
W20240430 20:11:44.676528 18699 leader_election.cc:341] T 73453a7fa30342e29db552b17b62f0c1 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Not found: Tablet not found: 73453a7fa30342e29db552b17b62f0c1
I20240430 20:11:44.680559 19489 ts_tablet_manager.cc:1909] T b76a4105a2974fe280e759349037d027 P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:44.681460 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 211) took 6192 ms. Trace:
I20240430 20:11:44.681581 19489 rpcz_store.cc:276] 0430 20:11:38.489158 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:38.608515 (+119357us) service_pool.cc:225] Handling call
0430 20:11:44.641346 (+6032831us) ts_tablet_manager.cc:1120] Deleting tablet b76a4105a2974fe280e759349037d027
0430 20:11:44.641359 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:44.662790 (+ 21431us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:44.671181 (+ 8391us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:44.681437 (+ 10256us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6032660}
I20240430 20:11:44.687275 20205 raft_consensus.cc:482] T 34130f2820064fa0898ef1f9d7de1c6d P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:44.687490 18988 tablet_bootstrap.cc:654] T 115c4a12440647a4bb46eb6824f7ea98 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:44.687430 20205 raft_consensus.cc:504] T 34130f2820064fa0898ef1f9d7de1c6d P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.687523 19001 catalog_manager.cc:4830] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): tablet b76a4105a2974fe280e759349037d027 (table default.Table_95220daaf6d44c08beff4ed6abdbea79 [id=75f0256acc45402d8b6809d92c38a1d8]) successfully deleted
I20240430 20:11:44.687840 19489 tablet_replica.cc:330] T 24dcc20832804229a2664873bf6ddaa7 P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:44.688037 19489 raft_consensus.cc:2229] T 24dcc20832804229a2664873bf6ddaa7 P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Raft consensus shutting down.
I20240430 20:11:44.688094 20205 leader_election.cc:288] T 34130f2820064fa0898ef1f9d7de1c6d P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:44.688436 19489 raft_consensus.cc:2258] T 24dcc20832804229a2664873bf6ddaa7 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240430 20:11:44.689118 18699 leader_election.cc:341] T 34130f2820064fa0898ef1f9d7de1c6d P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Not found: Tablet not found: 34130f2820064fa0898ef1f9d7de1c6d
I20240430 20:11:44.689811 19489 ts_tablet_manager.cc:1875] T 24dcc20832804229a2664873bf6ddaa7 P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240430 20:11:44.691047 19049 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7a0176f8a74c47efbbe3d3343b3dc896 has no consensus state
I20240430 20:11:44.694988 18985 tablet_bootstrap.cc:654] T a9f37f0f7ce8425f9b24699513d41d61 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:44.703070 18620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34130f2820064fa0898ef1f9d7de1c6d" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
W20240430 20:11:44.703272 18601 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40218: Illegal state: replica 5e1c6397f09747f091534faf93542710 is not leader of this config: current role FOLLOWER
I20240430 20:11:44.713353 18985 tablet_bootstrap.cc:492] T a9f37f0f7ce8425f9b24699513d41d61 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:44.694579 19489 ts_tablet_manager.cc:1888] T 24dcc20832804229a2664873bf6ddaa7 P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20240430 20:11:44.704883 18696 leader_election.cc:341] T 73453a7fa30342e29db552b17b62f0c1 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Not found: Tablet not found: 73453a7fa30342e29db552b17b62f0c1
I20240430 20:11:44.713832 18696 leader_election.cc:302] T 73453a7fa30342e29db552b17b62f0c1 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710
I20240430 20:11:44.713892 19489 log.cc:1196] T 24dcc20832804229a2664873bf6ddaa7 P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/24dcc20832804229a2664873bf6ddaa7
I20240430 20:11:44.714293 19489 ts_tablet_manager.cc:1909] T 24dcc20832804229a2664873bf6ddaa7 P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:44.714391 18696 leader_election.cc:302] T 34130f2820064fa0898ef1f9d7de1c6d P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710
I20240430 20:11:44.714982 20205 raft_consensus.cc:2726] T 73453a7fa30342e29db552b17b62f0c1 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:44.715140 20205 raft_consensus.cc:2726] T 34130f2820064fa0898ef1f9d7de1c6d P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:44.715255 18985 ts_tablet_manager.cc:1367] T a9f37f0f7ce8425f9b24699513d41d61 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.065s user 0.002s sys 0.004s
I20240430 20:11:44.716799 18985 raft_consensus.cc:348] T a9f37f0f7ce8425f9b24699513d41d61 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.717530 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 212) took 5720 ms. Trace:
I20240430 20:11:44.718469 18985 raft_consensus.cc:374] T a9f37f0f7ce8425f9b24699513d41d61 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:44.718711 19489 rpcz_store.cc:276] 0430 20:11:38.996637 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:38.996711 (+ 74us) service_pool.cc:225] Handling call
0430 20:11:44.687789 (+5691078us) ts_tablet_manager.cc:1120] Deleting tablet 24dcc20832804229a2664873bf6ddaa7
0430 20:11:44.687800 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:44.692536 (+ 4736us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:44.694470 (+ 1934us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:44.717507 (+ 23037us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5688920}
I20240430 20:11:44.719017 18985 raft_consensus.cc:729] T a9f37f0f7ce8425f9b24699513d41d61 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:44.719518 18988 tablet_bootstrap.cc:492] T 115c4a12440647a4bb46eb6824f7ea98 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:44.719553 19001 catalog_manager.cc:4830] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): tablet 24dcc20832804229a2664873bf6ddaa7 (table default.Table_e3160f5ced1b4718b479401ca4df675c [id=2d44ec94131148048021b2ee6366ba30]) successfully deleted
I20240430 20:11:44.719717 18988 ts_tablet_manager.cc:1367] T 115c4a12440647a4bb46eb6824f7ea98 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.053s user 0.008s sys 0.003s
W20240430 20:11:44.719753 18557 leader_election.cc:341] T f940d17b02b541289121463e05b0bb5f P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:44.720378 18750 raft_consensus.cc:3037] T a59ef6c545dd41a28265a4c1b8ce10e0 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:44.720492 18988 raft_consensus.cc:348] T 115c4a12440647a4bb46eb6824f7ea98 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.720767 18988 raft_consensus.cc:374] T 115c4a12440647a4bb46eb6824f7ea98 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:44.720885 18988 raft_consensus.cc:729] T 115c4a12440647a4bb46eb6824f7ea98 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:44.721185 18988 consensus_queue.cc:260] T 115c4a12440647a4bb46eb6824f7ea98 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.721915 18988 ts_tablet_manager.cc:1398] T 115c4a12440647a4bb46eb6824f7ea98 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240430 20:11:44.722244 18988 tablet_bootstrap.cc:492] T 2cdccf4be7094ca5b863dd5c3432d55d P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:44.723708 18988 tablet_bootstrap.cc:654] T 2cdccf4be7094ca5b863dd5c3432d55d P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:44.723989 18985 consensus_queue.cc:260] T a9f37f0f7ce8425f9b24699513d41d61 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.724638 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 220) took 5182 ms. Trace:
I20240430 20:11:44.725039 19489 rpcz_store.cc:276] 0430 20:11:39.542657 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:39.542721 (+ 64us) service_pool.cc:225] Handling call
0430 20:11:44.724452 (+5181731us) ts_tablet_manager.cc:1120] Deleting tablet de1dfe6c82d34a4d8d02e466c55327ff
0430 20:11:44.724466 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:44.724613 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5180597}
I20240430 20:11:44.725999 18985 ts_tablet_manager.cc:1398] T a9f37f0f7ce8425f9b24699513d41d61 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.011s user 0.001s sys 0.001s
W20240430 20:11:44.726735 19001 catalog_manager.cc:4810] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): delete failed for tablet de1dfe6c82d34a4d8d02e466c55327ff because tablet deleting was already in progress. No further retry: Already present: State transition of tablet de1dfe6c82d34a4d8d02e466c55327ff already in progress: creating tablet
I20240430 20:11:44.727692 18985 tablet_bootstrap.cc:492] T ae24e75a0d42441ebc97400ffa94145e P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:44.729003 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 221) took 5139 ms. Trace:
I20240430 20:11:44.729149 19489 rpcz_store.cc:276] 0430 20:11:39.589515 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:39.589606 (+ 91us) service_pool.cc:225] Handling call
0430 20:11:44.727256 (+5137650us) ts_tablet_manager.cc:1120] Deleting tablet 90a694bd5db649afa50f3c4e752939c1
0430 20:11:44.728845 (+ 1589us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:44.728983 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5137153}
W20240430 20:11:44.729754 19001 catalog_manager.cc:4810] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): delete failed for tablet 90a694bd5db649afa50f3c4e752939c1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 90a694bd5db649afa50f3c4e752939c1 already in progress: creating tablet
I20240430 20:11:44.730198 19489 tablet_replica.cc:330] T 53a899fa5a424a0d99f1b47925e8c7ad P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:44.730495 19489 raft_consensus.cc:2229] T 53a899fa5a424a0d99f1b47925e8c7ad P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:44.732153 18985 tablet_bootstrap.cc:654] T ae24e75a0d42441ebc97400ffa94145e P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:44.736395 19489 raft_consensus.cc:2258] T 53a899fa5a424a0d99f1b47925e8c7ad P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:44.737298 19489 ts_tablet_manager.cc:1875] T 53a899fa5a424a0d99f1b47925e8c7ad P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:44.737764 20198 raft_consensus.cc:482] T 30a521916bda46678be20a066bd1d3f3 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:44.737916 20198 raft_consensus.cc:504] T 30a521916bda46678be20a066bd1d3f3 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.738643 20198 leader_election.cc:288] T 30a521916bda46678be20a066bd1d3f3 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:44.738776 20198 raft_consensus.cc:482] T 5f02d8862e1f4ccfa9dbda73dbe18615 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:44.738901 20198 raft_consensus.cc:504] T 5f02d8862e1f4ccfa9dbda73dbe18615 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.739295 18620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30a521916bda46678be20a066bd1d3f3" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:44.739456 20198 leader_election.cc:288] T 5f02d8862e1f4ccfa9dbda73dbe18615 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:44.739493 18620 raft_consensus.cc:2454] T 30a521916bda46678be20a066bd1d3f3 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:44.740281 18475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30a521916bda46678be20a066bd1d3f3" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:44.740456 18696 leader_election.cc:302] T 30a521916bda46678be20a066bd1d3f3 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:44.740489 18475 raft_consensus.cc:2454] T 30a521916bda46678be20a066bd1d3f3 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:44.740796 18473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f02d8862e1f4ccfa9dbda73dbe18615" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:44.740937 18620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f02d8862e1f4ccfa9dbda73dbe18615" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:44.741775 20198 raft_consensus.cc:2781] T 30a521916bda46678be20a066bd1d3f3 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:44.741860 20198 raft_consensus.cc:482] T 30a521916bda46678be20a066bd1d3f3 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:44.741940 20198 raft_consensus.cc:3037] T 30a521916bda46678be20a066bd1d3f3 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:44.742414 18696 leader_election.cc:302] T 5f02d8862e1f4ccfa9dbda73dbe18615 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710
I20240430 20:11:44.742904 20205 raft_consensus.cc:2726] T 5f02d8862e1f4ccfa9dbda73dbe18615 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:44.744778 20198 raft_consensus.cc:504] T 30a521916bda46678be20a066bd1d3f3 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.746090 20198 leader_election.cc:288] T 30a521916bda46678be20a066bd1d3f3 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Requested vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:44.746856 18988 tablet_bootstrap.cc:492] T 2cdccf4be7094ca5b863dd5c3432d55d P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:44.746968 18620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30a521916bda46678be20a066bd1d3f3" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:44.747108 18988 ts_tablet_manager.cc:1367] T 2cdccf4be7094ca5b863dd5c3432d55d P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.000s
I20240430 20:11:44.747184 18620 raft_consensus.cc:3037] T 30a521916bda46678be20a066bd1d3f3 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:44.747803 18988 raft_consensus.cc:348] T 2cdccf4be7094ca5b863dd5c3432d55d P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.747974 18988 raft_consensus.cc:374] T 2cdccf4be7094ca5b863dd5c3432d55d P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:44.748018 18988 raft_consensus.cc:729] T 2cdccf4be7094ca5b863dd5c3432d55d P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:44.748242 18988 consensus_queue.cc:260] T 2cdccf4be7094ca5b863dd5c3432d55d P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.748713 18988 ts_tablet_manager.cc:1398] T 2cdccf4be7094ca5b863dd5c3432d55d P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240430 20:11:44.748973 18988 tablet_bootstrap.cc:492] T f7b0132c17d44a9ba17288e0321a7f11 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:44.750308 18988 tablet_bootstrap.cc:654] T f7b0132c17d44a9ba17288e0321a7f11 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:44.750289 18620 raft_consensus.cc:2454] T 30a521916bda46678be20a066bd1d3f3 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:44.750874 18983 tablet_bootstrap.cc:492] T 4f6f9ae722da49e58c9b72107f6da2fa P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:44.751149 18696 leader_election.cc:302] T 30a521916bda46678be20a066bd1d3f3 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:44.752413 18983 tablet_bootstrap.cc:654] T 4f6f9ae722da49e58c9b72107f6da2fa P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:44.753216 20487 raft_consensus.cc:482] T ae20f44df50343289b0a512a4ea53692 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:44.753340 20487 raft_consensus.cc:504] T ae20f44df50343289b0a512a4ea53692 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.754078 20487 leader_election.cc:288] T ae20f44df50343289b0a512a4ea53692 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:44.755306 18756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae20f44df50343289b0a512a4ea53692" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:44.755327 18475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae20f44df50343289b0a512a4ea53692" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:44.755597 18475 raft_consensus.cc:2454] T ae20f44df50343289b0a512a4ea53692 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 0.
I20240430 20:11:44.755944 20105 raft_consensus.cc:482] T 99203cc522de424184c6f97453f166fd P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:44.756013 18558 leader_election.cc:302] T ae20f44df50343289b0a512a4ea53692 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:44.756106 20105 raft_consensus.cc:504] T 99203cc522de424184c6f97453f166fd P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:44.756271 20487 raft_consensus.cc:2781] T ae20f44df50343289b0a512a4ea53692 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:44.756330 20487 raft_consensus.cc:482] T ae20f44df50343289b0a512a4ea53692 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:44.756387 20487 raft_consensus.cc:3037] T ae20f44df50343289b0a512a4ea53692 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:44.756872 20105 leader_election.cc:288] T 99203cc522de424184c6f97453f166fd P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:44.758958 20487 raft_consensus.cc:504] T ae20f44df50343289b0a512a4ea53692 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
W20240430 20:11:44.759225 18557 leader_election.cc:341] T ae20f44df50343289b0a512a4ea53692 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:44.759750 18756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99203cc522de424184c6f97453f166fd" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:44.760007 18763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae20f44df50343289b0a512a4ea53692" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
W20240430 20:11:44.760381 18417 leader_election.cc:341] T 99203cc522de424184c6f97453f166fd P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
W20240430 20:11:44.760566 18557 leader_election.cc:341] T ae20f44df50343289b0a512a4ea53692 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:44.760895 20198 raft_consensus.cc:2781] T 30a521916bda46678be20a066bd1d3f3 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:44.761035 18620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99203cc522de424184c6f97453f166fd" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:44.761114 20198 raft_consensus.cc:686] T 30a521916bda46678be20a066bd1d3f3 P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Becoming Leader. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Running, Role: LEADER
I20240430 20:11:44.761286 18620 raft_consensus.cc:2454] T 99203cc522de424184c6f97453f166fd P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 0.
I20240430 20:11:44.761364 20198 consensus_queue.cc:237] T 30a521916bda46678be20a066bd1d3f3 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.761495 20487 leader_election.cc:288] T ae20f44df50343289b0a512a4ea53692 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:44.761685 18415 leader_election.cc:302] T 99203cc522de424184c6f97453f166fd P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters: 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:44.762045 18475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae20f44df50343289b0a512a4ea53692" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:44.762182 20105 raft_consensus.cc:2781] T 99203cc522de424184c6f97453f166fd P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:44.762261 20105 raft_consensus.cc:482] T 99203cc522de424184c6f97453f166fd P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:44.762287 18475 raft_consensus.cc:3037] T ae20f44df50343289b0a512a4ea53692 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:44.762333 20105 raft_consensus.cc:3037] T 99203cc522de424184c6f97453f166fd P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:44.763341 18478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30a521916bda46678be20a066bd1d3f3" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:44.763586 18478 raft_consensus.cc:3037] T 30a521916bda46678be20a066bd1d3f3 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:44.765110 18475 raft_consensus.cc:2454] T ae20f44df50343289b0a512a4ea53692 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
I20240430 20:11:44.765224 20105 raft_consensus.cc:504] T 99203cc522de424184c6f97453f166fd P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:44.765647 18558 leader_election.cc:302] T ae20f44df50343289b0a512a4ea53692 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters: 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:44.766099 20487 raft_consensus.cc:2781] T ae20f44df50343289b0a512a4ea53692 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:44.766137 18478 raft_consensus.cc:2454] T 30a521916bda46678be20a066bd1d3f3 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:44.766398 18763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99203cc522de424184c6f97453f166fd" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:44.766731 20487 raft_consensus.cc:686] T ae20f44df50343289b0a512a4ea53692 P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Becoming Leader. State: Replica: 5e1c6397f09747f091534faf93542710, State: Running, Role: LEADER
W20240430 20:11:44.766944 18417 leader_election.cc:341] T 99203cc522de424184c6f97453f166fd P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:44.767236 20487 consensus_queue.cc:237] T ae20f44df50343289b0a512a4ea53692 P 5e1c6397f09747f091534faf93542710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.767663 20105 leader_election.cc:288] T 99203cc522de424184c6f97453f166fd P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:44.767483 18616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99203cc522de424184c6f97453f166fd" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:44.768649 18616 raft_consensus.cc:3037] T 99203cc522de424184c6f97453f166fd P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:44.771716 18616 raft_consensus.cc:2454] T 99203cc522de424184c6f97453f166fd P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 1.
I20240430 20:11:44.772230 18415 leader_election.cc:302] T 99203cc522de424184c6f97453f166fd P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters: 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:44.772847 20105 raft_consensus.cc:2781] T 99203cc522de424184c6f97453f166fd P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:44.773170 20105 raft_consensus.cc:686] T 99203cc522de424184c6f97453f166fd P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Becoming Leader. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Running, Role: LEADER
I20240430 20:11:44.773486 20105 consensus_queue.cc:237] T 99203cc522de424184c6f97453f166fd P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
W20240430 20:11:44.779728 19004 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 7a0176f8a74c47efbbe3d3343b3dc896 (table default.Table_94e5f2a77765434e8a76d9a558b23ae2 [id=73e5c9f0470d47849109819c063e1ce4], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 7a0176f8a74c47efbbe3d3343b3dc896 has no consensus state
I20240430 20:11:44.779767 19489 ts_tablet_manager.cc:1888] T 53a899fa5a424a0d99f1b47925e8c7ad P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:44.779877 19489 log.cc:1196] T 53a899fa5a424a0d99f1b47925e8c7ad P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/53a899fa5a424a0d99f1b47925e8c7ad
I20240430 20:11:44.780282 19489 ts_tablet_manager.cc:1909] T 53a899fa5a424a0d99f1b47925e8c7ad P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:44.781208 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 222) took 5093 ms. Trace:
I20240430 20:11:44.781328 19489 rpcz_store.cc:276] 0430 20:11:39.687466 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:39.687549 (+ 83us) service_pool.cc:225] Handling call
0430 20:11:44.730140 (+5042591us) ts_tablet_manager.cc:1120] Deleting tablet 53a899fa5a424a0d99f1b47925e8c7ad
0430 20:11:44.730153 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:44.752820 (+ 22667us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:44.779638 (+ 26818us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:44.781175 (+ 1537us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3686,"tablet-delete.queue_time_us":5042323}
I20240430 20:11:44.783380 20572 tablet.cc:1722] T ebf74809b52f47f3b0ed21b6ac0f541a P 3c51c33d5616482cafb68e26688820f4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240430 20:11:44.789785 20235 raft_consensus.cc:482] T 06d28b6f48a34b17b0d7fb3102df2f4c P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:44.790007 20235 raft_consensus.cc:504] T 06d28b6f48a34b17b0d7fb3102df2f4c P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.790887 20235 leader_election.cc:288] T 06d28b6f48a34b17b0d7fb3102df2f4c P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
W20240430 20:11:44.791752 18415 leader_election.cc:341] T 06d28b6f48a34b17b0d7fb3102df2f4c P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Not found: Tablet not found: 06d28b6f48a34b17b0d7fb3102df2f4c
I20240430 20:11:44.791775 18763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06d28b6f48a34b17b0d7fb3102df2f4c" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
W20240430 20:11:44.792513 18417 leader_election.cc:341] T 06d28b6f48a34b17b0d7fb3102df2f4c P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:44.792640 18417 leader_election.cc:302] T 06d28b6f48a34b17b0d7fb3102df2f4c P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a; no voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710
I20240430 20:11:44.793180 20235 raft_consensus.cc:2726] T 06d28b6f48a34b17b0d7fb3102df2f4c P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240430 20:11:44.796061 18557 consensus_peers.cc:485] T 680553d4d0bc4fcb839235c1832757e1 P 5e1c6397f09747f091534faf93542710 -> Peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Couldn't send request to peer 3c51c33d5616482cafb68e26688820f4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 680553d4d0bc4fcb839235c1832757e1. This is attempt 1: this message will repeat every 5th retry.
I20240430 20:11:44.796728 20205 raft_consensus.cc:482] T ae87f5e92a934e2182164c231bc76df0 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:44.796751 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 173) took 10007 ms. Trace:
I20240430 20:11:44.796831 19471 rpcz_store.cc:276] 0430 20:11:34.788913 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:34.805949 (+ 17036us) service_pool.cc:225] Handling call
0430 20:11:44.509959 (+9704010us) ts_tablet_manager.cc:1120] Deleting tablet 06d28b6f48a34b17b0d7fb3102df2f4c
0430 20:11:44.509968 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:44.545294 (+ 35326us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:44.628040 (+ 82746us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:44.796729 (+168689us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":9056431}
I20240430 20:11:44.796878 20205 raft_consensus.cc:504] T ae87f5e92a934e2182164c231bc76df0 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.797278 18999 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet 06d28b6f48a34b17b0d7fb3102df2f4c (table default.Table_4a09e48b08fa4c4387262a5b92b2a42d [id=565591f1f0e84bc89c8d366ef53ae969]) successfully deleted
I20240430 20:11:44.797561 20205 leader_election.cc:288] T ae87f5e92a934e2182164c231bc76df0 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:44.798830 18616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae87f5e92a934e2182164c231bc76df0" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
W20240430 20:11:44.799088 18699 leader_election.cc:341] T ae87f5e92a934e2182164c231bc76df0 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Not found: Tablet not found: ae87f5e92a934e2182164c231bc76df0
I20240430 20:11:44.799443 18696 leader_election.cc:302] T ae87f5e92a934e2182164c231bc76df0 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710
I20240430 20:11:44.799989 20205 raft_consensus.cc:2726] T ae87f5e92a934e2182164c231bc76df0 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:44.803946 20562 tablet.cc:1722] T ebf74809b52f47f3b0ed21b6ac0f541a P 5e1c6397f09747f091534faf93542710: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240430 20:11:44.805197 20563 tablet.cc:1722] T ebf74809b52f47f3b0ed21b6ac0f541a P 2efb542aa55e439b99ccb98d1c5c884a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240430 20:11:44.805684 19495 ts_tablet_manager.cc:1888] T 115c4a12440647a4bb46eb6824f7ea98 P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:44.805809 19495 log.cc:1196] T 115c4a12440647a4bb46eb6824f7ea98 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/115c4a12440647a4bb46eb6824f7ea98
I20240430 20:11:44.807183 19489 tablet_replica.cc:330] T aec9c09cc0454acab6006bb3e10982b7 P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:44.808578 19495 ts_tablet_manager.cc:1909] T 115c4a12440647a4bb46eb6824f7ea98 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:44.808621 19489 raft_consensus.cc:2229] T aec9c09cc0454acab6006bb3e10982b7 P 3c51c33d5616482cafb68e26688820f4 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:44.809393 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 344) took 1358 ms. Trace:
I20240430 20:11:44.826277 18985 tablet_bootstrap.cc:492] T ae24e75a0d42441ebc97400ffa94145e P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:44.826449 18985 ts_tablet_manager.cc:1367] T ae24e75a0d42441ebc97400ffa94145e P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.099s user 0.004s sys 0.008s
I20240430 20:11:44.827419 18985 raft_consensus.cc:348] T ae24e75a0d42441ebc97400ffa94145e P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.814661 19489 raft_consensus.cc:2258] T aec9c09cc0454acab6006bb3e10982b7 P 3c51c33d5616482cafb68e26688820f4 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:44.829524 19489 ts_tablet_manager.cc:1875] T aec9c09cc0454acab6006bb3e10982b7 P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:44.831650 20489 raft_consensus.cc:482] T 677481aee5fa4eb4b189322d7a3f8f2e P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:44.832055 20489 raft_consensus.cc:504] T 677481aee5fa4eb4b189322d7a3f8f2e P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.835794 20489 leader_election.cc:288] T 677481aee5fa4eb4b189322d7a3f8f2e P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
W20240430 20:11:44.840920 18588 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40218: Illegal state: replica 5e1c6397f09747f091534faf93542710 is not leader of this config: current role FOLLOWER
I20240430 20:11:44.845973 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 186) took 8897 ms. Trace:
I20240430 20:11:44.846107 18763 raft_consensus.cc:3037] T 5f02d8862e1f4ccfa9dbda73dbe18615 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:44.846215 19471 rpcz_store.cc:276] 0430 20:11:35.948669 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:35.948713 (+ 44us) service_pool.cc:225] Handling call
0430 20:11:44.845818 (+8897105us) ts_tablet_manager.cc:1120] Deleting tablet c66c142419ca4e1399ecd1c8d1e62a59
0430 20:11:44.845836 (+ 18us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:44.845946 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7834885}
I20240430 20:11:44.846554 19471 tablet_replica.cc:330] T 84568f90b9c54cf292099808d4cbf5dc P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:44.846807 19471 raft_consensus.cc:2229] T 84568f90b9c54cf292099808d4cbf5dc P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Raft consensus shutting down.
W20240430 20:11:44.847048 18999 catalog_manager.cc:4797] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): delete failed for tablet c66c142419ca4e1399ecd1c8d1e62a59 because the tablet was not found. No further retry: Not found: Tablet not found: c66c142419ca4e1399ecd1c8d1e62a59
I20240430 20:11:44.847182 19471 raft_consensus.cc:2258] T 84568f90b9c54cf292099808d4cbf5dc P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:44.849164 18750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "677481aee5fa4eb4b189322d7a3f8f2e" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:44.849448 20205 raft_consensus.cc:482] T d61c279d20b0471a967942525ceb4fcb P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:44.849574 20205 raft_consensus.cc:504] T d61c279d20b0471a967942525ceb4fcb P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.850293 20205 leader_election.cc:288] T d61c279d20b0471a967942525ceb4fcb P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:44.851256 18616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d61c279d20b0471a967942525ceb4fcb" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:44.851539 18616 raft_consensus.cc:2379] T d61c279d20b0471a967942525ceb4fcb P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c51c33d5616482cafb68e26688820f4 in current term 1: Already voted for candidate 2efb542aa55e439b99ccb98d1c5c884a in this term.
W20240430 20:11:44.855393 18557 leader_election.cc:341] T 677481aee5fa4eb4b189322d7a3f8f2e P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:44.833993 18482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "677481aee5fa4eb4b189322d7a3f8f2e" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
W20240430 20:11:44.865666 18558 leader_election.cc:341] T 677481aee5fa4eb4b189322d7a3f8f2e P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:44.865818 18558 leader_election.cc:302] T 677481aee5fa4eb4b189322d7a3f8f2e P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:44.862301 18985 raft_consensus.cc:374] T ae24e75a0d42441ebc97400ffa94145e P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:44.866794 20047 raft_consensus.cc:2726] T 677481aee5fa4eb4b189322d7a3f8f2e P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:44.866219 18985 raft_consensus.cc:729] T ae24e75a0d42441ebc97400ffa94145e P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:44.854146 18478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d61c279d20b0471a967942525ceb4fcb" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:44.869251 18699 leader_election.cc:302] T d61c279d20b0471a967942525ceb4fcb P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710
I20240430 20:11:44.869771 20198 raft_consensus.cc:3037] T d61c279d20b0471a967942525ceb4fcb P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:44.872676 20198 raft_consensus.cc:2726] T d61c279d20b0471a967942525ceb4fcb P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:44.874311 20235 raft_consensus.cc:482] T 2e79df8744e74b5991a79ce978ff0202 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:44.874467 20235 raft_consensus.cc:504] T 2e79df8744e74b5991a79ce978ff0202 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.875277 20235 leader_election.cc:288] T 2e79df8744e74b5991a79ce978ff0202 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:44.875764 18750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e79df8744e74b5991a79ce978ff0202" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:44.875995 18750 raft_consensus.cc:2454] T 2e79df8744e74b5991a79ce978ff0202 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 0.
I20240430 20:11:44.876458 18417 leader_election.cc:302] T 2e79df8744e74b5991a79ce978ff0202 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters:
I20240430 20:11:44.876789 20235 raft_consensus.cc:2781] T 2e79df8744e74b5991a79ce978ff0202 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:44.876868 20235 raft_consensus.cc:482] T 2e79df8744e74b5991a79ce978ff0202 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:44.878401 20235 raft_consensus.cc:3037] T 2e79df8744e74b5991a79ce978ff0202 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:44.884124 20235 raft_consensus.cc:504] T 2e79df8744e74b5991a79ce978ff0202 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.886152 18620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e79df8744e74b5991a79ce978ff0202" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:44.886191 18750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e79df8744e74b5991a79ce978ff0202" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:44.886359 18620 raft_consensus.cc:3037] T 2e79df8744e74b5991a79ce978ff0202 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:44.886389 18750 raft_consensus.cc:3037] T 2e79df8744e74b5991a79ce978ff0202 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:44.886852 20235 leader_election.cc:288] T 2e79df8744e74b5991a79ce978ff0202 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:44.887130 20198 raft_consensus.cc:482] T 5bd0362ff41b4faca0205d136d596737 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:44.887338 20198 raft_consensus.cc:504] T 5bd0362ff41b4faca0205d136d596737 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.888975 20198 leader_election.cc:288] T 5bd0362ff41b4faca0205d136d596737 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:44.889379 18473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bd0362ff41b4faca0205d136d596737" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:44.889717 18473 raft_consensus.cc:2379] T 5bd0362ff41b4faca0205d136d596737 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c51c33d5616482cafb68e26688820f4 in current term 1: Already voted for candidate 5e1c6397f09747f091534faf93542710 in this term.
I20240430 20:11:44.893595 20198 raft_consensus.cc:482] T abdaf243e5be4ad58036b23151b14fe4 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:44.893790 18750 raft_consensus.cc:2454] T 2e79df8744e74b5991a79ce978ff0202 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 1.
I20240430 20:11:44.894699 18417 leader_election.cc:302] T 2e79df8744e74b5991a79ce978ff0202 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters:
I20240430 20:11:44.895311 20235 raft_consensus.cc:2781] T 2e79df8744e74b5991a79ce978ff0202 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:44.895694 20235 raft_consensus.cc:686] T 2e79df8744e74b5991a79ce978ff0202 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Becoming Leader. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Running, Role: LEADER
I20240430 20:11:44.896574 20235 consensus_queue.cc:237] T 2e79df8744e74b5991a79ce978ff0202 P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.893847 20198 raft_consensus.cc:504] T abdaf243e5be4ad58036b23151b14fe4 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.905237 20198 leader_election.cc:288] T abdaf243e5be4ad58036b23151b14fe4 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:44.905319 18473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abdaf243e5be4ad58036b23151b14fe4" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:44.905567 18473 raft_consensus.cc:2379] T abdaf243e5be4ad58036b23151b14fe4 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c51c33d5616482cafb68e26688820f4 in current term 1: Already voted for candidate 5e1c6397f09747f091534faf93542710 in this term.
I20240430 20:11:44.915776 19051 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: Sending DeleteTablet for 3 replicas of tablet 9f0b341a31224bf1be525b84b9500304
I20240430 20:11:44.915997 19027 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: Sending DeleteTablet for 3 replicas of tablet e3e74e76ab2b4fe492ffcc1838656fd1
I20240430 20:11:44.920650 19027 catalog_manager.cc:6554] ReplaceTablet: tablet e3e74e76ab2b4fe492ffcc1838656fd1 deleted and replaced by tablet 7fceb89e8ee148999f747b08a15fd274
I20240430 20:11:44.921031 18733 tablet_service.cc:1514] Processing DeleteTablet for tablet e3e74e76ab2b4fe492ffcc1838656fd1 with delete_type TABLET_DATA_DELETED (replaced by tablet 7fceb89e8ee148999f747b08a15fd274) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:44.921031 18742 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f0b341a31224bf1be525b84b9500304 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:44 UTC) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:44.923632 18615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bd0362ff41b4faca0205d136d596737" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:44.809514 19495 rpcz_store.cc:276] 0430 20:11:43.450751 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:43.450862 (+ 111us) service_pool.cc:225] Handling call
0430 20:11:44.575366 (+1124504us) ts_tablet_manager.cc:1120] Deleting tablet 115c4a12440647a4bb46eb6824f7ea98
0430 20:11:44.575387 (+ 21us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:44.635961 (+ 60574us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:44.805563 (+169602us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:44.809368 (+ 3805us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1117698}
I20240430 20:11:44.926374 19495 tablet_replica.cc:330] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:44.926582 19495 raft_consensus.cc:2229] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 2efb542aa55e439b99ccb98d1c5c884a [term 2 LEADER]: Raft consensus shutting down.
I20240430 20:11:44.928651 19004 catalog_manager.cc:4830] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): tablet 115c4a12440647a4bb46eb6824f7ea98 (table default.Table_cd719a43207f4bf983e1b8e6d236bb16 [id=47e0b04b17834deb833a8f31c123edc5]) successfully deleted
I20240430 20:11:44.928972 18445 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f0b341a31224bf1be525b84b9500304 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:44 UTC) from {username='slave'} at 127.0.0.1:41680
W20240430 20:11:44.915776 19023 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet aaef85eea87f4443943fd3692bb2a38f has no consensus state
I20240430 20:11:44.929570 18450 tablet_service.cc:1514] Processing DeleteTablet for tablet e3e74e76ab2b4fe492ffcc1838656fd1 with delete_type TABLET_DATA_DELETED (replaced by tablet 7fceb89e8ee148999f747b08a15fd274) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:44.929775 19495 raft_consensus.cc:2258] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 2efb542aa55e439b99ccb98d1c5c884a [term 2 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:44.929657 19023 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_df40d2a666d94cada72ebdb96b7d7f9e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:44.930280 19023 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_df40d2a666d94cada72ebdb96b7d7f9e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:44.930377 19495 ts_tablet_manager.cc:1875] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:44.931310 19028 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: Sending DeleteTablet for 3 replicas of tablet 76d6e9d547ca4a848eae852944b1d117
I20240430 20:11:44.933023 18988 tablet_bootstrap.cc:492] T f7b0132c17d44a9ba17288e0321a7f11 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:44.933230 18988 ts_tablet_manager.cc:1367] T f7b0132c17d44a9ba17288e0321a7f11 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.184s user 0.025s sys 0.024s
I20240430 20:11:44.933115 19028 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_fac4c5c0ce954200838069be87be2ad9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:44.933735 19028 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fac4c5c0ce954200838069be87be2ad9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:44.934291 18742 tablet_service.cc:1514] Processing DeleteTablet for tablet 76d6e9d547ca4a848eae852944b1d117 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:44 UTC) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:44.934221 18988 raft_consensus.cc:348] T f7b0132c17d44a9ba17288e0321a7f11 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.934401 18988 raft_consensus.cc:374] T f7b0132c17d44a9ba17288e0321a7f11 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:44.934479 18988 raft_consensus.cc:729] T f7b0132c17d44a9ba17288e0321a7f11 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:44.934749 18988 consensus_queue.cc:260] T f7b0132c17d44a9ba17288e0321a7f11 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.934882 18450 tablet_service.cc:1514] Processing DeleteTablet for tablet 76d6e9d547ca4a848eae852944b1d117 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:44 UTC) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:44.935359 18988 ts_tablet_manager.cc:1398] T f7b0132c17d44a9ba17288e0321a7f11 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240430 20:11:44.935810 18988 tablet_bootstrap.cc:492] T 681cdd85ac6847a0a1aaded310456582 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:44.936864 20489 raft_consensus.cc:482] T 66dd767bd3f8494f94061fb89e5e5680 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:44.937047 20489 raft_consensus.cc:504] T 66dd767bd3f8494f94061fb89e5e5680 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.937469 18988 tablet_bootstrap.cc:654] T 681cdd85ac6847a0a1aaded310456582 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:44.937861 20489 leader_election.cc:288] T 66dd767bd3f8494f94061fb89e5e5680 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:44.943491 19033 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_6e886b67708e4dc1b9e1753b1dd00800" } new_table_name: "default.Table_ba8b218e936b4bfcb073f8182f474cca" modify_external_catalogs: true
I20240430 20:11:44.879842 18985 consensus_queue.cc:260] T ae24e75a0d42441ebc97400ffa94145e P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.947044 19051 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_6c7df3aa7d8741ae9ac67cf4af868788"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240430 20:11:44.947621 20547 raft_consensus.cc:482] T ae24e75a0d42441ebc97400ffa94145e P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:44.947772 20547 raft_consensus.cc:504] T ae24e75a0d42441ebc97400ffa94145e P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
W20240430 20:11:44.947847 19051 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6c7df3aa7d8741ae9ac67cf4af868788 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:44.948678 20547 leader_election.cc:288] T ae24e75a0d42441ebc97400ffa94145e P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:44.949646 18763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae24e75a0d42441ebc97400ffa94145e" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:44.951565 19471 ts_tablet_manager.cc:1875] T 84568f90b9c54cf292099808d4cbf5dc P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:44.952046 20198 raft_consensus.cc:482] T 50486644ec234933b9a0b45ee39a6685 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240430 20:11:44.858326 18557 consensus_peers.cc:485] T 53a899fa5a424a0d99f1b47925e8c7ad P 5e1c6397f09747f091534faf93542710 -> Peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Couldn't send request to peer 3c51c33d5616482cafb68e26688820f4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 53a899fa5a424a0d99f1b47925e8c7ad. This is attempt 1: this message will repeat every 5th retry.
W20240430 20:11:44.952199 18557 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.177.3:44867, user_credentials={real_user=slave}} blocked reactor thread for 93916.5us
I20240430 20:11:44.952198 20198 raft_consensus.cc:504] T 50486644ec234933b9a0b45ee39a6685 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.952780 20198 leader_election.cc:288] T 50486644ec234933b9a0b45ee39a6685 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:44.876237 18616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e79df8744e74b5991a79ce978ff0202" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:44.954419 19489 ts_tablet_manager.cc:1888] T aec9c09cc0454acab6006bb3e10982b7 P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 3.4
I20240430 20:11:44.954519 19489 log.cc:1196] T aec9c09cc0454acab6006bb3e10982b7 P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/aec9c09cc0454acab6006bb3e10982b7
I20240430 20:11:44.954912 19489 ts_tablet_manager.cc:1909] T aec9c09cc0454acab6006bb3e10982b7 P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:44.956918 18758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66dd767bd3f8494f94061fb89e5e5680" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:44.957304 18763 raft_consensus.cc:3037] T 115c4a12440647a4bb46eb6824f7ea98 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:44.957360 18473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50486644ec234933b9a0b45ee39a6685" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:44.957610 18473 raft_consensus.cc:2379] T 50486644ec234933b9a0b45ee39a6685 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c51c33d5616482cafb68e26688820f4 in current term 1: Already voted for candidate 5e1c6397f09747f091534faf93542710 in this term.
W20240430 20:11:44.959779 18557 consensus_peers.cc:485] T aec9c09cc0454acab6006bb3e10982b7 P 5e1c6397f09747f091534faf93542710 -> Peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Couldn't send request to peer 3c51c33d5616482cafb68e26688820f4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: aec9c09cc0454acab6006bb3e10982b7. This is attempt 1: this message will repeat every 5th retry.
W20240430 20:11:44.959964 18557 leader_election.cc:341] T 66dd767bd3f8494f94061fb89e5e5680 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
W20240430 20:11:44.961306 18417 leader_election.cc:341] T ae24e75a0d42441ebc97400ffa94145e P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:44.964174 18588 tablet_service.cc:1514] Processing DeleteTablet for tablet e3e74e76ab2b4fe492ffcc1838656fd1 with delete_type TABLET_DATA_DELETED (replaced by tablet 7fceb89e8ee148999f747b08a15fd274) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:44.954984 18619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae24e75a0d42441ebc97400ffa94145e" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:44.964715 20547 consensus_queue.cc:1035] T e87fb2f0361d4aa3ada27bbdfa88e94c P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:44.964854 18619 raft_consensus.cc:2454] T ae24e75a0d42441ebc97400ffa94145e P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 0.
I20240430 20:11:44.965621 20563 tablet.cc:1722] T 1462ecf1176c4e66be42abddbe36b00c P 2efb542aa55e439b99ccb98d1c5c884a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240430 20:11:44.966327 18415 leader_election.cc:302] T ae24e75a0d42441ebc97400ffa94145e P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters: 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:44.967427 20572 tablet.cc:1722] T 1462ecf1176c4e66be42abddbe36b00c P 3c51c33d5616482cafb68e26688820f4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240430 20:11:44.967756 20545 raft_consensus.cc:2781] T ae24e75a0d42441ebc97400ffa94145e P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:44.967832 20545 raft_consensus.cc:482] T ae24e75a0d42441ebc97400ffa94145e P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:44.967907 20545 raft_consensus.cc:3037] T ae24e75a0d42441ebc97400ffa94145e P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:44.971318 20545 raft_consensus.cc:504] T ae24e75a0d42441ebc97400ffa94145e P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:44.973042 20545 leader_election.cc:288] T ae24e75a0d42441ebc97400ffa94145e P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:44.973940 18758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae24e75a0d42441ebc97400ffa94145e" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:44.964485 18608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abdaf243e5be4ad58036b23151b14fe4" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
W20240430 20:11:44.974978 18417 leader_election.cc:341] T ae24e75a0d42441ebc97400ffa94145e P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:44.979776 18601 tablet_service.cc:1514] Processing DeleteTablet for tablet 76d6e9d547ca4a848eae852944b1d117 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:44 UTC) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:44.980223 18696 leader_election.cc:302] T 5bd0362ff41b4faca0205d136d596737 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710
I20240430 20:11:44.980473 20198 raft_consensus.cc:3037] T 5bd0362ff41b4faca0205d136d596737 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:44.980566 18588 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f0b341a31224bf1be525b84b9500304 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:44 UTC) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:44.981750 18473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66dd767bd3f8494f94061fb89e5e5680" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
W20240430 20:11:44.983594 18558 leader_election.cc:341] T 66dd767bd3f8494f94061fb89e5e5680 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:44.983767 18558 leader_election.cc:302] T 66dd767bd3f8494f94061fb89e5e5680 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:44.984622 20489 raft_consensus.cc:2726] T 66dd767bd3f8494f94061fb89e5e5680 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:44.985515 18623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae24e75a0d42441ebc97400ffa94145e" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:44.985724 18623 raft_consensus.cc:3037] T ae24e75a0d42441ebc97400ffa94145e P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:44.991556 19028 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_9c3d369e0c4f4bb393b7278a5fdcb2e8" } new_table_name: "default.Table_fc0a4540f98b4eea95099b8ab46ea235" modify_external_catalogs: true
I20240430 20:11:44.993544 18985 ts_tablet_manager.cc:1398] T ae24e75a0d42441ebc97400ffa94145e P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.167s user 0.004s sys 0.000s
I20240430 20:11:45.003980 18620 raft_consensus.cc:2454] T 2e79df8744e74b5991a79ce978ff0202 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 1.
I20240430 20:11:45.014528 18623 raft_consensus.cc:2454] T ae24e75a0d42441ebc97400ffa94145e P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 1.
I20240430 20:11:45.015122 18415 leader_election.cc:302] T ae24e75a0d42441ebc97400ffa94145e P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters: 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:45.015607 20105 raft_consensus.cc:2781] T ae24e75a0d42441ebc97400ffa94145e P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:45.015740 20105 raft_consensus.cc:686] T ae24e75a0d42441ebc97400ffa94145e P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Becoming Leader. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Running, Role: LEADER
I20240430 20:11:45.022851 20484 raft_consensus.cc:482] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader b632d6ef4c104a058a6db208024cab33)
I20240430 20:11:45.023113 20484 raft_consensus.cc:504] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b3f98793354c7aaa5c106e8ec31ed8" member_type: VOTER last_known_addr { host: "127.17.177.62" port: 36441 } } peers { permanent_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e" member_type: VOTER last_known_addr { host: "127.17.177.61" port: 34777 } } peers { permanent_uuid: "b632d6ef4c104a058a6db208024cab33" member_type: VOTER last_known_addr { host: "127.17.177.60" port: 37225 } }
I20240430 20:11:45.023612 19023 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_2e4e483bcd6c4dd6bdfcc938cb0e2da6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:45.024325 19023 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2e4e483bcd6c4dd6bdfcc938cb0e2da6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240430 20:11:45.025043 19946 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.17.177.1:47345 (request call id 38) took 814 ms (client timeout 1000 ms). Trace:
W20240430 20:11:45.025151 19946 rpcz_store.cc:269] 0430 20:11:44.210509 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:44.210573 (+ 64us) service_pool.cc:225] Handling call
0430 20:11:45.023816 (+813243us) spinlock_profiling.cc:243] Waited 811 ms on lock 0x616000070654. stack: 00007f6d23ac8756 00007f6d23ac8510 00007f6d22afb48d 0000000000955804 00000000009557cd 00007f6d3260715d 00007f6d32607253 00007f6d2f339d41 00007f6d36f2534d 00007f6d370ac668 00007f6d2de97786 00007f6d2de974b5 00007f6d2a7d004e 00007f6d2a7cef2e 00007f6d2a7d5405 00007f6d2a7d5d2f
0430 20:11:45.024966 (+ 1150us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1865811840}
W20240430 20:11:45.027221 19945 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.17.177.3:39881 (request call id 47) took 826 ms (client timeout 1000 ms). Trace:
W20240430 20:11:45.027395 19945 rpcz_store.cc:269] 0430 20:11:44.200902 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:44.200970 (+ 68us) service_pool.cc:225] Handling call
0430 20:11:45.025729 (+824759us) spinlock_profiling.cc:243] Waited 821 ms on lock 0x616000070654. stack: 00007f6d23ac8756 00007f6d23ac8510 00007f6d22afb48d 0000000000955804 00000000009557cd 00007f6d3260715d 00007f6d32607253 00007f6d2f339d41 00007f6d36f2534d 00007f6d370ac668 00007f6d2de97786 00007f6d2de974b5 00007f6d2a7d004e 00007f6d2a7cef2e 00007f6d2a7d5405 00007f6d2a7d5d2f
0430 20:11:45.027136 (+ 1407us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1887235840}
W20240430 20:11:45.030154 19942 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.17.177.2:50593 (request call id 40) took 1022 ms (client timeout 1000 ms). Trace:
W20240430 20:11:45.030306 19942 rpcz_store.cc:269] 0430 20:11:44.007264 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:44.083042 (+ 75778us) service_pool.cc:225] Handling call
0430 20:11:45.028266 (+945224us) spinlock_profiling.cc:243] Waited 937 ms on lock 0x616000070654. stack: 00007f6d23ac8756 00007f6d23ac8510 00007f6d22afb48d 0000000000955804 00000000009557cd 00007f6d3260715d 00007f6d32607253 00007f6d2f339d41 00007f6d36f2534d 00007f6d370ac668 00007f6d2de97786 00007f6d2de974b5 00007f6d2a7d004e 00007f6d2a7cef2e 00007f6d2a7d5405 00007f6d2a7d5d2f
0430 20:11:45.030068 (+ 1802us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2155688320}
I20240430 20:11:45.032034 20484 leader_election.cc:288] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 89df84ccf5394c9bb60d4e4ed872fc2e (127.17.177.61:34777), b632d6ef4c104a058a6db208024cab33 (127.17.177.60:37225)
I20240430 20:11:45.032984 19074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88b3f98793354c7aaa5c106e8ec31ed8" candidate_term: 6 candidate_status { last_received { term: 5 index: 696 } } ignore_live_leader: false dest_uuid: "b632d6ef4c104a058a6db208024cab33" is_pre_election: true
I20240430 20:11:45.033866 18983 tablet_bootstrap.cc:492] T 4f6f9ae722da49e58c9b72107f6da2fa P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:45.034001 18983 ts_tablet_manager.cc:1367] T 4f6f9ae722da49e58c9b72107f6da2fa P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.283s user 0.001s sys 0.014s
I20240430 20:11:45.034818 18983 raft_consensus.cc:348] T 4f6f9ae722da49e58c9b72107f6da2fa P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:45.035161 18983 raft_consensus.cc:374] T 4f6f9ae722da49e58c9b72107f6da2fa P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:45.035252 18983 raft_consensus.cc:729] T 4f6f9ae722da49e58c9b72107f6da2fa P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:45.035519 18983 consensus_queue.cc:260] T 4f6f9ae722da49e58c9b72107f6da2fa P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:45.036490 18983 ts_tablet_manager.cc:1398] T 4f6f9ae722da49e58c9b72107f6da2fa P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240430 20:11:45.051138 18235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88b3f98793354c7aaa5c106e8ec31ed8" candidate_term: 6 candidate_status { last_received { term: 5 index: 696 } } ignore_live_leader: false dest_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e" is_pre_election: true
I20240430 20:11:45.052071 19899 leader_election.cc:302] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88b3f98793354c7aaa5c106e8ec31ed8; no voters: 89df84ccf5394c9bb60d4e4ed872fc2e, b632d6ef4c104a058a6db208024cab33
I20240430 20:11:45.052675 20484 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240430 20:11:45.055157 19051 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_17cb787c6a4c4c9fa720ce8c4e326605" } new_table_name: "default.Table_c6c962da746e4fd8a4d7200d09f08e9e" modify_external_catalogs: true
I20240430 20:11:45.063855 18985 tablet_bootstrap.cc:492] T 4f6f9ae722da49e58c9b72107f6da2fa P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:45.015988 20105 consensus_queue.cc:237] T ae24e75a0d42441ebc97400ffa94145e P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:45.076050 19024 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet ca64de16d40f4a44b8b411f5526ba11a (table default.Table_1cf914acf7fe4e98b8dbc297b6372f21 [id=f4ba9536e9c141deb68f4f62b7e8fa70]). Expected version 1 got 0
I20240430 20:11:45.076318 19024 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 99203cc522de424184c6f97453f166fd (table default.Table_cb2a4b9ceaf4431facfff2c9bcf94c5d [id=3f8690a437ae49a3beb009fa7899d5d0]). Expected version 1 got 0
I20240430 20:11:45.076483 19024 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 2e79df8744e74b5991a79ce978ff0202 (table default.Table_276bbdc7cfe542d1a2936485a2aad1ec [id=f146a15e6e8a4acbb4c734c1c863da57]). Expected version 1 got 0
I20240430 20:11:45.076854 19024 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 2da900b6bbfa4fa99fe8494fd2e68da8 (table default.Table_19a91a7f17ff456ebb145e8dd89c0940 [id=67283dd3995a44c6a7a7bc8722e4f955]). Expected version 1 got 0
I20240430 20:11:45.077190 19024 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 1462ecf1176c4e66be42abddbe36b00c (table default.Table_b49a7942aa8c4f58acb94c35d1cdbb70 [id=6257559b0e9545c58848a76af0c5a5f7]). Expected version 1 got 0
I20240430 20:11:45.078830 19024 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet a59ef6c545dd41a28265a4c1b8ce10e0 (table default.Table_07e40af8f1c546048975c8edf88170d7 [id=86df5d161d8641c89070e7eed9a39360]). Expected version 2 got 0
I20240430 20:11:45.079937 19024 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet ebf74809b52f47f3b0ed21b6ac0f541a (table default.Table_b2d07520b9944a89ad62fca70396174e [id=1c8e0a3a44ec4a798770c0165ed3dffa]). Expected version 1 got 0
W20240430 20:11:45.081641 19028 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet dac141d1b70c48d785caddd7182811ee has no consensus state
I20240430 20:11:45.086822 18450 tablet_service.cc:1467] Processing CreateTablet for tablet 467cafc47106474e96f619b4043db233 (DEFAULT_TABLE table=default.Table_f45eee4d4b6a4baf83111f12dfd25a9f [id=ea6b9f7d86424c84a8ea05a5877afd06]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:45.087440 18450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 467cafc47106474e96f619b4043db233. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:45.090251 19027 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 1462ecf1176c4e66be42abddbe36b00c (table default.Table_b49a7942aa8c4f58acb94c35d1cdbb70 [id=6257559b0e9545c58848a76af0c5a5f7]). Expected version 1 got 0
I20240430 20:11:45.090965 19027 catalog_manager.cc:5484] T 681cdd85ac6847a0a1aaded310456582 P 5e1c6397f09747f091534faf93542710 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e1c6397f09747f091534faf93542710 (127.17.177.2). New cstate: current_term: 1 leader_uuid: "5e1c6397f09747f091534faf93542710" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } health_report { overall_health: UNKNOWN } } }
I20240430 20:11:45.091823 19027 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 2da900b6bbfa4fa99fe8494fd2e68da8 (table default.Table_19a91a7f17ff456ebb145e8dd89c0940 [id=67283dd3995a44c6a7a7bc8722e4f955]). Expected version 1 got 0
I20240430 20:11:45.092137 19027 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet ae20f44df50343289b0a512a4ea53692 (table default.Table_1844f6b1bda84812825384e14a2fd94e [id=71da9acc3cff4d95bbac4a686b708fd5]). Expected version 1 got 0
I20240430 20:11:45.092499 19027 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 677481aee5fa4eb4b189322d7a3f8f2e (table default.Table_1ce204e15fce4c469816cf4e7f80cac7 [id=d268c31d82df4871bc1a78aea5871b33]). Expected version 1 got 0
I20240430 20:11:45.093019 19027 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet ebf74809b52f47f3b0ed21b6ac0f541a (table default.Table_b2d07520b9944a89ad62fca70396174e [id=1c8e0a3a44ec4a798770c0165ed3dffa]). Expected version 1 got 0
I20240430 20:11:45.093509 19027 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 99203cc522de424184c6f97453f166fd (table default.Table_cb2a4b9ceaf4431facfff2c9bcf94c5d [id=3f8690a437ae49a3beb009fa7899d5d0]). Expected version 1 got 0
I20240430 20:11:45.093849 19027 catalog_manager.cc:5484] T 7ce09fcbe1ac457a9c05efaf000c3263 P 5e1c6397f09747f091534faf93542710 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e1c6397f09747f091534faf93542710 (127.17.177.2). New cstate: current_term: 1 leader_uuid: "5e1c6397f09747f091534faf93542710" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } health_report { overall_health: UNKNOWN } } }
I20240430 20:11:45.094372 19027 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 7ce09fcbe1ac457a9c05efaf000c3263 (table default.Table_edc4c27ed846454288f73f6a0d88cb69 [id=78fd3f516083489ab3b42d90dcef7e4d]). Expected version 1 got 0
I20240430 20:11:45.094797 19027 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 66dd767bd3f8494f94061fb89e5e5680 (table default.Table_b064d5659f7243f5b0ec167fb6c3f404 [id=5227206f44d04fa89b4842b899f33afb]). Expected version 2 got 0
I20240430 20:11:45.095113 19027 catalog_manager.cc:5484] T abdaf243e5be4ad58036b23151b14fe4 P 5e1c6397f09747f091534faf93542710 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e1c6397f09747f091534faf93542710 (127.17.177.2). New cstate: current_term: 1 leader_uuid: "5e1c6397f09747f091534faf93542710" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } health_report { overall_health: UNKNOWN } } }
I20240430 20:11:45.095752 19027 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet ca64de16d40f4a44b8b411f5526ba11a (table default.Table_1cf914acf7fe4e98b8dbc297b6372f21 [id=f4ba9536e9c141deb68f4f62b7e8fa70]). Expected version 1 got 0
I20240430 20:11:45.096359 19027 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet a59ef6c545dd41a28265a4c1b8ce10e0 (table default.Table_07e40af8f1c546048975c8edf88170d7 [id=86df5d161d8641c89070e7eed9a39360]). Expected version 2 got 0
I20240430 20:11:45.101722 18445 tablet_service.cc:1467] Processing CreateTablet for tablet 2e54ea9db8584db8abc36c1d3fd02c76 (DEFAULT_TABLE table=default.Table_75efd32c9420454f901c61882c8c29bc [id=0d741ba94928463a9929f799e0336fdb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:45.107874 18459 tablet_service.cc:1467] Processing CreateTablet for tablet 3fa6c1078c074ef4a7a408efa8c19467 (DEFAULT_TABLE table=default.Table_e311a584368b4233b6e5beb24587ce00 [id=65018c93b8e2409f984c52764e82dcb3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:45.108302 18454 tablet_service.cc:1467] Processing CreateTablet for tablet 37b2ea4be7494f0985808ade788dbe2b (DEFAULT_TABLE table=default.Table_5f6e0fce999b46de927f9271e5de5071 [id=0baad83a8c0f4aeebc7ddad068625a69]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:45.108808 18454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37b2ea4be7494f0985808ade788dbe2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:45.110323 18446 tablet_service.cc:1467] Processing CreateTablet for tablet aaef85eea87f4443943fd3692bb2a38f (DEFAULT_TABLE table=default.Table_3da5b4c8668c470483ae2a0c7773eaaf [id=e292d818ee3c45df90522f08eb2d6227]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:45.110806 18446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aaef85eea87f4443943fd3692bb2a38f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:45.112486 18445 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e54ea9db8584db8abc36c1d3fd02c76. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:45.113881 18449 tablet_service.cc:1467] Processing CreateTablet for tablet cdc07bb607374c6e8eb6ca35a86d0204 (DEFAULT_TABLE table=default.Table_fe491607e9e7480b9d9d2c8eb89ccac9 [id=ba205c2256d24f998fb16e6590b74cf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:45.114275 18449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdc07bb607374c6e8eb6ca35a86d0204. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:45.114583 18459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fa6c1078c074ef4a7a408efa8c19467. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:45.116432 18985 tablet_bootstrap.cc:654] T 4f6f9ae722da49e58c9b72107f6da2fa P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:45.119791 18457 tablet_service.cc:1467] Processing CreateTablet for tablet aed9a3eefe3f4136af1b9469c9e2e23c (DEFAULT_TABLE table=default.Table_457e03bc2d524accaa76649790c42fc8 [id=f26e395466f24585b9524a1ec2f77051]), partition=RANGE (key) PARTITION UNBOUNDED
W20240430 20:11:45.122434 18696 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.177.2:46027, user_credentials={real_user=slave}} blocked reactor thread for 142221us
I20240430 20:11:45.122903 18696 leader_election.cc:302] T abdaf243e5be4ad58036b23151b14fe4 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710
I20240430 20:11:45.125519 18457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aed9a3eefe3f4136af1b9469c9e2e23c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:45.126456 19023 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_ea4369e78de74045baa10ea2a61bfe38"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:45.127487 19023 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ea4369e78de74045baa10ea2a61bfe38 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:45.129369 18750 raft_consensus.cc:3037] T abdaf243e5be4ad58036b23151b14fe4 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:45.131443 19028 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_f86df7f6c95e4da48b80ae4733dc6994"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:45.132294 19028 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f86df7f6c95e4da48b80ae4733dc6994 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:45.134756 18756 raft_consensus.cc:3037] T 7ce09fcbe1ac457a9c05efaf000c3263 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:45.135200 18985 tablet_bootstrap.cc:492] T 4f6f9ae722da49e58c9b72107f6da2fa P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:45.135341 18985 ts_tablet_manager.cc:1367] T 4f6f9ae722da49e58c9b72107f6da2fa P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.072s user 0.004s sys 0.000s
I20240430 20:11:45.132450 18461 tablet_service.cc:1467] Processing CreateTablet for tablet a3b6e2ed50184e97914a703e2d73a285 (DEFAULT_TABLE table=default.Table_8d32c8a8f05e4b599dd6a713133ed82e [id=295c03e97c87495f8d71b255b2ec9065]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:45.137686 18461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3b6e2ed50184e97914a703e2d73a285. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:45.140810 20198 raft_consensus.cc:2726] T 5bd0362ff41b4faca0205d136d596737 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:45.146356 18988 tablet_bootstrap.cc:492] T 681cdd85ac6847a0a1aaded310456582 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:45.146653 18988 ts_tablet_manager.cc:1367] T 681cdd85ac6847a0a1aaded310456582 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.211s user 0.042s sys 0.044s
W20240430 20:11:45.146924 18696 leader_election.cc:341] T 50486644ec234933b9a0b45ee39a6685 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Not found: Tablet not found: 50486644ec234933b9a0b45ee39a6685
I20240430 20:11:45.147073 18696 leader_election.cc:302] T 50486644ec234933b9a0b45ee39a6685 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710
I20240430 20:11:45.147567 18988 raft_consensus.cc:348] T 681cdd85ac6847a0a1aaded310456582 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:45.147768 20198 raft_consensus.cc:3037] T 50486644ec234933b9a0b45ee39a6685 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:45.147742 20469 raft_consensus.cc:2726] T abdaf243e5be4ad58036b23151b14fe4 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:45.147753 18988 raft_consensus.cc:374] T 681cdd85ac6847a0a1aaded310456582 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:45.147853 18988 raft_consensus.cc:729] T 681cdd85ac6847a0a1aaded310456582 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:45.148053 18988 consensus_queue.cc:260] T 681cdd85ac6847a0a1aaded310456582 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:45.148640 18988 ts_tablet_manager.cc:1398] T 681cdd85ac6847a0a1aaded310456582 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240430 20:11:45.148998 18988 tablet_bootstrap.cc:492] T 7683caba80594627900a0185f772c131 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:45.149348 18457 tablet_service.cc:1467] Processing CreateTablet for tablet 118eb508b3294b55abbda27e98495353 (DEFAULT_TABLE table=default.Table_d770977de70d4cac8511d8d125c1b160 [id=d80cad6e0e734b8fa6d2011a30f9657b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:45.149775 18457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 118eb508b3294b55abbda27e98495353. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:45.150440 18733 tablet_service.cc:1467] Processing CreateTablet for tablet 2e54ea9db8584db8abc36c1d3fd02c76 (DEFAULT_TABLE table=default.Table_75efd32c9420454f901c61882c8c29bc [id=0d741ba94928463a9929f799e0336fdb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:45.150642 18726 tablet_service.cc:1467] Processing CreateTablet for tablet 37b2ea4be7494f0985808ade788dbe2b (DEFAULT_TABLE table=default.Table_5f6e0fce999b46de927f9271e5de5071 [id=0baad83a8c0f4aeebc7ddad068625a69]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:45.150848 18733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e54ea9db8584db8abc36c1d3fd02c76. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:45.151078 18742 tablet_service.cc:1467] Processing CreateTablet for tablet 467cafc47106474e96f619b4043db233 (DEFAULT_TABLE table=default.Table_f45eee4d4b6a4baf83111f12dfd25a9f [id=ea6b9f7d86424c84a8ea05a5877afd06]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:45.151152 18737 tablet_service.cc:1467] Processing CreateTablet for tablet aaef85eea87f4443943fd3692bb2a38f (DEFAULT_TABLE table=default.Table_3da5b4c8668c470483ae2a0c7773eaaf [id=e292d818ee3c45df90522f08eb2d6227]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:45.151368 18742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 467cafc47106474e96f619b4043db233. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:45.151665 18737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aaef85eea87f4443943fd3692bb2a38f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:45.152678 19495 ts_tablet_manager.cc:1888] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.2
I20240430 20:11:45.152781 19495 log.cc:1196] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/dd6083f12c7e4d1e8a2006b3ddf9aa3a
I20240430 20:11:45.153146 19495 ts_tablet_manager.cc:1909] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:45.153457 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 345) took 1702 ms. Trace:
I20240430 20:11:45.153534 19495 rpcz_store.cc:276] 0430 20:11:43.450882 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:43.451027 (+ 145us) service_pool.cc:225] Handling call
0430 20:11:44.926318 (+1475291us) ts_tablet_manager.cc:1120] Deleting tablet dd6083f12c7e4d1e8a2006b3ddf9aa3a
0430 20:11:44.926331 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:44.980795 (+ 54464us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:45.152569 (+171774us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:45.153436 (+ 867us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1475128}
I20240430 20:11:45.154098 19004 catalog_manager.cc:4830] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): tablet dd6083f12c7e4d1e8a2006b3ddf9aa3a (table default.Table_276f62c2338b46a0aab8905430e8b61b [id=789b538b500642d89721e6028b9ac48e]) successfully deleted
I20240430 20:11:45.154294 18744 tablet_service.cc:1467] Processing CreateTablet for tablet 118eb508b3294b55abbda27e98495353 (DEFAULT_TABLE table=default.Table_d770977de70d4cac8511d8d125c1b160 [id=d80cad6e0e734b8fa6d2011a30f9657b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:45.154469 18740 tablet_service.cc:1467] Processing CreateTablet for tablet cdc07bb607374c6e8eb6ca35a86d0204 (DEFAULT_TABLE table=default.Table_fe491607e9e7480b9d9d2c8eb89ccac9 [id=ba205c2256d24f998fb16e6590b74cf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:45.154667 18744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 118eb508b3294b55abbda27e98495353. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:45.154912 18739 tablet_service.cc:1467] Processing CreateTablet for tablet aed9a3eefe3f4136af1b9469c9e2e23c (DEFAULT_TABLE table=default.Table_457e03bc2d524accaa76649790c42fc8 [id=f26e395466f24585b9524a1ec2f77051]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:45.155210 18739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aed9a3eefe3f4136af1b9469c9e2e23c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:45.155905 18726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37b2ea4be7494f0985808ade788dbe2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:45.158854 18725 tablet_service.cc:1467] Processing CreateTablet for tablet a3b6e2ed50184e97914a703e2d73a285 (DEFAULT_TABLE table=default.Table_8d32c8a8f05e4b599dd6a713133ed82e [id=295c03e97c87495f8d71b255b2ec9065]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:45.159204 18725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3b6e2ed50184e97914a703e2d73a285. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:45.154294 18732 tablet_service.cc:1467] Processing CreateTablet for tablet 3fa6c1078c074ef4a7a408efa8c19467 (DEFAULT_TABLE table=default.Table_e311a584368b4233b6e5beb24587ce00 [id=65018c93b8e2409f984c52764e82dcb3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:45.163357 18732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fa6c1078c074ef4a7a408efa8c19467. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:45.164382 18988 tablet_bootstrap.cc:654] T 7683caba80594627900a0185f772c131 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:45.166857 18740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdc07bb607374c6e8eb6ca35a86d0204. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:45.169518 20469 raft_consensus.cc:482] T 2cdccf4be7094ca5b863dd5c3432d55d P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:45.169705 20469 raft_consensus.cc:504] T 2cdccf4be7094ca5b863dd5c3432d55d P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:45.170517 20469 leader_election.cc:288] T 2cdccf4be7094ca5b863dd5c3432d55d P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:45.172969 18988 tablet_bootstrap.cc:492] T 7683caba80594627900a0185f772c131 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:45.173218 18988 ts_tablet_manager.cc:1367] T 7683caba80594627900a0185f772c131 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.008s
I20240430 20:11:45.174738 18988 raft_consensus.cc:348] T 7683caba80594627900a0185f772c131 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:45.175055 18988 raft_consensus.cc:374] T 7683caba80594627900a0185f772c131 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:45.175251 18988 raft_consensus.cc:729] T 7683caba80594627900a0185f772c131 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:45.175494 18988 consensus_queue.cc:260] T 7683caba80594627900a0185f772c131 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:45.176069 18988 ts_tablet_manager.cc:1398] T 7683caba80594627900a0185f772c131 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240430 20:11:45.176417 18988 tablet_bootstrap.cc:492] T f940d17b02b541289121463e05b0bb5f P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:45.178081 18988 tablet_bootstrap.cc:654] T f940d17b02b541289121463e05b0bb5f P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:45.184772 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 223) took 5437 ms. Trace:
I20240430 20:11:45.185118 19489 rpcz_store.cc:276] 0430 20:11:39.747584 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:39.747663 (+ 79us) service_pool.cc:225] Handling call
0430 20:11:44.807121 (+5059458us) ts_tablet_manager.cc:1120] Deleting tablet aec9c09cc0454acab6006bb3e10982b7
0430 20:11:44.807133 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:44.951059 (+143926us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:44.954314 (+ 3255us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:45.184725 (+230411us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":37607,"tablet-delete.queue_time_us":5058719}
I20240430 20:11:45.185832 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 224) took 5426 ms. Trace:
I20240430 20:11:45.186376 19001 catalog_manager.cc:4830] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): tablet aec9c09cc0454acab6006bb3e10982b7 (table default.Table_ff91abfadbc746d6b01fdbfecd16fac5 [id=26d4860fb33047b7b812ebd4309f2e28]) successfully deleted
I20240430 20:11:45.186218 18988 tablet_bootstrap.cc:492] T f940d17b02b541289121463e05b0bb5f P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:45.186672 19489 rpcz_store.cc:276] 0430 20:11:39.759180 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:39.759244 (+ 64us) service_pool.cc:225] Handling call
0430 20:11:45.185687 (+5426443us) ts_tablet_manager.cc:1120] Deleting tablet 04b3dff86d5c48a09d03da58bd48f360
0430 20:11:45.185698 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:45.185813 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5426248}
I20240430 20:11:45.186843 18988 ts_tablet_manager.cc:1367] T f940d17b02b541289121463e05b0bb5f P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
W20240430 20:11:45.187151 19001 catalog_manager.cc:4810] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): delete failed for tablet 04b3dff86d5c48a09d03da58bd48f360 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 04b3dff86d5c48a09d03da58bd48f360 already in progress: creating tablet
I20240430 20:11:45.187559 19489 tablet_replica.cc:330] T 294644d7a16b4ca2bbb6d106bcb8c7ff P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:45.187759 19489 raft_consensus.cc:2229] T 294644d7a16b4ca2bbb6d106bcb8c7ff P 3c51c33d5616482cafb68e26688820f4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:45.187958 19489 raft_consensus.cc:2258] T 294644d7a16b4ca2bbb6d106bcb8c7ff P 3c51c33d5616482cafb68e26688820f4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:45.188086 18988 raft_consensus.cc:348] T f940d17b02b541289121463e05b0bb5f P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:45.188324 18988 raft_consensus.cc:374] T f940d17b02b541289121463e05b0bb5f P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:45.188421 18988 raft_consensus.cc:729] T f940d17b02b541289121463e05b0bb5f P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:45.188524 19489 ts_tablet_manager.cc:1875] T 294644d7a16b4ca2bbb6d106bcb8c7ff P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:45.188717 18988 consensus_queue.cc:260] T f940d17b02b541289121463e05b0bb5f P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:45.189271 18988 ts_tablet_manager.cc:1398] T f940d17b02b541289121463e05b0bb5f P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240430 20:11:45.189577 18988 tablet_bootstrap.cc:492] T 2da900b6bbfa4fa99fe8494fd2e68da8 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:45.191220 18988 tablet_bootstrap.cc:654] T 2da900b6bbfa4fa99fe8494fd2e68da8 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
W20240430 20:11:45.193287 18417 consensus_peers.cc:485] T e87fb2f0361d4aa3ada27bbdfa88e94c P 2efb542aa55e439b99ccb98d1c5c884a -> Peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Couldn't send request to peer 3c51c33d5616482cafb68e26688820f4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e87fb2f0361d4aa3ada27bbdfa88e94c. This is attempt 1: this message will repeat every 5th retry.
W20240430 20:11:45.194470 18699 leader_election.cc:341] T 2cdccf4be7094ca5b863dd5c3432d55d P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Not found: Tablet not found: 2cdccf4be7094ca5b863dd5c3432d55d
I20240430 20:11:45.199235 18988 tablet_bootstrap.cc:492] T 2da900b6bbfa4fa99fe8494fd2e68da8 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:45.199486 18988 ts_tablet_manager.cc:1367] T 2da900b6bbfa4fa99fe8494fd2e68da8 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.009s
I20240430 20:11:45.200320 18988 raft_consensus.cc:348] T 2da900b6bbfa4fa99fe8494fd2e68da8 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:45.200608 18988 raft_consensus.cc:374] T 2da900b6bbfa4fa99fe8494fd2e68da8 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:45.200721 18988 raft_consensus.cc:729] T 2da900b6bbfa4fa99fe8494fd2e68da8 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:45.199235 20198 raft_consensus.cc:2726] T 50486644ec234933b9a0b45ee39a6685 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:45.201292 18988 consensus_queue.cc:260] T 2da900b6bbfa4fa99fe8494fd2e68da8 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:45.203393 18988 ts_tablet_manager.cc:1398] T 2da900b6bbfa4fa99fe8494fd2e68da8 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240430 20:11:45.203732 18988 tablet_bootstrap.cc:492] T 99203cc522de424184c6f97453f166fd P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:45.205442 18988 tablet_bootstrap.cc:654] T 99203cc522de424184c6f97453f166fd P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:45.212348 20198 raft_consensus.cc:482] T 7ce09fcbe1ac457a9c05efaf000c3263 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:45.212597 20198 raft_consensus.cc:504] T 7ce09fcbe1ac457a9c05efaf000c3263 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:45.213500 20198 leader_election.cc:288] T 7ce09fcbe1ac457a9c05efaf000c3263 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:45.217347 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 343) took 1766 ms. Trace:
I20240430 20:11:45.217550 19495 rpcz_store.cc:276] 0430 20:11:43.450661 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:43.451138 (+ 477us) service_pool.cc:225] Handling call
0430 20:11:45.157952 (+1706814us) ts_tablet_manager.cc:1120] Deleting tablet b7141e215d6d44bc8406c80508190874
0430 20:11:45.217175 (+ 59223us) spinlock_profiling.cc:243] Waited 9.03 ms on lock 0x61800000fd18. stack: 00007f9c64594756 00007f9c64594510 00007f9c635c748d 0000000000955804 00000000009557cd 000000000095519a 00007f9c6458a19d 00007f9c6458b20e 00007f9c77a904ed 00007f9c77a0c6b3 00007f9c73b6da46 00007f9c73b6ed47 00007f9c73ba1814 00007f9c73b82b00 00007f9c73b8288d 0000000000a60bcc
0430 20:11:45.217187 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:45.217322 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":20760832,"tablet-delete.queue_time_us":1706622}
W20240430 20:11:45.218358 19004 catalog_manager.cc:4810] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): delete failed for tablet b7141e215d6d44bc8406c80508190874 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b7141e215d6d44bc8406c80508190874 already in progress: creating tablet
I20240430 20:11:45.224937 18473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ce09fcbe1ac457a9c05efaf000c3263" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:45.225541 19495 tablet_replica.cc:330] T 50486644ec234933b9a0b45ee39a6685 P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:45.226231 19495 raft_consensus.cc:2229] T 50486644ec234933b9a0b45ee39a6685 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:45.226620 19495 raft_consensus.cc:2258] T 50486644ec234933b9a0b45ee39a6685 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:45.239078 20424 catalog_manager.cc:1259] Loaded cluster ID: 889e0faede1f41d1ab74633d6826c0d4
I20240430 20:11:45.240485 20424 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: loading cluster ID for follower catalog manager: success
I20240430 20:11:45.043615 20489 raft_consensus.cc:482] T 4f6f9ae722da49e58c9b72107f6da2fa P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:45.247869 20489 raft_consensus.cc:504] T 4f6f9ae722da49e58c9b72107f6da2fa P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:45.248775 20489 leader_election.cc:288] T 4f6f9ae722da49e58c9b72107f6da2fa P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:45.054181 20562 tablet.cc:1722] T 1462ecf1176c4e66be42abddbe36b00c P 5e1c6397f09747f091534faf93542710: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240430 20:11:45.077625 18588 tablet_service.cc:1467] Processing CreateTablet for tablet 467cafc47106474e96f619b4043db233 (DEFAULT_TABLE table=default.Table_f45eee4d4b6a4baf83111f12dfd25a9f [id=ea6b9f7d86424c84a8ea05a5877afd06]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:45.249737 18588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 467cafc47106474e96f619b4043db233. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:45.099896 18601 tablet_service.cc:1467] Processing CreateTablet for tablet 2e54ea9db8584db8abc36c1d3fd02c76 (DEFAULT_TABLE table=default.Table_75efd32c9420454f901c61882c8c29bc [id=0d741ba94928463a9929f799e0336fdb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:45.255089 18601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e54ea9db8584db8abc36c1d3fd02c76. 1 dirs total, 0 dirs full, 0 dirs failed
W20240430 20:11:45.100406 18558 consensus_peers.cc:485] T a05e32a33aa7488eb287cc6314c09c44 P 5e1c6397f09747f091534faf93542710 -> Peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Couldn't send request to peer 2efb542aa55e439b99ccb98d1c5c884a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a05e32a33aa7488eb287cc6314c09c44. This is attempt 1: this message will repeat every 5th retry.
W20240430 20:11:45.258713 18558 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.177.1:41217, user_credentials={real_user=slave}} blocked reactor thread for 158335us
W20240430 20:11:45.259274 18558 consensus_peers.cc:485] T acf3b6fead41483a82c43dc0286fab7c P 5e1c6397f09747f091534faf93542710 -> Peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Couldn't send request to peer 2efb542aa55e439b99ccb98d1c5c884a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: acf3b6fead41483a82c43dc0286fab7c. This is attempt 1: this message will repeat every 5th retry.
I20240430 20:11:45.260167 20384 consensus_queue.cc:1035] T 681cdd85ac6847a0a1aaded310456582 P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:45.225308 18473 raft_consensus.cc:2454] T 7ce09fcbe1ac457a9c05efaf000c3263 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:45.261094 18617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ce09fcbe1ac457a9c05efaf000c3263" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
W20240430 20:11:45.148984 18557 consensus_peers.cc:485] T 681cdd85ac6847a0a1aaded310456582 P 5e1c6397f09747f091534faf93542710 -> Peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Couldn't send request to peer 3c51c33d5616482cafb68e26688820f4. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
W20240430 20:11:45.261874 18557 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.177.3:44867, user_credentials={real_user=slave}} blocked reactor thread for 112932us
I20240430 20:11:45.262015 18595 tablet_service.cc:1467] Processing CreateTablet for tablet 3fa6c1078c074ef4a7a408efa8c19467 (DEFAULT_TABLE table=default.Table_e311a584368b4233b6e5beb24587ce00 [id=65018c93b8e2409f984c52764e82dcb3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:45.262097 19495 ts_tablet_manager.cc:1875] T 50486644ec234933b9a0b45ee39a6685 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:45.262395 18595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fa6c1078c074ef4a7a408efa8c19467. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:45.263430 18586 tablet_service.cc:1467] Processing CreateTablet for tablet 37b2ea4be7494f0985808ade788dbe2b (DEFAULT_TABLE table=default.Table_5f6e0fce999b46de927f9271e5de5071 [id=0baad83a8c0f4aeebc7ddad068625a69]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:45.263751 18586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37b2ea4be7494f0985808ade788dbe2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:45.266254 18985 raft_consensus.cc:348] T 4f6f9ae722da49e58c9b72107f6da2fa P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:45.266464 18985 raft_consensus.cc:374] T 4f6f9ae722da49e58c9b72107f6da2fa P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:45.266511 18985 raft_consensus.cc:729] T 4f6f9ae722da49e58c9b72107f6da2fa P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:45.266752 18985 consensus_queue.cc:260] T 4f6f9ae722da49e58c9b72107f6da2fa P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:45.266906 18600 tablet_service.cc:1467] Processing CreateTablet for tablet aed9a3eefe3f4136af1b9469c9e2e23c (DEFAULT_TABLE table=default.Table_457e03bc2d524accaa76649790c42fc8 [id=f26e395466f24585b9524a1ec2f77051]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:45.266949 18602 tablet_service.cc:1467] Processing CreateTablet for tablet aaef85eea87f4443943fd3692bb2a38f (DEFAULT_TABLE table=default.Table_3da5b4c8668c470483ae2a0c7773eaaf [id=e292d818ee3c45df90522f08eb2d6227]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:45.267261 18600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aed9a3eefe3f4136af1b9469c9e2e23c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:45.267249 18985 ts_tablet_manager.cc:1398] T 4f6f9ae722da49e58c9b72107f6da2fa P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.132s user 0.002s sys 0.000s
I20240430 20:11:45.267525 18985 tablet_bootstrap.cc:492] T 677481aee5fa4eb4b189322d7a3f8f2e P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:45.267549 18603 tablet_service.cc:1467] Processing CreateTablet for tablet cdc07bb607374c6e8eb6ca35a86d0204 (DEFAULT_TABLE table=default.Table_fe491607e9e7480b9d9d2c8eb89ccac9 [id=ba205c2256d24f998fb16e6590b74cf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:45.267903 18603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdc07bb607374c6e8eb6ca35a86d0204. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:45.270800 18699 leader_election.cc:302] T 7ce09fcbe1ac457a9c05efaf000c3263 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters:
I20240430 20:11:45.270939 18590 tablet_service.cc:1467] Processing CreateTablet for tablet a3b6e2ed50184e97914a703e2d73a285 (DEFAULT_TABLE table=default.Table_8d32c8a8f05e4b599dd6a713133ed82e [id=295c03e97c87495f8d71b255b2ec9065]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:45.271188 18482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f6f9ae722da49e58c9b72107f6da2fa" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:45.271366 18590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3b6e2ed50184e97914a703e2d73a285. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:45.271382 18482 raft_consensus.cc:2454] T 4f6f9ae722da49e58c9b72107f6da2fa P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 0.
I20240430 20:11:45.271800 18985 tablet_bootstrap.cc:654] T 677481aee5fa4eb4b189322d7a3f8f2e P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:45.271984 20587 raft_consensus.cc:2781] T 7ce09fcbe1ac457a9c05efaf000c3263 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240430 20:11:45.272076 20587 raft_consensus.cc:482] T 7ce09fcbe1ac457a9c05efaf000c3263 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:45.272140 20587 raft_consensus.cc:3037] T 7ce09fcbe1ac457a9c05efaf000c3263 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Advancing to term 2
I20240430 20:11:45.274817 20047 consensus_queue.cc:1035] T abdaf243e5be4ad58036b23151b14fe4 P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:45.275039 18592 tablet_service.cc:1467] Processing CreateTablet for tablet 118eb508b3294b55abbda27e98495353 (DEFAULT_TABLE table=default.Table_d770977de70d4cac8511d8d125c1b160 [id=d80cad6e0e734b8fa6d2011a30f9657b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:45.275385 20047 consensus_queue.cc:1035] T 7ce09fcbe1ac457a9c05efaf000c3263 P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:45.275542 20198 raft_consensus.cc:482] T 5d09baed576e4136bf6d756ca39d08b2 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:45.275678 20198 raft_consensus.cc:504] T 5d09baed576e4136bf6d756ca39d08b2 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:45.275992 20587 raft_consensus.cc:504] T 7ce09fcbe1ac457a9c05efaf000c3263 P 3c51c33d5616482cafb68e26688820f4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:45.276353 20198 leader_election.cc:288] T 5d09baed576e4136bf6d756ca39d08b2 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:45.276433 20587 leader_election.cc:288] T 7ce09fcbe1ac457a9c05efaf000c3263 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 2 election: Requested vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
W20240430 20:11:45.276940 18589 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40218: Illegal state: replica 5e1c6397f09747f091534faf93542710 is not leader of this config: current role FOLLOWER
W20240430 20:11:45.277403 18696 leader_election.cc:341] T 2cdccf4be7094ca5b863dd5c3432d55d P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Not found: Tablet not found: 2cdccf4be7094ca5b863dd5c3432d55d
I20240430 20:11:45.277508 18696 leader_election.cc:302] T 2cdccf4be7094ca5b863dd5c3432d55d P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710
I20240430 20:11:45.277801 20205 raft_consensus.cc:2726] T 2cdccf4be7094ca5b863dd5c3432d55d P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:45.204085 20488 raft_consensus.cc:482] T f6cfd089f08b47409fb2a6cf4f6b0385 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:45.278185 20488 raft_consensus.cc:504] T f6cfd089f08b47409fb2a6cf4f6b0385 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:45.278966 20488 leader_election.cc:288] T f6cfd089f08b47409fb2a6cf4f6b0385 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:45.275400 18592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 118eb508b3294b55abbda27e98495353. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:45.283892 18602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aaef85eea87f4443943fd3692bb2a38f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:45.287514 18473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ce09fcbe1ac457a9c05efaf000c3263" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:45.287771 18473 raft_consensus.cc:3037] T 7ce09fcbe1ac457a9c05efaf000c3263 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Advancing to term 2
W20240430 20:11:45.291926 18699 leader_election.cc:341] T 5d09baed576e4136bf6d756ca39d08b2 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Not found: Tablet not found: 5d09baed576e4136bf6d756ca39d08b2
I20240430 20:11:45.294978 18473 raft_consensus.cc:2454] T 7ce09fcbe1ac457a9c05efaf000c3263 P 2efb542aa55e439b99ccb98d1c5c884a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 2.
I20240430 20:11:45.296608 18699 leader_election.cc:302] T 7ce09fcbe1ac457a9c05efaf000c3263 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters:
I20240430 20:11:45.297070 20587 raft_consensus.cc:2781] T 7ce09fcbe1ac457a9c05efaf000c3263 P 3c51c33d5616482cafb68e26688820f4 [term 2 FOLLOWER]: Leader election won for term 2
I20240430 20:11:45.297376 20587 raft_consensus.cc:686] T 7ce09fcbe1ac457a9c05efaf000c3263 P 3c51c33d5616482cafb68e26688820f4 [term 2 LEADER]: Becoming Leader. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Running, Role: LEADER
I20240430 20:11:45.297804 20587 consensus_queue.cc:237] T 7ce09fcbe1ac457a9c05efaf000c3263 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:45.307610 18762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6cfd089f08b47409fb2a6cf4f6b0385" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:45.307402 18763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f6f9ae722da49e58c9b72107f6da2fa" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
W20240430 20:11:45.310775 18557 leader_election.cc:341] T f6cfd089f08b47409fb2a6cf4f6b0385 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
W20240430 20:11:45.311153 18557 leader_election.cc:341] T 4f6f9ae722da49e58c9b72107f6da2fa P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:45.311473 18758 raft_consensus.cc:1226] T 7ce09fcbe1ac457a9c05efaf000c3263 P 3c51c33d5616482cafb68e26688820f4 [term 2 LEADER]: Rejecting Update request from peer 5e1c6397f09747f091534faf93542710 for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20240430 20:11:45.317726 19495 ts_tablet_manager.cc:1888] T 50486644ec234933b9a0b45ee39a6685 P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:45.317858 19495 log.cc:1196] T 50486644ec234933b9a0b45ee39a6685 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/50486644ec234933b9a0b45ee39a6685
I20240430 20:11:45.319022 18558 leader_election.cc:302] T 4f6f9ae722da49e58c9b72107f6da2fa P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters: 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:45.319725 20047 consensus_queue.cc:1035] T abdaf243e5be4ad58036b23151b14fe4 P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:45.322155 20489 raft_consensus.cc:2781] T 4f6f9ae722da49e58c9b72107f6da2fa P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:45.322288 20489 raft_consensus.cc:482] T 4f6f9ae722da49e58c9b72107f6da2fa P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:45.322384 20489 raft_consensus.cc:3037] T 4f6f9ae722da49e58c9b72107f6da2fa P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:45.326082 20489 raft_consensus.cc:504] T 4f6f9ae722da49e58c9b72107f6da2fa P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:45.326866 20047 consensus_queue.cc:1035] T 5bd0362ff41b4faca0205d136d596737 P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:45.327505 20587 raft_consensus.cc:482] T f7b0132c17d44a9ba17288e0321a7f11 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:45.327661 20587 raft_consensus.cc:504] T f7b0132c17d44a9ba17288e0321a7f11 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:45.328279 20587 leader_election.cc:288] T f7b0132c17d44a9ba17288e0321a7f11 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:45.327790 20593 consensus_queue.cc:1046] T 7ce09fcbe1ac457a9c05efaf000c3263 P 5e1c6397f09747f091534faf93542710 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240430 20:11:45.328466 20593 consensus_queue.cc:1225] T 7ce09fcbe1ac457a9c05efaf000c3263 P 5e1c6397f09747f091534faf93542710 [LEADER]: Peer 3c51c33d5616482cafb68e26688820f4 log is divergent from this leader: its last log entry 2.1 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 0
I20240430 20:11:45.328606 20593 raft_consensus.cc:3032] T 7ce09fcbe1ac457a9c05efaf000c3263 P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Stepping down as leader of term 1
I20240430 20:11:45.328675 20593 raft_consensus.cc:729] T 7ce09fcbe1ac457a9c05efaf000c3263 P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Running, Role: LEADER
I20240430 20:11:45.328971 20593 consensus_queue.cc:260] T 7ce09fcbe1ac457a9c05efaf000c3263 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:45.329417 20593 raft_consensus.cc:3037] T 7ce09fcbe1ac457a9c05efaf000c3263 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Advancing to term 2
W20240430 20:11:45.329434 18699 leader_election.cc:341] T f7b0132c17d44a9ba17288e0321a7f11 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Not found: Tablet not found: f7b0132c17d44a9ba17288e0321a7f11
I20240430 20:11:45.330070 19495 ts_tablet_manager.cc:1909] T 50486644ec234933b9a0b45ee39a6685 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:45.330866 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 347) took 1879 ms. Trace:
I20240430 20:11:45.330963 19495 rpcz_store.cc:276] 0430 20:11:43.451048 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:43.451208 (+ 160us) service_pool.cc:225] Handling call
0430 20:11:45.218729 (+1767521us) ts_tablet_manager.cc:1120] Deleting tablet 50486644ec234933b9a0b45ee39a6685
0430 20:11:45.218743 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:45.280423 (+ 61680us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:45.317626 (+ 37203us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:45.330843 (+ 13217us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1767376}
I20240430 20:11:45.331411 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 349) took 1830 ms. Trace:
I20240430 20:11:45.331488 19004 catalog_manager.cc:4830] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): tablet 50486644ec234933b9a0b45ee39a6685 (table default.Table_a2e85efc6b454524b0b122102ec07002 [id=86d795a82e674d859104741ebd04f4e8]) successfully deleted
I20240430 20:11:45.331496 19495 rpcz_store.cc:276] 0430 20:11:43.501367 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:43.504259 (+ 2892us) service_pool.cc:225] Handling call
0430 20:11:45.331274 (+1827015us) ts_tablet_manager.cc:1120] Deleting tablet 489dc838744a49eba542009c86688aa5
0430 20:11:45.331285 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:45.331394 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1826101}
I20240430 20:11:45.331771 19495 tablet_replica.cc:330] T 5bd0362ff41b4faca0205d136d596737 P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
W20240430 20:11:45.332041 19004 catalog_manager.cc:4810] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): delete failed for tablet 489dc838744a49eba542009c86688aa5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 489dc838744a49eba542009c86688aa5 already in progress: creating tablet
W20240430 20:11:45.327579 18558 consensus_peers.cc:485] T 115c4a12440647a4bb46eb6824f7ea98 P 5e1c6397f09747f091534faf93542710 -> Peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Couldn't send request to peer 2efb542aa55e439b99ccb98d1c5c884a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 115c4a12440647a4bb46eb6824f7ea98. This is attempt 1: this message will repeat every 5th retry.
I20240430 20:11:45.331967 19495 raft_consensus.cc:2229] T 5bd0362ff41b4faca0205d136d596737 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus shutting down.
W20240430 20:11:45.332851 18557 consensus_peers.cc:485] T 294644d7a16b4ca2bbb6d106bcb8c7ff P 5e1c6397f09747f091534faf93542710 -> Peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Couldn't send request to peer 3c51c33d5616482cafb68e26688820f4. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240430 20:11:45.333009 19495 raft_consensus.cc:2258] T 5bd0362ff41b4faca0205d136d596737 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:45.333698 19495 ts_tablet_manager.cc:1875] T 5bd0362ff41b4faca0205d136d596737 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:45.338807 20488 consensus_queue.cc:1035] T 5bd0362ff41b4faca0205d136d596737 P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:45.346992 18985 tablet_bootstrap.cc:492] T 677481aee5fa4eb4b189322d7a3f8f2e P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:45.347271 18985 ts_tablet_manager.cc:1367] T 677481aee5fa4eb4b189322d7a3f8f2e P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.080s user 0.006s sys 0.000s
I20240430 20:11:45.349704 18474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6cfd089f08b47409fb2a6cf4f6b0385" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:45.349980 18985 raft_consensus.cc:348] T 677481aee5fa4eb4b189322d7a3f8f2e P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:45.350178 18985 raft_consensus.cc:374] T 677481aee5fa4eb4b189322d7a3f8f2e P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:45.358877 18985 raft_consensus.cc:729] T 677481aee5fa4eb4b189322d7a3f8f2e P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:45.359634 18985 consensus_queue.cc:260] T 677481aee5fa4eb4b189322d7a3f8f2e P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:45.360845 18985 ts_tablet_manager.cc:1398] T 677481aee5fa4eb4b189322d7a3f8f2e P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20240430 20:11:45.361649 18985 tablet_bootstrap.cc:492] T 06e82c706b9c4f35bf53faddf57b3fd7 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:45.363500 20489 leader_election.cc:288] T 4f6f9ae722da49e58c9b72107f6da2fa P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:45.364367 18750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f6f9ae722da49e58c9b72107f6da2fa" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
W20240430 20:11:45.364977 18557 leader_election.cc:341] T 4f6f9ae722da49e58c9b72107f6da2fa P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:45.366819 18619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ce09fcbe1ac457a9c05efaf000c3263" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:45.367156 18619 raft_consensus.cc:2396] T 7ce09fcbe1ac457a9c05efaf000c3263 P 5e1c6397f09747f091534faf93542710 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 3c51c33d5616482cafb68e26688820f4 for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20240430 20:11:45.367902 18619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7b0132c17d44a9ba17288e0321a7f11" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:45.368088 18619 raft_consensus.cc:2454] T f7b0132c17d44a9ba17288e0321a7f11 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:45.371068 18473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f6f9ae722da49e58c9b72107f6da2fa" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:45.371279 18473 raft_consensus.cc:3037] T 4f6f9ae722da49e58c9b72107f6da2fa P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:45.374053 18473 raft_consensus.cc:2454] T 4f6f9ae722da49e58c9b72107f6da2fa P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
W20240430 20:11:45.380384 18558 leader_election.cc:341] T f6cfd089f08b47409fb2a6cf4f6b0385 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:45.380384 18696 leader_election.cc:302] T f7b0132c17d44a9ba17288e0321a7f11 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a
I20240430 20:11:45.380535 18558 leader_election.cc:302] T f6cfd089f08b47409fb2a6cf4f6b0385 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:45.380849 20587 raft_consensus.cc:2781] T f7b0132c17d44a9ba17288e0321a7f11 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:45.380932 20587 raft_consensus.cc:482] T f7b0132c17d44a9ba17288e0321a7f11 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:45.381017 20587 raft_consensus.cc:3037] T f7b0132c17d44a9ba17288e0321a7f11 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:45.383610 20587 raft_consensus.cc:504] T f7b0132c17d44a9ba17288e0321a7f11 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:45.384363 20587 leader_election.cc:288] T f7b0132c17d44a9ba17288e0321a7f11 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Requested vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:45.385198 20593 raft_consensus.cc:2726] T f6cfd089f08b47409fb2a6cf4f6b0385 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:45.385624 20489 raft_consensus.cc:482] T eb76f60448774fe1967fffe7c6873cfc P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:45.385787 20489 raft_consensus.cc:504] T eb76f60448774fe1967fffe7c6873cfc P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
W20240430 20:11:45.386011 18699 leader_election.cc:341] T f7b0132c17d44a9ba17288e0321a7f11 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Not found: Tablet not found: f7b0132c17d44a9ba17288e0321a7f11
I20240430 20:11:45.386740 20489 leader_election.cc:288] T eb76f60448774fe1967fffe7c6873cfc P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:45.390195 18750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb76f60448774fe1967fffe7c6873cfc" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
W20240430 20:11:45.391214 18557 leader_election.cc:341] T eb76f60448774fe1967fffe7c6873cfc P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
W20240430 20:11:45.393575 18558 consensus_peers.cc:485] T 5bd0362ff41b4faca0205d136d596737 P 5e1c6397f09747f091534faf93542710 -> Peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Couldn't send request to peer 2efb542aa55e439b99ccb98d1c5c884a. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240430 20:11:45.394109 18558 leader_election.cc:302] T 4f6f9ae722da49e58c9b72107f6da2fa P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters: 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:45.394454 20593 raft_consensus.cc:2781] T 4f6f9ae722da49e58c9b72107f6da2fa P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:45.394701 20593 raft_consensus.cc:686] T 4f6f9ae722da49e58c9b72107f6da2fa P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Becoming Leader. State: Replica: 5e1c6397f09747f091534faf93542710, State: Running, Role: LEADER
W20240430 20:11:45.395303 18696 leader_election.cc:341] T 5d09baed576e4136bf6d756ca39d08b2 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Not found: Tablet not found: 5d09baed576e4136bf6d756ca39d08b2
I20240430 20:11:45.395444 18696 leader_election.cc:302] T 5d09baed576e4136bf6d756ca39d08b2 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710
I20240430 20:11:45.396135 18474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb76f60448774fe1967fffe7c6873cfc" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:45.396262 20587 raft_consensus.cc:2726] T 5d09baed576e4136bf6d756ca39d08b2 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240430 20:11:45.395983 18623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7b0132c17d44a9ba17288e0321a7f11" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:45.396895 18623 raft_consensus.cc:3037] T f7b0132c17d44a9ba17288e0321a7f11 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:45.397033 20593 consensus_queue.cc:237] T 4f6f9ae722da49e58c9b72107f6da2fa P 5e1c6397f09747f091534faf93542710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:45.400650 18623 raft_consensus.cc:2454] T f7b0132c17d44a9ba17288e0321a7f11 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:45.352020 19495 ts_tablet_manager.cc:1888] T 5bd0362ff41b4faca0205d136d596737 P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:45.401667 19495 log.cc:1196] T 5bd0362ff41b4faca0205d136d596737 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/5bd0362ff41b4faca0205d136d596737
I20240430 20:11:45.402060 19495 ts_tablet_manager.cc:1909] T 5bd0362ff41b4faca0205d136d596737 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:45.404510 18985 tablet_bootstrap.cc:654] T 06e82c706b9c4f35bf53faddf57b3fd7 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:45.409829 18983 tablet_bootstrap.cc:492] T 2873af6da928463f9809864b8eaf0c4a P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:45.411742 18983 tablet_bootstrap.cc:654] T 2873af6da928463f9809864b8eaf0c4a P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:45.412904 20424 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: acquiring CA information for follower catalog manager: success
I20240430 20:11:45.414793 18116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskNG3HqZ/build/asan/bin/kudu pid 18934
I20240430 20:11:45.416378 19023 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_673d1448679f4b15a5ee2a1369c8f8ea"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:45.417135 19023 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_673d1448679f4b15a5ee2a1369c8f8ea in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:45.427884 19028 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_123c6be593f44fc2ac4115d192c43e68"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:45.428544 19028 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_123c6be593f44fc2ac4115d192c43e68 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:45.435712 18696 leader_election.cc:302] T f7b0132c17d44a9ba17288e0321a7f11 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a
I20240430 20:11:45.436015 20587 raft_consensus.cc:2781] T f7b0132c17d44a9ba17288e0321a7f11 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:45.436138 20587 raft_consensus.cc:686] T f7b0132c17d44a9ba17288e0321a7f11 P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Becoming Leader. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Running, Role: LEADER
I20240430 20:11:45.436405 20587 consensus_queue.cc:237] T f7b0132c17d44a9ba17288e0321a7f11 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
W20240430 20:11:45.436596 18558 leader_election.cc:341] T eb76f60448774fe1967fffe7c6873cfc P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:45.436714 18558 leader_election.cc:302] T eb76f60448774fe1967fffe7c6873cfc P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:45.437739 20488 raft_consensus.cc:2726] T eb76f60448774fe1967fffe7c6873cfc P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:45.442566 19023 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_1755296c0a63478699d9b8387f606a8e" } modify_external_catalogs: true
I20240430 20:11:45.442953 19023 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_1755296c0a63478699d9b8387f606a8e" } modify_external_catalogs: true
I20240430 20:11:45.247603 19471 ts_tablet_manager.cc:1888] T 84568f90b9c54cf292099808d4cbf5dc P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:45.446453 19471 log.cc:1196] T 84568f90b9c54cf292099808d4cbf5dc P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/84568f90b9c54cf292099808d4cbf5dc
I20240430 20:11:45.446431 19028 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_3f42877693b04aaa837c89b8983d5a72" } modify_external_catalogs: true
I20240430 20:11:45.446676 19028 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_3f42877693b04aaa837c89b8983d5a72" } modify_external_catalogs: true
I20240430 20:11:45.447031 19471 ts_tablet_manager.cc:1909] T 84568f90b9c54cf292099808d4cbf5dc P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:45.447444 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 187) took 9498 ms. Trace:
I20240430 20:11:45.447530 19471 rpcz_store.cc:276] 0430 20:11:35.948993 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:35.949036 (+ 43us) service_pool.cc:225] Handling call
0430 20:11:44.846489 (+8897453us) ts_tablet_manager.cc:1120] Deleting tablet 84568f90b9c54cf292099808d4cbf5dc
0430 20:11:44.846511 (+ 22us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:44.955598 (+109087us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:45.247429 (+291831us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:45.447421 (+199992us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7835362}
I20240430 20:11:45.449224 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 346) took 1998 ms. Trace:
I20240430 20:11:45.449337 19495 rpcz_store.cc:276] 0430 20:11:43.450969 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:43.516796 (+ 65827us) service_pool.cc:225] Handling call
0430 20:11:45.331723 (+1814927us) ts_tablet_manager.cc:1120] Deleting tablet 5bd0362ff41b4faca0205d136d596737
0430 20:11:45.331734 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:45.338063 (+ 6329us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:45.351903 (+ 13840us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:45.449194 (+ 97291us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":46085,"tablet-delete.queue_time_us":1814575}
I20240430 20:11:45.449831 19004 catalog_manager.cc:4830] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): tablet 5bd0362ff41b4faca0205d136d596737 (table default.Table_fe1773cf77654e2a86e95444437b7201 [id=fe2222c2b62f4360a5ade622ae8b23e5]) successfully deleted
I20240430 20:11:45.450165 19471 tablet_replica.cc:330] T 680553d4d0bc4fcb839235c1832757e1 P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:45.450385 19471 raft_consensus.cc:2229] T 680553d4d0bc4fcb839235c1832757e1 P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Raft consensus shutting down.
I20240430 20:11:45.455538 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 362) took 1109 ms. Trace:
I20240430 20:11:45.456138 19471 raft_consensus.cc:2258] T 680553d4d0bc4fcb839235c1832757e1 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:45.456591 18999 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet 84568f90b9c54cf292099808d4cbf5dc (table default.Table_0034641cfa4d42a68ff4341e87fcb396 [id=53c496365b43473fa7e61ca706cdd984]) successfully deleted
I20240430 20:11:45.456657 19495 rpcz_store.cc:276] 0430 20:11:44.346322 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:44.346381 (+ 59us) service_pool.cc:225] Handling call
0430 20:11:45.455403 (+1109022us) ts_tablet_manager.cc:1120] Deleting tablet c1017a50d1d646c486c0b9a75cb88ce4
0430 20:11:45.455413 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:45.455515 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1108753}
W20240430 20:11:45.457221 19004 catalog_manager.cc:4810] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): delete failed for tablet c1017a50d1d646c486c0b9a75cb88ce4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c1017a50d1d646c486c0b9a75cb88ce4 already in progress: creating tablet
I20240430 20:11:45.458061 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 363) took 1092 ms. Trace:
I20240430 20:11:45.457494 20489 consensus_queue.cc:1035] T 7683caba80594627900a0185f772c131 P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:45.460165 19489 ts_tablet_manager.cc:1888] T 294644d7a16b4ca2bbb6d106bcb8c7ff P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.3
I20240430 20:11:45.460325 19489 log.cc:1196] T 294644d7a16b4ca2bbb6d106bcb8c7ff P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/294644d7a16b4ca2bbb6d106bcb8c7ff
I20240430 20:11:45.460832 19489 ts_tablet_manager.cc:1909] T 294644d7a16b4ca2bbb6d106bcb8c7ff P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:45.461680 20205 raft_consensus.cc:482] T 2da900b6bbfa4fa99fe8494fd2e68da8 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:45.461820 20205 raft_consensus.cc:504] T 2da900b6bbfa4fa99fe8494fd2e68da8 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:45.462522 20205 leader_election.cc:288] T 2da900b6bbfa4fa99fe8494fd2e68da8 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:45.464032 18623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2da900b6bbfa4fa99fe8494fd2e68da8" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:45.464313 18623 raft_consensus.cc:2379] T 2da900b6bbfa4fa99fe8494fd2e68da8 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c51c33d5616482cafb68e26688820f4 in current term 1: Already voted for candidate 2efb542aa55e439b99ccb98d1c5c884a in this term.
I20240430 20:11:45.464653 19471 ts_tablet_manager.cc:1875] T 680553d4d0bc4fcb839235c1832757e1 P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:45.463908 18474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2da900b6bbfa4fa99fe8494fd2e68da8" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:45.465790 19495 rpcz_store.cc:276] 0430 20:11:44.365606 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:44.366867 (+ 1261us) service_pool.cc:225] Handling call
0430 20:11:45.457919 (+1091052us) ts_tablet_manager.cc:1120] Deleting tablet 024069ab786749028c020abd1c836cf5
0430 20:11:45.457931 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:45.458041 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1090858}
W20240430 20:11:45.466358 19004 catalog_manager.cc:4810] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): delete failed for tablet 024069ab786749028c020abd1c836cf5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 024069ab786749028c020abd1c836cf5 already in progress: creating tablet
I20240430 20:11:45.466449 18699 leader_election.cc:302] T 2da900b6bbfa4fa99fe8494fd2e68da8 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710
I20240430 20:11:45.466970 20205 raft_consensus.cc:3037] T 2da900b6bbfa4fa99fe8494fd2e68da8 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:45.469452 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 364) took 1103 ms. Trace:
I20240430 20:11:45.469627 20205 raft_consensus.cc:2726] T 2da900b6bbfa4fa99fe8494fd2e68da8 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:45.469549 19495 rpcz_store.cc:276] 0430 20:11:44.365751 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:44.365808 (+ 57us) service_pool.cc:225] Handling call
0430 20:11:45.469290 (+1103482us) ts_tablet_manager.cc:1120] Deleting tablet f6cfd089f08b47409fb2a6cf4f6b0385
0430 20:11:45.469302 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:45.469424 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1101974}
W20240430 20:11:45.470084 19004 catalog_manager.cc:4810] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): delete failed for tablet f6cfd089f08b47409fb2a6cf4f6b0385 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f6cfd089f08b47409fb2a6cf4f6b0385 already in progress: creating tablet
W20240430 20:11:45.472285 19004 catalog_manager.cc:4810] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): delete failed for tablet a37d917bcc45468f9e3ee805640fba98 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a37d917bcc45468f9e3ee805640fba98 already in progress: creating tablet
I20240430 20:11:45.480930 18758 raft_consensus.cc:3037] T 7683caba80594627900a0185f772c131 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:45.484983 20487 consensus_queue.cc:1035] T 7683caba80594627900a0185f772c131 P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240430 20:11:45.492102 19004 catalog_manager.cc:4810] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): delete failed for tablet a445fdfde01c4371a5e6b3daea5b00ab because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a445fdfde01c4371a5e6b3daea5b00ab already in progress: creating tablet
W20240430 20:11:45.492900 19004 catalog_manager.cc:4810] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): delete failed for tablet 9f0b341a31224bf1be525b84b9500304 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9f0b341a31224bf1be525b84b9500304 already in progress: creating tablet
I20240430 20:11:45.495745 19495 tablet_replica.cc:330] T e3e74e76ab2b4fe492ffcc1838656fd1 P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:45.495811 18983 tablet_bootstrap.cc:492] T 2873af6da928463f9809864b8eaf0c4a P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:45.495978 18983 ts_tablet_manager.cc:1367] T 2873af6da928463f9809864b8eaf0c4a P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.086s user 0.002s sys 0.001s
I20240430 20:11:45.496815 19495 raft_consensus.cc:2229] T e3e74e76ab2b4fe492ffcc1838656fd1 P 2efb542aa55e439b99ccb98d1c5c884a [term 3 LEADER]: Raft consensus shutting down.
I20240430 20:11:45.496799 18983 raft_consensus.cc:348] T 2873af6da928463f9809864b8eaf0c4a P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:45.497013 18983 raft_consensus.cc:374] T 2873af6da928463f9809864b8eaf0c4a P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:45.497081 18983 raft_consensus.cc:729] T 2873af6da928463f9809864b8eaf0c4a P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:45.497299 18983 consensus_queue.cc:260] T 2873af6da928463f9809864b8eaf0c4a P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:45.497789 18983 ts_tablet_manager.cc:1398] T 2873af6da928463f9809864b8eaf0c4a P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240430 20:11:45.498049 18983 tablet_bootstrap.cc:492] T fbdeb4714b8c4b93ab72822c43329aae P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:45.498059 19495 raft_consensus.cc:2258] T e3e74e76ab2b4fe492ffcc1838656fd1 P 2efb542aa55e439b99ccb98d1c5c884a [term 3 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:45.499543 19495 ts_tablet_manager.cc:1875] T e3e74e76ab2b4fe492ffcc1838656fd1 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:45.509094 18985 tablet_bootstrap.cc:492] T 06e82c706b9c4f35bf53faddf57b3fd7 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:45.509277 18985 ts_tablet_manager.cc:1367] T 06e82c706b9c4f35bf53faddf57b3fd7 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.148s user 0.003s sys 0.000s
I20240430 20:11:45.510087 18985 raft_consensus.cc:348] T 06e82c706b9c4f35bf53faddf57b3fd7 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:45.510290 18985 raft_consensus.cc:374] T 06e82c706b9c4f35bf53faddf57b3fd7 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:45.510346 18985 raft_consensus.cc:729] T 06e82c706b9c4f35bf53faddf57b3fd7 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:45.510640 18985 consensus_queue.cc:260] T 06e82c706b9c4f35bf53faddf57b3fd7 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:45.510903 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 225) took 5747 ms. Trace:
I20240430 20:11:45.511011 19489 rpcz_store.cc:276] 0430 20:11:39.763790 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:39.774040 (+ 10250us) service_pool.cc:225] Handling call
0430 20:11:45.187513 (+5413473us) ts_tablet_manager.cc:1120] Deleting tablet 294644d7a16b4ca2bbb6d106bcb8c7ff
0430 20:11:45.187522 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:45.236996 (+ 49474us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:45.460007 (+223011us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:45.510870 (+ 50863us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5376019}
I20240430 20:11:45.511479 18985 ts_tablet_manager.cc:1398] T 06e82c706b9c4f35bf53faddf57b3fd7 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240430 20:11:45.511808 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 226) took 5700 ms. Trace:
I20240430 20:11:45.511905 19489 rpcz_store.cc:276] 0430 20:11:39.811460 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:39.811542 (+ 82us) service_pool.cc:225] Handling call
0430 20:11:45.511639 (+5700097us) ts_tablet_manager.cc:1120] Deleting tablet 1f4e860484784aa9a2d28af2e9fb611c
0430 20:11:45.511653 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:45.511790 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5699353}
I20240430 20:11:45.512372 19489 tablet_replica.cc:330] T d61c279d20b0471a967942525ceb4fcb P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:45.512560 19489 raft_consensus.cc:2229] T d61c279d20b0471a967942525ceb4fcb P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:45.512724 19489 raft_consensus.cc:2258] T d61c279d20b0471a967942525ceb4fcb P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:45.513267 19489 ts_tablet_manager.cc:1875] T d61c279d20b0471a967942525ceb4fcb P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240430 20:11:45.513942 18417 consensus_peers.cc:485] T d61c279d20b0471a967942525ceb4fcb P 2efb542aa55e439b99ccb98d1c5c884a -> Peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Couldn't send request to peer 3c51c33d5616482cafb68e26688820f4. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240430 20:11:45.515290 18985 tablet_bootstrap.cc:492] T eb76f60448774fe1967fffe7c6873cfc P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:45.515290 20384 consensus_queue.cc:1035] T f940d17b02b541289121463e05b0bb5f P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:45.515566 19001 catalog_manager.cc:4830] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): tablet 294644d7a16b4ca2bbb6d106bcb8c7ff (table default.Table_190b2d86515a44898fa38a49517b9df1 [id=97d8a36ddfae4bb5b99c04a7a08544af]) successfully deleted
I20240430 20:11:45.516477 19471 ts_tablet_manager.cc:1888] T 680553d4d0bc4fcb839235c1832757e1 P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240430 20:11:45.516661 19471 log.cc:1196] T 680553d4d0bc4fcb839235c1832757e1 P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/680553d4d0bc4fcb839235c1832757e1
I20240430 20:11:45.517071 19471 ts_tablet_manager.cc:1909] T 680553d4d0bc4fcb839235c1832757e1 P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:45.518378 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 194) took 9548 ms. Trace:
I20240430 20:11:45.518520 19471 rpcz_store.cc:276] 0430 20:11:35.969571 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:35.969642 (+ 71us) service_pool.cc:225] Handling call
0430 20:11:45.448464 (+9478822us) ts_tablet_manager.cc:1120] Deleting tablet 680553d4d0bc4fcb839235c1832757e1
0430 20:11:45.450083 (+ 1619us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:45.503138 (+ 53055us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:45.516370 (+ 13232us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:45.518355 (+ 1985us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7961010}
I20240430 20:11:45.519464 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 188) took 9570 ms. Trace:
W20240430 20:11:45.519971 19001 catalog_manager.cc:4810] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): delete failed for tablet 1f4e860484784aa9a2d28af2e9fb611c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1f4e860484784aa9a2d28af2e9fb611c already in progress: creating tablet
I20240430 20:11:45.520184 19471 rpcz_store.cc:276] 0430 20:11:35.949279 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:35.949323 (+ 44us) service_pool.cc:225] Handling call
0430 20:11:45.519331 (+9570008us) ts_tablet_manager.cc:1120] Deleting tablet 6db55bb510b24b0cae897936211e15f0
0430 20:11:45.519342 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:45.519446 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":8029144}
I20240430 20:11:45.520867 19471 tablet_replica.cc:330] T e87fb2f0361d4aa3ada27bbdfa88e94c P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:45.520870 18988 tablet_bootstrap.cc:492] T 99203cc522de424184c6f97453f166fd P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:45.521016 18988 ts_tablet_manager.cc:1367] T 99203cc522de424184c6f97453f166fd P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.317s user 0.032s sys 0.095s
I20240430 20:11:45.521827 18988 raft_consensus.cc:348] T 99203cc522de424184c6f97453f166fd P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:45.522049 18988 raft_consensus.cc:374] T 99203cc522de424184c6f97453f166fd P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:45.522104 18988 raft_consensus.cc:729] T 99203cc522de424184c6f97453f166fd P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
W20240430 20:11:45.522202 18999 catalog_manager.cc:4810] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): delete failed for tablet 6db55bb510b24b0cae897936211e15f0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6db55bb510b24b0cae897936211e15f0 already in progress: creating tablet
I20240430 20:11:45.522348 19471 raft_consensus.cc:2229] T e87fb2f0361d4aa3ada27bbdfa88e94c P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:45.522316 18988 consensus_queue.cc:260] T 99203cc522de424184c6f97453f166fd P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:45.522522 19471 raft_consensus.cc:2258] T e87fb2f0361d4aa3ada27bbdfa88e94c P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:45.519845 18983 tablet_bootstrap.cc:654] T fbdeb4714b8c4b93ab72822c43329aae P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:45.522917 18988 ts_tablet_manager.cc:1398] T 99203cc522de424184c6f97453f166fd P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240430 20:11:45.526762 18988 tablet_bootstrap.cc:492] T a9f37f0f7ce8425f9b24699513d41d61 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:45.528399 18988 tablet_bootstrap.cc:654] T a9f37f0f7ce8425f9b24699513d41d61 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:45.531982 19471 ts_tablet_manager.cc:1875] T e87fb2f0361d4aa3ada27bbdfa88e94c P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:45.527878 20105 raft_consensus.cc:482] T a9f37f0f7ce8425f9b24699513d41d61 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:45.540879 20105 raft_consensus.cc:504] T a9f37f0f7ce8425f9b24699513d41d61 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:45.541996 20105 leader_election.cc:288] T a9f37f0f7ce8425f9b24699513d41d61 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:45.542779 18985 tablet_bootstrap.cc:654] T eb76f60448774fe1967fffe7c6873cfc P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:45.549970 18623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9f37f0f7ce8425f9b24699513d41d61" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:45.550225 18623 raft_consensus.cc:2454] T a9f37f0f7ce8425f9b24699513d41d61 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 0.
I20240430 20:11:45.551802 18756 raft_consensus.cc:3037] T f940d17b02b541289121463e05b0bb5f P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:45.555704 18415 leader_election.cc:302] T a9f37f0f7ce8425f9b24699513d41d61 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:45.556025 20545 raft_consensus.cc:2781] T a9f37f0f7ce8425f9b24699513d41d61 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:45.556102 20545 raft_consensus.cc:482] T a9f37f0f7ce8425f9b24699513d41d61 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:45.556160 20545 raft_consensus.cc:3037] T a9f37f0f7ce8425f9b24699513d41d61 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:45.565465 19471 ts_tablet_manager.cc:1888] T e87fb2f0361d4aa3ada27bbdfa88e94c P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:45.567433 18985 tablet_bootstrap.cc:492] T eb76f60448774fe1967fffe7c6873cfc P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:45.567684 18985 ts_tablet_manager.cc:1367] T eb76f60448774fe1967fffe7c6873cfc P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.055s user 0.002s sys 0.002s
I20240430 20:11:45.568485 18985 raft_consensus.cc:348] T eb76f60448774fe1967fffe7c6873cfc P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:45.568759 18985 raft_consensus.cc:374] T eb76f60448774fe1967fffe7c6873cfc P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:45.568850 18985 raft_consensus.cc:729] T eb76f60448774fe1967fffe7c6873cfc P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:45.569140 18985 consensus_queue.cc:260] T eb76f60448774fe1967fffe7c6873cfc P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:45.569839 18985 ts_tablet_manager.cc:1398] T eb76f60448774fe1967fffe7c6873cfc P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240430 20:11:45.570081 19471 log.cc:1196] T e87fb2f0361d4aa3ada27bbdfa88e94c P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/e87fb2f0361d4aa3ada27bbdfa88e94c
I20240430 20:11:45.570557 19471 ts_tablet_manager.cc:1909] T e87fb2f0361d4aa3ada27bbdfa88e94c P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:45.571208 18985 tablet_bootstrap.cc:492] T fbdeb4714b8c4b93ab72822c43329aae P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:45.571614 20538 consensus_queue.cc:1035] T 2da900b6bbfa4fa99fe8494fd2e68da8 P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:45.572917 18985 tablet_bootstrap.cc:654] T fbdeb4714b8c4b93ab72822c43329aae P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:45.573930 20545 raft_consensus.cc:504] T a9f37f0f7ce8425f9b24699513d41d61 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:45.575237 20545 leader_election.cc:288] T a9f37f0f7ce8425f9b24699513d41d61 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:45.575771 18750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9f37f0f7ce8425f9b24699513d41d61" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:45.575920 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 199) took 6917 ms. Trace:
I20240430 20:11:45.576073 19471 rpcz_store.cc:276] 0430 20:11:38.658923 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:38.659001 (+ 78us) service_pool.cc:225] Handling call
0430 20:11:45.520818 (+6861817us) ts_tablet_manager.cc:1120] Deleting tablet e87fb2f0361d4aa3ada27bbdfa88e94c
0430 20:11:45.520828 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:45.546970 (+ 26142us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:45.565321 (+ 18351us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:45.575895 (+ 10574us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6861512}
W20240430 20:11:45.576391 18417 leader_election.cc:341] T a9f37f0f7ce8425f9b24699513d41d61 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:45.576776 18999 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet e87fb2f0361d4aa3ada27bbdfa88e94c (table default.Table_2f122c3c1c5949108231e55f6326e934 [id=02a19d9d05264236b35244d4e9998d36]) successfully deleted
I20240430 20:11:45.576910 18623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9f37f0f7ce8425f9b24699513d41d61" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:45.577100 18623 raft_consensus.cc:3037] T a9f37f0f7ce8425f9b24699513d41d61 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:45.577299 20384 consensus_queue.cc:1035] T f940d17b02b541289121463e05b0bb5f P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:45.579882 18623 raft_consensus.cc:2454] T a9f37f0f7ce8425f9b24699513d41d61 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 1.
I20240430 20:11:45.580353 18415 leader_election.cc:302] T a9f37f0f7ce8425f9b24699513d41d61 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters: 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:45.580893 19471 tablet_replica.cc:330] T b76a4105a2974fe280e759349037d027 P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:45.581668 20545 raft_consensus.cc:2781] T a9f37f0f7ce8425f9b24699513d41d61 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:45.581972 20545 raft_consensus.cc:686] T a9f37f0f7ce8425f9b24699513d41d61 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Becoming Leader. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Running, Role: LEADER
I20240430 20:11:45.582298 20545 consensus_queue.cc:237] T a9f37f0f7ce8425f9b24699513d41d61 P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:45.583899 19471 raft_consensus.cc:2229] T b76a4105a2974fe280e759349037d027 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:45.584184 19471 raft_consensus.cc:2258] T b76a4105a2974fe280e759349037d027 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:45.584195 19489 ts_tablet_manager.cc:1888] T d61c279d20b0471a967942525ceb4fcb P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:45.584302 19489 log.cc:1196] T d61c279d20b0471a967942525ceb4fcb P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/d61c279d20b0471a967942525ceb4fcb
I20240430 20:11:45.584614 19489 ts_tablet_manager.cc:1909] T d61c279d20b0471a967942525ceb4fcb P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:45.585064 19471 ts_tablet_manager.cc:1875] T b76a4105a2974fe280e759349037d027 P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:45.585400 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 231) took 5761 ms. Trace:
I20240430 20:11:45.585505 19489 rpcz_store.cc:276] 0430 20:11:39.824036 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:39.824101 (+ 65us) service_pool.cc:225] Handling call
0430 20:11:45.512321 (+5688220us) ts_tablet_manager.cc:1120] Deleting tablet d61c279d20b0471a967942525ceb4fcb
0430 20:11:45.512332 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:45.553532 (+ 41200us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:45.584073 (+ 30541us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:45.585380 (+ 1307us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5681801}
I20240430 20:11:45.585867 19489 tablet_replica.cc:330] T e2d90ff17e574f88b2061b1c117d4b7d P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:45.586035 19489 raft_consensus.cc:2229] T e2d90ff17e574f88b2061b1c117d4b7d P 3c51c33d5616482cafb68e26688820f4 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:45.586086 19001 catalog_manager.cc:4830] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): tablet d61c279d20b0471a967942525ceb4fcb (table default.Table_1177828cad1f4a24bd397619dad96170 [id=a95fed1bc3fa4865bf9a35dfc51b659c]) successfully deleted
I20240430 20:11:45.586205 19489 raft_consensus.cc:2258] T e2d90ff17e574f88b2061b1c117d4b7d P 3c51c33d5616482cafb68e26688820f4 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:45.587574 18758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9f37f0f7ce8425f9b24699513d41d61" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
W20240430 20:11:45.588325 18417 leader_election.cc:341] T a9f37f0f7ce8425f9b24699513d41d61 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:45.595798 19049 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 3a03b6997921408cb5a65f4cae223e9c (table default.Table_744b5c96ea014bdb8feae740356a790e [id=83b1ac2e9c3d4a10b02f96ebb17faf44]). Expected version 1 got 0
I20240430 20:11:45.599279 19049 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet ae20f44df50343289b0a512a4ea53692 (table default.Table_1844f6b1bda84812825384e14a2fd94e [id=71da9acc3cff4d95bbac4a686b708fd5]). Expected version 1 got 0
I20240430 20:11:45.600607 19489 ts_tablet_manager.cc:1875] T e2d90ff17e574f88b2061b1c117d4b7d P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:45.602025 18985 tablet_bootstrap.cc:492] T fbdeb4714b8c4b93ab72822c43329aae P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:45.603690 19049 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet eb6c96047b114ebba73db7a371f2da8e (table default.Table_7d5f6f18cbd64c869739872a7dff9be2 [id=e67d619e36c0485f8f22b4a81cb4cb6c]). Expected version 1 got 0
I20240430 20:11:45.605724 19049 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 62732af95c6a4c749c5b726f0460dc72 (table default.Table_e3160f5ced1b4718b479401ca4df675c [id=2d44ec94131148048021b2ee6366ba30]). Expected version 3 got 0
I20240430 20:11:45.606746 18985 ts_tablet_manager.cc:1367] T fbdeb4714b8c4b93ab72822c43329aae P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.036s user 0.002s sys 0.002s
I20240430 20:11:45.607262 19049 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet a0baee91f5b44945b595d0e234af018e (table default.Table_5ed3ea587cf843db9a1ff1ffab40b678 [id=b3d1c2f97c4648518dbea83a46cadd85]). Expected version 2 got 0
I20240430 20:11:45.607725 18985 raft_consensus.cc:348] T fbdeb4714b8c4b93ab72822c43329aae P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:45.608680 18985 raft_consensus.cc:374] T fbdeb4714b8c4b93ab72822c43329aae P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:45.608744 18985 raft_consensus.cc:729] T fbdeb4714b8c4b93ab72822c43329aae P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:45.609011 19049 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 266fd23786bc435db5083370f3330ed0 (table default.Table_cf60b652e9dc49e7a948ce66fd1cbffe [id=9734bb8dd05a400ca4244b3829f01dee]). Expected version 1 got 0
I20240430 20:11:45.608985 18985 consensus_queue.cc:260] T fbdeb4714b8c4b93ab72822c43329aae P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:45.609524 18985 ts_tablet_manager.cc:1398] T fbdeb4714b8c4b93ab72822c43329aae P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240430 20:11:45.609870 18985 tablet_bootstrap.cc:492] T f6cfd089f08b47409fb2a6cf4f6b0385 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:45.614738 19049 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet ed0c37a71ede4bfc9eb5ad1a4365f25e (table default.Table_a2e85efc6b454524b0b122102ec07002 [id=86d795a82e674d859104741ebd04f4e8]). Expected version 1 got 0
I20240430 20:11:45.625802 18985 tablet_bootstrap.cc:654] T f6cfd089f08b47409fb2a6cf4f6b0385 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:45.625577 19471 ts_tablet_manager.cc:1888] T b76a4105a2974fe280e759349037d027 P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:45.627110 19471 log.cc:1196] T b76a4105a2974fe280e759349037d027 P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/b76a4105a2974fe280e759349037d027
I20240430 20:11:45.627470 19471 ts_tablet_manager.cc:1909] T b76a4105a2974fe280e759349037d027 P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:45.628470 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 201) took 6969 ms. Trace:
I20240430 20:11:45.628620 19471 rpcz_store.cc:276] 0430 20:11:38.659158 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:38.659215 (+ 57us) service_pool.cc:225] Handling call
0430 20:11:45.580838 (+6921623us) ts_tablet_manager.cc:1120] Deleting tablet b76a4105a2974fe280e759349037d027
0430 20:11:45.580850 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:45.605776 (+ 24926us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:45.625445 (+ 19669us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:45.628446 (+ 3001us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2720,"tablet-delete.queue_time_us":6921412}
I20240430 20:11:45.629292 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 200) took 6970 ms. Trace:
I20240430 20:11:45.629446 19471 rpcz_store.cc:276] 0430 20:11:38.659049 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:38.659367 (+ 318us) service_pool.cc:225] Handling call
0430 20:11:45.629160 (+6969793us) ts_tablet_manager.cc:1120] Deleting tablet 558e71ec370d46ca8fbdcec5b57eb2ec
0430 20:11:45.629173 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:45.629275 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6969552}
I20240430 20:11:45.629981 19471 tablet_replica.cc:330] T 24dcc20832804229a2664873bf6ddaa7 P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:45.630250 19471 raft_consensus.cc:2229] T 24dcc20832804229a2664873bf6ddaa7 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:45.630546 19471 raft_consensus.cc:2258] T 24dcc20832804229a2664873bf6ddaa7 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:45.631342 19471 ts_tablet_manager.cc:1875] T 24dcc20832804229a2664873bf6ddaa7 P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:45.629257 18999 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet b76a4105a2974fe280e759349037d027 (table default.Table_95220daaf6d44c08beff4ed6abdbea79 [id=75f0256acc45402d8b6809d92c38a1d8]) successfully deleted
W20240430 20:11:45.642728 18999 catalog_manager.cc:4810] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): delete failed for tablet 558e71ec370d46ca8fbdcec5b57eb2ec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 558e71ec370d46ca8fbdcec5b57eb2ec already in progress: creating tablet
I20240430 20:11:45.644418 19049 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 3e5225f1f5f544ecbf8dfa470af3daca (table default.Table_c7c87a8e019944cab3ec99e9563ac01a [id=abf3765074eb4d7d9d2bcc4ab721e6bd]). Expected version 1 got 0
I20240430 20:11:45.649734 19049 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 1701a15b115741aea13f8327e3d26b34 (table default.Table_3776a4b9d4f24aa6b7f882bb4d0116ef [id=7127dbd368864f7ba80a755549fe7ede]). Expected version 1 got 0
I20240430 20:11:45.650068 20593 consensus_queue.cc:1035] T ae20f44df50343289b0a512a4ea53692 P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:45.652096 19049 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 12795670f21447a8964fe1efc6e92398 (table default.Table_896bbae1f6944005b0272ca8ebc97218 [id=d5cea90cbc2c48df9d8b80da1eccc949]). Expected version 3 got 0
I20240430 20:11:45.660224 20593 raft_consensus.cc:482] T 2873af6da928463f9809864b8eaf0c4a P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:45.660421 20593 raft_consensus.cc:504] T 2873af6da928463f9809864b8eaf0c4a P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:45.661111 20593 leader_election.cc:288] T 2873af6da928463f9809864b8eaf0c4a P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:45.662158 18474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2873af6da928463f9809864b8eaf0c4a" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
W20240430 20:11:45.662947 18558 leader_election.cc:341] T 2873af6da928463f9809864b8eaf0c4a P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:45.672293 19049 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet b285e3eb16d148779c74e057458d8461 (table default.Table_7fb60e4b56f444009ab5da219238eec7 [id=d1655c507b434f129ad7ae0b04365e51]). Expected version 1 got 0
I20240430 20:11:45.673115 20587 consensus_queue.cc:1035] T 30a521916bda46678be20a066bd1d3f3 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:45.673832 20424 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240430 20:11:45.675370 19049 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 130007acc8ba4e709c3eb85fabe75903 (table default.Table_78987875650940c293247bc43db345ab [id=36f478a20a1f4a53a52fc29ade62b7f6]). Expected version 1 got 0
I20240430 20:11:45.676465 19049 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet a482f4622f774d60b4efd6827b8be7d7 (table default.Table_e686cdfcbcc046749fa5b51578a64794 [id=c3dc9f05316d46d6a3ef4d03f40877ec]). Expected version 1 got 0
I20240430 20:11:45.677192 19049 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet f5eaf2413e1d4e69b56fae61f768f0ce (table default.Table_1280cfe32d5b41abb4d3eb1bd70641bf [id=8489b872074a42b6abb713e420968141]). Expected version 1 got 0
I20240430 20:11:45.679913 19049 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet ca64de16d40f4a44b8b411f5526ba11a (table default.Table_1cf914acf7fe4e98b8dbc297b6372f21 [id=f4ba9536e9c141deb68f4f62b7e8fa70]). Expected version 1 got 0
I20240430 20:11:45.684568 20205 consensus_queue.cc:1035] T 30a521916bda46678be20a066bd1d3f3 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:45.687081 19049 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 66dd767bd3f8494f94061fb89e5e5680 (table default.Table_b064d5659f7243f5b0ec167fb6c3f404 [id=5227206f44d04fa89b4842b899f33afb]). Expected version 2 got 0
I20240430 20:11:45.689334 19471 ts_tablet_manager.cc:1888] T 24dcc20832804229a2664873bf6ddaa7 P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:45.689659 19471 log.cc:1196] T 24dcc20832804229a2664873bf6ddaa7 P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/24dcc20832804229a2664873bf6ddaa7
I20240430 20:11:45.690847 19489 ts_tablet_manager.cc:1888] T e2d90ff17e574f88b2061b1c117d4b7d P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 3.3
I20240430 20:11:45.691403 19471 ts_tablet_manager.cc:1909] T 24dcc20832804229a2664873bf6ddaa7 P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:45.692080 19049 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 22fef8bb6b034db098c3372ae992f7a7 (table default.Table_4440df38610340758047115845b42b95 [id=7d186ca20c094cc19aef792fe1c094e9]). Expected version 1 got 0
I20240430 20:11:45.692513 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 202) took 6785 ms. Trace:
I20240430 20:11:45.692793 19489 log.cc:1196] T e2d90ff17e574f88b2061b1c117d4b7d P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/e2d90ff17e574f88b2061b1c117d4b7d
I20240430 20:11:45.693022 19049 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 330453ddeea6456e8ef93fa92608b0d3 (table default.Table_c1f36f7df5ae4e7c8eb8b4e99b89748b [id=a50abf28790d4284a76ac6ee22c02606]). Expected version 1 got 0
I20240430 20:11:45.693722 19489 ts_tablet_manager.cc:1909] T e2d90ff17e574f88b2061b1c117d4b7d P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:45.694012 19049 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 94fbd4a57a1e438891fcb456f6b40be3 (table default.Table_25ddc3adef6b4ccf8b9c00cc6772031c [id=4fa623ffdcf347639150a355806038af]). Expected version 1 got 0
I20240430 20:11:45.695504 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 232) took 5598 ms. Trace:
I20240430 20:11:45.698841 19049 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 2e79df8744e74b5991a79ce978ff0202 (table default.Table_276bbdc7cfe542d1a2936485a2aad1ec [id=f146a15e6e8a4acbb4c734c1c863da57]). Expected version 1 got 0
I20240430 20:11:45.696503 19471 rpcz_store.cc:276] 0430 20:11:38.907157 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:38.907240 (+ 83us) service_pool.cc:225] Handling call
0430 20:11:45.629934 (+6722694us) ts_tablet_manager.cc:1120] Deleting tablet 24dcc20832804229a2664873bf6ddaa7
0430 20:11:45.629944 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:45.638128 (+ 8184us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:45.688285 (+ 50157us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:45.692449 (+ 4164us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6722483}
I20240430 20:11:45.699513 19049 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet acd23f0a63d5448aaa7ed78422e67bad (table default.Table_972f13be4dca4cd19b9a04534dc59f51 [id=69f37db4de634a3185613e40013e0ed7]). Expected version 1 got 0
I20240430 20:11:45.701833 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 211) took 6129 ms. Trace:
I20240430 20:11:45.701665 18999 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet 24dcc20832804229a2664873bf6ddaa7 (table default.Table_e3160f5ced1b4718b479401ca4df675c [id=2d44ec94131148048021b2ee6366ba30]) successfully deleted
I20240430 20:11:45.702486 19471 rpcz_store.cc:276] 0430 20:11:39.572228 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:39.572298 (+ 70us) service_pool.cc:225] Handling call
0430 20:11:45.701670 (+6129372us) ts_tablet_manager.cc:1120] Deleting tablet de1dfe6c82d34a4d8d02e466c55327ff
0430 20:11:45.701684 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:45.701807 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6116911}
I20240430 20:11:45.702797 19049 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 821c55edb3414d72b1489eea97d3f73a (table default.Table_4d68c38d9ba7452a829f8318b2b267b8 [id=d06e3e9045b1429aaeebd1fd67d841de]). Expected version 1 got 0
W20240430 20:11:45.703404 18999 catalog_manager.cc:4810] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): delete failed for tablet de1dfe6c82d34a4d8d02e466c55327ff because tablet deleting was already in progress. No further retry: Already present: State transition of tablet de1dfe6c82d34a4d8d02e466c55327ff already in progress: creating tablet
I20240430 20:11:45.703564 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 212) took 6112 ms. Trace:
I20240430 20:11:45.703648 19471 rpcz_store.cc:276] 0430 20:11:39.591181 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:39.592066 (+ 885us) service_pool.cc:225] Handling call
0430 20:11:45.703420 (+6111354us) ts_tablet_manager.cc:1120] Deleting tablet 90a694bd5db649afa50f3c4e752939c1
0430 20:11:45.703432 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:45.703546 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6088250}
W20240430 20:11:45.704170 18999 catalog_manager.cc:4810] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): delete failed for tablet 90a694bd5db649afa50f3c4e752939c1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 90a694bd5db649afa50f3c4e752939c1 already in progress: creating tablet
I20240430 20:11:45.704697 19471 tablet_replica.cc:330] T 53a899fa5a424a0d99f1b47925e8c7ad P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:45.704504 19489 rpcz_store.cc:276] 0430 20:11:40.096923 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:40.097011 (+ 88us) service_pool.cc:225] Handling call
0430 20:11:45.585800 (+5488789us) ts_tablet_manager.cc:1120] Deleting tablet e2d90ff17e574f88b2061b1c117d4b7d
0430 20:11:45.585812 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:45.612109 (+ 26297us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:45.690722 (+ 78613us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:45.695478 (+ 4756us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5488501}
I20240430 20:11:45.704952 19471 raft_consensus.cc:2229] T 53a899fa5a424a0d99f1b47925e8c7ad P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Raft consensus shutting down.
I20240430 20:11:45.706393 19471 raft_consensus.cc:2258] T 53a899fa5a424a0d99f1b47925e8c7ad P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:45.706400 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 237) took 5354 ms. Trace:
I20240430 20:11:45.706718 19489 rpcz_store.cc:276] 0430 20:11:40.352178 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:40.354025 (+ 1847us) service_pool.cc:225] Handling call
0430 20:11:45.705889 (+5351864us) ts_tablet_manager.cc:1120] Deleting tablet 898b2c91adf34545808df9c786e218c9
0430 20:11:45.705904 (+ 15us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:45.706374 (+ 470us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5347702}
W20240430 20:11:45.707561 19001 catalog_manager.cc:4810] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): delete failed for tablet 898b2c91adf34545808df9c786e218c9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 898b2c91adf34545808df9c786e218c9 already in progress: creating tablet
I20240430 20:11:45.707911 19049 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 5df364628561475391c9a5e10ff5a05f (table default.Table_3c8f9f7b22f64fb1bdc3f1550987233a [id=7fd6f8576f4c4cbc8ffcdb4ab95e15dc]). Expected version 1 got 0
I20240430 20:11:45.708294 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 243) took 5279 ms. Trace:
I20240430 20:11:45.708981 19049 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 69f88532952a4591bf1577541c3430e6 (table default.Table_1177828cad1f4a24bd397619dad96170 [id=a95fed1bc3fa4865bf9a35dfc51b659c]). Expected version 2 got 0
I20240430 20:11:45.709841 19471 ts_tablet_manager.cc:1875] T 53a899fa5a424a0d99f1b47925e8c7ad P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:45.710043 19489 rpcz_store.cc:276] 0430 20:11:40.428931 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:40.429108 (+ 177us) service_pool.cc:225] Handling call
0430 20:11:45.708193 (+5279085us) ts_tablet_manager.cc:1120] Deleting tablet 26ed12f498644d15bf45e884dcad172e
0430 20:11:45.708205 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:45.708276 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5278839}
I20240430 20:11:45.712339 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 238) took 5284 ms. Trace:
W20240430 20:11:45.711957 19001 catalog_manager.cc:4797] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): delete failed for tablet 26ed12f498644d15bf45e884dcad172e because the tablet was not found. No further retry: Not found: Tablet not found: 26ed12f498644d15bf45e884dcad172e
I20240430 20:11:45.713012 19049 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 1050ab528c7d41559831652a1437d358 (table default.Table_acddc4cbf6334546b5fb3e4752fef2a5 [id=274cb15fd751410e83327ac4e8dc4629]). Expected version 2 got 0
I20240430 20:11:45.713325 19489 rpcz_store.cc:276] 0430 20:11:40.428319 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:40.429188 (+ 869us) service_pool.cc:225] Handling call
0430 20:11:45.712223 (+5283035us) ts_tablet_manager.cc:1120] Deleting tablet 802336602d8f4ac4bd7e0872d6f3dae9
0430 20:11:45.712238 (+ 15us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:45.712319 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5282800}
I20240430 20:11:45.715306 19049 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 99203cc522de424184c6f97453f166fd (table default.Table_cb2a4b9ceaf4431facfff2c9bcf94c5d [id=3f8690a437ae49a3beb009fa7899d5d0]). Expected version 1 got 0
I20240430 20:11:45.716249 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 244) took 5287 ms. Trace:
I20240430 20:11:45.717765 19489 rpcz_store.cc:276] 0430 20:11:40.429022 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:40.429396 (+ 374us) service_pool.cc:225] Handling call
0430 20:11:45.716124 (+5286728us) ts_tablet_manager.cc:1120] Deleting tablet cace1398492f4d968dcb6899385d527c
0430 20:11:45.716135 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:45.716219 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5286580}
I20240430 20:11:45.719219 19489 tablet_replica.cc:330] T 5bd0362ff41b4faca0205d136d596737 P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
W20240430 20:11:45.715886 19001 catalog_manager.cc:4797] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): delete failed for tablet 802336602d8f4ac4bd7e0872d6f3dae9 because the tablet was not found. No further retry: Not found: Tablet not found: 802336602d8f4ac4bd7e0872d6f3dae9
W20240430 20:11:45.719646 19001 catalog_manager.cc:4797] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): delete failed for tablet cace1398492f4d968dcb6899385d527c because the tablet was not found. No further retry: Not found: Tablet not found: cace1398492f4d968dcb6899385d527c
I20240430 20:11:45.720721 19049 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet dbb427aefa6b47b0805d1751fea4efcb (table default.Table_4130caf534004edfb413056d7f6cfd52 [id=da9ad991953a40c4b77b31317597ad1d]). Expected version 4 got 0
I20240430 20:11:45.721195 19489 raft_consensus.cc:2229] T 5bd0362ff41b4faca0205d136d596737 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:45.726271 20105 consensus_queue.cc:1035] T 99203cc522de424184c6f97453f166fd P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:45.731781 19489 raft_consensus.cc:2258] T 5bd0362ff41b4faca0205d136d596737 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:45.735787 19049 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet e041756c962b4a9f97c2cb314e7d8c05 (table default.Table_de88cafd92b149f9b545369bff71033c [id=c4bf4806f3d84579b16a3038d8d982c3]). Expected version 1 got 0
I20240430 20:11:45.735971 18758 raft_consensus.cc:3037] T 99203cc522de424184c6f97453f166fd P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:45.737186 19049 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 086b3dfa84c045a18c8c4f177dce606a (table default.Table_df19cb8a6f4d46699350ec5b69c14e5a [id=197923633fc54a4582207942b51530e2]). Expected version 2 got 0
I20240430 20:11:45.737915 19489 ts_tablet_manager.cc:1875] T 5bd0362ff41b4faca0205d136d596737 P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:45.740166 20538 consensus_queue.cc:1035] T 99203cc522de424184c6f97453f166fd P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:45.741856 19049 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 488d0e609b224674932f890939664df2 (table default.Table_58f35a2120874472978ca0e5f816e86c [id=ff18001c6d814afc8c1768d0c84a6bac]). Expected version 1 got 0
I20240430 20:11:45.742913 19049 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet a59ef6c545dd41a28265a4c1b8ce10e0 (table default.Table_07e40af8f1c546048975c8edf88170d7 [id=86df5d161d8641c89070e7eed9a39360]). Expected version 2 got 0
I20240430 20:11:45.744289 19049 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet fbdeb4714b8c4b93ab72822c43329aae (table default.Table_c12ca6938d424fbfa2e1df5cdb00fdbc [id=0fb93f36bc17408ea56bc4cafcfd8580]). Expected version 1 got 0
I20240430 20:11:45.747718 19049 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 772eba5f24c549c791cad75565858285 (table default.Table_5fc77d5f60ae44f4bb4ca9f7f183672c [id=6ca223ca6354449c959d85ed2ac6a9c8]). Expected version 1 got 0
I20240430 20:11:45.752020 19049 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet c6e3d09402574f1e8fe9e4160c70a3ee (table default.Table_656db97355f64d7c8aa29b1352dfe9c8 [id=450f5d1c7e084a1bb88d035c0ccfcf82]). Expected version 1 got 0
I20240430 20:11:45.753093 19471 ts_tablet_manager.cc:1888] T 53a899fa5a424a0d99f1b47925e8c7ad P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:45.753227 19471 log.cc:1196] T 53a899fa5a424a0d99f1b47925e8c7ad P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/53a899fa5a424a0d99f1b47925e8c7ad
I20240430 20:11:45.753683 19471 ts_tablet_manager.cc:1909] T 53a899fa5a424a0d99f1b47925e8c7ad P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:45.754034 19049 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 1462ecf1176c4e66be42abddbe36b00c (table default.Table_b49a7942aa8c4f58acb94c35d1cdbb70 [id=6257559b0e9545c58848a76af0c5a5f7]). Expected version 1 got 0
I20240430 20:11:45.754868 19049 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 78ba48dadef548708153878810cc46f8 (table default.Table_b9983e101a6547649eb44ee04b5e1147 [id=29769be2b9ec4ecba9de5c0a65692f85]). Expected version 1 got 0
I20240430 20:11:45.757899 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 213) took 6071 ms. Trace:
I20240430 20:11:45.758008 19471 rpcz_store.cc:276] 0430 20:11:39.686110 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:39.686174 (+ 64us) service_pool.cc:225] Handling call
0430 20:11:45.704640 (+6018466us) ts_tablet_manager.cc:1120] Deleting tablet 53a899fa5a424a0d99f1b47925e8c7ad
0430 20:11:45.704650 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:45.721043 (+ 16393us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:45.752976 (+ 31933us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:45.757873 (+ 4897us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6017439}
I20240430 20:11:45.759011 19471 tablet_replica.cc:330] T aec9c09cc0454acab6006bb3e10982b7 P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:45.759217 19471 raft_consensus.cc:2229] T aec9c09cc0454acab6006bb3e10982b7 P 5e1c6397f09747f091534faf93542710 [term 3 LEADER]: Raft consensus shutting down.
I20240430 20:11:45.760671 19471 raft_consensus.cc:2258] T aec9c09cc0454acab6006bb3e10982b7 P 5e1c6397f09747f091534faf93542710 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:45.761639 19471 ts_tablet_manager.cc:1875] T aec9c09cc0454acab6006bb3e10982b7 P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:45.759265 19049 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 47bf64ecc41b445da4ff697e141243dd (table default.Table_f1ad29d5986d4b799cfc67afb2b0dfdf [id=838fc8b61a864dea8877a6f144e4cb4f]). Expected version 2 got 0
I20240430 20:11:45.769167 19049 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 7ce09fcbe1ac457a9c05efaf000c3263 (table default.Table_edc4c27ed846454288f73f6a0d88cb69 [id=78fd3f516083489ab3b42d90dcef7e4d]). Expected version 1 got 0
I20240430 20:11:45.771070 19049 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 331c031d44854d679a27d10d9151d0ea (table default.Table_763f0800e8bb4fa69976886416250f42 [id=61c8a5b99ef34a0baeeb343fef7cfd05]). Expected version 1 got 0
I20240430 20:11:45.771417 19049 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet ebf74809b52f47f3b0ed21b6ac0f541a (table default.Table_b2d07520b9944a89ad62fca70396174e [id=1c8e0a3a44ec4a798770c0165ed3dffa]). Expected version 1 got 0
I20240430 20:11:45.771780 19049 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 0368d37dd9e2452982fb81f539c6805b (table default.Table_404b186c544e4b30bb36cb77d1a14120 [id=4330626e342a48699ea939add5c91b11]). Expected version 1 got 0
I20240430 20:11:45.772424 19049 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 2da900b6bbfa4fa99fe8494fd2e68da8 (table default.Table_19a91a7f17ff456ebb145e8dd89c0940 [id=67283dd3995a44c6a7a7bc8722e4f955]). Expected version 1 got 0
I20240430 20:11:45.773155 20105 consensus_queue.cc:1035] T d61c279d20b0471a967942525ceb4fcb P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:45.776403 19049 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 8e91920e7e36418d8c27d1f0bfd9b981 (table default.Table_e860191f401c434e8cf0085ef41952b9 [id=f393bcc9669c4ea985a2d40deb161f0c]). Expected version 1 got 0
I20240430 20:11:45.779680 19489 ts_tablet_manager.cc:1888] T 5bd0362ff41b4faca0205d136d596737 P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:45.780143 19049 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 677481aee5fa4eb4b189322d7a3f8f2e (table default.Table_1ce204e15fce4c469816cf4e7f80cac7 [id=d268c31d82df4871bc1a78aea5871b33]). Expected version 1 got 0
I20240430 20:11:45.780561 19049 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 5b26319488644d6a8b6be60460a5e7af (table default.Table_69892a8264a4477db48ee6248758e046 [id=5a1c4a7be2fc465fac89a50854eb3322]). Expected version 3 got 0
I20240430 20:11:45.781075 19029 catalog_manager.cc:6029] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: default.Table_938c56d9ae2f4ff6b3264a53b65d0977 [id=1df4e95702cd46f59ab21b101049a9ed] alter complete (version 2)
W20240430 20:11:45.783156 19029 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.17.177.1:50353 (request call id 94) took 2278 ms (client timeout 1000 ms). Trace:
W20240430 20:11:45.783306 19029 rpcz_store.cc:269] 0430 20:11:43.505187 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:43.544367 (+ 39180us) service_pool.cc:225] Handling call
0430 20:11:45.783103 (+2238736us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:43.900444 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:44.189073 (+288629us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:44.189083 (+ 10us) write_op.cc:427] Acquired schema lock
0430 20:11:44.189087 (+ 4us) tablet.cc:660] Decoding operations
0430 20:11:44.189230 (+ 143us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:44.189252 (+ 22us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:44.189260 (+ 8us) tablet.cc:683] Acquiring locks for 1 operations
0430 20:11:44.189357 (+ 97us) tablet.cc:699] Row locks acquired
0430 20:11:44.189361 (+ 4us) write_op.cc:260] PREPARE: finished
0430 20:11:44.189425 (+ 64us) write_op.cc:270] Start()
0430 20:11:44.189475 (+ 50us) write_op.cc:276] Timestamp: P: 1714507904189375 usec, L: 0
0430 20:11:44.189482 (+ 7us) op_driver.cc:331] REPLICATION: starting
0430 20:11:44.189776 (+ 294us) log.cc:842] Serialized 356 byte log entry
0430 20:11:44.228043 (+ 38267us) op_driver.cc:444] REPLICATION: finished
0430 20:11:44.228188 (+ 145us) write_op.cc:301] APPLY: starting
0430 20:11:44.228212 (+ 24us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:44.228286 (+ 74us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:44.228323 (+ 37us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:44.228399 (+ 76us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:44.228454 (+ 55us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0430 20:11:44.228467 (+ 13us) write_op.cc:312] APPLY: finished
0430 20:11:44.228568 (+ 101us) log.cc:842] Serialized 23 byte log entry
0430 20:11:44.228642 (+ 74us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:44.228667 (+ 25us) write_op.cc:446] Released schema lock
0430 20:11:44.228676 (+ 9us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:44.228743 (+ 67us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:44.228748 (+ 5us) write_op.cc:446] Released schema lock
Related trace 'op':
0430 20:11:44.611330 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:44.611478 (+ 148us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:44.611484 (+ 6us) write_op.cc:427] Acquired schema lock
0430 20:11:44.611486 (+ 2us) tablet.cc:660] Decoding operations
0430 20:11:44.611853 (+ 367us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:44.611868 (+ 15us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:44.611872 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0430 20:11:44.611944 (+ 72us) tablet.cc:699] Row locks acquired
0430 20:11:44.611947 (+ 3us) write_op.cc:260] PREPARE: finished
0430 20:11:44.611981 (+ 34us) write_op.cc:270] Start()
0430 20:11:44.612015 (+ 34us) write_op.cc:276] Timestamp: P: 1714507904611948 usec, L: 0
0430 20:11:44.612018 (+ 3us) op_driver.cc:331] REPLICATION: starting
0430 20:11:44.612263 (+ 245us) log.cc:842] Serialized 356 byte log entry
0430 20:11:44.618297 (+ 6034us) op_driver.cc:444] REPLICATION: finished
0430 20:11:44.618718 (+ 421us) write_op.cc:301] APPLY: starting
0430 20:11:44.618735 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:44.618781 (+ 46us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:44.618807 (+ 26us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:44.618864 (+ 57us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:44.618902 (+ 38us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0430 20:11:44.618909 (+ 7us) write_op.cc:312] APPLY: finished
0430 20:11:44.618993 (+ 84us) log.cc:842] Serialized 23 byte log entry
0430 20:11:44.619059 (+ 66us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:44.619082 (+ 23us) write_op.cc:446] Released schema lock
0430 20:11:44.619090 (+ 8us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:44.619159 (+ 69us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:44.619163 (+ 4us) write_op.cc:446] Released schema lock
Metrics: {"mutex_wait_us":182,"reported_tablets":10,"child_traces":[["op",{"apply.queue_time_us":82,"apply.run_cpu_time_us":619,"apply.run_wall_time_us":618,"num_ops":1,"prepare.queue_time_us":122929,"prepare.run_cpu_time_us":1100,"prepare.run_wall_time_us":311274,"raft.queue_time_us":1327,"raft.run_cpu_time_us":324,"raft.run_wall_time_us":5519,"replication_time_us":38523}],["op",{"apply.queue_time_us":349,"apply.run_cpu_time_us":497,"apply.run_wall_time_us":509,"num_ops":1,"prepare.queue_time_us":161386,"prepare.run_cpu_time_us":891,"prepare.run_wall_time_us":26337,"raft.queue_time_us":1631,"raft.run_cpu_time_us":254,"raft.run_wall_time_us":254,"replication_time_us":6254}]]}
I20240430 20:11:45.790603 18445 tablet_service.cc:1514] Processing DeleteTablet for tablet 32c823d1700949ab8b22adcd83b7bfd1 with delete_type TABLET_DATA_DELETED (replaced by tablet 024069ab786749028c020abd1c836cf5) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:45.790689 18461 tablet_service.cc:1514] Processing DeleteTablet for tablet ae87f5e92a934e2182164c231bc76df0 with delete_type TABLET_DATA_DELETED (replaced by tablet 4fa9575d96a84bef876467e21dada803) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:45.790954 18446 tablet_service.cc:1514] Processing DeleteTablet for tablet dd6083f12c7e4d1e8a2006b3ddf9aa3a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:43 UTC) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:45.791103 18457 tablet_service.cc:1514] Processing DeleteTablet for tablet 115c4a12440647a4bb46eb6824f7ea98 with delete_type TABLET_DATA_DELETED (Sending initial creation of tablet) from {username='slave'} at 127.0.0.1:41680
W20240430 20:11:45.791934 19026 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.17.177.2:41749 (request call id 110) took 2030 ms (client timeout 1000 ms). Trace:
W20240430 20:11:45.792053 19026 rpcz_store.cc:269] 0430 20:11:43.761086 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:43.881718 (+120632us) service_pool.cc:225] Handling call
0430 20:11:43.884375 (+ 2657us) spinlock_profiling.cc:243] Waited 1.26 ms on lock 0x616000040654. stack: 00007feb16c74756 00007feb16c74510 00007feb15ca748d 0000000000955804 00000000009557cd 00007feb257b315d 00007feb257b3253 00007feb224e5d41 00007feb2a0d134d 00007feb2a258668 00007feb21043786 00007feb210434b5 00007feb1d97c04e 00007feb1d97af2e 00007feb1d981405 00007feb1d981d2f
0430 20:11:45.791870 (+1907495us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:44.586708 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:44.586829 (+ 121us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:44.586835 (+ 6us) write_op.cc:427] Acquired schema lock
0430 20:11:44.586838 (+ 3us) tablet.cc:660] Decoding operations
0430 20:11:44.586946 (+ 108us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:44.586960 (+ 14us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:44.586965 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0430 20:11:44.587028 (+ 63us) tablet.cc:699] Row locks acquired
0430 20:11:44.587030 (+ 2us) write_op.cc:260] PREPARE: finished
0430 20:11:44.587066 (+ 36us) write_op.cc:270] Start()
0430 20:11:44.587100 (+ 34us) write_op.cc:276] Timestamp: P: 1714507904587035 usec, L: 0
0430 20:11:44.587104 (+ 4us) op_driver.cc:331] REPLICATION: starting
0430 20:11:44.587290 (+ 186us) log.cc:842] Serialized 530 byte log entry
0430 20:11:44.602150 (+ 14860us) op_driver.cc:444] REPLICATION: finished
0430 20:11:44.603298 (+ 1148us) write_op.cc:301] APPLY: starting
0430 20:11:44.603315 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:44.603367 (+ 52us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:44.603409 (+ 42us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:44.603466 (+ 57us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:44.603502 (+ 36us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0430 20:11:44.603510 (+ 8us) write_op.cc:312] APPLY: finished
0430 20:11:44.603606 (+ 96us) log.cc:842] Serialized 23 byte log entry
0430 20:11:44.603679 (+ 73us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:44.603701 (+ 22us) write_op.cc:446] Released schema lock
0430 20:11:44.603708 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:44.603765 (+ 57us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:44.603769 (+ 4us) write_op.cc:446] Released schema lock
Metrics: {"reported_tablets":16,"spinlock_wait_cycles":2906368,"child_traces":[["op",{"apply.queue_time_us":1085,"apply.run_cpu_time_us":525,"apply.run_wall_time_us":523,"num_ops":1,"prepare.queue_time_us":138276,"prepare.run_cpu_time_us":806,"prepare.run_wall_time_us":24682,"raft.queue_time_us":97,"raft.run_cpu_time_us":268,"raft.run_wall_time_us":268,"replication_time_us":15020}]]}
I20240430 20:11:45.793396 19026 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_cfb6af12a3114f96837868eaa33be09b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240430 20:11:45.794641 18590 tablet_service.cc:1514] Processing DeleteTablet for tablet 50486644ec234933b9a0b45ee39a6685 with delete_type TABLET_DATA_DELETED (replaced by tablet ed0c37a71ede4bfc9eb5ad1a4365f25e) from {username='slave'} at 127.0.0.1:40218
W20240430 20:11:45.795536 19026 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cfb6af12a3114f96837868eaa33be09b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:45.806389 19489 log.cc:1196] T 5bd0362ff41b4faca0205d136d596737 P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/5bd0362ff41b4faca0205d136d596737
I20240430 20:11:45.807483 19028 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: Sending DeleteTablet for 3 replicas of tablet c6378f1495144c458af49b3c19a2b67c
I20240430 20:11:45.808099 19028 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:51664 (request call id 388) took 1035 ms. Trace:
I20240430 20:11:45.808207 19028 rpcz_store.cc:276] 0430 20:11:44.772583 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:45.446244 (+673661us) service_pool.cc:225] Handling call
0430 20:11:45.446822 (+ 578us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0430 20:11:45.446985 (+ 163us) catalog_manager.cc:2749] Modifying in-memory table state
0430 20:11:45.447043 (+ 58us) catalog_manager.cc:2758] Locking tablets
0430 20:11:45.447185 (+ 142us) catalog_manager.cc:2772] Removing table and tablets from system table
0430 20:11:45.485917 (+ 38732us) catalog_manager.cc:2790] Removing table from by-name map
0430 20:11:45.486016 (+ 99us) catalog_manager.cc:2806] Committing in-memory state
0430 20:11:45.807423 (+321407us) catalog_manager.cc:2810] Aborting table tasks
0430 20:11:45.808039 (+ 616us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:45.447939 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:45.448069 (+ 130us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:45.448074 (+ 5us) write_op.cc:427] Acquired schema lock
0430 20:11:45.448077 (+ 3us) tablet.cc:660] Decoding operations
0430 20:11:45.448198 (+ 121us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:45.448212 (+ 14us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:45.448216 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
0430 20:11:45.448331 (+ 115us) tablet.cc:699] Row locks acquired
0430 20:11:45.448333 (+ 2us) write_op.cc:260] PREPARE: finished
0430 20:11:45.448368 (+ 35us) write_op.cc:270] Start()
0430 20:11:45.448398 (+ 30us) write_op.cc:276] Timestamp: P: 1714507905448349 usec, L: 0
0430 20:11:45.448401 (+ 3us) op_driver.cc:331] REPLICATION: starting
0430 20:11:45.448585 (+ 184us) log.cc:842] Serialized 728 byte log entry
0430 20:11:45.467414 (+ 18829us) op_driver.cc:444] REPLICATION: finished
0430 20:11:45.477784 (+ 10370us) write_op.cc:301] APPLY: starting
0430 20:11:45.477809 (+ 25us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:45.477866 (+ 57us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:45.477921 (+ 55us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:45.478004 (+ 83us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:45.478045 (+ 41us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0430 20:11:45.478054 (+ 9us) write_op.cc:312] APPLY: finished
0430 20:11:45.478148 (+ 94us) log.cc:842] Serialized 29 byte log entry
0430 20:11:45.478453 (+ 305us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:45.478479 (+ 26us) write_op.cc:446] Released schema lock
0430 20:11:45.478487 (+ 8us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:45.478544 (+ 57us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:45.478547 (+ 3us) write_op.cc:446] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":124,"apply.run_cpu_time_us":674,"apply.run_wall_time_us":17110,"num_ops":2,"prepare.queue_time_us":54,"prepare.run_cpu_time_us":721,"prepare.run_wall_time_us":721,"replication_time_us":18987}]]}
I20240430 20:11:45.809008 19028 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_74b89f328a354c9a937566cdaf2a3377" } modify_external_catalogs: true
I20240430 20:11:45.809203 19028 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_74b89f328a354c9a937566cdaf2a3377" } modify_external_catalogs: true
I20240430 20:11:45.810796 18457 tablet_service.cc:1514] Processing DeleteTablet for tablet c6378f1495144c458af49b3c19a2b67c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:45 UTC) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:45.811151 19026 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_640bb0fa95ca44d49921276d665b9ca3" } new_table_name: "default.Table_2ec9e316bc3347b189a13b7a8bbcf829" modify_external_catalogs: true
I20240430 20:11:45.812780 18603 tablet_service.cc:1514] Processing DeleteTablet for tablet c6378f1495144c458af49b3c19a2b67c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:45 UTC) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:45.814716 20631 consensus_queue.cc:1035] T 2da900b6bbfa4fa99fe8494fd2e68da8 P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:45.824208 19023 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: Sending DeleteTablet for 3 replicas of tablet df92eb85261d47d88cbf16d29d9ad30e
W20240430 20:11:45.824363 19051 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6b51cbc731794083a88fe4bed1ddeebf consensus state has no leader
I20240430 20:11:45.824599 19051 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:51664 (request call id 382) took 1172 ms. Trace:
I20240430 20:11:45.824571 20629 consensus_queue.cc:1035] T 2e79df8744e74b5991a79ce978ff0202 P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:45.824975 19023 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:51664 (request call id 387) took 1080 ms. Trace:
I20240430 20:11:45.825510 18457 tablet_service.cc:1514] Processing DeleteTablet for tablet df92eb85261d47d88cbf16d29d9ad30e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:45 UTC) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:45.825117 19023 rpcz_store.cc:276] 0430 20:11:44.744912 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:45.442379 (+697467us) service_pool.cc:225] Handling call
0430 20:11:45.443176 (+ 797us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0430 20:11:45.443338 (+ 162us) catalog_manager.cc:2749] Modifying in-memory table state
0430 20:11:45.443409 (+ 71us) catalog_manager.cc:2758] Locking tablets
0430 20:11:45.443563 (+ 154us) catalog_manager.cc:2772] Removing table and tablets from system table
0430 20:11:45.464642 (+ 21079us) catalog_manager.cc:2790] Removing table from by-name map
0430 20:11:45.464747 (+ 105us) catalog_manager.cc:2806] Committing in-memory state
0430 20:11:45.824118 (+359371us) catalog_manager.cc:2810] Aborting table tasks
0430 20:11:45.824892 (+ 774us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:45.444416 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:45.444584 (+ 168us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:45.444592 (+ 8us) write_op.cc:427] Acquired schema lock
0430 20:11:45.444597 (+ 5us) tablet.cc:660] Decoding operations
0430 20:11:45.444746 (+ 149us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:45.444760 (+ 14us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:45.444768 (+ 8us) tablet.cc:683] Acquiring locks for 2 operations
0430 20:11:45.444951 (+ 183us) tablet.cc:699] Row locks acquired
0430 20:11:45.444955 (+ 4us) write_op.cc:260] PREPARE: finished
0430 20:11:45.445001 (+ 46us) write_op.cc:270] Start()
0430 20:11:45.445040 (+ 39us) write_op.cc:276] Timestamp: P: 1714507905444980 usec, L: 0
0430 20:11:45.445046 (+ 6us) op_driver.cc:331] REPLICATION: starting
0430 20:11:45.445234 (+ 188us) log.cc:842] Serialized 728 byte log entry
0430 20:11:45.451645 (+ 6411us) op_driver.cc:444] REPLICATION: finished
0430 20:11:45.462961 (+ 11316us) write_op.cc:301] APPLY: starting
0430 20:11:45.462984 (+ 23us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:45.463039 (+ 55us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:45.463067 (+ 28us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:45.463145 (+ 78us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:45.463188 (+ 43us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0430 20:11:45.463198 (+ 10us) write_op.cc:312] APPLY: finished
0430 20:11:45.463290 (+ 92us) log.cc:842] Serialized 29 byte log entry
0430 20:11:45.463695 (+ 405us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:45.463719 (+ 24us) write_op.cc:446] Released schema lock
0430 20:11:45.463727 (+ 8us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:45.463788 (+ 61us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:45.463791 (+ 3us) write_op.cc:446] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":11236,"apply.run_cpu_time_us":575,"apply.run_wall_time_us":900,"num_ops":2,"prepare.queue_time_us":94,"prepare.run_cpu_time_us":911,"prepare.run_wall_time_us":912,"replication_time_us":6575}]]}
I20240430 20:11:45.825999 18985 tablet_bootstrap.cc:492] T f6cfd089f08b47409fb2a6cf4f6b0385 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:45.826282 18985 ts_tablet_manager.cc:1367] T f6cfd089f08b47409fb2a6cf4f6b0385 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.216s user 0.013s sys 0.017s
I20240430 20:11:45.826198 19023 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_67d51131549440cc99629dda924693f7" } modify_external_catalogs: true
I20240430 20:11:45.824693 19051 rpcz_store.cc:276] 0430 20:11:44.651638 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:45.032561 (+380923us) service_pool.cc:225] Handling call
0430 20:11:45.055043 (+ 22482us) spinlock_profiling.cc:243] Waited 22.2 ms on lock 0x616000040654. stack: 00007feb16c74756 00007feb16c74510 00007feb15ca748d 0000000000955804 00000000009557cd 00007feb257b315d 00007feb257b3253 00007feb224e5d41 00007feb2a0d134d 00007feb2a25ff46 00007feb21049c26 00007feb21049955 00007feb1d97c04e 00007feb1d97af2e 00007feb1d981405 00007feb1d981d2f
0430 20:11:45.055581 (+ 538us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0430 20:11:45.055948 (+ 367us) catalog_manager.cc:3613] Apply alter schema
0430 20:11:45.056167 (+ 219us) catalog_manager.cc:3636] Acquired catalog manager lock
0430 20:11:45.056286 (+ 119us) catalog_manager.cc:3783] Updating metadata on disk
0430 20:11:45.240213 (+183927us) catalog_manager.cc:3814] Committing alterations to in-memory state
0430 20:11:45.824518 (+584305us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:45.060529 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:45.060598 (+ 69us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:45.060603 (+ 5us) write_op.cc:427] Acquired schema lock
0430 20:11:45.060606 (+ 3us) tablet.cc:660] Decoding operations
0430 20:11:45.060672 (+ 66us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:45.060679 (+ 7us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:45.060684 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0430 20:11:45.060723 (+ 39us) tablet.cc:699] Row locks acquired
0430 20:11:45.060725 (+ 2us) write_op.cc:260] PREPARE: finished
0430 20:11:45.060746 (+ 21us) write_op.cc:270] Start()
0430 20:11:45.060765 (+ 19us) write_op.cc:276] Timestamp: P: 1714507905060724 usec, L: 0
0430 20:11:45.060768 (+ 3us) op_driver.cc:331] REPLICATION: starting
0430 20:11:45.060894 (+ 126us) log.cc:842] Serialized 405 byte log entry
0430 20:11:45.122026 (+ 61132us) op_driver.cc:444] REPLICATION: finished
0430 20:11:45.239064 (+117038us) write_op.cc:301] APPLY: starting
0430 20:11:45.239097 (+ 33us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:45.239165 (+ 68us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:45.239255 (+ 90us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:45.239352 (+ 97us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:45.239433 (+ 81us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0430 20:11:45.239446 (+ 13us) write_op.cc:312] APPLY: finished
0430 20:11:45.239621 (+ 175us) log.cc:842] Serialized 23 byte log entry
0430 20:11:45.240046 (+ 425us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:45.240075 (+ 29us) write_op.cc:446] Released schema lock
0430 20:11:45.240085 (+ 10us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:45.240338 (+ 253us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:45.240345 (+ 7us) write_op.cc:446] Released schema lock
Metrics: {"spinlock_wait_cycles":50956672,"child_traces":[["op",{"apply.queue_time_us":116879,"apply.run_cpu_time_us":917,"apply.run_wall_time_us":1386,"num_ops":1,"prepare.queue_time_us":3569,"prepare.run_cpu_time_us":420,"prepare.run_wall_time_us":419,"replication_time_us":61217}]]}
I20240430 20:11:45.826442 19023 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_67d51131549440cc99629dda924693f7" } modify_external_catalogs: true
I20240430 20:11:45.826771 19051 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_0c5d4686f2ed40fb8d62c31b0166dc05" } new_table_name: "default.Table_19df33614a6c4f94b6419a5b441e3ef8" modify_external_catalogs: true
I20240430 20:11:45.827167 18985 raft_consensus.cc:348] T f6cfd089f08b47409fb2a6cf4f6b0385 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:45.827394 18985 raft_consensus.cc:374] T f6cfd089f08b47409fb2a6cf4f6b0385 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:45.827494 18985 raft_consensus.cc:729] T f6cfd089f08b47409fb2a6cf4f6b0385 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:45.827742 18985 consensus_queue.cc:260] T f6cfd089f08b47409fb2a6cf4f6b0385 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:45.829211 18603 tablet_service.cc:1514] Processing DeleteTablet for tablet df92eb85261d47d88cbf16d29d9ad30e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:45 UTC) from {username='slave'} at 127.0.0.1:40218
W20240430 20:11:45.830367 19033 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f1073fc792954c5795a05aa7d0ec24df consensus state has no leader
I20240430 20:11:45.830909 19489 ts_tablet_manager.cc:1909] T 5bd0362ff41b4faca0205d136d596737 P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:45.830554 19033 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:51664 (request call id 379) took 1208 ms. Trace:
I20240430 20:11:45.801851 18988 tablet_bootstrap.cc:492] T a9f37f0f7ce8425f9b24699513d41d61 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:45.831570 18988 ts_tablet_manager.cc:1367] T a9f37f0f7ce8425f9b24699513d41d61 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.305s user 0.062s sys 0.073s
I20240430 20:11:45.796550 18810 heartbeater.cc:347] Connected to a master server at 127.17.177.60:37225
I20240430 20:11:45.831415 19033 rpcz_store.cc:276] 0430 20:11:44.621945 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:44.943242 (+321297us) service_pool.cc:225] Handling call
0430 20:11:44.943821 (+ 579us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0430 20:11:44.944107 (+ 286us) catalog_manager.cc:3613] Apply alter schema
0430 20:11:44.944352 (+ 245us) catalog_manager.cc:3636] Acquired catalog manager lock
0430 20:11:44.944467 (+ 115us) catalog_manager.cc:3783] Updating metadata on disk
0430 20:11:45.002555 (+ 58088us) catalog_manager.cc:3814] Committing alterations to in-memory state
0430 20:11:45.830501 (+827946us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:44.993558 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:44.993700 (+ 142us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:44.993707 (+ 7us) write_op.cc:427] Acquired schema lock
0430 20:11:44.993709 (+ 2us) tablet.cc:660] Decoding operations
0430 20:11:44.993865 (+ 156us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:44.993886 (+ 21us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:44.993894 (+ 8us) tablet.cc:683] Acquiring locks for 1 operations
0430 20:11:44.994011 (+ 117us) tablet.cc:699] Row locks acquired
0430 20:11:44.994015 (+ 4us) write_op.cc:260] PREPARE: finished
0430 20:11:44.994071 (+ 56us) write_op.cc:270] Start()
0430 20:11:44.994117 (+ 46us) write_op.cc:276] Timestamp: P: 1714507904994047 usec, L: 0
0430 20:11:44.994123 (+ 6us) op_driver.cc:331] REPLICATION: starting
0430 20:11:44.994379 (+ 256us) log.cc:842] Serialized 405 byte log entry
0430 20:11:44.998915 (+ 4536us) op_driver.cc:444] REPLICATION: finished
0430 20:11:44.999173 (+ 258us) write_op.cc:301] APPLY: starting
0430 20:11:44.999193 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:44.999248 (+ 55us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:44.999288 (+ 40us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:44.999353 (+ 65us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:44.999404 (+ 51us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0430 20:11:44.999417 (+ 13us) write_op.cc:312] APPLY: finished
0430 20:11:44.999516 (+ 99us) log.cc:842] Serialized 23 byte log entry
0430 20:11:44.999587 (+ 71us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:44.999615 (+ 28us) write_op.cc:446] Released schema lock
0430 20:11:44.999624 (+ 9us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:44.999694 (+ 70us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:44.999700 (+ 6us) write_op.cc:446] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":197,"apply.run_cpu_time_us":588,"apply.run_wall_time_us":587,"num_ops":1,"prepare.queue_time_us":48521,"prepare.run_cpu_time_us":1101,"prepare.run_wall_time_us":17361,"raft.queue_time_us":156,"raft.run_cpu_time_us":391,"raft.run_wall_time_us":391,"replication_time_us":4755}]]}
I20240430 20:11:45.831907 18810 heartbeater.cc:503] Master 127.17.177.60:37225 was elected leader, sending a full tablet report...
I20240430 20:11:45.832235 19033 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_4130caf534004edfb413056d7f6cfd52" } modify_external_catalogs: true
I20240430 20:11:45.832417 19033 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_4130caf534004edfb413056d7f6cfd52" } modify_external_catalogs: true
I20240430 20:11:45.832399 18988 raft_consensus.cc:348] T a9f37f0f7ce8425f9b24699513d41d61 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:45.832599 18988 raft_consensus.cc:374] T a9f37f0f7ce8425f9b24699513d41d61 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:45.832661 18988 raft_consensus.cc:729] T a9f37f0f7ce8425f9b24699513d41d61 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:45.810197 18732 tablet_service.cc:1514] Processing DeleteTablet for tablet c6378f1495144c458af49b3c19a2b67c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:45 UTC) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:45.826753 18740 tablet_service.cc:1514] Processing DeleteTablet for tablet df92eb85261d47d88cbf16d29d9ad30e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:45 UTC) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:45.832926 18988 consensus_queue.cc:260] T a9f37f0f7ce8425f9b24699513d41d61 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:45.833489 18988 ts_tablet_manager.cc:1398] T a9f37f0f7ce8425f9b24699513d41d61 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240430 20:11:45.833779 18988 tablet_bootstrap.cc:492] T 06d28b6f48a34b17b0d7fb3102df2f4c P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:45.846570 18988 tablet_bootstrap.cc:654] T 06d28b6f48a34b17b0d7fb3102df2f4c P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:45.850904 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 245) took 5421 ms. Trace:
I20240430 20:11:45.851043 19489 rpcz_store.cc:276] 0430 20:11:40.429136 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:40.429453 (+ 317us) service_pool.cc:225] Handling call
0430 20:11:45.719165 (+5289712us) ts_tablet_manager.cc:1120] Deleting tablet 5bd0362ff41b4faca0205d136d596737
0430 20:11:45.719177 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:45.748897 (+ 29720us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:45.779519 (+ 30622us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:45.850876 (+ 71357us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5289538}
I20240430 20:11:45.851495 19001 catalog_manager.cc:4830] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): tablet 5bd0362ff41b4faca0205d136d596737 (table default.Table_fe1773cf77654e2a86e95444437b7201 [id=fe2222c2b62f4360a5ade622ae8b23e5]) successfully deleted
I20240430 20:11:45.854282 19023 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: Sending DeleteTablet for 3 replicas of tablet 47e98935eb21449abd3d4d24327778d5
I20240430 20:11:45.855194 19023 master_service.cc:946] ReplaceTablet: received request to replace tablet f7b0132c17d44a9ba17288e0321a7f11 from {username='slave'} at 127.0.0.1:51664
I20240430 20:11:45.855973 18983 tablet_bootstrap.cc:492] T fbdeb4714b8c4b93ab72822c43329aae P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:45.856186 18983 ts_tablet_manager.cc:1367] T fbdeb4714b8c4b93ab72822c43329aae P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.358s user 0.002s sys 0.007s
I20240430 20:11:45.856758 19028 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: Sending DeleteTablet for 3 replicas of tablet fcc16db4d6be42edb7bd1ce18fe0190d
I20240430 20:11:45.857127 18983 raft_consensus.cc:348] T fbdeb4714b8c4b93ab72822c43329aae P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:45.857374 18983 raft_consensus.cc:374] T fbdeb4714b8c4b93ab72822c43329aae P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:45.857430 18983 raft_consensus.cc:729] T fbdeb4714b8c4b93ab72822c43329aae P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:45.857465 19028 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_ae525cb900474e458017ab35d825fc1c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240430 20:11:45.857507 19023 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_dfbb6c73df814ce49886f7b8f0e0968d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240430 20:11:45.857656 18983 consensus_queue.cc:260] T fbdeb4714b8c4b93ab72822c43329aae P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
W20240430 20:11:45.858090 19023 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dfbb6c73df814ce49886f7b8f0e0968d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:45.859256 18457 tablet_service.cc:1514] Processing DeleteTablet for tablet 47e98935eb21449abd3d4d24327778d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:45 UTC) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:45.859464 18446 tablet_service.cc:1514] Processing DeleteTablet for tablet fcc16db4d6be42edb7bd1ce18fe0190d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:45 UTC) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:45.859629 19033 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: Sending DeleteTablet for 3 replicas of tablet dbb427aefa6b47b0805d1751fea4efcb
I20240430 20:11:45.860390 18732 tablet_service.cc:1514] Processing DeleteTablet for tablet fcc16db4d6be42edb7bd1ce18fe0190d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:45 UTC) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:45.860500 18740 tablet_service.cc:1514] Processing DeleteTablet for tablet 47e98935eb21449abd3d4d24327778d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:45 UTC) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:45.860364 19033 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51664:
name: "default.Table_8db024fb9ef241ed86cb98d1c3939607"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240430 20:11:45.860663 18739 tablet_service.cc:1514] Processing DeleteTablet for tablet dbb427aefa6b47b0805d1751fea4efcb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:45 UTC) from {username='slave'} at 127.0.0.1:59830
W20240430 20:11:45.860970 19033 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8db024fb9ef241ed86cb98d1c3939607 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:45.861493 18446 tablet_service.cc:1514] Processing DeleteTablet for tablet dbb427aefa6b47b0805d1751fea4efcb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:45 UTC) from {username='slave'} at 127.0.0.1:41680
I20240430 20:11:45.862540 18603 tablet_service.cc:1514] Processing DeleteTablet for tablet 47e98935eb21449abd3d4d24327778d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:45 UTC) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:45.863036 18603 tablet_service.cc:1514] Processing DeleteTablet for tablet fcc16db4d6be42edb7bd1ce18fe0190d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:45 UTC) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:45.863724 18603 tablet_service.cc:1514] Processing DeleteTablet for tablet dbb427aefa6b47b0805d1751fea4efcb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:45 UTC) from {username='slave'} at 127.0.0.1:40218
I20240430 20:11:45.864735 18988 tablet_bootstrap.cc:492] T 06d28b6f48a34b17b0d7fb3102df2f4c P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:45.864960 18988 ts_tablet_manager.cc:1367] T 06d28b6f48a34b17b0d7fb3102df2f4c P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.031s user 0.005s sys 0.005s
I20240430 20:11:45.865792 18988 raft_consensus.cc:348] T 06d28b6f48a34b17b0d7fb3102df2f4c P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:45.866047 18988 raft_consensus.cc:374] T 06d28b6f48a34b17b0d7fb3102df2f4c P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:45.866132 18988 raft_consensus.cc:729] T 06d28b6f48a34b17b0d7fb3102df2f4c P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:45.866397 18988 consensus_queue.cc:260] T 06d28b6f48a34b17b0d7fb3102df2f4c P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
W20240430 20:11:45.858096 19028 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ae525cb900474e458017ab35d825fc1c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:45.873917 18988 ts_tablet_manager.cc:1398] T 06d28b6f48a34b17b0d7fb3102df2f4c P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
W20240430 20:11:45.874337 19001 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet f1073fc792954c5795a05aa7d0ec24df (table default.Table_ba8b218e936b4bfcb073f8182f474cca [id=aedf11fda8914a3b843a57f8a7b69a53], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet f1073fc792954c5795a05aa7d0ec24df consensus state has no leader
I20240430 20:11:45.878238 18739 tablet_service.cc:1514] Processing DeleteTablet for tablet 84568f90b9c54cf292099808d4cbf5dc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:34 UTC) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:45.879364 18740 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f4e860484784aa9a2d28af2e9fb611c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:39 UTC) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:45.880313 18739 tablet_service.cc:1514] Processing DeleteTablet for tablet 6db55bb510b24b0cae897936211e15f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:34 UTC) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:45.880983 18739 tablet_service.cc:1514] Processing DeleteTablet for tablet 558e71ec370d46ca8fbdcec5b57eb2ec with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:37 UTC) from {username='slave'} at 127.0.0.1:59830
W20240430 20:11:45.881884 19026 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 89b774967624498fa2be499878b71124 consensus state has no leader
I20240430 20:11:45.882059 18739 tablet_service.cc:1514] Processing DeleteTablet for tablet 90a694bd5db649afa50f3c4e752939c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:39 UTC) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:45.883486 18740 tablet_service.cc:1514] Processing DeleteTablet for tablet 04b3dff86d5c48a09d03da58bd48f360 with delete_type TABLET_DATA_DELETED (replaced by tablet 89b774967624498fa2be499878b71124) from {username='slave'} at 127.0.0.1:59830
I20240430 20:11:45.883623 18983 ts_tablet_manager.cc:1398] T fbdeb4714b8c4b93ab72822c43329aae P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.027s user 0.001s sys 0.001s
I20240430 20:11:45.887555 18983 tablet_bootstrap.cc:492] T c6378f1495144c458af49b3c19a2b67c P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:45.889096 19033 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51664:
table { table_name: "default.Table_c1669746f9114bf7b37a6f6a68fed20e" } new_table_name: "default.Table_0b4ca4cd0af148399dfa21f2d32114e3" modify_external_catalogs: true
I20240430 20:11:45.890231 19135 generic_service.cc:196] Checking for leaks (request via RPC)
I20240430 20:11:45.890908 18988 tablet_bootstrap.cc:492] T ae20f44df50343289b0a512a4ea53692 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:45.894520 18988 tablet_bootstrap.cc:654] T ae20f44df50343289b0a512a4ea53692 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:45.895069 18983 tablet_bootstrap.cc:654] T c6378f1495144c458af49b3c19a2b67c P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:45.904542 18988 tablet_bootstrap.cc:492] T ae20f44df50343289b0a512a4ea53692 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:45.905455 18988 ts_tablet_manager.cc:1367] T ae20f44df50343289b0a512a4ea53692 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.003s
I20240430 20:11:45.906332 18988 raft_consensus.cc:348] T ae20f44df50343289b0a512a4ea53692 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
W20240430 20:11:45.906701 18603 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40218: Illegal state: replica 5e1c6397f09747f091534faf93542710 is not leader of this config: current role FOLLOWER
I20240430 20:11:45.907363 18988 raft_consensus.cc:374] T ae20f44df50343289b0a512a4ea53692 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:45.907655 18988 raft_consensus.cc:729] T ae20f44df50343289b0a512a4ea53692 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:45.908102 18988 consensus_queue.cc:260] T ae20f44df50343289b0a512a4ea53692 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:45.913575 18985 ts_tablet_manager.cc:1398] T f6cfd089f08b47409fb2a6cf4f6b0385 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.087s user 0.002s sys 0.000s
I20240430 20:11:45.914005 18985 tablet_bootstrap.cc:492] T 66dd767bd3f8494f94061fb89e5e5680 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:45.916869 18985 tablet_bootstrap.cc:654] T 66dd767bd3f8494f94061fb89e5e5680 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:45.920073 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 239) took 5491 ms. Trace:
I20240430 20:11:45.920197 19489 rpcz_store.cc:276] 0430 20:11:40.428502 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:40.430651 (+ 2149us) service_pool.cc:225] Handling call
0430 20:11:45.919949 (+5489298us) ts_tablet_manager.cc:1120] Deleting tablet dd9b893d48684cfea7c242b93ecba01f
0430 20:11:45.919961 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:45.920048 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5489063}
I20240430 20:11:45.924041 18756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2873af6da928463f9809864b8eaf0c4a" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:45.925958 18983 tablet_bootstrap.cc:492] T c6378f1495144c458af49b3c19a2b67c P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:45.926260 18983 ts_tablet_manager.cc:1367] T c6378f1495144c458af49b3c19a2b67c P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.039s user 0.003s sys 0.000s
I20240430 20:11:45.926620 19471 ts_tablet_manager.cc:1888] T aec9c09cc0454acab6006bb3e10982b7 P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 3.4
I20240430 20:11:45.927193 19471 log.cc:1196] T aec9c09cc0454acab6006bb3e10982b7 P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/aec9c09cc0454acab6006bb3e10982b7
I20240430 20:11:45.928221 19471 ts_tablet_manager.cc:1909] T aec9c09cc0454acab6006bb3e10982b7 P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:45.928745 18988 ts_tablet_manager.cc:1398] T ae20f44df50343289b0a512a4ea53692 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.023s user 0.001s sys 0.001s
I20240430 20:11:45.929157 18988 tablet_bootstrap.cc:492] T 06e82c706b9c4f35bf53faddf57b3fd7 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:45.932327 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 214) took 6185 ms. Trace:
I20240430 20:11:45.932600 19471 rpcz_store.cc:276] 0430 20:11:39.746820 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:39.752297 (+ 5477us) service_pool.cc:225] Handling call
0430 20:11:45.758948 (+6006651us) ts_tablet_manager.cc:1120] Deleting tablet aec9c09cc0454acab6006bb3e10982b7
0430 20:11:45.758962 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:45.793729 (+ 34767us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:45.926461 (+132732us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:45.932302 (+ 5841us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6006406}
I20240430 20:11:45.933228 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 215) took 6176 ms. Trace:
I20240430 20:11:45.933355 19471 rpcz_store.cc:276] 0430 20:11:39.757301 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:39.757509 (+ 208us) service_pool.cc:225] Handling call
0430 20:11:45.933100 (+6175591us) ts_tablet_manager.cc:1120] Deleting tablet 04b3dff86d5c48a09d03da58bd48f360
0430 20:11:45.933109 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:45.933213 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6175386}
I20240430 20:11:45.934764 19471 tablet_replica.cc:330] T 294644d7a16b4ca2bbb6d106bcb8c7ff P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:45.934819 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 240) took 5506 ms. Trace:
I20240430 20:11:45.934911 19489 rpcz_store.cc:276] 0430 20:11:40.428631 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:40.430913 (+ 2282us) service_pool.cc:225] Handling call
0430 20:11:45.934715 (+5503802us) ts_tablet_manager.cc:1120] Deleting tablet 34ab40d8d91e401db299ddee7175ba28
0430 20:11:45.934724 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:45.934799 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5503601}
I20240430 20:11:45.935207 19471 raft_consensus.cc:2229] T 294644d7a16b4ca2bbb6d106bcb8c7ff P 5e1c6397f09747f091534faf93542710 [term 2 LEADER]: Raft consensus shutting down.
I20240430 20:11:45.935539 18756 raft_consensus.cc:3037] T ae20f44df50343289b0a512a4ea53692 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:45.936010 19471 raft_consensus.cc:2258] T 294644d7a16b4ca2bbb6d106bcb8c7ff P 5e1c6397f09747f091534faf93542710 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:45.936182 18988 tablet_bootstrap.cc:654] T 06e82c706b9c4f35bf53faddf57b3fd7 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:45.936632 19471 ts_tablet_manager.cc:1875] T 294644d7a16b4ca2bbb6d106bcb8c7ff P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:45.937606 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 241) took 5508 ms. Trace:
I20240430 20:11:45.937700 19489 rpcz_store.cc:276] 0430 20:11:40.428736 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:40.431144 (+ 2408us) service_pool.cc:225] Handling call
0430 20:11:45.937509 (+5506365us) ts_tablet_manager.cc:1120] Deleting tablet 5bde72e2791a497db7dd08686ca36d80
0430 20:11:45.937519 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:45.937589 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5506150}
I20240430 20:11:45.938759 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 242) took 5509 ms. Trace:
I20240430 20:11:45.938845 19489 rpcz_store.cc:276] 0430 20:11:40.428826 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:40.431387 (+ 2561us) service_pool.cc:225] Handling call
0430 20:11:45.938676 (+5507289us) ts_tablet_manager.cc:1120] Deleting tablet 77285afcafe04a829e485998bbbe00f0
0430 20:11:45.938685 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:45.938744 (+ 59us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5507130}
I20240430 20:11:45.941114 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 246) took 5509 ms. Trace:
I20240430 20:11:45.941207 19489 rpcz_store.cc:276] 0430 20:11:40.432100 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:40.432149 (+ 49us) service_pool.cc:225] Handling call
0430 20:11:45.940832 (+5508683us) ts_tablet_manager.cc:1120] Deleting tablet d4d6c37b578c4fe1aa666b0a6b0bc7c9
0430 20:11:45.940992 (+ 160us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:45.941095 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5508478}
I20240430 20:11:45.927201 18983 raft_consensus.cc:348] T c6378f1495144c458af49b3c19a2b67c P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:45.942713 18983 raft_consensus.cc:374] T c6378f1495144c458af49b3c19a2b67c P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:45.942775 18983 raft_consensus.cc:729] T c6378f1495144c458af49b3c19a2b67c P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:45.944185 18985 tablet_bootstrap.cc:492] T 66dd767bd3f8494f94061fb89e5e5680 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:45.944360 18985 ts_tablet_manager.cc:1367] T 66dd767bd3f8494f94061fb89e5e5680 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.030s user 0.003s sys 0.000s
I20240430 20:11:45.945245 18985 raft_consensus.cc:348] T 66dd767bd3f8494f94061fb89e5e5680 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:45.945447 18985 raft_consensus.cc:374] T 66dd767bd3f8494f94061fb89e5e5680 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:45.945502 18985 raft_consensus.cc:729] T 66dd767bd3f8494f94061fb89e5e5680 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:45.945724 18985 consensus_queue.cc:260] T 66dd767bd3f8494f94061fb89e5e5680 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:45.946223 18985 ts_tablet_manager.cc:1398] T 66dd767bd3f8494f94061fb89e5e5680 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240430 20:11:45.943015 18983 consensus_queue.cc:260] T c6378f1495144c458af49b3c19a2b67c P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:45.948539 18985 tablet_bootstrap.cc:492] T 2873af6da928463f9809864b8eaf0c4a P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:45.950016 18985 tablet_bootstrap.cc:654] T 2873af6da928463f9809864b8eaf0c4a P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:45.951435 18983 ts_tablet_manager.cc:1398] T c6378f1495144c458af49b3c19a2b67c P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.025s user 0.002s sys 0.000s
I20240430 20:11:45.955447 18988 tablet_bootstrap.cc:492] T 06e82c706b9c4f35bf53faddf57b3fd7 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:45.955688 18988 ts_tablet_manager.cc:1367] T 06e82c706b9c4f35bf53faddf57b3fd7 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.027s user 0.006s sys 0.002s
I20240430 20:11:45.957147 18988 raft_consensus.cc:348] T 06e82c706b9c4f35bf53faddf57b3fd7 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:45.957394 18988 raft_consensus.cc:374] T 06e82c706b9c4f35bf53faddf57b3fd7 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:45.957541 18988 raft_consensus.cc:729] T 06e82c706b9c4f35bf53faddf57b3fd7 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:45.957818 18988 consensus_queue.cc:260] T 06e82c706b9c4f35bf53faddf57b3fd7 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:45.963292 18983 tablet_bootstrap.cc:492] T 3ec3993042f5498b818bdf32d5087928 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:45.963896 19471 ts_tablet_manager.cc:1888] T 294644d7a16b4ca2bbb6d106bcb8c7ff P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.3
I20240430 20:11:45.964020 19471 log.cc:1196] T 294644d7a16b4ca2bbb6d106bcb8c7ff P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/294644d7a16b4ca2bbb6d106bcb8c7ff
I20240430 20:11:45.964382 19471 ts_tablet_manager.cc:1909] T 294644d7a16b4ca2bbb6d106bcb8c7ff P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:45.964886 18983 tablet_bootstrap.cc:654] T 3ec3993042f5498b818bdf32d5087928 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:45.965247 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 216) took 6191 ms. Trace:
I20240430 20:11:45.965337 19471 rpcz_store.cc:276] 0430 20:11:39.773438 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:39.773506 (+ 68us) service_pool.cc:225] Handling call
0430 20:11:45.934715 (+6161209us) ts_tablet_manager.cc:1120] Deleting tablet 294644d7a16b4ca2bbb6d106bcb8c7ff
0430 20:11:45.934724 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:45.954115 (+ 19391us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:45.963779 (+ 9664us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:45.965222 (+ 1443us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6160960}
I20240430 20:11:45.965973 18985 tablet_bootstrap.cc:492] T 2873af6da928463f9809864b8eaf0c4a P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:45.966099 18985 ts_tablet_manager.cc:1367] T 2873af6da928463f9809864b8eaf0c4a P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.002s
I20240430 20:11:45.966746 18985 raft_consensus.cc:348] T 2873af6da928463f9809864b8eaf0c4a P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:45.966972 18985 raft_consensus.cc:374] T 2873af6da928463f9809864b8eaf0c4a P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:45.967022 18985 raft_consensus.cc:729] T 2873af6da928463f9809864b8eaf0c4a P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:45.967243 18985 consensus_queue.cc:260] T 2873af6da928463f9809864b8eaf0c4a P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:45.967769 18985 ts_tablet_manager.cc:1398] T 2873af6da928463f9809864b8eaf0c4a P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240430 20:11:45.967891 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 217) took 6194 ms. Trace:
I20240430 20:11:45.968043 19471 rpcz_store.cc:276] 0430 20:11:39.773607 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:39.773833 (+ 226us) service_pool.cc:225] Handling call
0430 20:11:45.967766 (+6193933us) ts_tablet_manager.cc:1120] Deleting tablet eed6338ad3d64859af087ee15e1ec798
0430 20:11:45.967777 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:45.967868 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6193743}
I20240430 20:11:45.968281 18988 ts_tablet_manager.cc:1398] T 06e82c706b9c4f35bf53faddf57b3fd7 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s
I20240430 20:11:45.968722 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 218) took 6161 ms. Trace:
I20240430 20:11:45.968847 19471 rpcz_store.cc:276] 0430 20:11:39.830583 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:39.831015 (+ 432us) service_pool.cc:225] Handling call
0430 20:11:45.968627 (+6137612us) ts_tablet_manager.cc:1120] Deleting tablet 5bde72e2791a497db7dd08686ca36d80
0430 20:11:45.968636 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:45.968701 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6113253}
I20240430 20:11:45.969314 19471 tablet_replica.cc:330] T e6eac92dfa5f4456bc49571878f0f8dd P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:45.970032 18985 tablet_bootstrap.cc:492] T de1dfe6c82d34a4d8d02e466c55327ff P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:45.970322 19471 raft_consensus.cc:2229] T e6eac92dfa5f4456bc49571878f0f8dd P 5e1c6397f09747f091534faf93542710 [term 2 LEADER]: Raft consensus shutting down.
I20240430 20:11:45.970721 19471 raft_consensus.cc:2258] T e6eac92dfa5f4456bc49571878f0f8dd P 5e1c6397f09747f091534faf93542710 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:45.971383 19471 ts_tablet_manager.cc:1875] T e6eac92dfa5f4456bc49571878f0f8dd P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:45.976974 20198 raft_consensus.cc:482] T 681cdd85ac6847a0a1aaded310456582 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:45.977150 20198 raft_consensus.cc:504] T 681cdd85ac6847a0a1aaded310456582 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:45.977874 20198 leader_election.cc:288] T 681cdd85ac6847a0a1aaded310456582 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:45.978786 18474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "681cdd85ac6847a0a1aaded310456582" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:45.983122 18985 tablet_bootstrap.cc:654] T de1dfe6c82d34a4d8d02e466c55327ff P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:45.983553 19489 tablet_replica.cc:330] T a9e0659564894b0e8cb7f600bb07a085 P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:45.983772 19489 raft_consensus.cc:2229] T a9e0659564894b0e8cb7f600bb07a085 P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Raft consensus shutting down.
I20240430 20:11:45.984251 19489 raft_consensus.cc:2258] T a9e0659564894b0e8cb7f600bb07a085 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:45.984282 18623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "681cdd85ac6847a0a1aaded310456582" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:45.984877 19489 ts_tablet_manager.cc:1875] T a9e0659564894b0e8cb7f600bb07a085 P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:45.984926 18696 leader_election.cc:302] T 681cdd85ac6847a0a1aaded310456582 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710
I20240430 20:11:45.985167 20198 raft_consensus.cc:2726] T 681cdd85ac6847a0a1aaded310456582 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:45.992398 19489 ts_tablet_manager.cc:1888] T a9e0659564894b0e8cb7f600bb07a085 P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:45.992640 19489 log.cc:1196] T a9e0659564894b0e8cb7f600bb07a085 P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/a9e0659564894b0e8cb7f600bb07a085
W20240430 20:11:45.993014 18557 leader_election.cc:341] T 2873af6da928463f9809864b8eaf0c4a P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:45.993156 19489 ts_tablet_manager.cc:1909] T a9e0659564894b0e8cb7f600bb07a085 P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:45.993219 18557 leader_election.cc:302] T 2873af6da928463f9809864b8eaf0c4a P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:45.995437 20384 raft_consensus.cc:2726] T 2873af6da928463f9809864b8eaf0c4a P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:45.995766 20384 consensus_queue.cc:1035] T ae20f44df50343289b0a512a4ea53692 P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:46.000267 18983 tablet_bootstrap.cc:492] T 3ec3993042f5498b818bdf32d5087928 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:46.000505 18983 ts_tablet_manager.cc:1367] T 3ec3993042f5498b818bdf32d5087928 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.037s user 0.003s sys 0.000s
I20240430 20:11:46.001286 18983 raft_consensus.cc:348] T 3ec3993042f5498b818bdf32d5087928 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.001566 18983 raft_consensus.cc:374] T 3ec3993042f5498b818bdf32d5087928 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:46.001658 18983 raft_consensus.cc:729] T 3ec3993042f5498b818bdf32d5087928 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:46.001950 18983 consensus_queue.cc:260] T 3ec3993042f5498b818bdf32d5087928 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.002668 18983 ts_tablet_manager.cc:1398] T 3ec3993042f5498b818bdf32d5087928 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240430 20:11:46.002993 18983 tablet_bootstrap.cc:492] T 95383d9d5a234a4d9d841a8b114428b0 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:46.004608 18983 tablet_bootstrap.cc:654] T 95383d9d5a234a4d9d841a8b114428b0 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:46.005700 19495 ts_tablet_manager.cc:1888] T e3e74e76ab2b4fe492ffcc1838656fd1 P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 3.2
I20240430 20:11:46.007584 18756 raft_consensus.cc:3037] T 681cdd85ac6847a0a1aaded310456582 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:46.007879 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 248) took 5575 ms. Trace:
I20240430 20:11:46.007993 19489 rpcz_store.cc:276] 0430 20:11:40.432319 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:40.432379 (+ 60us) service_pool.cc:225] Handling call
0430 20:11:45.942004 (+5509625us) ts_tablet_manager.cc:1120] Deleting tablet a9e0659564894b0e8cb7f600bb07a085
0430 20:11:45.983466 (+ 41462us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:45.988816 (+ 5350us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:45.992296 (+ 3480us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:46.007851 (+ 15555us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5509448}
I20240430 20:11:46.009593 19471 ts_tablet_manager.cc:1888] T e6eac92dfa5f4456bc49571878f0f8dd P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.1
I20240430 20:11:46.009995 19471 log.cc:1196] T e6eac92dfa5f4456bc49571878f0f8dd P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/e6eac92dfa5f4456bc49571878f0f8dd
I20240430 20:11:46.010505 19471 ts_tablet_manager.cc:1909] T e6eac92dfa5f4456bc49571878f0f8dd P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:46.011500 19495 log.cc:1196] T e3e74e76ab2b4fe492ffcc1838656fd1 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/e3e74e76ab2b4fe492ffcc1838656fd1
I20240430 20:11:46.011732 18988 tablet_bootstrap.cc:492] T 677481aee5fa4eb4b189322d7a3f8f2e P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:46.014012 18988 tablet_bootstrap.cc:654] T 677481aee5fa4eb4b189322d7a3f8f2e P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:46.014637 19495 ts_tablet_manager.cc:1909] T e3e74e76ab2b4fe492ffcc1838656fd1 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:46.014811 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 247) took 5582 ms. Trace:
I20240430 20:11:46.014904 19489 rpcz_store.cc:276] 0430 20:11:40.432208 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:40.432584 (+ 376us) service_pool.cc:225] Handling call
0430 20:11:46.014702 (+5582118us) ts_tablet_manager.cc:1120] Deleting tablet 15674aeb56b74210a4a2809608fd4457
0430 20:11:46.014710 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:46.014791 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5581936}
I20240430 20:11:46.015389 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 249) took 5582 ms. Trace:
I20240430 20:11:46.015465 19489 rpcz_store.cc:276] 0430 20:11:40.433054 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:40.433100 (+ 46us) service_pool.cc:225] Handling call
0430 20:11:46.015305 (+5582205us) ts_tablet_manager.cc:1120] Deleting tablet eed6338ad3d64859af087ee15e1ec798
0430 20:11:46.015314 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:46.015374 (+ 60us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5582006}
I20240430 20:11:46.015933 19489 tablet_replica.cc:330] T e6eac92dfa5f4456bc49571878f0f8dd P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:46.016105 19489 raft_consensus.cc:2229] T e6eac92dfa5f4456bc49571878f0f8dd P 3c51c33d5616482cafb68e26688820f4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:46.016307 19489 raft_consensus.cc:2258] T e6eac92dfa5f4456bc49571878f0f8dd P 3c51c33d5616482cafb68e26688820f4 [term 2 FOLLOWER]: Raft consensus is shut down!
W20240430 20:11:46.016817 18417 consensus_peers.cc:485] T ae24e75a0d42441ebc97400ffa94145e P 2efb542aa55e439b99ccb98d1c5c884a -> Peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Couldn't send request to peer 3c51c33d5616482cafb68e26688820f4. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240430 20:11:46.016842 19489 ts_tablet_manager.cc:1875] T e6eac92dfa5f4456bc49571878f0f8dd P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:46.017149 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 368) took 1088 ms. Trace:
I20240430 20:11:46.017256 19495 rpcz_store.cc:276] 0430 20:11:44.928695 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:44.929512 (+ 817us) service_pool.cc:225] Handling call
0430 20:11:45.495689 (+566177us) ts_tablet_manager.cc:1120] Deleting tablet e3e74e76ab2b4fe492ffcc1838656fd1
0430 20:11:45.495700 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:45.514512 (+ 18812us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:46.005555 (+491043us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:46.017123 (+ 11568us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":566000}
I20240430 20:11:46.017606 19495 tablet_replica.cc:330] T 76d6e9d547ca4a848eae852944b1d117 P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:46.017820 19495 raft_consensus.cc:2229] T 76d6e9d547ca4a848eae852944b1d117 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:46.018019 19495 raft_consensus.cc:2258] T 76d6e9d547ca4a848eae852944b1d117 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:46.019052 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 219) took 6165 ms. Trace:
I20240430 20:11:46.019614 19471 rpcz_store.cc:276] 0430 20:11:39.853513 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:39.853583 (+ 70us) service_pool.cc:225] Handling call
0430 20:11:45.969262 (+6115679us) ts_tablet_manager.cc:1120] Deleting tablet e6eac92dfa5f4456bc49571878f0f8dd
0430 20:11:45.969270 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:45.999542 (+ 30272us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:46.009467 (+ 9925us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:46.019027 (+ 9560us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6113694}
I20240430 20:11:46.020257 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 220) took 6157 ms. Trace:
I20240430 20:11:46.020676 19471 rpcz_store.cc:276] 0430 20:11:39.863183 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:39.863245 (+ 62us) service_pool.cc:225] Handling call
0430 20:11:46.020139 (+6156894us) ts_tablet_manager.cc:1120] Deleting tablet 1f4e860484784aa9a2d28af2e9fb611c
0430 20:11:46.020149 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:46.020244 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6155222}
I20240430 20:11:46.021178 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 225) took 6150 ms. Trace:
I20240430 20:11:46.021514 19471 rpcz_store.cc:276] 0430 20:11:39.870628 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:39.870679 (+ 51us) service_pool.cc:225] Handling call
0430 20:11:46.021103 (+6150424us) ts_tablet_manager.cc:1120] Deleting tablet 2a1e50f99c5d4b44bdbb51b58a92bc75
0430 20:11:46.021112 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:46.021164 (+ 52us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6139881}
I20240430 20:11:46.021937 19471 tablet_replica.cc:330] T a9e0659564894b0e8cb7f600bb07a085 P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:46.025053 19471 raft_consensus.cc:2229] T a9e0659564894b0e8cb7f600bb07a085 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:46.025305 19471 raft_consensus.cc:2258] T a9e0659564894b0e8cb7f600bb07a085 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:46.025940 19471 ts_tablet_manager.cc:1875] T a9e0659564894b0e8cb7f600bb07a085 P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:46.030047 19495 ts_tablet_manager.cc:1875] T 76d6e9d547ca4a848eae852944b1d117 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:46.030567 18988 tablet_bootstrap.cc:492] T 677481aee5fa4eb4b189322d7a3f8f2e P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:46.030794 18988 ts_tablet_manager.cc:1367] T 677481aee5fa4eb4b189322d7a3f8f2e P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.003s
I20240430 20:11:46.031901 18988 raft_consensus.cc:348] T 677481aee5fa4eb4b189322d7a3f8f2e P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.032178 18988 raft_consensus.cc:374] T 677481aee5fa4eb4b189322d7a3f8f2e P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:46.031555 20545 consensus_queue.cc:1035] T 2e79df8744e74b5991a79ce978ff0202 P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:46.032320 18988 raft_consensus.cc:729] T 677481aee5fa4eb4b189322d7a3f8f2e P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:46.032596 18988 consensus_queue.cc:260] T 677481aee5fa4eb4b189322d7a3f8f2e P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.035296 18988 ts_tablet_manager.cc:1398] T 677481aee5fa4eb4b189322d7a3f8f2e P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240430 20:11:46.035636 18988 tablet_bootstrap.cc:492] T 4f6f9ae722da49e58c9b72107f6da2fa P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:46.036674 18983 tablet_bootstrap.cc:492] T 95383d9d5a234a4d9d841a8b114428b0 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:46.037021 18988 tablet_bootstrap.cc:654] T 4f6f9ae722da49e58c9b72107f6da2fa P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:46.046530 20545 raft_consensus.cc:482] T 677481aee5fa4eb4b189322d7a3f8f2e P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:46.046823 18983 ts_tablet_manager.cc:1367] T 95383d9d5a234a4d9d841a8b114428b0 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.044s user 0.004s sys 0.000s
I20240430 20:11:46.046756 20545 raft_consensus.cc:504] T 677481aee5fa4eb4b189322d7a3f8f2e P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.047427 20545 leader_election.cc:288] T 677481aee5fa4eb4b189322d7a3f8f2e P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:46.047649 18983 raft_consensus.cc:348] T 95383d9d5a234a4d9d841a8b114428b0 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.047835 18983 raft_consensus.cc:374] T 95383d9d5a234a4d9d841a8b114428b0 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:46.047883 18983 raft_consensus.cc:729] T 95383d9d5a234a4d9d841a8b114428b0 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:46.048261 18623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "677481aee5fa4eb4b189322d7a3f8f2e" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:46.048461 18623 raft_consensus.cc:2454] T 677481aee5fa4eb4b189322d7a3f8f2e P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 0.
I20240430 20:11:46.049104 18415 leader_election.cc:302] T 677481aee5fa4eb4b189322d7a3f8f2e P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:46.049057 18983 consensus_queue.cc:260] T 95383d9d5a234a4d9d841a8b114428b0 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.049659 18988 tablet_bootstrap.cc:492] T 4f6f9ae722da49e58c9b72107f6da2fa P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:46.049623 18758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "677481aee5fa4eb4b189322d7a3f8f2e" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:46.049795 18988 ts_tablet_manager.cc:1367] T 4f6f9ae722da49e58c9b72107f6da2fa P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.003s
I20240430 20:11:46.049875 18758 raft_consensus.cc:2454] T 677481aee5fa4eb4b189322d7a3f8f2e P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 0.
I20240430 20:11:46.050253 20545 raft_consensus.cc:2781] T 677481aee5fa4eb4b189322d7a3f8f2e P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:46.050348 20545 raft_consensus.cc:482] T 677481aee5fa4eb4b189322d7a3f8f2e P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:46.050424 20545 raft_consensus.cc:3037] T 677481aee5fa4eb4b189322d7a3f8f2e P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:46.050380 18988 raft_consensus.cc:348] T 4f6f9ae722da49e58c9b72107f6da2fa P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.050554 18988 raft_consensus.cc:374] T 4f6f9ae722da49e58c9b72107f6da2fa P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:46.050632 18988 raft_consensus.cc:729] T 4f6f9ae722da49e58c9b72107f6da2fa P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:46.038673 19471 ts_tablet_manager.cc:1888] T a9e0659564894b0e8cb7f600bb07a085 P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:46.050837 19471 log.cc:1196] T a9e0659564894b0e8cb7f600bb07a085 P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/a9e0659564894b0e8cb7f600bb07a085
I20240430 20:11:46.051198 19471 ts_tablet_manager.cc:1909] T a9e0659564894b0e8cb7f600bb07a085 P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:46.053061 20631 raft_consensus.cc:482] T eb76f60448774fe1967fffe7c6873cfc P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:46.053182 20631 raft_consensus.cc:504] T eb76f60448774fe1967fffe7c6873cfc P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.053311 20545 raft_consensus.cc:504] T 677481aee5fa4eb4b189322d7a3f8f2e P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.053753 20631 leader_election.cc:288] T eb76f60448774fe1967fffe7c6873cfc P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:46.053855 20545 leader_election.cc:288] T 677481aee5fa4eb4b189322d7a3f8f2e P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:46.054554 18623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb76f60448774fe1967fffe7c6873cfc" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:46.054569 18619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "677481aee5fa4eb4b189322d7a3f8f2e" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:46.054781 18623 raft_consensus.cc:2454] T eb76f60448774fe1967fffe7c6873cfc P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 0.
I20240430 20:11:46.054817 18619 raft_consensus.cc:3037] T 677481aee5fa4eb4b189322d7a3f8f2e P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:46.055178 18415 leader_election.cc:302] T eb76f60448774fe1967fffe7c6873cfc P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:46.058032 18619 raft_consensus.cc:2454] T 677481aee5fa4eb4b189322d7a3f8f2e P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 1.
I20240430 20:11:46.058564 18415 leader_election.cc:302] T 677481aee5fa4eb4b189322d7a3f8f2e P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:46.058727 18988 consensus_queue.cc:260] T 4f6f9ae722da49e58c9b72107f6da2fa P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.058852 20631 raft_consensus.cc:2781] T eb76f60448774fe1967fffe7c6873cfc P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:46.058928 20631 raft_consensus.cc:482] T eb76f60448774fe1967fffe7c6873cfc P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:46.058984 20631 raft_consensus.cc:3037] T eb76f60448774fe1967fffe7c6873cfc P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:46.059274 18988 ts_tablet_manager.cc:1398] T 4f6f9ae722da49e58c9b72107f6da2fa P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.009s user 0.001s sys 0.000s
I20240430 20:11:46.059556 18988 tablet_bootstrap.cc:492] T ae24e75a0d42441ebc97400ffa94145e P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:46.060775 18983 ts_tablet_manager.cc:1398] T 95383d9d5a234a4d9d841a8b114428b0 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.014s user 0.001s sys 0.001s
I20240430 20:11:46.060894 18988 tablet_bootstrap.cc:654] T ae24e75a0d42441ebc97400ffa94145e P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:46.061465 20631 raft_consensus.cc:504] T eb76f60448774fe1967fffe7c6873cfc P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.062067 20631 leader_election.cc:288] T eb76f60448774fe1967fffe7c6873cfc P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:46.062202 20631 raft_consensus.cc:2781] T 677481aee5fa4eb4b189322d7a3f8f2e P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:46.062923 20631 raft_consensus.cc:686] T 677481aee5fa4eb4b189322d7a3f8f2e P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Becoming Leader. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Running, Role: LEADER
I20240430 20:11:46.063436 20631 consensus_queue.cc:237] T 677481aee5fa4eb4b189322d7a3f8f2e P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.064499 18623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb76f60448774fe1967fffe7c6873cfc" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:46.064711 18623 raft_consensus.cc:3037] T eb76f60448774fe1967fffe7c6873cfc P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:46.065934 18985 tablet_bootstrap.cc:492] T de1dfe6c82d34a4d8d02e466c55327ff P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:46.066076 18985 ts_tablet_manager.cc:1367] T de1dfe6c82d34a4d8d02e466c55327ff P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.096s user 0.002s sys 0.004s
I20240430 20:11:46.066107 18758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "677481aee5fa4eb4b189322d7a3f8f2e" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:46.066277 18758 raft_consensus.cc:3037] T 677481aee5fa4eb4b189322d7a3f8f2e P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:46.066771 18985 raft_consensus.cc:348] T de1dfe6c82d34a4d8d02e466c55327ff P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.066933 18985 raft_consensus.cc:374] T de1dfe6c82d34a4d8d02e466c55327ff P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:46.066974 18985 raft_consensus.cc:729] T de1dfe6c82d34a4d8d02e466c55327ff P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:46.067535 18623 raft_consensus.cc:2454] T eb76f60448774fe1967fffe7c6873cfc P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 1.
I20240430 20:11:46.067469 18756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb76f60448774fe1967fffe7c6873cfc" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:46.067883 18985 consensus_queue.cc:260] T de1dfe6c82d34a4d8d02e466c55327ff P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.068151 18415 leader_election.cc:302] T eb76f60448774fe1967fffe7c6873cfc P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:46.068370 20630 raft_consensus.cc:2781] T eb76f60448774fe1967fffe7c6873cfc P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:46.068434 18985 ts_tablet_manager.cc:1398] T de1dfe6c82d34a4d8d02e466c55327ff P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240430 20:11:46.068485 20630 raft_consensus.cc:686] T eb76f60448774fe1967fffe7c6873cfc P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Becoming Leader. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Running, Role: LEADER
I20240430 20:11:46.068676 20630 consensus_queue.cc:237] T eb76f60448774fe1967fffe7c6873cfc P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.068897 18985 tablet_bootstrap.cc:492] T a0baee91f5b44945b595d0e234af018e P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:46.069398 19489 ts_tablet_manager.cc:1888] T e6eac92dfa5f4456bc49571878f0f8dd P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.1
I20240430 20:11:46.070369 18758 raft_consensus.cc:2454] T 677481aee5fa4eb4b189322d7a3f8f2e P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 1.
I20240430 20:11:46.070892 19489 log.cc:1196] T e6eac92dfa5f4456bc49571878f0f8dd P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/e6eac92dfa5f4456bc49571878f0f8dd
I20240430 20:11:46.071319 19489 ts_tablet_manager.cc:1909] T e6eac92dfa5f4456bc49571878f0f8dd P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:46.072117 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 250) took 5638 ms. Trace:
I20240430 20:11:46.072389 18985 tablet_bootstrap.cc:654] T a0baee91f5b44945b595d0e234af018e P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:46.073436 19489 rpcz_store.cc:276] 0430 20:11:40.433650 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:40.433698 (+ 48us) service_pool.cc:225] Handling call
0430 20:11:46.015875 (+5582177us) ts_tablet_manager.cc:1120] Deleting tablet e6eac92dfa5f4456bc49571878f0f8dd
0430 20:11:46.015887 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:46.051919 (+ 36032us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:46.062515 (+ 10596us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:46.072098 (+ 9583us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5581998}
I20240430 20:11:46.073884 19489 tablet_replica.cc:330] T 2a1e50f99c5d4b44bdbb51b58a92bc75 P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:46.074157 19489 raft_consensus.cc:2229] T 2a1e50f99c5d4b44bdbb51b58a92bc75 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:46.074394 19489 raft_consensus.cc:2258] T 2a1e50f99c5d4b44bdbb51b58a92bc75 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:46.075040 19489 ts_tablet_manager.cc:1875] T 2a1e50f99c5d4b44bdbb51b58a92bc75 P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240430 20:11:46.077227 18417 leader_election.cc:341] T eb76f60448774fe1967fffe7c6873cfc P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:46.077395 18758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb76f60448774fe1967fffe7c6873cfc" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
W20240430 20:11:46.078660 18417 leader_election.cc:341] T eb76f60448774fe1967fffe7c6873cfc P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:46.080340 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 226) took 6199 ms. Trace:
I20240430 20:11:46.080495 19471 rpcz_store.cc:276] 0430 20:11:39.880837 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:39.880943 (+ 106us) service_pool.cc:225] Handling call
0430 20:11:46.021898 (+6140955us) ts_tablet_manager.cc:1120] Deleting tablet a9e0659564894b0e8cb7f600bb07a085
0430 20:11:46.021906 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:46.035554 (+ 13648us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:46.038549 (+ 2995us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:46.080318 (+ 41769us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":25,"tablet-delete.queue_time_us":6140448}
I20240430 20:11:46.088214 19489 ts_tablet_manager.cc:1888] T 2a1e50f99c5d4b44bdbb51b58a92bc75 P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:46.089217 18985 tablet_bootstrap.cc:492] T a0baee91f5b44945b595d0e234af018e P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:46.089474 18985 ts_tablet_manager.cc:1367] T a0baee91f5b44945b595d0e234af018e P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.001s
I20240430 20:11:46.090633 18985 raft_consensus.cc:348] T a0baee91f5b44945b595d0e234af018e P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.090893 18985 raft_consensus.cc:374] T a0baee91f5b44945b595d0e234af018e P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:46.090883 18983 tablet_bootstrap.cc:492] T c6e3d09402574f1e8fe9e4160c70a3ee P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:46.090893 18619 raft_consensus.cc:1261] T 7ce09fcbe1ac457a9c05efaf000c3263 P 5e1c6397f09747f091534faf93542710 [term 2 FOLLOWER]: Refusing update from remote peer 3c51c33d5616482cafb68e26688820f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 1. (term mismatch)
I20240430 20:11:46.091030 18619 pending_rounds.cc:76] T 7ce09fcbe1ac457a9c05efaf000c3263 P 5e1c6397f09747f091534faf93542710: Aborting all ops after (but not including) 0
I20240430 20:11:46.091068 18619 pending_rounds.cc:97] T 7ce09fcbe1ac457a9c05efaf000c3263 P 5e1c6397f09747f091534faf93542710: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20240430 20:11:46.091114 18619 raft_consensus.cc:2866] T 7ce09fcbe1ac457a9c05efaf000c3263 P 5e1c6397f09747f091534faf93542710 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240430 20:11:46.091199 18985 raft_consensus.cc:729] T a0baee91f5b44945b595d0e234af018e P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:46.091543 18985 consensus_queue.cc:260] T a0baee91f5b44945b595d0e234af018e P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.092129 18985 ts_tablet_manager.cc:1398] T a0baee91f5b44945b595d0e234af018e P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240430 20:11:46.092417 18985 tablet_bootstrap.cc:492] T 95383d9d5a234a4d9d841a8b114428b0 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:46.094027 18985 tablet_bootstrap.cc:654] T 95383d9d5a234a4d9d841a8b114428b0 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:46.094465 18983 tablet_bootstrap.cc:654] T c6e3d09402574f1e8fe9e4160c70a3ee P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
W20240430 20:11:46.095265 18415 consensus_peers.cc:485] T e87fb2f0361d4aa3ada27bbdfa88e94c P 2efb542aa55e439b99ccb98d1c5c884a -> Peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Couldn't send request to peer 5e1c6397f09747f091534faf93542710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e87fb2f0361d4aa3ada27bbdfa88e94c. This is attempt 1: this message will repeat every 5th retry.
I20240430 20:11:46.097256 19471 tablet_replica.cc:330] T d61c279d20b0471a967942525ceb4fcb P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:46.097456 19471 raft_consensus.cc:2229] T d61c279d20b0471a967942525ceb4fcb P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:46.097666 19471 raft_consensus.cc:2258] T d61c279d20b0471a967942525ceb4fcb P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:46.098975 19471 ts_tablet_manager.cc:1875] T d61c279d20b0471a967942525ceb4fcb P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:46.091830 20205 consensus_queue.cc:1035] T 7ce09fcbe1ac457a9c05efaf000c3263 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:46.107815 19489 log.cc:1196] T 2a1e50f99c5d4b44bdbb51b58a92bc75 P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/2a1e50f99c5d4b44bdbb51b58a92bc75
I20240430 20:11:46.108937 19489 ts_tablet_manager.cc:1909] T 2a1e50f99c5d4b44bdbb51b58a92bc75 P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:46.111521 19471 ts_tablet_manager.cc:1888] T d61c279d20b0471a967942525ceb4fcb P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:46.111728 19471 log.cc:1196] T d61c279d20b0471a967942525ceb4fcb P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/d61c279d20b0471a967942525ceb4fcb
I20240430 20:11:46.112102 19471 ts_tablet_manager.cc:1909] T d61c279d20b0471a967942525ceb4fcb P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:46.112489 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 227) took 6229 ms. Trace:
I20240430 20:11:46.112594 19471 rpcz_store.cc:276] 0430 20:11:39.882833 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:39.882896 (+ 63us) service_pool.cc:225] Handling call
0430 20:11:46.080981 (+6198085us) ts_tablet_manager.cc:1120] Deleting tablet d61c279d20b0471a967942525ceb4fcb
0430 20:11:46.097203 (+ 16222us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:46.103844 (+ 6641us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:46.111394 (+ 7550us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:46.112471 (+ 1077us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6196846}
I20240430 20:11:46.113163 19471 tablet_replica.cc:330] T e2d90ff17e574f88b2061b1c117d4b7d P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:46.113569 19471 raft_consensus.cc:2229] T e2d90ff17e574f88b2061b1c117d4b7d P 5e1c6397f09747f091534faf93542710 [term 3 LEADER]: Raft consensus shutting down.
I20240430 20:11:46.116348 19471 raft_consensus.cc:2258] T e2d90ff17e574f88b2061b1c117d4b7d P 5e1c6397f09747f091534faf93542710 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:46.117309 19471 ts_tablet_manager.cc:1875] T e2d90ff17e574f88b2061b1c117d4b7d P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:46.126734 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 251) took 5692 ms. Trace:
I20240430 20:11:46.127940 18985 tablet_bootstrap.cc:492] T 95383d9d5a234a4d9d841a8b114428b0 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:46.128155 19489 rpcz_store.cc:276] 0430 20:11:40.433919 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:40.433957 (+ 38us) service_pool.cc:225] Handling call
0430 20:11:46.073834 (+5639877us) ts_tablet_manager.cc:1120] Deleting tablet 2a1e50f99c5d4b44bdbb51b58a92bc75
0430 20:11:46.073845 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:46.083791 (+ 9946us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:46.088089 (+ 4298us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:46.126711 (+ 38622us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5639700}
I20240430 20:11:46.128990 19489 tablet_replica.cc:330] T f5546f9edeea44d7a6c5cb954f5157ba P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:46.129412 19489 raft_consensus.cc:2229] T f5546f9edeea44d7a6c5cb954f5157ba P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:46.129634 18985 ts_tablet_manager.cc:1367] T 95383d9d5a234a4d9d841a8b114428b0 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.037s user 0.000s sys 0.006s
I20240430 20:11:46.130789 19489 raft_consensus.cc:2258] T f5546f9edeea44d7a6c5cb954f5157ba P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:46.135890 18983 tablet_bootstrap.cc:492] T c6e3d09402574f1e8fe9e4160c70a3ee P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:46.136219 18983 ts_tablet_manager.cc:1367] T c6e3d09402574f1e8fe9e4160c70a3ee P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.045s user 0.005s sys 0.000s
I20240430 20:11:46.137025 18983 raft_consensus.cc:348] T c6e3d09402574f1e8fe9e4160c70a3ee P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.137307 18983 raft_consensus.cc:374] T c6e3d09402574f1e8fe9e4160c70a3ee P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:46.137396 18983 raft_consensus.cc:729] T c6e3d09402574f1e8fe9e4160c70a3ee P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:46.137683 18983 consensus_queue.cc:260] T c6e3d09402574f1e8fe9e4160c70a3ee P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.138286 19471 ts_tablet_manager.cc:1888] T e2d90ff17e574f88b2061b1c117d4b7d P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 3.3
I20240430 20:11:46.142251 18988 tablet_bootstrap.cc:492] T ae24e75a0d42441ebc97400ffa94145e P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:46.142442 18988 ts_tablet_manager.cc:1367] T ae24e75a0d42441ebc97400ffa94145e P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.083s user 0.005s sys 0.009s
I20240430 20:11:46.146751 19489 ts_tablet_manager.cc:1875] T f5546f9edeea44d7a6c5cb954f5157ba P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:46.147888 18474 raft_consensus.cc:1261] T 7ce09fcbe1ac457a9c05efaf000c3263 P 2efb542aa55e439b99ccb98d1c5c884a [term 2 FOLLOWER]: Refusing update from remote peer 3c51c33d5616482cafb68e26688820f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20240430 20:11:46.148212 18988 raft_consensus.cc:348] T ae24e75a0d42441ebc97400ffa94145e P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.148417 18988 raft_consensus.cc:374] T ae24e75a0d42441ebc97400ffa94145e P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:46.148466 18988 raft_consensus.cc:729] T ae24e75a0d42441ebc97400ffa94145e P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:46.148670 18988 consensus_queue.cc:260] T ae24e75a0d42441ebc97400ffa94145e P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.148756 20205 consensus_queue.cc:1035] T 7ce09fcbe1ac457a9c05efaf000c3263 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:46.149166 18988 ts_tablet_manager.cc:1398] T ae24e75a0d42441ebc97400ffa94145e P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240430 20:11:46.149422 18988 tablet_bootstrap.cc:492] T eb76f60448774fe1967fffe7c6873cfc P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:46.150451 18985 raft_consensus.cc:348] T 95383d9d5a234a4d9d841a8b114428b0 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.150743 18985 raft_consensus.cc:374] T 95383d9d5a234a4d9d841a8b114428b0 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:46.150808 18985 raft_consensus.cc:729] T 95383d9d5a234a4d9d841a8b114428b0 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:46.151252 18988 tablet_bootstrap.cc:654] T eb76f60448774fe1967fffe7c6873cfc P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:46.151474 18985 consensus_queue.cc:260] T 95383d9d5a234a4d9d841a8b114428b0 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.152043 18985 ts_tablet_manager.cc:1398] T 95383d9d5a234a4d9d841a8b114428b0 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.022s user 0.001s sys 0.000s
I20240430 20:11:46.153838 20469 raft_consensus.cc:482] T ae24e75a0d42441ebc97400ffa94145e P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:46.153952 20469 raft_consensus.cc:504] T ae24e75a0d42441ebc97400ffa94145e P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.154538 20469 leader_election.cc:288] T ae24e75a0d42441ebc97400ffa94145e P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:46.138427 18983 ts_tablet_manager.cc:1398] T c6e3d09402574f1e8fe9e4160c70a3ee P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240430 20:11:46.154929 18623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae24e75a0d42441ebc97400ffa94145e" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:46.154971 18478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae24e75a0d42441ebc97400ffa94145e" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:46.155074 18983 tablet_bootstrap.cc:492] T de1dfe6c82d34a4d8d02e466c55327ff P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:46.155148 18623 raft_consensus.cc:2379] T ae24e75a0d42441ebc97400ffa94145e P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c51c33d5616482cafb68e26688820f4 in current term 1: Already voted for candidate 2efb542aa55e439b99ccb98d1c5c884a in this term.
I20240430 20:11:46.155726 18699 leader_election.cc:302] T ae24e75a0d42441ebc97400ffa94145e P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710
I20240430 20:11:46.155983 20469 raft_consensus.cc:3037] T ae24e75a0d42441ebc97400ffa94145e P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:46.158444 20469 raft_consensus.cc:2726] T ae24e75a0d42441ebc97400ffa94145e P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:46.155066 18985 tablet_bootstrap.cc:492] T c6378f1495144c458af49b3c19a2b67c P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:46.159389 18983 tablet_bootstrap.cc:654] T de1dfe6c82d34a4d8d02e466c55327ff P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:46.161274 18985 tablet_bootstrap.cc:654] T c6378f1495144c458af49b3c19a2b67c P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:46.161743 19489 ts_tablet_manager.cc:1888] T f5546f9edeea44d7a6c5cb954f5157ba P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:46.163738 19471 log.cc:1196] T e2d90ff17e574f88b2061b1c117d4b7d P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/e2d90ff17e574f88b2061b1c117d4b7d
I20240430 20:11:46.164136 19471 ts_tablet_manager.cc:1909] T e2d90ff17e574f88b2061b1c117d4b7d P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:46.164882 19489 log.cc:1196] T f5546f9edeea44d7a6c5cb954f5157ba P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/f5546f9edeea44d7a6c5cb954f5157ba
I20240430 20:11:46.165210 19489 ts_tablet_manager.cc:1909] T f5546f9edeea44d7a6c5cb954f5157ba P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:46.165932 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 252) took 5730 ms. Trace:
I20240430 20:11:46.166046 19489 rpcz_store.cc:276] 0430 20:11:40.435075 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:40.435122 (+ 47us) service_pool.cc:225] Handling call
0430 20:11:46.128939 (+5693817us) ts_tablet_manager.cc:1120] Deleting tablet f5546f9edeea44d7a6c5cb954f5157ba
0430 20:11:46.128950 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:46.155455 (+ 26505us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:46.161615 (+ 6160us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:46.165910 (+ 4295us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5693559}
I20240430 20:11:46.166355 19489 tablet_replica.cc:330] T 4e69ca12bd10494fb013614741bc166c P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:46.166539 19489 raft_consensus.cc:2229] T 4e69ca12bd10494fb013614741bc166c P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:46.166707 19489 raft_consensus.cc:2258] T 4e69ca12bd10494fb013614741bc166c P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:46.167218 19489 ts_tablet_manager.cc:1875] T 4e69ca12bd10494fb013614741bc166c P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:46.172988 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 228) took 6090 ms. Trace:
I20240430 20:11:46.173714 19471 rpcz_store.cc:276] 0430 20:11:40.082070 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:40.082447 (+ 377us) service_pool.cc:225] Handling call
0430 20:11:46.113114 (+6030667us) ts_tablet_manager.cc:1120] Deleting tablet e2d90ff17e574f88b2061b1c117d4b7d
0430 20:11:46.113125 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:46.129023 (+ 15898us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:46.138173 (+ 9150us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:46.172965 (+ 34792us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6001680}
I20240430 20:11:46.174804 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 233) took 5847 ms. Trace:
I20240430 20:11:46.175773 19471 rpcz_store.cc:276] 0430 20:11:40.327562 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:40.329474 (+ 1912us) service_pool.cc:225] Handling call
0430 20:11:46.174679 (+5845205us) ts_tablet_manager.cc:1120] Deleting tablet 898b2c91adf34545808df9c786e218c9
0430 20:11:46.174689 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:46.174789 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5844625}
I20240430 20:11:46.176470 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 234) took 5746 ms. Trace:
I20240430 20:11:46.177165 18983 tablet_bootstrap.cc:492] T de1dfe6c82d34a4d8d02e466c55327ff P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:46.177688 19471 rpcz_store.cc:276] 0430 20:11:40.429927 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:40.429996 (+ 69us) service_pool.cc:225] Handling call
0430 20:11:46.176363 (+5746367us) ts_tablet_manager.cc:1120] Deleting tablet d4d6c37b578c4fe1aa666b0a6b0bc7c9
0430 20:11:46.176369 (+ 6us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:46.176455 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5746140}
I20240430 20:11:46.177881 18983 ts_tablet_manager.cc:1367] T de1dfe6c82d34a4d8d02e466c55327ff P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.023s user 0.001s sys 0.005s
I20240430 20:11:46.179482 18983 raft_consensus.cc:348] T de1dfe6c82d34a4d8d02e466c55327ff P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.179737 18983 raft_consensus.cc:374] T de1dfe6c82d34a4d8d02e466c55327ff P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:46.179828 18983 raft_consensus.cc:729] T de1dfe6c82d34a4d8d02e466c55327ff P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:46.180060 18983 consensus_queue.cc:260] T de1dfe6c82d34a4d8d02e466c55327ff P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.182631 18983 ts_tablet_manager.cc:1398] T de1dfe6c82d34a4d8d02e466c55327ff P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240430 20:11:46.182957 18983 tablet_bootstrap.cc:492] T 0a16d98f700c46e29f6dd8568752ef29 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:46.184331 18983 tablet_bootstrap.cc:654] T 0a16d98f700c46e29f6dd8568752ef29 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:46.186525 19495 ts_tablet_manager.cc:1888] T 76d6e9d547ca4a848eae852944b1d117 P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:46.188361 19495 log.cc:1196] T 76d6e9d547ca4a848eae852944b1d117 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/76d6e9d547ca4a848eae852944b1d117
I20240430 20:11:46.189402 18988 tablet_bootstrap.cc:492] T eb76f60448774fe1967fffe7c6873cfc P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:46.189564 18988 ts_tablet_manager.cc:1367] T eb76f60448774fe1967fffe7c6873cfc P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.040s user 0.003s sys 0.001s
I20240430 20:11:46.189608 19495 ts_tablet_manager.cc:1909] T 76d6e9d547ca4a848eae852944b1d117 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:46.190276 18988 raft_consensus.cc:348] T eb76f60448774fe1967fffe7c6873cfc P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.190481 18988 raft_consensus.cc:374] T eb76f60448774fe1967fffe7c6873cfc P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:46.190534 18988 raft_consensus.cc:729] T eb76f60448774fe1967fffe7c6873cfc P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:46.190812 18988 consensus_queue.cc:260] T eb76f60448774fe1967fffe7c6873cfc P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.192175 19495 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41680 (request call id 370) took 1257 ms. Trace:
I20240430 20:11:46.192225 19471 tablet_replica.cc:330] T 5809e7c9c2204ba293426c24129fb16e P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:46.192535 19471 raft_consensus.cc:2229] T 5809e7c9c2204ba293426c24129fb16e P 5e1c6397f09747f091534faf93542710 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:46.192790 19471 raft_consensus.cc:2258] T 5809e7c9c2204ba293426c24129fb16e P 5e1c6397f09747f091534faf93542710 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:46.194002 19471 ts_tablet_manager.cc:1875] T 5809e7c9c2204ba293426c24129fb16e P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:46.194729 20631 raft_consensus.cc:482] T ca64de16d40f4a44b8b411f5526ba11a P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5e1c6397f09747f091534faf93542710)
I20240430 20:11:46.195065 20631 raft_consensus.cc:504] T ca64de16d40f4a44b8b411f5526ba11a P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.196122 18756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca64de16d40f4a44b8b411f5526ba11a" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:46.196312 20631 leader_election.cc:288] T ca64de16d40f4a44b8b411f5526ba11a P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:46.196362 18756 raft_consensus.cc:2379] T ca64de16d40f4a44b8b411f5526ba11a P 3c51c33d5616482cafb68e26688820f4 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2efb542aa55e439b99ccb98d1c5c884a in current term 2: Already voted for candidate 3c51c33d5616482cafb68e26688820f4 in this term.
I20240430 20:11:46.196436 19495 rpcz_store.cc:276] 0430 20:11:44.934649 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:44.934833 (+ 184us) service_pool.cc:225] Handling call
0430 20:11:46.017558 (+1082725us) ts_tablet_manager.cc:1120] Deleting tablet 76d6e9d547ca4a848eae852944b1d117
0430 20:11:46.017567 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:46.041509 (+ 23942us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:46.186361 (+144852us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:46.192153 (+ 5792us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1082555}
I20240430 20:11:46.196985 18619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca64de16d40f4a44b8b411f5526ba11a" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:46.197348 18619 raft_consensus.cc:2454] T ca64de16d40f4a44b8b411f5526ba11a P 5e1c6397f09747f091534faf93542710 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 2.
I20240430 20:11:46.197790 18415 leader_election.cc:302] T ca64de16d40f4a44b8b411f5526ba11a P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:46.198421 18988 ts_tablet_manager.cc:1398] T eb76f60448774fe1967fffe7c6873cfc P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.009s user 0.000s sys 0.002s
I20240430 20:11:46.199038 20545 raft_consensus.cc:2781] T ca64de16d40f4a44b8b411f5526ba11a P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240430 20:11:46.199112 20545 raft_consensus.cc:482] T ca64de16d40f4a44b8b411f5526ba11a P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 5e1c6397f09747f091534faf93542710)
I20240430 20:11:46.199182 20545 raft_consensus.cc:3037] T ca64de16d40f4a44b8b411f5526ba11a P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Advancing to term 2
I20240430 20:11:46.204248 18988 tablet_bootstrap.cc:492] T 2873af6da928463f9809864b8eaf0c4a P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:46.204918 19489 ts_tablet_manager.cc:1888] T 4e69ca12bd10494fb013614741bc166c P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:46.205813 19471 ts_tablet_manager.cc:1888] T 5809e7c9c2204ba293426c24129fb16e P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.2
I20240430 20:11:46.205997 19471 log.cc:1196] T 5809e7c9c2204ba293426c24129fb16e P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/5809e7c9c2204ba293426c24129fb16e
I20240430 20:11:46.207561 20545 raft_consensus.cc:504] T ca64de16d40f4a44b8b411f5526ba11a P 2efb542aa55e439b99ccb98d1c5c884a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.208243 19471 ts_tablet_manager.cc:1909] T 5809e7c9c2204ba293426c24129fb16e P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:46.208223 20545 leader_election.cc:288] T ca64de16d40f4a44b8b411f5526ba11a P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:46.209168 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 235) took 5736 ms. Trace:
I20240430 20:11:46.209306 19471 rpcz_store.cc:276] 0430 20:11:40.473181 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:40.475377 (+ 2196us) service_pool.cc:225] Handling call
0430 20:11:46.184503 (+5709126us) ts_tablet_manager.cc:1120] Deleting tablet 5809e7c9c2204ba293426c24129fb16e
0430 20:11:46.192160 (+ 7657us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:46.202717 (+ 10557us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:46.205712 (+ 2995us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:46.209146 (+ 3434us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5708910}
I20240430 20:11:46.211283 18619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca64de16d40f4a44b8b411f5526ba11a" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:46.213917 18619 raft_consensus.cc:2454] T ca64de16d40f4a44b8b411f5526ba11a P 5e1c6397f09747f091534faf93542710 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 2.
W20240430 20:11:46.215147 18558 consensus_peers.cc:485] T 76d6e9d547ca4a848eae852944b1d117 P 5e1c6397f09747f091534faf93542710 -> Peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Couldn't send request to peer 2efb542aa55e439b99ccb98d1c5c884a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 76d6e9d547ca4a848eae852944b1d117. This is attempt 1: this message will repeat every 5th retry.
I20240430 20:11:46.215420 18983 tablet_bootstrap.cc:492] T 0a16d98f700c46e29f6dd8568752ef29 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:46.215524 18415 leader_election.cc:302] T ca64de16d40f4a44b8b411f5526ba11a P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:46.216024 18983 ts_tablet_manager.cc:1367] T 0a16d98f700c46e29f6dd8568752ef29 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.033s user 0.002s sys 0.001s
I20240430 20:11:46.216701 18983 raft_consensus.cc:348] T 0a16d98f700c46e29f6dd8568752ef29 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.216965 18983 raft_consensus.cc:374] T 0a16d98f700c46e29f6dd8568752ef29 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:46.217068 18983 raft_consensus.cc:729] T 0a16d98f700c46e29f6dd8568752ef29 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:46.217335 18983 consensus_queue.cc:260] T 0a16d98f700c46e29f6dd8568752ef29 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.218091 20631 raft_consensus.cc:2781] T ca64de16d40f4a44b8b411f5526ba11a P 2efb542aa55e439b99ccb98d1c5c884a [term 2 FOLLOWER]: Leader election won for term 2
I20240430 20:11:46.218220 20631 raft_consensus.cc:686] T ca64de16d40f4a44b8b411f5526ba11a P 2efb542aa55e439b99ccb98d1c5c884a [term 2 LEADER]: Becoming Leader. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Running, Role: LEADER
I20240430 20:11:46.218446 20631 consensus_queue.cc:237] T ca64de16d40f4a44b8b411f5526ba11a P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.225510 18983 ts_tablet_manager.cc:1398] T 0a16d98f700c46e29f6dd8568752ef29 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20240430 20:11:46.233884 18983 tablet_bootstrap.cc:492] T a0baee91f5b44945b595d0e234af018e P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:46.235683 18983 tablet_bootstrap.cc:654] T a0baee91f5b44945b595d0e234af018e P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:46.236413 18985 tablet_bootstrap.cc:492] T c6378f1495144c458af49b3c19a2b67c P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:46.236554 18985 ts_tablet_manager.cc:1367] T c6378f1495144c458af49b3c19a2b67c P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.082s user 0.005s sys 0.000s
I20240430 20:11:46.237253 18985 raft_consensus.cc:348] T c6378f1495144c458af49b3c19a2b67c P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.237455 18985 raft_consensus.cc:374] T c6378f1495144c458af49b3c19a2b67c P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:46.237515 18985 raft_consensus.cc:729] T c6378f1495144c458af49b3c19a2b67c P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:46.237720 18985 consensus_queue.cc:260] T c6378f1495144c458af49b3c19a2b67c P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.238245 18985 ts_tablet_manager.cc:1398] T c6378f1495144c458af49b3c19a2b67c P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240430 20:11:46.238509 18985 tablet_bootstrap.cc:492] T 3ec3993042f5498b818bdf32d5087928 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:46.240088 18985 tablet_bootstrap.cc:654] T 3ec3993042f5498b818bdf32d5087928 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:46.241299 20545 consensus_queue.cc:1035] T ae24e75a0d42441ebc97400ffa94145e P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:46.205704 18988 tablet_bootstrap.cc:654] T 2873af6da928463f9809864b8eaf0c4a P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:46.211046 18756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca64de16d40f4a44b8b411f5526ba11a" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:46.243643 18756 raft_consensus.cc:2379] T ca64de16d40f4a44b8b411f5526ba11a P 3c51c33d5616482cafb68e26688820f4 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 2efb542aa55e439b99ccb98d1c5c884a in current term 2: Already voted for candidate 3c51c33d5616482cafb68e26688820f4 in this term.
I20240430 20:11:46.247598 19489 log.cc:1196] T 4e69ca12bd10494fb013614741bc166c P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/4e69ca12bd10494fb013614741bc166c
I20240430 20:11:46.248200 19489 ts_tablet_manager.cc:1909] T 4e69ca12bd10494fb013614741bc166c P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:46.249451 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 253) took 5814 ms. Trace:
I20240430 20:11:46.251092 19489 rpcz_store.cc:276] 0430 20:11:40.435200 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:40.435249 (+ 49us) service_pool.cc:225] Handling call
0430 20:11:46.166308 (+5731059us) ts_tablet_manager.cc:1120] Deleting tablet 4e69ca12bd10494fb013614741bc166c
0430 20:11:46.166318 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:46.170138 (+ 3820us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:46.204793 (+ 34655us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:46.249430 (+ 44637us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5730866}
I20240430 20:11:46.252311 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 254) took 5817 ms. Trace:
I20240430 20:11:46.252996 19489 rpcz_store.cc:276] 0430 20:11:40.435357 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:40.435474 (+ 117us) service_pool.cc:225] Handling call
0430 20:11:46.252203 (+5816729us) ts_tablet_manager.cc:1120] Deleting tablet 0fe1247349344863a6273a769ce8ddf6
0430 20:11:46.252215 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:46.252293 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5816565}
I20240430 20:11:46.253867 19489 tablet_replica.cc:330] T 0a9066d895744c5bb7d9532b2940603c P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:46.256700 19489 raft_consensus.cc:2229] T 0a9066d895744c5bb7d9532b2940603c P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Raft consensus shutting down.
I20240430 20:11:46.257607 18985 tablet_bootstrap.cc:492] T 3ec3993042f5498b818bdf32d5087928 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:46.258054 19489 raft_consensus.cc:2258] T 0a9066d895744c5bb7d9532b2940603c P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:46.259032 18985 ts_tablet_manager.cc:1367] T 3ec3993042f5498b818bdf32d5087928 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.021s user 0.005s sys 0.000s
I20240430 20:11:46.258850 18988 tablet_bootstrap.cc:492] T 2873af6da928463f9809864b8eaf0c4a P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:46.259258 18988 ts_tablet_manager.cc:1367] T 2873af6da928463f9809864b8eaf0c4a P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.055s user 0.007s sys 0.002s
I20240430 20:11:46.259991 18988 raft_consensus.cc:348] T 2873af6da928463f9809864b8eaf0c4a P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.260257 18988 raft_consensus.cc:374] T 2873af6da928463f9809864b8eaf0c4a P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:46.260367 18988 raft_consensus.cc:729] T 2873af6da928463f9809864b8eaf0c4a P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:46.260653 18988 consensus_queue.cc:260] T 2873af6da928463f9809864b8eaf0c4a P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.261363 18988 ts_tablet_manager.cc:1398] T 2873af6da928463f9809864b8eaf0c4a P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240430 20:11:46.262048 19489 ts_tablet_manager.cc:1875] T 0a9066d895744c5bb7d9532b2940603c P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:46.262491 18988 tablet_bootstrap.cc:492] T fbdeb4714b8c4b93ab72822c43329aae P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:46.276223 18988 tablet_bootstrap.cc:654] T fbdeb4714b8c4b93ab72822c43329aae P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:46.263301 18985 raft_consensus.cc:348] T 3ec3993042f5498b818bdf32d5087928 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.277410 18985 raft_consensus.cc:374] T 3ec3993042f5498b818bdf32d5087928 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:46.277535 18985 raft_consensus.cc:729] T 3ec3993042f5498b818bdf32d5087928 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:46.277839 18985 consensus_queue.cc:260] T 3ec3993042f5498b818bdf32d5087928 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.278425 18985 ts_tablet_manager.cc:1398] T 3ec3993042f5498b818bdf32d5087928 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.016s user 0.002s sys 0.000s
I20240430 20:11:46.282737 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 236) took 5801 ms. Trace:
I20240430 20:11:46.282923 18985 tablet_bootstrap.cc:492] T b22aaedb0a6c4fd9ad0ea8fa019001e1 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:46.283609 19471 rpcz_store.cc:276] 0430 20:11:40.481210 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:40.481529 (+ 319us) service_pool.cc:225] Handling call
0430 20:11:46.209800 (+5728271us) ts_tablet_manager.cc:1120] Deleting tablet f3711610610542128c4a84e17bce4a3e
0430 20:11:46.209810 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:46.282711 (+ 72901us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5728060}
I20240430 20:11:46.284364 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 237) took 5793 ms. Trace:
I20240430 20:11:46.284637 18985 tablet_bootstrap.cc:654] T b22aaedb0a6c4fd9ad0ea8fa019001e1 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:46.286701 18988 tablet_bootstrap.cc:492] T fbdeb4714b8c4b93ab72822c43329aae P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:46.286855 18988 ts_tablet_manager.cc:1367] T fbdeb4714b8c4b93ab72822c43329aae P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.000s
I20240430 20:11:46.287397 20631 raft_consensus.cc:482] T f6cfd089f08b47409fb2a6cf4f6b0385 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:46.287580 18988 raft_consensus.cc:348] T fbdeb4714b8c4b93ab72822c43329aae P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.287765 18988 raft_consensus.cc:374] T fbdeb4714b8c4b93ab72822c43329aae P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:46.287817 18988 raft_consensus.cc:729] T fbdeb4714b8c4b93ab72822c43329aae P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:46.287621 20631 raft_consensus.cc:504] T f6cfd089f08b47409fb2a6cf4f6b0385 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.288049 18988 consensus_queue.cc:260] T fbdeb4714b8c4b93ab72822c43329aae P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.288576 18988 ts_tablet_manager.cc:1398] T fbdeb4714b8c4b93ab72822c43329aae P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240430 20:11:46.288851 18756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6cfd089f08b47409fb2a6cf4f6b0385" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
W20240430 20:11:46.289479 18417 leader_election.cc:341] T f6cfd089f08b47409fb2a6cf4f6b0385 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:46.290326 18619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6cfd089f08b47409fb2a6cf4f6b0385" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:46.290778 18619 raft_consensus.cc:2454] T f6cfd089f08b47409fb2a6cf4f6b0385 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 0.
I20240430 20:11:46.291290 19489 ts_tablet_manager.cc:1888] T 0a9066d895744c5bb7d9532b2940603c P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:46.291312 18415 leader_election.cc:302] T f6cfd089f08b47409fb2a6cf4f6b0385 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters: 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:46.291693 20545 raft_consensus.cc:2781] T f6cfd089f08b47409fb2a6cf4f6b0385 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:46.291787 20545 raft_consensus.cc:482] T f6cfd089f08b47409fb2a6cf4f6b0385 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:46.291846 20545 raft_consensus.cc:3037] T f6cfd089f08b47409fb2a6cf4f6b0385 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:46.294430 20545 raft_consensus.cc:504] T f6cfd089f08b47409fb2a6cf4f6b0385 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.299331 20631 leader_election.cc:288] T f6cfd089f08b47409fb2a6cf4f6b0385 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:46.301134 18619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6cfd089f08b47409fb2a6cf4f6b0385" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:46.301496 19489 log.cc:1196] T 0a9066d895744c5bb7d9532b2940603c P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/0a9066d895744c5bb7d9532b2940603c
I20240430 20:11:46.301477 18619 raft_consensus.cc:3037] T f6cfd089f08b47409fb2a6cf4f6b0385 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:46.295470 20205 consensus_queue.cc:1035] T f7b0132c17d44a9ba17288e0321a7f11 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240430 20:11:46.297478 18696 consensus_peers.cc:485] T 5809e7c9c2204ba293426c24129fb16e P 3c51c33d5616482cafb68e26688820f4 -> Peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Couldn't send request to peer 5e1c6397f09747f091534faf93542710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5809e7c9c2204ba293426c24129fb16e. This is attempt 1: this message will repeat every 5th retry.
I20240430 20:11:46.301982 19489 ts_tablet_manager.cc:1909] T 0a9066d895744c5bb7d9532b2940603c P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:46.300472 18756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6cfd089f08b47409fb2a6cf4f6b0385" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:46.288852 18988 tablet_bootstrap.cc:492] T f6cfd089f08b47409fb2a6cf4f6b0385 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
W20240430 20:11:46.303699 18417 leader_election.cc:341] T f6cfd089f08b47409fb2a6cf4f6b0385 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:46.304165 18988 tablet_bootstrap.cc:654] T f6cfd089f08b47409fb2a6cf4f6b0385 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:46.305859 18985 tablet_bootstrap.cc:492] T b22aaedb0a6c4fd9ad0ea8fa019001e1 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:46.306028 18985 ts_tablet_manager.cc:1367] T b22aaedb0a6c4fd9ad0ea8fa019001e1 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.000s
I20240430 20:11:46.306934 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 255) took 5871 ms. Trace:
I20240430 20:11:46.308169 18619 raft_consensus.cc:2454] T f6cfd089f08b47409fb2a6cf4f6b0385 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 1.
I20240430 20:11:46.308634 20631 raft_consensus.cc:482] T 66dd767bd3f8494f94061fb89e5e5680 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:46.309031 20631 raft_consensus.cc:504] T 66dd767bd3f8494f94061fb89e5e5680 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.309604 18415 leader_election.cc:302] T f6cfd089f08b47409fb2a6cf4f6b0385 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters: 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:46.310156 20629 raft_consensus.cc:2781] T f6cfd089f08b47409fb2a6cf4f6b0385 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:46.310254 20629 raft_consensus.cc:686] T f6cfd089f08b47409fb2a6cf4f6b0385 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Becoming Leader. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Running, Role: LEADER
I20240430 20:11:46.310485 20629 consensus_queue.cc:237] T f6cfd089f08b47409fb2a6cf4f6b0385 P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.311209 18985 raft_consensus.cc:348] T b22aaedb0a6c4fd9ad0ea8fa019001e1 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.311365 18985 raft_consensus.cc:374] T b22aaedb0a6c4fd9ad0ea8fa019001e1 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:46.311412 18985 raft_consensus.cc:729] T b22aaedb0a6c4fd9ad0ea8fa019001e1 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:46.311589 18985 consensus_queue.cc:260] T b22aaedb0a6c4fd9ad0ea8fa019001e1 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.312103 18985 ts_tablet_manager.cc:1398] T b22aaedb0a6c4fd9ad0ea8fa019001e1 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240430 20:11:46.312430 18985 tablet_bootstrap.cc:492] T 0a16d98f700c46e29f6dd8568752ef29 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:46.314303 18985 tablet_bootstrap.cc:654] T 0a16d98f700c46e29f6dd8568752ef29 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:46.314657 19489 rpcz_store.cc:276] 0430 20:11:40.435498 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:40.435660 (+ 162us) service_pool.cc:225] Handling call
0430 20:11:46.253822 (+5818162us) ts_tablet_manager.cc:1120] Deleting tablet 0a9066d895744c5bb7d9532b2940603c
0430 20:11:46.253830 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:46.273712 (+ 19882us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:46.291176 (+ 17464us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:46.306908 (+ 15732us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5817970}
I20240430 20:11:46.315511 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 256) took 5879 ms. Trace:
I20240430 20:11:46.316546 19489 rpcz_store.cc:276] 0430 20:11:40.435625 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:40.435671 (+ 46us) service_pool.cc:225] Handling call
0430 20:11:46.315363 (+5879692us) ts_tablet_manager.cc:1120] Deleting tablet b688e03ea71940c69bda5520b62c1fc3
0430 20:11:46.315380 (+ 17us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:46.315491 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5879383}
I20240430 20:11:46.316850 18756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66dd767bd3f8494f94061fb89e5e5680" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:46.317077 19489 tablet_replica.cc:330] T 5809e7c9c2204ba293426c24129fb16e P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:46.317283 19489 raft_consensus.cc:2229] T 5809e7c9c2204ba293426c24129fb16e P 3c51c33d5616482cafb68e26688820f4 [term 2 LEADER]: Raft consensus shutting down.
W20240430 20:11:46.317471 18417 leader_election.cc:341] T 66dd767bd3f8494f94061fb89e5e5680 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:46.317687 20631 leader_election.cc:288] T 66dd767bd3f8494f94061fb89e5e5680 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:46.317982 18619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66dd767bd3f8494f94061fb89e5e5680" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:46.318281 18619 raft_consensus.cc:2454] T 66dd767bd3f8494f94061fb89e5e5680 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 0.
I20240430 20:11:46.318740 18415 leader_election.cc:302] T 66dd767bd3f8494f94061fb89e5e5680 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters: 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:46.318938 18988 tablet_bootstrap.cc:492] T f6cfd089f08b47409fb2a6cf4f6b0385 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:46.309918 20545 leader_election.cc:288] T f6cfd089f08b47409fb2a6cf4f6b0385 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:46.319272 18988 ts_tablet_manager.cc:1367] T f6cfd089f08b47409fb2a6cf4f6b0385 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.030s user 0.003s sys 0.000s
I20240430 20:11:46.319679 20545 raft_consensus.cc:2781] T 66dd767bd3f8494f94061fb89e5e5680 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:46.319804 20545 raft_consensus.cc:482] T 66dd767bd3f8494f94061fb89e5e5680 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:46.319923 20545 raft_consensus.cc:3037] T 66dd767bd3f8494f94061fb89e5e5680 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:46.320031 18988 raft_consensus.cc:348] T f6cfd089f08b47409fb2a6cf4f6b0385 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.320225 18988 raft_consensus.cc:374] T f6cfd089f08b47409fb2a6cf4f6b0385 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:46.320288 18988 raft_consensus.cc:729] T f6cfd089f08b47409fb2a6cf4f6b0385 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:46.320525 18988 consensus_queue.cc:260] T f6cfd089f08b47409fb2a6cf4f6b0385 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.321015 18988 ts_tablet_manager.cc:1398] T f6cfd089f08b47409fb2a6cf4f6b0385 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240430 20:11:46.321275 18988 tablet_bootstrap.cc:492] T 66dd767bd3f8494f94061fb89e5e5680 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:46.322845 18988 tablet_bootstrap.cc:654] T 66dd767bd3f8494f94061fb89e5e5680 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:46.286831 19471 rpcz_store.cc:276] 0430 20:11:40.490857 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:40.490921 (+ 64us) service_pool.cc:225] Handling call
0430 20:11:46.284236 (+5793315us) ts_tablet_manager.cc:1120] Deleting tablet d7c1fc7e81364fb890bbc13d7fb6848b
0430 20:11:46.284249 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:46.284347 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5793122}
I20240430 20:11:46.324673 19489 raft_consensus.cc:2258] T 5809e7c9c2204ba293426c24129fb16e P 3c51c33d5616482cafb68e26688820f4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:46.325349 19489 ts_tablet_manager.cc:1875] T 5809e7c9c2204ba293426c24129fb16e P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:46.325397 19471 tablet_replica.cc:330] T fd40967ab4a04001819e16c06a6c4300 P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:46.326210 19471 raft_consensus.cc:2229] T fd40967ab4a04001819e16c06a6c4300 P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Raft consensus shutting down.
I20240430 20:11:46.328486 19471 raft_consensus.cc:2258] T fd40967ab4a04001819e16c06a6c4300 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:46.326810 20545 raft_consensus.cc:504] T 66dd767bd3f8494f94061fb89e5e5680 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.331092 18756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66dd767bd3f8494f94061fb89e5e5680" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
W20240430 20:11:46.331728 18417 leader_election.cc:341] T 66dd767bd3f8494f94061fb89e5e5680 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:46.332801 19471 ts_tablet_manager.cc:1875] T fd40967ab4a04001819e16c06a6c4300 P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:46.334534 18619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66dd767bd3f8494f94061fb89e5e5680" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:46.334734 18619 raft_consensus.cc:3037] T 66dd767bd3f8494f94061fb89e5e5680 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:46.337479 18619 raft_consensus.cc:2454] T 66dd767bd3f8494f94061fb89e5e5680 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 1.
I20240430 20:11:46.339444 18988 tablet_bootstrap.cc:492] T 66dd767bd3f8494f94061fb89e5e5680 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:46.339583 18988 ts_tablet_manager.cc:1367] T 66dd767bd3f8494f94061fb89e5e5680 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.000s
I20240430 20:11:46.340332 18988 raft_consensus.cc:348] T 66dd767bd3f8494f94061fb89e5e5680 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.340530 18988 raft_consensus.cc:374] T 66dd767bd3f8494f94061fb89e5e5680 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:46.340582 18988 raft_consensus.cc:729] T 66dd767bd3f8494f94061fb89e5e5680 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:46.341734 20545 leader_election.cc:288] T 66dd767bd3f8494f94061fb89e5e5680 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:46.345288 19489 ts_tablet_manager.cc:1888] T 5809e7c9c2204ba293426c24129fb16e P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.2
I20240430 20:11:46.345163 18415 leader_election.cc:302] T 66dd767bd3f8494f94061fb89e5e5680 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters: 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:46.345827 20545 raft_consensus.cc:2781] T 66dd767bd3f8494f94061fb89e5e5680 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:46.347503 20545 raft_consensus.cc:686] T 66dd767bd3f8494f94061fb89e5e5680 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Becoming Leader. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Running, Role: LEADER
I20240430 20:11:46.347971 20545 consensus_queue.cc:237] T 66dd767bd3f8494f94061fb89e5e5680 P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.352761 19489 log.cc:1196] T 5809e7c9c2204ba293426c24129fb16e P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/5809e7c9c2204ba293426c24129fb16e
I20240430 20:11:46.353430 19489 ts_tablet_manager.cc:1909] T 5809e7c9c2204ba293426c24129fb16e P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:46.354787 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 257) took 5882 ms. Trace:
I20240430 20:11:46.358731 18985 tablet_bootstrap.cc:492] T 0a16d98f700c46e29f6dd8568752ef29 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:46.346019 18988 consensus_queue.cc:260] T 66dd767bd3f8494f94061fb89e5e5680 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.358915 18985 ts_tablet_manager.cc:1367] T 0a16d98f700c46e29f6dd8568752ef29 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.047s user 0.008s sys 0.000s
I20240430 20:11:46.359400 18988 ts_tablet_manager.cc:1398] T 66dd767bd3f8494f94061fb89e5e5680 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.020s user 0.002s sys 0.000s
I20240430 20:11:46.359804 18988 tablet_bootstrap.cc:492] T de1dfe6c82d34a4d8d02e466c55327ff P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:46.359717 18985 raft_consensus.cc:348] T 0a16d98f700c46e29f6dd8568752ef29 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.359923 18985 raft_consensus.cc:374] T 0a16d98f700c46e29f6dd8568752ef29 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:46.359982 18985 raft_consensus.cc:729] T 0a16d98f700c46e29f6dd8568752ef29 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:46.360119 20205 raft_consensus.cc:482] T 2873af6da928463f9809864b8eaf0c4a P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:46.360203 18985 consensus_queue.cc:260] T 0a16d98f700c46e29f6dd8568752ef29 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.360248 20205 raft_consensus.cc:504] T 2873af6da928463f9809864b8eaf0c4a P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.360774 18985 ts_tablet_manager.cc:1398] T 0a16d98f700c46e29f6dd8568752ef29 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240430 20:11:46.360918 20205 leader_election.cc:288] T 2873af6da928463f9809864b8eaf0c4a P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:46.361250 18985 tablet_bootstrap.cc:492] T 47bf64ecc41b445da4ff697e141243dd P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:46.362820 18988 tablet_bootstrap.cc:654] T de1dfe6c82d34a4d8d02e466c55327ff P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:46.362820 18985 tablet_bootstrap.cc:654] T 47bf64ecc41b445da4ff697e141243dd P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:46.362752 18474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2873af6da928463f9809864b8eaf0c4a" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:46.363512 18474 raft_consensus.cc:2454] T 2873af6da928463f9809864b8eaf0c4a P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:46.359544 19489 rpcz_store.cc:276] 0430 20:11:40.472718 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:40.472885 (+ 167us) service_pool.cc:225] Handling call
0430 20:11:46.317027 (+5844142us) ts_tablet_manager.cc:1120] Deleting tablet 5809e7c9c2204ba293426c24129fb16e
0430 20:11:46.317037 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:46.341401 (+ 24364us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:46.345193 (+ 3792us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:46.354765 (+ 9572us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5838660}
I20240430 20:11:46.364301 18699 leader_election.cc:302] T 2873af6da928463f9809864b8eaf0c4a P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters:
I20240430 20:11:46.370206 20198 raft_consensus.cc:2781] T 2873af6da928463f9809864b8eaf0c4a P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:46.370250 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 258) took 5891 ms. Trace:
I20240430 20:11:46.370307 20198 raft_consensus.cc:482] T 2873af6da928463f9809864b8eaf0c4a P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:46.370383 20198 raft_consensus.cc:3037] T 2873af6da928463f9809864b8eaf0c4a P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:46.373131 20198 raft_consensus.cc:504] T 2873af6da928463f9809864b8eaf0c4a P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.374581 18620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2873af6da928463f9809864b8eaf0c4a" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:46.374814 18620 raft_consensus.cc:3037] T 2873af6da928463f9809864b8eaf0c4a P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:46.374785 18478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2873af6da928463f9809864b8eaf0c4a" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:46.374990 18478 raft_consensus.cc:3037] T 2873af6da928463f9809864b8eaf0c4a P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:46.377408 18620 raft_consensus.cc:2454] T 2873af6da928463f9809864b8eaf0c4a P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:46.377554 18478 raft_consensus.cc:2454] T 2873af6da928463f9809864b8eaf0c4a P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:46.378062 18696 leader_election.cc:302] T 2873af6da928463f9809864b8eaf0c4a P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:46.378321 20469 raft_consensus.cc:2781] T 2873af6da928463f9809864b8eaf0c4a P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election won for term 1
W20240430 20:11:46.378423 18699 consensus_peers.cc:485] T f7b0132c17d44a9ba17288e0321a7f11 P 3c51c33d5616482cafb68e26688820f4 -> Peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Couldn't send request to peer 2efb542aa55e439b99ccb98d1c5c884a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f7b0132c17d44a9ba17288e0321a7f11. This is attempt 1: this message will repeat every 5th retry.
I20240430 20:11:46.378444 20469 raft_consensus.cc:686] T 2873af6da928463f9809864b8eaf0c4a P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Becoming Leader. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Running, Role: LEADER
I20240430 20:11:46.378554 20198 leader_election.cc:288] T 2873af6da928463f9809864b8eaf0c4a P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Requested vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:46.378780 20469 consensus_queue.cc:237] T 2873af6da928463f9809864b8eaf0c4a P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.379992 18988 tablet_bootstrap.cc:492] T de1dfe6c82d34a4d8d02e466c55327ff P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:46.380132 18988 ts_tablet_manager.cc:1367] T de1dfe6c82d34a4d8d02e466c55327ff P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.002s
I20240430 20:11:46.381253 18988 raft_consensus.cc:348] T de1dfe6c82d34a4d8d02e466c55327ff P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.381439 18988 raft_consensus.cc:374] T de1dfe6c82d34a4d8d02e466c55327ff P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:46.381490 18988 raft_consensus.cc:729] T de1dfe6c82d34a4d8d02e466c55327ff P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:46.381711 18988 consensus_queue.cc:260] T de1dfe6c82d34a4d8d02e466c55327ff P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.384748 18988 ts_tablet_manager.cc:1398] T de1dfe6c82d34a4d8d02e466c55327ff P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240430 20:11:46.386701 18988 tablet_bootstrap.cc:492] T a0baee91f5b44945b595d0e234af018e P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:46.370317 19489 rpcz_store.cc:276] 0430 20:11:40.478558 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:40.478641 (+ 83us) service_pool.cc:225] Handling call
0430 20:11:46.370110 (+5891469us) ts_tablet_manager.cc:1120] Deleting tablet f3711610610542128c4a84e17bce4a3e
0430 20:11:46.370122 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:46.370230 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5891251}
I20240430 20:11:46.387601 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 259) took 5897 ms. Trace:
I20240430 20:11:46.387770 19489 rpcz_store.cc:276] 0430 20:11:40.490288 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:40.490976 (+ 688us) service_pool.cc:225] Handling call
0430 20:11:46.387467 (+5896491us) ts_tablet_manager.cc:1120] Deleting tablet d7c1fc7e81364fb890bbc13d7fb6848b
0430 20:11:46.387479 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:46.387584 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5896270}
I20240430 20:11:46.388270 19489 tablet_replica.cc:330] T fd40967ab4a04001819e16c06a6c4300 P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:46.388325 18988 tablet_bootstrap.cc:654] T a0baee91f5b44945b595d0e234af018e P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:46.388579 19489 raft_consensus.cc:2229] T fd40967ab4a04001819e16c06a6c4300 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:46.388805 19489 raft_consensus.cc:2258] T fd40967ab4a04001819e16c06a6c4300 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:46.389428 19489 ts_tablet_manager.cc:1875] T fd40967ab4a04001819e16c06a6c4300 P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:46.396004 18983 tablet_bootstrap.cc:492] T a0baee91f5b44945b595d0e234af018e P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:46.396203 18983 ts_tablet_manager.cc:1367] T a0baee91f5b44945b595d0e234af018e P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.162s user 0.012s sys 0.028s
I20240430 20:11:46.396315 18619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2873af6da928463f9809864b8eaf0c4a" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:46.396564 18619 raft_consensus.cc:2362] T 2873af6da928463f9809864b8eaf0c4a P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1. Re-sending same reply.
I20240430 20:11:46.397845 18983 raft_consensus.cc:348] T a0baee91f5b44945b595d0e234af018e P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.398037 18983 raft_consensus.cc:374] T a0baee91f5b44945b595d0e234af018e P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:46.398078 18983 raft_consensus.cc:729] T a0baee91f5b44945b595d0e234af018e P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:46.398250 18983 consensus_queue.cc:260] T a0baee91f5b44945b595d0e234af018e P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.401734 18988 tablet_bootstrap.cc:492] T a0baee91f5b44945b595d0e234af018e P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:46.401885 18988 ts_tablet_manager.cc:1367] T a0baee91f5b44945b595d0e234af018e P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.000s
I20240430 20:11:46.402310 18983 ts_tablet_manager.cc:1398] T a0baee91f5b44945b595d0e234af018e P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.006s user 0.002s sys 0.002s
I20240430 20:11:46.402626 18988 raft_consensus.cc:348] T a0baee91f5b44945b595d0e234af018e P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.402822 18988 raft_consensus.cc:374] T a0baee91f5b44945b595d0e234af018e P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:46.402884 18988 raft_consensus.cc:729] T a0baee91f5b44945b595d0e234af018e P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:46.403112 18988 consensus_queue.cc:260] T a0baee91f5b44945b595d0e234af018e P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.403725 18988 ts_tablet_manager.cc:1398] T a0baee91f5b44945b595d0e234af018e P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240430 20:11:46.404014 18988 tablet_bootstrap.cc:492] T 3ec3993042f5498b818bdf32d5087928 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:46.404379 18983 tablet_bootstrap.cc:492] T b22aaedb0a6c4fd9ad0ea8fa019001e1 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:46.406064 18983 tablet_bootstrap.cc:654] T b22aaedb0a6c4fd9ad0ea8fa019001e1 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:46.407297 18988 tablet_bootstrap.cc:654] T 3ec3993042f5498b818bdf32d5087928 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:46.407868 20545 raft_consensus.cc:482] T fbdeb4714b8c4b93ab72822c43329aae P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:46.408087 20545 raft_consensus.cc:504] T fbdeb4714b8c4b93ab72822c43329aae P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.409232 18750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbdeb4714b8c4b93ab72822c43329aae" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:46.409436 18750 raft_consensus.cc:2454] T fbdeb4714b8c4b93ab72822c43329aae P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 0.
I20240430 20:11:46.409817 18417 leader_election.cc:302] T fbdeb4714b8c4b93ab72822c43329aae P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters:
I20240430 20:11:46.410180 20631 raft_consensus.cc:2781] T fbdeb4714b8c4b93ab72822c43329aae P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:46.410255 20631 raft_consensus.cc:482] T fbdeb4714b8c4b93ab72822c43329aae P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:46.410316 20631 raft_consensus.cc:3037] T fbdeb4714b8c4b93ab72822c43329aae P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:46.413053 20631 raft_consensus.cc:504] T fbdeb4714b8c4b93ab72822c43329aae P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.414080 18750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbdeb4714b8c4b93ab72822c43329aae" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:46.414268 18750 raft_consensus.cc:3037] T fbdeb4714b8c4b93ab72822c43329aae P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:46.414567 20631 leader_election.cc:288] T fbdeb4714b8c4b93ab72822c43329aae P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:46.414965 20545 leader_election.cc:288] T fbdeb4714b8c4b93ab72822c43329aae P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:46.414901 18623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbdeb4714b8c4b93ab72822c43329aae" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:46.415099 18623 raft_consensus.cc:3037] T fbdeb4714b8c4b93ab72822c43329aae P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:46.416177 19471 ts_tablet_manager.cc:1888] T fd40967ab4a04001819e16c06a6c4300 P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:46.416270 19471 log.cc:1196] T fd40967ab4a04001819e16c06a6c4300 P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/fd40967ab4a04001819e16c06a6c4300
I20240430 20:11:46.416568 19471 ts_tablet_manager.cc:1909] T fd40967ab4a04001819e16c06a6c4300 P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:46.416929 18750 raft_consensus.cc:2454] T fbdeb4714b8c4b93ab72822c43329aae P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 1.
I20240430 20:11:46.417407 18417 leader_election.cc:302] T fbdeb4714b8c4b93ab72822c43329aae P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters:
I20240430 20:11:46.417428 20683 raft_consensus.cc:482] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader b632d6ef4c104a058a6db208024cab33)
I20240430 20:11:46.417604 20683 raft_consensus.cc:504] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b3f98793354c7aaa5c106e8ec31ed8" member_type: VOTER last_known_addr { host: "127.17.177.62" port: 36441 } } peers { permanent_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e" member_type: VOTER last_known_addr { host: "127.17.177.61" port: 34777 } } peers { permanent_uuid: "b632d6ef4c104a058a6db208024cab33" member_type: VOTER last_known_addr { host: "127.17.177.60" port: 37225 } }
I20240430 20:11:46.417764 20545 raft_consensus.cc:2781] T fbdeb4714b8c4b93ab72822c43329aae P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:46.418251 18623 raft_consensus.cc:2454] T fbdeb4714b8c4b93ab72822c43329aae P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 1.
I20240430 20:11:46.418493 20545 raft_consensus.cc:686] T fbdeb4714b8c4b93ab72822c43329aae P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Becoming Leader. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Running, Role: LEADER
I20240430 20:11:46.414932 18619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbdeb4714b8c4b93ab72822c43329aae" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:46.418890 20545 consensus_queue.cc:237] T fbdeb4714b8c4b93ab72822c43329aae P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.419071 18619 raft_consensus.cc:2362] T fbdeb4714b8c4b93ab72822c43329aae P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 1. Re-sending same reply.
I20240430 20:11:46.420480 18235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88b3f98793354c7aaa5c106e8ec31ed8" candidate_term: 6 candidate_status { last_received { term: 5 index: 749 } } ignore_live_leader: false dest_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e" is_pre_election: true
I20240430 20:11:46.420756 18235 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 89df84ccf5394c9bb60d4e4ed872fc2e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88b3f98793354c7aaa5c106e8ec31ed8 in term 5.
I20240430 20:11:46.421447 19899 leader_election.cc:302] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b3f98793354c7aaa5c106e8ec31ed8, 89df84ccf5394c9bb60d4e4ed872fc2e; no voters:
I20240430 20:11:46.423868 20198 raft_consensus.cc:482] T 06e82c706b9c4f35bf53faddf57b3fd7 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:46.424082 20198 raft_consensus.cc:504] T 06e82c706b9c4f35bf53faddf57b3fd7 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.424804 20198 leader_election.cc:288] T 06e82c706b9c4f35bf53faddf57b3fd7 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:46.426455 18474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06e82c706b9c4f35bf53faddf57b3fd7" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:46.426776 18474 raft_consensus.cc:2454] T 06e82c706b9c4f35bf53faddf57b3fd7 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:46.427207 18699 leader_election.cc:302] T 06e82c706b9c4f35bf53faddf57b3fd7 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters:
I20240430 20:11:46.428577 18619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06e82c706b9c4f35bf53faddf57b3fd7" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:46.428699 20469 raft_consensus.cc:2781] T 06e82c706b9c4f35bf53faddf57b3fd7 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:46.428846 20469 raft_consensus.cc:482] T 06e82c706b9c4f35bf53faddf57b3fd7 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:46.428970 20469 raft_consensus.cc:3037] T 06e82c706b9c4f35bf53faddf57b3fd7 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:46.429121 18619 raft_consensus.cc:2454] T 06e82c706b9c4f35bf53faddf57b3fd7 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:46.430681 18985 tablet_bootstrap.cc:492] T 47bf64ecc41b445da4ff697e141243dd P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:46.431013 18985 ts_tablet_manager.cc:1367] T 47bf64ecc41b445da4ff697e141243dd P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.070s user 0.004s sys 0.000s
I20240430 20:11:46.432128 18985 raft_consensus.cc:348] T 47bf64ecc41b445da4ff697e141243dd P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.432615 20686 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240430 20:11:46.432734 20686 raft_consensus.cc:482] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8 [term 5 FOLLOWER]: Starting leader election (detected failure of leader b632d6ef4c104a058a6db208024cab33)
I20240430 20:11:46.432878 20686 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8 [term 5 FOLLOWER]: Advancing to term 6
I20240430 20:11:46.434675 18985 raft_consensus.cc:374] T 47bf64ecc41b445da4ff697e141243dd P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:46.435012 18985 raft_consensus.cc:729] T 47bf64ecc41b445da4ff697e141243dd P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:46.435474 18985 consensus_queue.cc:260] T 47bf64ecc41b445da4ff697e141243dd P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.436043 20469 raft_consensus.cc:504] T 06e82c706b9c4f35bf53faddf57b3fd7 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.436714 18985 ts_tablet_manager.cc:1398] T 47bf64ecc41b445da4ff697e141243dd P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240430 20:11:46.437018 18985 tablet_bootstrap.cc:492] T 8e33df70a1504dbd852fe0b3c3f61622 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:46.438802 20683 leader_election.cc:288] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 89df84ccf5394c9bb60d4e4ed872fc2e (127.17.177.61:34777), b632d6ef4c104a058a6db208024cab33 (127.17.177.60:37225)
I20240430 20:11:46.439962 18988 tablet_bootstrap.cc:492] T 3ec3993042f5498b818bdf32d5087928 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:46.440168 18988 ts_tablet_manager.cc:1367] T 3ec3993042f5498b818bdf32d5087928 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.036s user 0.004s sys 0.002s
I20240430 20:11:46.440234 20685 raft_consensus.cc:482] T 00000000000000000000000000000000 P 89df84ccf5394c9bb60d4e4ed872fc2e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader b632d6ef4c104a058a6db208024cab33)
I20240430 20:11:46.440421 20685 raft_consensus.cc:504] T 00000000000000000000000000000000 P 89df84ccf5394c9bb60d4e4ed872fc2e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b3f98793354c7aaa5c106e8ec31ed8" member_type: VOTER last_known_addr { host: "127.17.177.62" port: 36441 } } peers { permanent_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e" member_type: VOTER last_known_addr { host: "127.17.177.61" port: 34777 } } peers { permanent_uuid: "b632d6ef4c104a058a6db208024cab33" member_type: VOTER last_known_addr { host: "127.17.177.60" port: 37225 } }
I20240430 20:11:46.440683 18983 tablet_bootstrap.cc:492] T b22aaedb0a6c4fd9ad0ea8fa019001e1 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:46.440837 18983 ts_tablet_manager.cc:1367] T b22aaedb0a6c4fd9ad0ea8fa019001e1 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.037s user 0.004s sys 0.000s
I20240430 20:11:46.441246 18988 raft_consensus.cc:348] T 3ec3993042f5498b818bdf32d5087928 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.441547 18988 raft_consensus.cc:374] T 3ec3993042f5498b818bdf32d5087928 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:46.441654 18988 raft_consensus.cc:729] T 3ec3993042f5498b818bdf32d5087928 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:46.441608 18983 raft_consensus.cc:348] T b22aaedb0a6c4fd9ad0ea8fa019001e1 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.441941 18983 raft_consensus.cc:374] T b22aaedb0a6c4fd9ad0ea8fa019001e1 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:46.442001 18983 raft_consensus.cc:729] T b22aaedb0a6c4fd9ad0ea8fa019001e1 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:46.442032 18988 consensus_queue.cc:260] T 3ec3993042f5498b818bdf32d5087928 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.443459 18619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06e82c706b9c4f35bf53faddf57b3fd7" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:46.444070 18619 raft_consensus.cc:3037] T 06e82c706b9c4f35bf53faddf57b3fd7 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:46.445408 18988 ts_tablet_manager.cc:1398] T 3ec3993042f5498b818bdf32d5087928 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240430 20:11:46.445698 18988 tablet_bootstrap.cc:492] T c6378f1495144c458af49b3c19a2b67c P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:46.446046 18983 consensus_queue.cc:260] T b22aaedb0a6c4fd9ad0ea8fa019001e1 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.447934 18619 raft_consensus.cc:2454] T 06e82c706b9c4f35bf53faddf57b3fd7 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:46.448612 18696 leader_election.cc:302] T 06e82c706b9c4f35bf53faddf57b3fd7 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:46.449059 20198 raft_consensus.cc:2781] T 06e82c706b9c4f35bf53faddf57b3fd7 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:46.449182 20198 raft_consensus.cc:686] T 06e82c706b9c4f35bf53faddf57b3fd7 P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Becoming Leader. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Running, Role: LEADER
I20240430 20:11:46.449469 20198 consensus_queue.cc:237] T 06e82c706b9c4f35bf53faddf57b3fd7 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.450276 18988 tablet_bootstrap.cc:654] T c6378f1495144c458af49b3c19a2b67c P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:46.451959 20686 raft_consensus.cc:504] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b3f98793354c7aaa5c106e8ec31ed8" member_type: VOTER last_known_addr { host: "127.17.177.62" port: 36441 } } peers { permanent_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e" member_type: VOTER last_known_addr { host: "127.17.177.61" port: 34777 } } peers { permanent_uuid: "b632d6ef4c104a058a6db208024cab33" member_type: VOTER last_known_addr { host: "127.17.177.60" port: 37225 } }
I20240430 20:11:46.452178 18983 ts_tablet_manager.cc:1398] T b22aaedb0a6c4fd9ad0ea8fa019001e1 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240430 20:11:46.454452 20205 raft_consensus.cc:482] T 06d28b6f48a34b17b0d7fb3102df2f4c P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:46.454649 20205 raft_consensus.cc:504] T 06d28b6f48a34b17b0d7fb3102df2f4c P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.454738 18235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88b3f98793354c7aaa5c106e8ec31ed8" candidate_term: 6 candidate_status { last_received { term: 5 index: 749 } } ignore_live_leader: false dest_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e"
I20240430 20:11:46.454946 18235 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 89df84ccf5394c9bb60d4e4ed872fc2e [term 5 FOLLOWER]: Advancing to term 6
I20240430 20:11:46.454964 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 238) took 5938 ms. Trace:
I20240430 20:11:46.455071 19471 rpcz_store.cc:276] 0430 20:11:40.516999 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:40.517825 (+ 826us) service_pool.cc:225] Handling call
0430 20:11:46.325328 (+5807503us) ts_tablet_manager.cc:1120] Deleting tablet fd40967ab4a04001819e16c06a6c4300
0430 20:11:46.325340 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:46.357004 (+ 31664us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:46.416062 (+ 59058us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:46.454939 (+ 38877us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5749684}
I20240430 20:11:46.455274 20205 leader_election.cc:288] T 06d28b6f48a34b17b0d7fb3102df2f4c P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:46.455510 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 239) took 5884 ms. Trace:
I20240430 20:11:46.455579 19471 rpcz_store.cc:276] 0430 20:11:40.571329 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:40.571409 (+ 80us) service_pool.cc:225] Handling call
0430 20:11:46.455384 (+5883975us) ts_tablet_manager.cc:1120] Deleting tablet e4cffa295d8c401aa8415ce7e8f56526
0430 20:11:46.455394 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:46.455493 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5876746}
I20240430 20:11:46.456022 18478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06d28b6f48a34b17b0d7fb3102df2f4c" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:46.456212 18478 raft_consensus.cc:2454] T 06d28b6f48a34b17b0d7fb3102df2f4c P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
W20240430 20:11:46.456633 18696 leader_election.cc:341] T 06d28b6f48a34b17b0d7fb3102df2f4c P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Not found: Tablet not found: 06d28b6f48a34b17b0d7fb3102df2f4c
I20240430 20:11:46.456799 18699 leader_election.cc:302] T 06d28b6f48a34b17b0d7fb3102df2f4c P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters: 5e1c6397f09747f091534faf93542710
I20240430 20:11:46.456998 19471 tablet_replica.cc:330] T a9f37f0f7ce8425f9b24699513d41d61 P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:46.457149 20587 raft_consensus.cc:482] T 4f6f9ae722da49e58c9b72107f6da2fa P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:46.457235 19471 raft_consensus.cc:2229] T a9f37f0f7ce8425f9b24699513d41d61 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:46.457265 20587 raft_consensus.cc:504] T 4f6f9ae722da49e58c9b72107f6da2fa P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.457367 19471 raft_consensus.cc:2258] T a9f37f0f7ce8425f9b24699513d41d61 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:46.457733 18235 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 89df84ccf5394c9bb60d4e4ed872fc2e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b3f98793354c7aaa5c106e8ec31ed8 in term 6.
I20240430 20:11:46.457805 20587 leader_election.cc:288] T 4f6f9ae722da49e58c9b72107f6da2fa P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:46.457944 20587 raft_consensus.cc:2781] T 06d28b6f48a34b17b0d7fb3102df2f4c P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:46.457970 19471 ts_tablet_manager.cc:1875] T a9f37f0f7ce8425f9b24699513d41d61 P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:46.458003 20587 raft_consensus.cc:482] T 06d28b6f48a34b17b0d7fb3102df2f4c P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:46.458068 20587 raft_consensus.cc:3037] T 06d28b6f48a34b17b0d7fb3102df2f4c P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:46.459198 20685 leader_election.cc:288] T 00000000000000000000000000000000 P 89df84ccf5394c9bb60d4e4ed872fc2e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 88b3f98793354c7aaa5c106e8ec31ed8 (127.17.177.62:36441), b632d6ef4c104a058a6db208024cab33 (127.17.177.60:37225)
I20240430 20:11:46.459589 18983 tablet_bootstrap.cc:492] T 283e5e73cbc04daba0de19f4541f9fd0 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:46.460723 20587 raft_consensus.cc:504] T 06d28b6f48a34b17b0d7fb3102df2f4c P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.461264 18983 tablet_bootstrap.cc:654] T 283e5e73cbc04daba0de19f4541f9fd0 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:46.461321 20587 leader_election.cc:288] T 06d28b6f48a34b17b0d7fb3102df2f4c P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Requested vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:46.462484 18478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f6f9ae722da49e58c9b72107f6da2fa" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:46.462651 18985 tablet_bootstrap.cc:654] T 8e33df70a1504dbd852fe0b3c3f61622 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:46.462756 18478 raft_consensus.cc:2379] T 4f6f9ae722da49e58c9b72107f6da2fa P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c51c33d5616482cafb68e26688820f4 in current term 1: Already voted for candidate 5e1c6397f09747f091534faf93542710 in this term.
I20240430 20:11:46.463171 18474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06d28b6f48a34b17b0d7fb3102df2f4c" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:46.463330 18474 raft_consensus.cc:3037] T 06d28b6f48a34b17b0d7fb3102df2f4c P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:46.464062 19899 leader_election.cc:302] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b3f98793354c7aaa5c106e8ec31ed8, 89df84ccf5394c9bb60d4e4ed872fc2e; no voters:
I20240430 20:11:46.464457 20683 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8 [term 6 FOLLOWER]: Leader election won for term 6
I20240430 20:11:46.466017 18474 raft_consensus.cc:2454] T 06d28b6f48a34b17b0d7fb3102df2f4c P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:46.467038 20686 leader_election.cc:288] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8 [CANDIDATE]: Term 6 election: Requested vote from peers 89df84ccf5394c9bb60d4e4ed872fc2e (127.17.177.61:34777), b632d6ef4c104a058a6db208024cab33 (127.17.177.60:37225)
I20240430 20:11:46.467212 19980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e" candidate_term: 6 candidate_status { last_received { term: 5 index: 749 } } ignore_live_leader: false dest_uuid: "88b3f98793354c7aaa5c106e8ec31ed8" is_pre_election: true
I20240430 20:11:46.467267 18623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f6f9ae722da49e58c9b72107f6da2fa" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
W20240430 20:11:46.467759 18696 leader_election.cc:341] T 06d28b6f48a34b17b0d7fb3102df2f4c P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Not found: Tablet not found: 06d28b6f48a34b17b0d7fb3102df2f4c
I20240430 20:11:46.467893 18699 leader_election.cc:302] T 06d28b6f48a34b17b0d7fb3102df2f4c P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters: 5e1c6397f09747f091534faf93542710
I20240430 20:11:46.467954 18696 leader_election.cc:302] T 4f6f9ae722da49e58c9b72107f6da2fa P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710
I20240430 20:11:46.468469 20205 raft_consensus.cc:2781] T 06d28b6f48a34b17b0d7fb3102df2f4c P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:46.468775 19489 ts_tablet_manager.cc:1888] T fd40967ab4a04001819e16c06a6c4300 P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:46.471312 19489 log.cc:1196] T fd40967ab4a04001819e16c06a6c4300 P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/fd40967ab4a04001819e16c06a6c4300
I20240430 20:11:46.471724 19489 ts_tablet_manager.cc:1909] T fd40967ab4a04001819e16c06a6c4300 P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:46.476627 18985 tablet_bootstrap.cc:492] T 8e33df70a1504dbd852fe0b3c3f61622 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:46.476806 18985 ts_tablet_manager.cc:1367] T 8e33df70a1504dbd852fe0b3c3f61622 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.040s user 0.003s sys 0.000s
I20240430 20:11:46.477205 20205 raft_consensus.cc:686] T 06d28b6f48a34b17b0d7fb3102df2f4c P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Becoming Leader. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Running, Role: LEADER
I20240430 20:11:46.477437 20693 raft_consensus.cc:3037] T 4f6f9ae722da49e58c9b72107f6da2fa P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:46.477764 20205 consensus_queue.cc:237] T 06d28b6f48a34b17b0d7fb3102df2f4c P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.477766 18985 raft_consensus.cc:348] T 8e33df70a1504dbd852fe0b3c3f61622 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.477993 18985 raft_consensus.cc:374] T 8e33df70a1504dbd852fe0b3c3f61622 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:46.478066 18985 raft_consensus.cc:729] T 8e33df70a1504dbd852fe0b3c3f61622 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:46.478292 18985 consensus_queue.cc:260] T 8e33df70a1504dbd852fe0b3c3f61622 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.479476 18988 tablet_bootstrap.cc:492] T c6378f1495144c458af49b3c19a2b67c P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:46.479614 18988 ts_tablet_manager.cc:1367] T c6378f1495144c458af49b3c19a2b67c P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.034s user 0.004s sys 0.002s
I20240430 20:11:46.480273 18988 raft_consensus.cc:348] T c6378f1495144c458af49b3c19a2b67c P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.481288 20683 raft_consensus.cc:686] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8 [term 6 LEADER]: Becoming Leader. State: Replica: 88b3f98793354c7aaa5c106e8ec31ed8, State: Running, Role: LEADER
I20240430 20:11:46.482002 18985 ts_tablet_manager.cc:1398] T 8e33df70a1504dbd852fe0b3c3f61622 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240430 20:11:46.481956 20683 consensus_queue.cc:237] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 748, Committed index: 748, Last appended: 5.749, Last appended by leader: 749, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b3f98793354c7aaa5c106e8ec31ed8" member_type: VOTER last_known_addr { host: "127.17.177.62" port: 36441 } } peers { permanent_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e" member_type: VOTER last_known_addr { host: "127.17.177.61" port: 34777 } } peers { permanent_uuid: "b632d6ef4c104a058a6db208024cab33" member_type: VOTER last_known_addr { host: "127.17.177.60" port: 37225 } }
I20240430 20:11:46.482283 18985 tablet_bootstrap.cc:492] T 246fb63a5e9842df8263800e28a5935c P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:46.483255 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 260) took 5984 ms. Trace:
I20240430 20:11:46.483346 19489 rpcz_store.cc:276] 0430 20:11:40.498433 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:40.498497 (+ 64us) service_pool.cc:225] Handling call
0430 20:11:46.388225 (+5889728us) ts_tablet_manager.cc:1120] Deleting tablet fd40967ab4a04001819e16c06a6c4300
0430 20:11:46.388235 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:46.392661 (+ 4426us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:46.468663 (+ 76002us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:46.483233 (+ 14570us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5889487}
I20240430 20:11:46.483812 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 261) took 5938 ms. Trace:
I20240430 20:11:46.483901 19489 rpcz_store.cc:276] 0430 20:11:40.545063 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:40.545125 (+ 62us) service_pool.cc:225] Handling call
0430 20:11:46.483674 (+5938549us) ts_tablet_manager.cc:1120] Deleting tablet e4cffa295d8c401aa8415ce7e8f56526
0430 20:11:46.483685 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:46.483796 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5892966}
I20240430 20:11:46.485519 20686 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88b3f98793354c7aaa5c106e8ec31ed8. Latest consensus state: current_term: 6 leader_uuid: "88b3f98793354c7aaa5c106e8ec31ed8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b3f98793354c7aaa5c106e8ec31ed8" member_type: VOTER last_known_addr { host: "127.17.177.62" port: 36441 } } peers { permanent_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e" member_type: VOTER last_known_addr { host: "127.17.177.61" port: 34777 } } peers { permanent_uuid: "b632d6ef4c104a058a6db208024cab33" member_type: VOTER last_known_addr { host: "127.17.177.60" port: 37225 } } }
I20240430 20:11:46.485977 20686 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8 [sys.catalog]: This master's current role is: LEADER
I20240430 20:11:46.486510 19489 tablet_replica.cc:330] T a9f37f0f7ce8425f9b24699513d41d61 P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:46.480484 18988 raft_consensus.cc:374] T c6378f1495144c458af49b3c19a2b67c P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:46.486709 18988 raft_consensus.cc:729] T c6378f1495144c458af49b3c19a2b67c P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:46.486991 19489 raft_consensus.cc:2229] T a9f37f0f7ce8425f9b24699513d41d61 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:46.487337 19489 raft_consensus.cc:2258] T a9f37f0f7ce8425f9b24699513d41d61 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:46.488063 19489 ts_tablet_manager.cc:1875] T a9f37f0f7ce8425f9b24699513d41d61 P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:46.494897 19471 ts_tablet_manager.cc:1888] T a9f37f0f7ce8425f9b24699513d41d61 P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:46.486932 18988 consensus_queue.cc:260] T c6378f1495144c458af49b3c19a2b67c P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.497494 18988 ts_tablet_manager.cc:1398] T c6378f1495144c458af49b3c19a2b67c P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.018s user 0.001s sys 0.001s
I20240430 20:11:46.497822 18988 tablet_bootstrap.cc:492] T 95383d9d5a234a4d9d841a8b114428b0 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:46.499473 18985 tablet_bootstrap.cc:654] T 246fb63a5e9842df8263800e28a5935c P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:46.500721 18988 tablet_bootstrap.cc:654] T 95383d9d5a234a4d9d841a8b114428b0 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:46.501351 18474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06e82c706b9c4f35bf53faddf57b3fd7" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:46.501614 18474 raft_consensus.cc:3037] T 06e82c706b9c4f35bf53faddf57b3fd7 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:46.503780 18474 raft_consensus.cc:2454] T 06e82c706b9c4f35bf53faddf57b3fd7 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:46.506558 19471 log.cc:1196] T a9f37f0f7ce8425f9b24699513d41d61 P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/a9f37f0f7ce8425f9b24699513d41d61
I20240430 20:11:46.508631 19471 ts_tablet_manager.cc:1909] T a9f37f0f7ce8425f9b24699513d41d61 P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:46.504865 20469 leader_election.cc:288] T 06e82c706b9c4f35bf53faddf57b3fd7 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Requested vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:46.509670 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 250) took 5773 ms. Trace:
I20240430 20:11:46.509776 19471 rpcz_store.cc:276] 0430 20:11:40.735713 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:40.748109 (+ 12396us) service_pool.cc:225] Handling call
0430 20:11:46.456948 (+5708839us) ts_tablet_manager.cc:1120] Deleting tablet a9f37f0f7ce8425f9b24699513d41d61
0430 20:11:46.456958 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:46.461796 (+ 4838us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:46.494767 (+ 32971us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:46.509641 (+ 14874us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5685506}
I20240430 20:11:46.510298 20545 raft_consensus.cc:482] T b22aaedb0a6c4fd9ad0ea8fa019001e1 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:46.510499 20545 raft_consensus.cc:504] T b22aaedb0a6c4fd9ad0ea8fa019001e1 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.511204 20545 leader_election.cc:288] T b22aaedb0a6c4fd9ad0ea8fa019001e1 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:46.511788 18750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b22aaedb0a6c4fd9ad0ea8fa019001e1" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
W20240430 20:11:46.512343 18417 leader_election.cc:341] T b22aaedb0a6c4fd9ad0ea8fa019001e1 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:46.513031 20693 raft_consensus.cc:2726] T 4f6f9ae722da49e58c9b72107f6da2fa P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240430 20:11:46.513203 18557 consensus_peers.cc:485] T 5bd0362ff41b4faca0205d136d596737 P 5e1c6397f09747f091534faf93542710 -> Peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Couldn't send request to peer 3c51c33d5616482cafb68e26688820f4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5bd0362ff41b4faca0205d136d596737. This is attempt 1: this message will repeat every 5th retry.
I20240430 20:11:46.516465 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 251) took 5780 ms. Trace:
I20240430 20:11:46.516561 19471 rpcz_store.cc:276] 0430 20:11:40.735844 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:40.748183 (+ 12339us) service_pool.cc:225] Handling call
0430 20:11:46.516314 (+5768131us) ts_tablet_manager.cc:1120] Deleting tablet 4dac8b3838954d839094021f1c0503da
0430 20:11:46.516325 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:46.516445 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5744777}
I20240430 20:11:46.516829 18619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b22aaedb0a6c4fd9ad0ea8fa019001e1" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:46.517022 19471 tablet_replica.cc:330] T 34130f2820064fa0898ef1f9d7de1c6d P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:46.517133 18619 raft_consensus.cc:2454] T b22aaedb0a6c4fd9ad0ea8fa019001e1 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 0.
I20240430 20:11:46.517290 19471 raft_consensus.cc:2229] T 34130f2820064fa0898ef1f9d7de1c6d P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:46.517467 19471 raft_consensus.cc:2258] T 34130f2820064fa0898ef1f9d7de1c6d P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:46.517598 18415 leader_election.cc:302] T b22aaedb0a6c4fd9ad0ea8fa019001e1 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters: 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:46.518038 20545 raft_consensus.cc:2781] T b22aaedb0a6c4fd9ad0ea8fa019001e1 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:46.518078 19471 ts_tablet_manager.cc:1875] T 34130f2820064fa0898ef1f9d7de1c6d P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:46.518105 20545 raft_consensus.cc:482] T b22aaedb0a6c4fd9ad0ea8fa019001e1 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:46.518157 20545 raft_consensus.cc:3037] T b22aaedb0a6c4fd9ad0ea8fa019001e1 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:46.519234 18985 tablet_bootstrap.cc:492] T 246fb63a5e9842df8263800e28a5935c P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:46.519363 18985 ts_tablet_manager.cc:1367] T 246fb63a5e9842df8263800e28a5935c P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.037s user 0.008s sys 0.000s
I20240430 20:11:46.520102 18985 raft_consensus.cc:348] T 246fb63a5e9842df8263800e28a5935c P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.520277 18985 raft_consensus.cc:374] T 246fb63a5e9842df8263800e28a5935c P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:46.520320 18985 raft_consensus.cc:729] T 246fb63a5e9842df8263800e28a5935c P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:46.520364 18988 tablet_bootstrap.cc:492] T 95383d9d5a234a4d9d841a8b114428b0 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:46.520480 18988 ts_tablet_manager.cc:1367] T 95383d9d5a234a4d9d841a8b114428b0 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.002s
I20240430 20:11:46.520679 20545 raft_consensus.cc:504] T b22aaedb0a6c4fd9ad0ea8fa019001e1 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.520948 18985 consensus_queue.cc:260] T 246fb63a5e9842df8263800e28a5935c P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.521273 20545 leader_election.cc:288] T b22aaedb0a6c4fd9ad0ea8fa019001e1 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:46.521631 18985 ts_tablet_manager.cc:1398] T 246fb63a5e9842df8263800e28a5935c P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240430 20:11:46.522060 18985 tablet_bootstrap.cc:492] T a482f4622f774d60b4efd6827b8be7d7 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:46.522128 18623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b22aaedb0a6c4fd9ad0ea8fa019001e1" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:46.522300 18623 raft_consensus.cc:3037] T b22aaedb0a6c4fd9ad0ea8fa019001e1 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:46.521199 18988 raft_consensus.cc:348] T 95383d9d5a234a4d9d841a8b114428b0 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.523515 18988 raft_consensus.cc:374] T 95383d9d5a234a4d9d841a8b114428b0 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:46.523579 18988 raft_consensus.cc:729] T 95383d9d5a234a4d9d841a8b114428b0 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:46.523797 18988 consensus_queue.cc:260] T 95383d9d5a234a4d9d841a8b114428b0 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.524271 18988 ts_tablet_manager.cc:1398] T 95383d9d5a234a4d9d841a8b114428b0 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240430 20:11:46.524538 18988 tablet_bootstrap.cc:492] T f5eaf2413e1d4e69b56fae61f768f0ce P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:46.525003 18623 raft_consensus.cc:2454] T b22aaedb0a6c4fd9ad0ea8fa019001e1 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 1.
I20240430 20:11:46.526512 18415 leader_election.cc:302] T b22aaedb0a6c4fd9ad0ea8fa019001e1 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:46.527377 18985 tablet_bootstrap.cc:654] T a482f4622f774d60b4efd6827b8be7d7 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:46.527616 20545 raft_consensus.cc:2781] T b22aaedb0a6c4fd9ad0ea8fa019001e1 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:46.527889 20545 raft_consensus.cc:686] T b22aaedb0a6c4fd9ad0ea8fa019001e1 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Becoming Leader. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Running, Role: LEADER
I20240430 20:11:46.528012 18756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b22aaedb0a6c4fd9ad0ea8fa019001e1" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
W20240430 20:11:46.528560 18417 leader_election.cc:341] T b22aaedb0a6c4fd9ad0ea8fa019001e1 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:46.528218 20545 consensus_queue.cc:237] T b22aaedb0a6c4fd9ad0ea8fa019001e1 P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.529851 18988 tablet_bootstrap.cc:654] T f5eaf2413e1d4e69b56fae61f768f0ce P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:46.506279 19489 ts_tablet_manager.cc:1888] T a9f37f0f7ce8425f9b24699513d41d61 P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20240430 20:11:46.531996 18696 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.177.2:46027, user_credentials={real_user=slave}} blocked reactor thread for 64045.3us
I20240430 20:11:46.531308 19489 log.cc:1196] T a9f37f0f7ce8425f9b24699513d41d61 P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/a9f37f0f7ce8425f9b24699513d41d61
I20240430 20:11:46.534901 19489 ts_tablet_manager.cc:1909] T a9f37f0f7ce8425f9b24699513d41d61 P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:46.533257 18983 tablet_bootstrap.cc:492] T 283e5e73cbc04daba0de19f4541f9fd0 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:46.535706 18983 ts_tablet_manager.cc:1367] T 283e5e73cbc04daba0de19f4541f9fd0 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.076s user 0.003s sys 0.002s
I20240430 20:11:46.536669 18983 raft_consensus.cc:348] T 283e5e73cbc04daba0de19f4541f9fd0 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.538857 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 272) took 5824 ms. Trace:
I20240430 20:11:46.538960 19489 rpcz_store.cc:276] 0430 20:11:40.713954 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:40.714430 (+ 476us) service_pool.cc:225] Handling call
0430 20:11:46.486461 (+5772031us) ts_tablet_manager.cc:1120] Deleting tablet a9f37f0f7ce8425f9b24699513d41d61
0430 20:11:46.486473 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:46.503100 (+ 16627us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:46.506186 (+ 3086us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:46.538826 (+ 32640us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5734729}
I20240430 20:11:46.541481 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 273) took 5827 ms. Trace:
I20240430 20:11:46.541582 19489 rpcz_store.cc:276] 0430 20:11:40.714141 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:40.715636 (+ 1495us) service_pool.cc:225] Handling call
0430 20:11:46.541326 (+5825690us) ts_tablet_manager.cc:1120] Deleting tablet 4dac8b3838954d839094021f1c0503da
0430 20:11:46.541338 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:46.541459 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5789354}
I20240430 20:11:46.542871 18983 raft_consensus.cc:374] T 283e5e73cbc04daba0de19f4541f9fd0 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:46.544682 18983 raft_consensus.cc:729] T 283e5e73cbc04daba0de19f4541f9fd0 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:46.545269 19471 ts_tablet_manager.cc:1888] T 34130f2820064fa0898ef1f9d7de1c6d P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:46.545397 19471 log.cc:1196] T 34130f2820064fa0898ef1f9d7de1c6d P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/34130f2820064fa0898ef1f9d7de1c6d
I20240430 20:11:46.545296 18983 consensus_queue.cc:260] T 283e5e73cbc04daba0de19f4541f9fd0 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.546041 19489 tablet_replica.cc:330] T 34130f2820064fa0898ef1f9d7de1c6d P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:46.546245 19489 raft_consensus.cc:2229] T 34130f2820064fa0898ef1f9d7de1c6d P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Raft consensus shutting down.
W20240430 20:11:46.546439 18116 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskNG3HqZ/build/asan/bin/kudu pid 18934: Timed out: CheckLeaks RPC to 127.17.177.60:37225 timed out after 1.000s (SENT)
I20240430 20:11:46.546695 18116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNG3HqZ/build/asan/bin/kudu with pid 18934
I20240430 20:11:46.548228 19489 raft_consensus.cc:2258] T 34130f2820064fa0898ef1f9d7de1c6d P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:46.548779 19489 ts_tablet_manager.cc:1875] T 34130f2820064fa0898ef1f9d7de1c6d P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:46.548410 20489 raft_consensus.cc:482] T de1dfe6c82d34a4d8d02e466c55327ff P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:46.561630 20489 raft_consensus.cc:504] T de1dfe6c82d34a4d8d02e466c55327ff P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.562361 20489 leader_election.cc:288] T de1dfe6c82d34a4d8d02e466c55327ff P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:46.545778 19471 ts_tablet_manager.cc:1909] T 34130f2820064fa0898ef1f9d7de1c6d P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:46.563074 18474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de1dfe6c82d34a4d8d02e466c55327ff" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:46.563331 18474 raft_consensus.cc:2454] T de1dfe6c82d34a4d8d02e466c55327ff P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 0.
I20240430 20:11:46.563674 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 252) took 5767 ms. Trace:
I20240430 20:11:46.563648 18756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de1dfe6c82d34a4d8d02e466c55327ff" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:46.563859 18756 raft_consensus.cc:2454] T de1dfe6c82d34a4d8d02e466c55327ff P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 0.
I20240430 20:11:46.563761 19471 rpcz_store.cc:276] 0430 20:11:40.796157 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:40.800682 (+ 4525us) service_pool.cc:225] Handling call
0430 20:11:46.516975 (+5716293us) ts_tablet_manager.cc:1120] Deleting tablet 34130f2820064fa0898ef1f9d7de1c6d
0430 20:11:46.516985 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:46.542038 (+ 25053us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:46.545152 (+ 3114us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:46.563650 (+ 18498us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5716001}
I20240430 20:11:46.564211 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 254) took 5757 ms. Trace:
I20240430 20:11:46.564778 18557 leader_election.cc:302] T de1dfe6c82d34a4d8d02e466c55327ff P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:46.565377 18983 ts_tablet_manager.cc:1398] T 283e5e73cbc04daba0de19f4541f9fd0 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.029s user 0.001s sys 0.001s
I20240430 20:11:46.565477 20489 raft_consensus.cc:2781] T de1dfe6c82d34a4d8d02e466c55327ff P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:46.565565 20489 raft_consensus.cc:482] T de1dfe6c82d34a4d8d02e466c55327ff P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:46.565624 20489 raft_consensus.cc:3037] T de1dfe6c82d34a4d8d02e466c55327ff P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:46.568470 20489 raft_consensus.cc:504] T de1dfe6c82d34a4d8d02e466c55327ff P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.569036 18983 tablet_bootstrap.cc:492] T f5eaf2413e1d4e69b56fae61f768f0ce P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:46.569726 18756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de1dfe6c82d34a4d8d02e466c55327ff" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:46.569939 18756 raft_consensus.cc:3037] T de1dfe6c82d34a4d8d02e466c55327ff P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:46.570962 18983 tablet_bootstrap.cc:654] T f5eaf2413e1d4e69b56fae61f768f0ce P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:46.572717 18756 raft_consensus.cc:2454] T de1dfe6c82d34a4d8d02e466c55327ff P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
I20240430 20:11:46.573227 18557 leader_election.cc:302] T de1dfe6c82d34a4d8d02e466c55327ff P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:46.573515 20488 raft_consensus.cc:2781] T de1dfe6c82d34a4d8d02e466c55327ff P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:46.573621 20488 raft_consensus.cc:686] T de1dfe6c82d34a4d8d02e466c55327ff P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Becoming Leader. State: Replica: 5e1c6397f09747f091534faf93542710, State: Running, Role: LEADER
I20240430 20:11:46.573874 20488 consensus_queue.cc:237] T de1dfe6c82d34a4d8d02e466c55327ff P 5e1c6397f09747f091534faf93542710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.576493 18474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de1dfe6c82d34a4d8d02e466c55327ff" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:46.576696 18474 raft_consensus.cc:3037] T de1dfe6c82d34a4d8d02e466c55327ff P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:46.579325 18474 raft_consensus.cc:2454] T de1dfe6c82d34a4d8d02e466c55327ff P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
I20240430 20:11:46.580257 20047 consensus_queue.cc:1035] T 4f6f9ae722da49e58c9b72107f6da2fa P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:46.581311 20489 leader_election.cc:288] T de1dfe6c82d34a4d8d02e466c55327ff P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:46.582101 18988 tablet_bootstrap.cc:492] T f5eaf2413e1d4e69b56fae61f768f0ce P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:46.582252 18988 ts_tablet_manager.cc:1367] T f5eaf2413e1d4e69b56fae61f768f0ce P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.058s user 0.003s sys 0.000s
I20240430 20:11:46.582278 20489 raft_consensus.cc:482] T 283e5e73cbc04daba0de19f4541f9fd0 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:46.582505 20489 raft_consensus.cc:504] T 283e5e73cbc04daba0de19f4541f9fd0 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.583127 18988 raft_consensus.cc:348] T f5eaf2413e1d4e69b56fae61f768f0ce P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.583321 18988 raft_consensus.cc:374] T f5eaf2413e1d4e69b56fae61f768f0ce P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:46.583393 18988 raft_consensus.cc:729] T f5eaf2413e1d4e69b56fae61f768f0ce P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:46.583683 18988 consensus_queue.cc:260] T f5eaf2413e1d4e69b56fae61f768f0ce P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.584365 18988 ts_tablet_manager.cc:1398] T f5eaf2413e1d4e69b56fae61f768f0ce P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240430 20:11:46.585280 18756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "283e5e73cbc04daba0de19f4541f9fd0" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:46.585430 18988 tablet_bootstrap.cc:492] T 22fef8bb6b034db098c3372ae992f7a7 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
W20240430 20:11:46.585880 18557 leader_election.cc:341] T 283e5e73cbc04daba0de19f4541f9fd0 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:46.587313 18474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "283e5e73cbc04daba0de19f4541f9fd0" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
W20240430 20:11:46.587821 18558 leader_election.cc:341] T 283e5e73cbc04daba0de19f4541f9fd0 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:46.587939 18558 leader_election.cc:302] T 283e5e73cbc04daba0de19f4541f9fd0 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:46.588270 20047 raft_consensus.cc:2726] T 283e5e73cbc04daba0de19f4541f9fd0 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:46.588353 18988 tablet_bootstrap.cc:654] T 22fef8bb6b034db098c3372ae992f7a7 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:46.588723 20489 leader_election.cc:288] T 283e5e73cbc04daba0de19f4541f9fd0 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:46.589082 18985 tablet_bootstrap.cc:492] T a482f4622f774d60b4efd6827b8be7d7 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:46.589236 18985 ts_tablet_manager.cc:1367] T a482f4622f774d60b4efd6827b8be7d7 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.067s user 0.004s sys 0.007s
I20240430 20:11:46.590101 18985 raft_consensus.cc:348] T a482f4622f774d60b4efd6827b8be7d7 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.590312 18985 raft_consensus.cc:374] T a482f4622f774d60b4efd6827b8be7d7 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:46.590371 18985 raft_consensus.cc:729] T a482f4622f774d60b4efd6827b8be7d7 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:46.564298 19471 rpcz_store.cc:276] 0430 20:11:40.806733 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:40.806772 (+ 39us) service_pool.cc:225] Handling call
0430 20:11:46.564073 (+5757301us) ts_tablet_manager.cc:1120] Deleting tablet 436579f473844c6ca1038b496201a550
0430 20:11:46.564082 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:46.564195 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5757061}
I20240430 20:11:46.593201 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 255) took 5722 ms. Trace:
I20240430 20:11:46.598749 19471 rpcz_store.cc:276] 0430 20:11:40.871205 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:40.871420 (+ 215us) service_pool.cc:225] Handling call
0430 20:11:46.593071 (+5721651us) ts_tablet_manager.cc:1120] Deleting tablet f7d3dea34cf54fe2b949d1b7ca68577a
0430 20:11:46.593084 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:46.593186 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5720197}
I20240430 20:11:46.599673 19471 tablet_replica.cc:330] T e6c0c808bfc94be3837ab835292b3c1b P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:46.600143 19471 raft_consensus.cc:2229] T e6c0c808bfc94be3837ab835292b3c1b P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Raft consensus shutting down.
I20240430 20:11:46.600256 18988 tablet_bootstrap.cc:492] T 22fef8bb6b034db098c3372ae992f7a7 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:46.600533 18988 ts_tablet_manager.cc:1367] T 22fef8bb6b034db098c3372ae992f7a7 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.004s
I20240430 20:11:46.601101 18985 consensus_queue.cc:260] T a482f4622f774d60b4efd6827b8be7d7 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.601749 18988 raft_consensus.cc:348] T 22fef8bb6b034db098c3372ae992f7a7 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.601913 20693 raft_consensus.cc:482] T 66dd767bd3f8494f94061fb89e5e5680 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:46.601934 18988 raft_consensus.cc:374] T 22fef8bb6b034db098c3372ae992f7a7 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:46.602066 20693 raft_consensus.cc:504] T 66dd767bd3f8494f94061fb89e5e5680 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.602401 18985 ts_tablet_manager.cc:1398] T a482f4622f774d60b4efd6827b8be7d7 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.013s user 0.001s sys 0.001s
I20240430 20:11:46.602825 20693 leader_election.cc:288] T 66dd767bd3f8494f94061fb89e5e5680 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:46.603888 18985 tablet_bootstrap.cc:492] T 04b3dff86d5c48a09d03da58bd48f360 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:46.603829 18474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66dd767bd3f8494f94061fb89e5e5680" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:46.602034 18988 raft_consensus.cc:729] T 22fef8bb6b034db098c3372ae992f7a7 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:46.607141 18988 consensus_queue.cc:260] T 22fef8bb6b034db098c3372ae992f7a7 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.607686 18988 ts_tablet_manager.cc:1398] T 22fef8bb6b034db098c3372ae992f7a7 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.007s user 0.000s sys 0.001s
I20240430 20:11:46.608364 18988 tablet_bootstrap.cc:492] T 246fb63a5e9842df8263800e28a5935c P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:46.609019 18985 tablet_bootstrap.cc:654] T 04b3dff86d5c48a09d03da58bd48f360 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:46.609454 19471 raft_consensus.cc:2258] T e6c0c808bfc94be3837ab835292b3c1b P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:46.610323 19471 ts_tablet_manager.cc:1875] T e6c0c808bfc94be3837ab835292b3c1b P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:46.614414 19489 ts_tablet_manager.cc:1888] T 34130f2820064fa0898ef1f9d7de1c6d P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:46.607226 18619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66dd767bd3f8494f94061fb89e5e5680" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:46.614842 18619 raft_consensus.cc:2379] T 66dd767bd3f8494f94061fb89e5e5680 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c51c33d5616482cafb68e26688820f4 in current term 1: Already voted for candidate 2efb542aa55e439b99ccb98d1c5c884a in this term.
I20240430 20:11:46.611899 20489 consensus_queue.cc:1035] T 4f6f9ae722da49e58c9b72107f6da2fa P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:46.619385 18985 tablet_bootstrap.cc:492] T 04b3dff86d5c48a09d03da58bd48f360 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:46.620095 18985 ts_tablet_manager.cc:1367] T 04b3dff86d5c48a09d03da58bd48f360 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.000s
I20240430 20:11:46.621366 18985 raft_consensus.cc:348] T 04b3dff86d5c48a09d03da58bd48f360 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.622452 18985 raft_consensus.cc:374] T 04b3dff86d5c48a09d03da58bd48f360 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:46.622628 18985 raft_consensus.cc:729] T 04b3dff86d5c48a09d03da58bd48f360 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:46.606058 18983 tablet_bootstrap.cc:492] T f5eaf2413e1d4e69b56fae61f768f0ce P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:46.625196 18983 ts_tablet_manager.cc:1367] T f5eaf2413e1d4e69b56fae61f768f0ce P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.056s user 0.004s sys 0.000s
I20240430 20:11:46.626860 19489 log.cc:1196] T 34130f2820064fa0898ef1f9d7de1c6d P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/34130f2820064fa0898ef1f9d7de1c6d
I20240430 20:11:46.615705 18696 leader_election.cc:302] T 66dd767bd3f8494f94061fb89e5e5680 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710
I20240430 20:11:46.627527 20693 raft_consensus.cc:3037] T 66dd767bd3f8494f94061fb89e5e5680 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:46.627280 18983 raft_consensus.cc:348] T f5eaf2413e1d4e69b56fae61f768f0ce P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.627737 18983 raft_consensus.cc:374] T f5eaf2413e1d4e69b56fae61f768f0ce P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:46.627807 18983 raft_consensus.cc:729] T f5eaf2413e1d4e69b56fae61f768f0ce P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:46.617168 18988 tablet_bootstrap.cc:654] T 246fb63a5e9842df8263800e28a5935c P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:46.628099 19489 ts_tablet_manager.cc:1909] T 34130f2820064fa0898ef1f9d7de1c6d P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:46.628624 20587 raft_consensus.cc:482] T 3ec3993042f5498b818bdf32d5087928 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:46.628700 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 274) took 5842 ms. Trace:
I20240430 20:11:46.628800 19489 rpcz_store.cc:276] 0430 20:11:40.786715 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:40.786799 (+ 84us) service_pool.cc:225] Handling call
0430 20:11:46.545985 (+5759186us) ts_tablet_manager.cc:1120] Deleting tablet 34130f2820064fa0898ef1f9d7de1c6d
0430 20:11:46.545997 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:46.558995 (+ 12998us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:46.614297 (+ 55302us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:46.628675 (+ 14378us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5758901}
I20240430 20:11:46.629309 18983 consensus_queue.cc:260] T f5eaf2413e1d4e69b56fae61f768f0ce P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.634438 20693 raft_consensus.cc:2726] T 66dd767bd3f8494f94061fb89e5e5680 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:46.634968 18983 ts_tablet_manager.cc:1398] T f5eaf2413e1d4e69b56fae61f768f0ce P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240430 20:11:46.635280 18983 tablet_bootstrap.cc:492] T 22fef8bb6b034db098c3372ae992f7a7 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:46.636791 18983 tablet_bootstrap.cc:654] T 22fef8bb6b034db098c3372ae992f7a7 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:46.632349 20587 raft_consensus.cc:504] T 3ec3993042f5498b818bdf32d5087928 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.637810 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 275) took 5847 ms. Trace:
I20240430 20:11:46.637899 19489 rpcz_store.cc:276] 0430 20:11:40.790833 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:40.791073 (+ 240us) service_pool.cc:225] Handling call
0430 20:11:46.637622 (+5846549us) ts_tablet_manager.cc:1120] Deleting tablet 436579f473844c6ca1038b496201a550
0430 20:11:46.637634 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:46.637789 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5846344}
I20240430 20:11:46.638201 20587 leader_election.cc:288] T 3ec3993042f5498b818bdf32d5087928 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:46.642124 18478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ec3993042f5498b818bdf32d5087928" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:46.642346 18478 raft_consensus.cc:2454] T 3ec3993042f5498b818bdf32d5087928 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:46.642870 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 276) took 5776 ms. Trace:
I20240430 20:11:46.642957 19489 rpcz_store.cc:276] 0430 20:11:40.866342 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:40.866678 (+ 336us) service_pool.cc:225] Handling call
0430 20:11:46.642705 (+5776027us) ts_tablet_manager.cc:1120] Deleting tablet f7d3dea34cf54fe2b949d1b7ca68577a
0430 20:11:46.642715 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:46.642850 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5775756}
I20240430 20:11:46.645614 18699 leader_election.cc:302] T 3ec3993042f5498b818bdf32d5087928 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters:
I20240430 20:11:46.645946 20587 raft_consensus.cc:2781] T 3ec3993042f5498b818bdf32d5087928 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:46.646027 20587 raft_consensus.cc:482] T 3ec3993042f5498b818bdf32d5087928 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:46.646087 20587 raft_consensus.cc:3037] T 3ec3993042f5498b818bdf32d5087928 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:46.646274 18619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ec3993042f5498b818bdf32d5087928" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:46.646479 18619 raft_consensus.cc:2454] T 3ec3993042f5498b818bdf32d5087928 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:46.646785 19489 tablet_replica.cc:330] T 5d09baed576e4136bf6d756ca39d08b2 P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:46.647004 19489 raft_consensus.cc:2229] T 5d09baed576e4136bf6d756ca39d08b2 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:46.647140 19489 raft_consensus.cc:2258] T 5d09baed576e4136bf6d756ca39d08b2 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:46.647671 19489 ts_tablet_manager.cc:1875] T 5d09baed576e4136bf6d756ca39d08b2 P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:46.648905 20587 raft_consensus.cc:504] T 3ec3993042f5498b818bdf32d5087928 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.650746 18478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ec3993042f5498b818bdf32d5087928" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:46.651029 18478 raft_consensus.cc:3037] T 3ec3993042f5498b818bdf32d5087928 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:46.654352 18478 raft_consensus.cc:2454] T 3ec3993042f5498b818bdf32d5087928 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:46.655297 18699 leader_election.cc:302] T 3ec3993042f5498b818bdf32d5087928 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters:
I20240430 20:11:46.655591 20693 raft_consensus.cc:2781] T 3ec3993042f5498b818bdf32d5087928 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:46.655710 20693 raft_consensus.cc:686] T 3ec3993042f5498b818bdf32d5087928 P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Becoming Leader. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Running, Role: LEADER
I20240430 20:11:46.629004 18985 consensus_queue.cc:260] T 04b3dff86d5c48a09d03da58bd48f360 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.655983 20693 consensus_queue.cc:237] T 3ec3993042f5498b818bdf32d5087928 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.657068 18985 ts_tablet_manager.cc:1398] T 04b3dff86d5c48a09d03da58bd48f360 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.037s user 0.001s sys 0.001s
I20240430 20:11:46.657191 19471 ts_tablet_manager.cc:1888] T e6c0c808bfc94be3837ab835292b3c1b P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240430 20:11:46.657335 19471 log.cc:1196] T e6c0c808bfc94be3837ab835292b3c1b P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/e6c0c808bfc94be3837ab835292b3c1b
I20240430 20:11:46.657371 18985 tablet_bootstrap.cc:492] T 283e5e73cbc04daba0de19f4541f9fd0 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:46.657696 19471 ts_tablet_manager.cc:1909] T e6c0c808bfc94be3837ab835292b3c1b P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:46.657862 20488 raft_consensus.cc:482] T 95383d9d5a234a4d9d841a8b114428b0 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:46.657991 20488 raft_consensus.cc:504] T 95383d9d5a234a4d9d841a8b114428b0 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.658078 18619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ec3993042f5498b818bdf32d5087928" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:46.658218 18619 raft_consensus.cc:3037] T 3ec3993042f5498b818bdf32d5087928 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:46.659081 20488 leader_election.cc:288] T 95383d9d5a234a4d9d841a8b114428b0 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:46.659029 18756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95383d9d5a234a4d9d841a8b114428b0" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:46.659215 18756 raft_consensus.cc:2454] T 95383d9d5a234a4d9d841a8b114428b0 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 0.
I20240430 20:11:46.659638 18557 leader_election.cc:302] T 95383d9d5a234a4d9d841a8b114428b0 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:46.659765 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 257) took 5505 ms. Trace:
I20240430 20:11:46.659490 20695 raft_consensus.cc:482] T c6378f1495144c458af49b3c19a2b67c P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:46.659849 19471 rpcz_store.cc:276] 0430 20:11:41.154405 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:41.154464 (+ 59us) service_pool.cc:225] Handling call
0430 20:11:46.599612 (+5445148us) ts_tablet_manager.cc:1120] Deleting tablet e6c0c808bfc94be3837ab835292b3c1b
0430 20:11:46.599625 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:46.639326 (+ 39701us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:46.642501 (+ 3175us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:46.659746 (+ 17245us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5444944}
I20240430 20:11:46.659983 20695 raft_consensus.cc:504] T c6378f1495144c458af49b3c19a2b67c P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.660225 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 261) took 5288 ms. Trace:
I20240430 20:11:46.660283 19471 rpcz_store.cc:276] 0430 20:11:41.371548 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:41.371770 (+ 222us) service_pool.cc:225] Handling call
0430 20:11:46.660088 (+5288318us) ts_tablet_manager.cc:1120] Deleting tablet 3b7fdc6ec7e046c5af894f9a6eda17b0
0430 20:11:46.660098 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:46.660213 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5288133}
I20240430 20:11:46.660507 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 262) took 5272 ms. Trace:
I20240430 20:11:46.660559 19471 rpcz_store.cc:276] 0430 20:11:41.388410 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:41.388497 (+ 87us) service_pool.cc:225] Handling call
0430 20:11:46.660452 (+5271955us) ts_tablet_manager.cc:1120] Deleting tablet 26ed12f498644d15bf45e884dcad172e
0430 20:11:46.660459 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:46.660497 (+ 38us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5271742}
I20240430 20:11:46.660627 20695 leader_election.cc:288] T c6378f1495144c458af49b3c19a2b67c P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:46.660760 19471 tablet_replica.cc:330] T 95383d9d5a234a4d9d841a8b114428b0 P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:46.659348 20587 leader_election.cc:288] T 3ec3993042f5498b818bdf32d5087928 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Requested vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:46.660935 19471 raft_consensus.cc:2229] T 95383d9d5a234a4d9d841a8b114428b0 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:46.660977 18619 raft_consensus.cc:2454] T 3ec3993042f5498b818bdf32d5087928 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:46.661052 19471 raft_consensus.cc:2258] T 95383d9d5a234a4d9d841a8b114428b0 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:46.661543 19471 ts_tablet_manager.cc:1875] T 95383d9d5a234a4d9d841a8b114428b0 P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:46.661772 20587 raft_consensus.cc:482] T eb76f60448774fe1967fffe7c6873cfc P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:46.661895 20587 raft_consensus.cc:504] T eb76f60448774fe1967fffe7c6873cfc P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.661978 18473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6378f1495144c458af49b3c19a2b67c" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:46.662164 18473 raft_consensus.cc:2454] T c6378f1495144c458af49b3c19a2b67c P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:46.662444 18619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6378f1495144c458af49b3c19a2b67c" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:46.662657 18619 raft_consensus.cc:2454] T c6378f1495144c458af49b3c19a2b67c P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:46.663098 20587 leader_election.cc:288] T eb76f60448774fe1967fffe7c6873cfc P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:46.663483 18473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb76f60448774fe1967fffe7c6873cfc" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:46.664270 18473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95383d9d5a234a4d9d841a8b114428b0" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:46.664480 18473 raft_consensus.cc:2454] T 95383d9d5a234a4d9d841a8b114428b0 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 0.
I20240430 20:11:46.664836 18699 leader_election.cc:302] T c6378f1495144c458af49b3c19a2b67c P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters:
I20240430 20:11:46.665141 20587 raft_consensus.cc:2781] T c6378f1495144c458af49b3c19a2b67c P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:46.665241 20587 raft_consensus.cc:482] T c6378f1495144c458af49b3c19a2b67c P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:46.665334 20587 raft_consensus.cc:3037] T c6378f1495144c458af49b3c19a2b67c P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:46.666168 18619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb76f60448774fe1967fffe7c6873cfc" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:46.666397 18619 raft_consensus.cc:2379] T eb76f60448774fe1967fffe7c6873cfc P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c51c33d5616482cafb68e26688820f4 in current term 1: Already voted for candidate 2efb542aa55e439b99ccb98d1c5c884a in this term.
I20240430 20:11:46.667143 18696 leader_election.cc:302] T eb76f60448774fe1967fffe7c6873cfc P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710
W20240430 20:11:46.667146 19899 connection.cc:535] server connection from 127.0.0.1:33764 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240430 20:11:46.667420 20695 raft_consensus.cc:3037] T eb76f60448774fe1967fffe7c6873cfc P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
W20240430 20:11:46.667632 18417 connection.cc:535] server connection from 127.0.0.1:41680 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240430 20:11:46.667869 18558 connection.cc:535] server connection from 127.0.0.1:40218 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240430 20:11:46.668236 18697 connection.cc:535] server connection from 127.0.0.1:59830 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240430 20:11:46.668267 18212 connection.cc:535] server connection from 127.0.0.1:43312 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240430 20:11:46.669296 18415 connection.cc:535] client connection to 127.17.177.60:37225 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240430 20:11:46.669370 19895 connection.cc:535] client connection to 127.17.177.60:37225 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240430 20:11:46.669628 19895 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240430 20:11:46.670189 20695 raft_consensus.cc:2726] T eb76f60448774fe1967fffe7c6873cfc P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240430 20:11:46.670482 18818 connection.cc:535] client connection to 127.17.177.60:37225 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240430 20:11:46.671751 18818 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.17.177.60:37225)
W20240430 20:11:46.672106 18818 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.17.177.60:37225)
W20240430 20:11:46.672405 18211 connection.cc:535] client connection to 127.17.177.60:37225 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240430 20:11:46.672562 18211 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240430 20:11:46.675371 18818 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.17.177.60:37225)
W20240430 20:11:46.675583 18818 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.17.177.60:37225)
W20240430 20:11:46.679991 18696 connection.cc:535] client connection to 127.17.177.60:37225 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240430 20:11:46.689257 19489 ts_tablet_manager.cc:1888] T 5d09baed576e4136bf6d756ca39d08b2 P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:46.689441 19489 log.cc:1196] T 5d09baed576e4136bf6d756ca39d08b2 P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/5d09baed576e4136bf6d756ca39d08b2
I20240430 20:11:46.689759 19489 ts_tablet_manager.cc:1909] T 5d09baed576e4136bf6d756ca39d08b2 P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:46.690083 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 277) took 5707 ms. Trace:
I20240430 20:11:46.690160 19489 rpcz_store.cc:276] 0430 20:11:40.983119 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:40.983179 (+ 60us) service_pool.cc:225] Handling call
0430 20:11:46.646731 (+5663552us) ts_tablet_manager.cc:1120] Deleting tablet 5d09baed576e4136bf6d756ca39d08b2
0430 20:11:46.646742 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:46.664006 (+ 17264us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:46.689060 (+ 25054us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:46.690055 (+ 995us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5663265}
W20240430 20:11:46.690426 18697 connection.cc:441] server connection from 127.0.0.1:59830 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 277) could send its response
I20240430 20:11:46.690793 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 278) took 5696 ms. Trace:
I20240430 20:11:46.690865 19489 rpcz_store.cc:276] 0430 20:11:40.994197 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:40.994482 (+ 285us) service_pool.cc:225] Handling call
0430 20:11:46.690672 (+5696190us) ts_tablet_manager.cc:1120] Deleting tablet fcc9698abda4491b98e1e879589913a8
0430 20:11:46.690682 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:46.690778 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5695913}
W20240430 20:11:46.691040 18697 connection.cc:441] server connection from 127.0.0.1:59830 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 278) could send its response
I20240430 20:11:46.693527 20587 raft_consensus.cc:504] T c6378f1495144c458af49b3c19a2b67c P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.694393 20587 leader_election.cc:288] T c6378f1495144c458af49b3c19a2b67c P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Requested vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:46.695061 18473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6378f1495144c458af49b3c19a2b67c" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:46.695281 18473 raft_consensus.cc:3037] T c6378f1495144c458af49b3c19a2b67c P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:46.698045 18473 raft_consensus.cc:2454] T c6378f1495144c458af49b3c19a2b67c P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:46.698691 18699 leader_election.cc:302] T c6378f1495144c458af49b3c19a2b67c P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters:
I20240430 20:11:46.699702 19489 tablet_replica.cc:330] T 2f61e3f88a6448dfa323204bd8d14cc5 P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:46.699981 19489 raft_consensus.cc:2229] T 2f61e3f88a6448dfa323204bd8d14cc5 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:46.700160 19489 raft_consensus.cc:2258] T 2f61e3f88a6448dfa323204bd8d14cc5 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:46.700647 20695 raft_consensus.cc:2781] T c6378f1495144c458af49b3c19a2b67c P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:46.700807 20695 raft_consensus.cc:686] T c6378f1495144c458af49b3c19a2b67c P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Becoming Leader. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Running, Role: LEADER
I20240430 20:11:46.700953 19489 ts_tablet_manager.cc:1875] T 2f61e3f88a6448dfa323204bd8d14cc5 P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:46.701069 20695 consensus_queue.cc:237] T c6378f1495144c458af49b3c19a2b67c P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.705097 19471 ts_tablet_manager.cc:1888] T 95383d9d5a234a4d9d841a8b114428b0 P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:46.705227 19471 log.cc:1196] T 95383d9d5a234a4d9d841a8b114428b0 P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/95383d9d5a234a4d9d841a8b114428b0
I20240430 20:11:46.713053 19471 ts_tablet_manager.cc:1909] T 95383d9d5a234a4d9d841a8b114428b0 P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:46.713977 18619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6378f1495144c458af49b3c19a2b67c" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:46.714203 18619 raft_consensus.cc:3037] T c6378f1495144c458af49b3c19a2b67c P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
W20240430 20:11:46.716068 20713 negotiation.cc:337] Failed RPC negotiation. Trace:
0430 20:11:46.671304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.177.60:37225 (local address 127.17.177.1:41545)
0430 20:11:46.715667 (+ 44363us) negotiation.cc:107] Waiting for socket to connect
0430 20:11:46.715787 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.177.60:37225: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":44209,"threads_started":1}
W20240430 20:11:46.716435 18211 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.177.60:37225, user_credentials={real_user=slave}} blocked reactor thread for 43884.7us
I20240430 20:11:46.718250 18619 raft_consensus.cc:2454] T c6378f1495144c458af49b3c19a2b67c P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:46.718747 19489 ts_tablet_manager.cc:1888] T 2f61e3f88a6448dfa323204bd8d14cc5 P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:46.718883 19489 log.cc:1196] T 2f61e3f88a6448dfa323204bd8d14cc5 P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/2f61e3f88a6448dfa323204bd8d14cc5
I20240430 20:11:46.719571 19489 ts_tablet_manager.cc:1909] T 2f61e3f88a6448dfa323204bd8d14cc5 P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:46.719874 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 279) took 5722 ms. Trace:
I20240430 20:11:46.719954 19489 rpcz_store.cc:276] 0430 20:11:40.997515 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:40.997894 (+ 379us) service_pool.cc:225] Handling call
0430 20:11:46.696576 (+5698682us) ts_tablet_manager.cc:1120] Deleting tablet 2f61e3f88a6448dfa323204bd8d14cc5
0430 20:11:46.699616 (+ 3040us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:46.704653 (+ 5037us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:46.714775 (+ 10122us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:46.719858 (+ 5083us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5698482}
W20240430 20:11:46.720185 18697 connection.cc:441] server connection from 127.0.0.1:59830 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 279) could send its response
I20240430 20:11:46.720467 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 269) took 5196 ms. Trace:
I20240430 20:11:46.720592 19471 rpcz_store.cc:276] 0430 20:11:41.523781 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:41.526653 (+ 2872us) service_pool.cc:225] Handling call
0430 20:11:46.660723 (+5134070us) ts_tablet_manager.cc:1120] Deleting tablet 95383d9d5a234a4d9d841a8b114428b0
0430 20:11:46.660730 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:46.681636 (+ 20906us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:46.704994 (+ 23358us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:46.720447 (+ 15453us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5133841}
I20240430 20:11:46.721386 18983 tablet_bootstrap.cc:492] T 22fef8bb6b034db098c3372ae992f7a7 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:46.721537 18983 ts_tablet_manager.cc:1367] T 22fef8bb6b034db098c3372ae992f7a7 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.086s user 0.003s sys 0.001s
I20240430 20:11:46.722235 18983 raft_consensus.cc:348] T 22fef8bb6b034db098c3372ae992f7a7 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.722422 18983 raft_consensus.cc:374] T 22fef8bb6b034db098c3372ae992f7a7 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:46.722467 18983 raft_consensus.cc:729] T 22fef8bb6b034db098c3372ae992f7a7 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
W20240430 20:11:46.722443 18417 consensus_peers.cc:485] T a9f37f0f7ce8425f9b24699513d41d61 P 2efb542aa55e439b99ccb98d1c5c884a -> Peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Couldn't send request to peer 3c51c33d5616482cafb68e26688820f4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a9f37f0f7ce8425f9b24699513d41d61. This is attempt 1: this message will repeat every 5th retry.
I20240430 20:11:46.722671 18983 consensus_queue.cc:260] T 22fef8bb6b034db098c3372ae992f7a7 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.723336 18983 ts_tablet_manager.cc:1398] T 22fef8bb6b034db098c3372ae992f7a7 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240430 20:11:46.723678 18983 tablet_bootstrap.cc:492] T 8e33df70a1504dbd852fe0b3c3f61622 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:46.724629 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 280) took 5713 ms. Trace:
I20240430 20:11:46.724723 19489 rpcz_store.cc:276] 0430 20:11:41.010309 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:41.010388 (+ 79us) service_pool.cc:225] Handling call
0430 20:11:46.723710 (+5713322us) ts_tablet_manager.cc:1120] Deleting tablet 7dfdecba959543a795dfcd3eeda29e58
0430 20:11:46.723720 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:46.723809 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5713084}
W20240430 20:11:46.724962 18697 connection.cc:441] server connection from 127.0.0.1:59830 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 280) could send its response
I20240430 20:11:46.725459 18983 tablet_bootstrap.cc:654] T 8e33df70a1504dbd852fe0b3c3f61622 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:46.725550 19489 tablet_replica.cc:330] T e6c0c808bfc94be3837ab835292b3c1b P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:46.725762 19489 raft_consensus.cc:2229] T e6c0c808bfc94be3837ab835292b3c1b P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:46.725957 19489 raft_consensus.cc:2258] T e6c0c808bfc94be3837ab835292b3c1b P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:46.726505 19489 ts_tablet_manager.cc:1875] T e6c0c808bfc94be3837ab835292b3c1b P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240430 20:11:46.730510 18558 connection.cc:441] server connection from 127.0.0.1:40218 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 269) could send its response
I20240430 20:11:46.730993 19471 tablet_replica.cc:330] T 283e5e73cbc04daba0de19f4541f9fd0 P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:46.731417 19471 raft_consensus.cc:2229] T 283e5e73cbc04daba0de19f4541f9fd0 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:46.734833 19471 raft_consensus.cc:2258] T 283e5e73cbc04daba0de19f4541f9fd0 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:46.736429 19471 ts_tablet_manager.cc:1875] T 283e5e73cbc04daba0de19f4541f9fd0 P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:46.742699 18985 tablet_bootstrap.cc:654] T 283e5e73cbc04daba0de19f4541f9fd0 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:46.754238 18983 tablet_bootstrap.cc:492] T 8e33df70a1504dbd852fe0b3c3f61622 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:46.754621 18983 ts_tablet_manager.cc:1367] T 8e33df70a1504dbd852fe0b3c3f61622 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.031s user 0.006s sys 0.005s
I20240430 20:11:46.755625 18983 raft_consensus.cc:348] T 8e33df70a1504dbd852fe0b3c3f61622 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
W20240430 20:11:46.755852 19895 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.177.60:37225, user_credentials={real_user=slave}} blocked reactor thread for 86275.8us
I20240430 20:11:46.756146 18983 raft_consensus.cc:374] T 8e33df70a1504dbd852fe0b3c3f61622 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:46.756263 18983 raft_consensus.cc:729] T 8e33df70a1504dbd852fe0b3c3f61622 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
W20240430 20:11:46.757278 18211 leader_election.cc:334] T 00000000000000000000000000000000 P 89df84ccf5394c9bb60d4e4ed872fc2e [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer b632d6ef4c104a058a6db208024cab33 (127.17.177.60:37225): Network error: Client connection negotiation failed: client connection to 127.17.177.60:37225: connect: Connection refused (error 111)
I20240430 20:11:46.757580 18211 leader_election.cc:302] T 00000000000000000000000000000000 P 89df84ccf5394c9bb60d4e4ed872fc2e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89df84ccf5394c9bb60d4e4ed872fc2e; no voters: 88b3f98793354c7aaa5c106e8ec31ed8, b632d6ef4c104a058a6db208024cab33
W20240430 20:11:46.757803 18415 consensus_peers.cc:485] T d61c279d20b0471a967942525ceb4fcb P 2efb542aa55e439b99ccb98d1c5c884a -> Peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Couldn't send request to peer 5e1c6397f09747f091534faf93542710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d61c279d20b0471a967942525ceb4fcb. This is attempt 1: this message will repeat every 5th retry.
I20240430 20:11:46.758185 20685 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 89df84ccf5394c9bb60d4e4ed872fc2e [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240430 20:11:46.758493 18983 consensus_queue.cc:260] T 8e33df70a1504dbd852fe0b3c3f61622 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.759383 18983 ts_tablet_manager.cc:1398] T 8e33df70a1504dbd852fe0b3c3f61622 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240430 20:11:46.759714 18983 tablet_bootstrap.cc:492] T 04b3dff86d5c48a09d03da58bd48f360 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:46.762845 18988 tablet_bootstrap.cc:492] T 246fb63a5e9842df8263800e28a5935c P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:46.765271 18983 tablet_bootstrap.cc:654] T 04b3dff86d5c48a09d03da58bd48f360 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:46.761317 19471 ts_tablet_manager.cc:1888] T 283e5e73cbc04daba0de19f4541f9fd0 P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:46.766332 19471 log.cc:1196] T 283e5e73cbc04daba0de19f4541f9fd0 P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/283e5e73cbc04daba0de19f4541f9fd0
I20240430 20:11:46.766757 19471 ts_tablet_manager.cc:1909] T 283e5e73cbc04daba0de19f4541f9fd0 P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:46.767905 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 275) took 4639 ms. Trace:
I20240430 20:11:46.771322 20695 raft_consensus.cc:482] T 32c823d1700949ab8b22adcd83b7bfd1 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2efb542aa55e439b99ccb98d1c5c884a)
I20240430 20:11:46.771479 20695 raft_consensus.cc:504] T 32c823d1700949ab8b22adcd83b7bfd1 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.772150 20695 leader_election.cc:288] T 32c823d1700949ab8b22adcd83b7bfd1 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:46.772228 19471 rpcz_store.cc:276] 0430 20:11:42.128749 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:42.129434 (+ 685us) service_pool.cc:225] Handling call
0430 20:11:46.730936 (+4601502us) ts_tablet_manager.cc:1120] Deleting tablet 283e5e73cbc04daba0de19f4541f9fd0
0430 20:11:46.730948 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:46.740738 (+ 9790us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:46.761195 (+ 20457us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:46.767878 (+ 6683us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4601288}
W20240430 20:11:46.772509 18558 connection.cc:441] server connection from 127.0.0.1:40218 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 275) could send its response
W20240430 20:11:46.772789 19895 leader_election.cc:334] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer b632d6ef4c104a058a6db208024cab33 (127.17.177.60:37225): Network error: Client connection negotiation failed: client connection to 127.17.177.60:37225: connect: Connection refused (error 111)
I20240430 20:11:46.772845 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 277) took 4631 ms. Trace:
I20240430 20:11:46.772919 19471 rpcz_store.cc:276] 0430 20:11:42.141166 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:42.141223 (+ 57us) service_pool.cc:225] Handling call
0430 20:11:46.772711 (+4631488us) ts_tablet_manager.cc:1120] Deleting tablet 9c73d0a6bc414d3b9900570848d19a51
0430 20:11:46.772722 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:46.772829 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4631257}
W20240430 20:11:46.773183 18558 connection.cc:441] server connection from 127.0.0.1:40218 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 277) could send its response
I20240430 20:11:46.773151 18619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32c823d1700949ab8b22adcd83b7bfd1" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
W20240430 20:11:46.773224 19895 leader_election.cc:334] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer b632d6ef4c104a058a6db208024cab33 (127.17.177.60:37225): Network error: Client connection negotiation failed: client connection to 127.17.177.60:37225: connect: Connection refused (error 111)
I20240430 20:11:46.773579 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 276) took 4632 ms. Trace:
I20240430 20:11:46.773702 19471 rpcz_store.cc:276] 0430 20:11:42.141024 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:42.141526 (+ 502us) service_pool.cc:225] Handling call
0430 20:11:46.773468 (+4631942us) ts_tablet_manager.cc:1120] Deleting tablet 2aa2a489116940dea3173cad16d94fa5
0430 20:11:46.773477 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:46.773564 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4631787}
W20240430 20:11:46.773968 18558 connection.cc:441] server connection from 127.0.0.1:40218 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 276) could send its response
I20240430 20:11:46.774273 19471 tablet_replica.cc:330] T f7b0132c17d44a9ba17288e0321a7f11 P 5e1c6397f09747f091534faf93542710: stopping tablet replica
W20240430 20:11:46.774749 18699 leader_election.cc:341] T 32c823d1700949ab8b22adcd83b7bfd1 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Not found: Tablet not found: 32c823d1700949ab8b22adcd83b7bfd1
I20240430 20:11:46.774791 19471 raft_consensus.cc:2229] T f7b0132c17d44a9ba17288e0321a7f11 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:46.774878 18699 leader_election.cc:302] T 32c823d1700949ab8b22adcd83b7bfd1 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710
I20240430 20:11:46.774966 19471 raft_consensus.cc:2258] T f7b0132c17d44a9ba17288e0321a7f11 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:46.775373 20695 raft_consensus.cc:2726] T 32c823d1700949ab8b22adcd83b7bfd1 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240430 20:11:46.775697 18985 tablet_bootstrap.cc:492] T 283e5e73cbc04daba0de19f4541f9fd0 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:46.775851 18985 ts_tablet_manager.cc:1367] T 283e5e73cbc04daba0de19f4541f9fd0 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.119s user 0.001s sys 0.003s
I20240430 20:11:46.776582 18985 raft_consensus.cc:348] T 283e5e73cbc04daba0de19f4541f9fd0 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.776789 18985 raft_consensus.cc:374] T 283e5e73cbc04daba0de19f4541f9fd0 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:46.776841 18985 raft_consensus.cc:729] T 283e5e73cbc04daba0de19f4541f9fd0 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:46.779525 19489 ts_tablet_manager.cc:1888] T e6c0c808bfc94be3837ab835292b3c1b P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240430 20:11:46.779649 19489 log.cc:1196] T e6c0c808bfc94be3837ab835292b3c1b P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/e6c0c808bfc94be3837ab835292b3c1b
I20240430 20:11:46.779994 19489 ts_tablet_manager.cc:1909] T e6c0c808bfc94be3837ab835292b3c1b P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:46.780761 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 281) took 5628 ms. Trace:
I20240430 20:11:46.780841 19489 rpcz_store.cc:276] 0430 20:11:41.152041 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:41.152114 (+ 73us) service_pool.cc:225] Handling call
0430 20:11:46.725500 (+5573386us) ts_tablet_manager.cc:1120] Deleting tablet e6c0c808bfc94be3837ab835292b3c1b
0430 20:11:46.725511 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:46.763206 (+ 37695us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:46.779404 (+ 16198us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:46.780737 (+ 1333us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5573173}
W20240430 20:11:46.781080 18697 connection.cc:441] server connection from 127.0.0.1:59830 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 281) could send its response
I20240430 20:11:46.783077 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 285) took 5412 ms. Trace:
I20240430 20:11:46.783165 19489 rpcz_store.cc:276] 0430 20:11:41.370630 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:41.406668 (+ 36038us) service_pool.cc:225] Handling call
0430 20:11:46.782938 (+5376270us) ts_tablet_manager.cc:1120] Deleting tablet 3b7fdc6ec7e046c5af894f9a6eda17b0
0430 20:11:46.782951 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:46.783059 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5376031}
W20240430 20:11:46.783390 18697 connection.cc:441] server connection from 127.0.0.1:59830 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 285) could send its response
I20240430 20:11:46.784035 19489 tablet_replica.cc:330] T 95383d9d5a234a4d9d841a8b114428b0 P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:46.786541 19489 raft_consensus.cc:2229] T 95383d9d5a234a4d9d841a8b114428b0 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:46.786828 19489 raft_consensus.cc:2258] T 95383d9d5a234a4d9d841a8b114428b0 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:46.787688 19489 ts_tablet_manager.cc:1875] T 95383d9d5a234a4d9d841a8b114428b0 P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:46.793525 18983 tablet_bootstrap.cc:492] T 04b3dff86d5c48a09d03da58bd48f360 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:46.793689 18983 ts_tablet_manager.cc:1367] T 04b3dff86d5c48a09d03da58bd48f360 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.034s user 0.001s sys 0.001s
I20240430 20:11:46.794394 18983 raft_consensus.cc:348] T 04b3dff86d5c48a09d03da58bd48f360 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.794632 18983 raft_consensus.cc:374] T 04b3dff86d5c48a09d03da58bd48f360 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:46.794692 18983 raft_consensus.cc:729] T 04b3dff86d5c48a09d03da58bd48f360 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:46.794936 18983 consensus_queue.cc:260] T 04b3dff86d5c48a09d03da58bd48f360 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.797606 18983 ts_tablet_manager.cc:1398] T 04b3dff86d5c48a09d03da58bd48f360 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240430 20:11:46.797921 18983 tablet_bootstrap.cc:492] T a482f4622f774d60b4efd6827b8be7d7 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:46.798631 19471 ts_tablet_manager.cc:1875] T f7b0132c17d44a9ba17288e0321a7f11 P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:46.800659 18983 tablet_bootstrap.cc:654] T a482f4622f774d60b4efd6827b8be7d7 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:46.806775 18985 consensus_queue.cc:260] T 283e5e73cbc04daba0de19f4541f9fd0 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.809741 20489 raft_consensus.cc:482] T f5eaf2413e1d4e69b56fae61f768f0ce P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:46.809901 20489 raft_consensus.cc:504] T f5eaf2413e1d4e69b56fae61f768f0ce P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.810534 20489 leader_election.cc:288] T f5eaf2413e1d4e69b56fae61f768f0ce P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:46.811272 18474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5eaf2413e1d4e69b56fae61f768f0ce" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:46.812111 18756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5eaf2413e1d4e69b56fae61f768f0ce" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:46.812322 18756 raft_consensus.cc:2454] T f5eaf2413e1d4e69b56fae61f768f0ce P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 0.
W20240430 20:11:46.812834 18558 leader_election.cc:341] T f5eaf2413e1d4e69b56fae61f768f0ce P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:46.813004 18988 ts_tablet_manager.cc:1367] T 246fb63a5e9842df8263800e28a5935c P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.205s user 0.020s sys 0.003s
I20240430 20:11:46.813359 18557 leader_election.cc:302] T f5eaf2413e1d4e69b56fae61f768f0ce P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a
I20240430 20:11:46.813692 18988 raft_consensus.cc:348] T 246fb63a5e9842df8263800e28a5935c P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.813871 18988 raft_consensus.cc:374] T 246fb63a5e9842df8263800e28a5935c P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:46.813918 18988 raft_consensus.cc:729] T 246fb63a5e9842df8263800e28a5935c P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:46.814158 18988 consensus_queue.cc:260] T 246fb63a5e9842df8263800e28a5935c P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.817238 20489 raft_consensus.cc:2781] T f5eaf2413e1d4e69b56fae61f768f0ce P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:46.817341 20489 raft_consensus.cc:482] T f5eaf2413e1d4e69b56fae61f768f0ce P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:46.817399 20489 raft_consensus.cc:3037] T f5eaf2413e1d4e69b56fae61f768f0ce P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:46.820384 20489 raft_consensus.cc:504] T f5eaf2413e1d4e69b56fae61f768f0ce P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
W20240430 20:11:46.822101 18415 consensus_peers.cc:485] T a9f37f0f7ce8425f9b24699513d41d61 P 2efb542aa55e439b99ccb98d1c5c884a -> Peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Couldn't send request to peer 5e1c6397f09747f091534faf93542710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a9f37f0f7ce8425f9b24699513d41d61. This is attempt 1: this message will repeat every 5th retry.
I20240430 20:11:46.822993 18756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5eaf2413e1d4e69b56fae61f768f0ce" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:46.823199 18756 raft_consensus.cc:3037] T f5eaf2413e1d4e69b56fae61f768f0ce P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:46.825450 20587 raft_consensus.cc:482] T 22fef8bb6b034db098c3372ae992f7a7 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:46.825599 20587 raft_consensus.cc:504] T 22fef8bb6b034db098c3372ae992f7a7 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.825927 18756 raft_consensus.cc:2454] T f5eaf2413e1d4e69b56fae61f768f0ce P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
I20240430 20:11:46.826181 20587 leader_election.cc:288] T 22fef8bb6b034db098c3372ae992f7a7 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:46.826886 18474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22fef8bb6b034db098c3372ae992f7a7" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:46.827337 18619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22fef8bb6b034db098c3372ae992f7a7" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:46.827548 18619 raft_consensus.cc:2454] T 22fef8bb6b034db098c3372ae992f7a7 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:46.827873 20047 raft_consensus.cc:482] T a0baee91f5b44945b595d0e234af018e P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:46.827983 20047 raft_consensus.cc:504] T a0baee91f5b44945b595d0e234af018e P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.828550 20047 leader_election.cc:288] T a0baee91f5b44945b595d0e234af018e P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:46.828876 18696 leader_election.cc:302] T 22fef8bb6b034db098c3372ae992f7a7 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:46.829269 20587 raft_consensus.cc:2781] T 22fef8bb6b034db098c3372ae992f7a7 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:46.829286 18557 leader_election.cc:302] T f5eaf2413e1d4e69b56fae61f768f0ce P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:46.829339 20587 raft_consensus.cc:482] T 22fef8bb6b034db098c3372ae992f7a7 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:46.829396 20587 raft_consensus.cc:3037] T 22fef8bb6b034db098c3372ae992f7a7 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:46.829530 20047 raft_consensus.cc:2781] T f5eaf2413e1d4e69b56fae61f768f0ce P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:46.829629 20047 raft_consensus.cc:686] T f5eaf2413e1d4e69b56fae61f768f0ce P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Becoming Leader. State: Replica: 5e1c6397f09747f091534faf93542710, State: Running, Role: LEADER
I20240430 20:11:46.829847 20047 consensus_queue.cc:237] T f5eaf2413e1d4e69b56fae61f768f0ce P 5e1c6397f09747f091534faf93542710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.831604 18474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0baee91f5b44945b595d0e234af018e" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:46.831809 18474 raft_consensus.cc:2454] T a0baee91f5b44945b595d0e234af018e P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 0.
I20240430 20:11:46.831938 20587 raft_consensus.cc:504] T 22fef8bb6b034db098c3372ae992f7a7 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.832468 18558 leader_election.cc:302] T a0baee91f5b44945b595d0e234af018e P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:46.833092 20488 raft_consensus.cc:2781] T a0baee91f5b44945b595d0e234af018e P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:46.833173 20488 raft_consensus.cc:482] T a0baee91f5b44945b595d0e234af018e P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:46.833235 20488 raft_consensus.cc:3037] T a0baee91f5b44945b595d0e234af018e P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
W20240430 20:11:46.832144 18699 leader_election.cc:341] T 22fef8bb6b034db098c3372ae992f7a7 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:46.834642 18619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22fef8bb6b034db098c3372ae992f7a7" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:46.834806 18619 raft_consensus.cc:3037] T 22fef8bb6b034db098c3372ae992f7a7 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:46.836061 20488 raft_consensus.cc:504] T a0baee91f5b44945b595d0e234af018e P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.836623 20488 leader_election.cc:288] T a0baee91f5b44945b595d0e234af018e P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:46.837319 18619 raft_consensus.cc:2454] T 22fef8bb6b034db098c3372ae992f7a7 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:46.837363 18474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0baee91f5b44945b595d0e234af018e" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:46.837538 18474 raft_consensus.cc:3037] T a0baee91f5b44945b595d0e234af018e P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:46.837875 20587 leader_election.cc:288] T 22fef8bb6b034db098c3372ae992f7a7 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Requested vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:46.838279 18988 ts_tablet_manager.cc:1398] T 246fb63a5e9842df8263800e28a5935c P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.025s user 0.002s sys 0.000s
I20240430 20:11:46.839165 18750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0baee91f5b44945b595d0e234af018e" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:46.839335 18750 raft_consensus.cc:3037] T a0baee91f5b44945b595d0e234af018e P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:46.840587 18696 leader_election.cc:302] T 22fef8bb6b034db098c3372ae992f7a7 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:46.840950 19489 ts_tablet_manager.cc:1888] T 95383d9d5a234a4d9d841a8b114428b0 P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:46.841056 19489 log.cc:1196] T 95383d9d5a234a4d9d841a8b114428b0 P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/95383d9d5a234a4d9d841a8b114428b0
I20240430 20:11:46.841116 18474 raft_consensus.cc:2454] T a0baee91f5b44945b595d0e234af018e P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
I20240430 20:11:46.841365 19489 ts_tablet_manager.cc:1909] T 95383d9d5a234a4d9d841a8b114428b0 P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:46.841755 18558 leader_election.cc:302] T a0baee91f5b44945b595d0e234af018e P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:46.841820 18750 raft_consensus.cc:2454] T a0baee91f5b44945b595d0e234af018e P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
I20240430 20:11:46.842088 18756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0baee91f5b44945b595d0e234af018e" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:46.842214 20587 raft_consensus.cc:2781] T 22fef8bb6b034db098c3372ae992f7a7 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:46.842149 18473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22fef8bb6b034db098c3372ae992f7a7" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:46.842300 18756 raft_consensus.cc:2362] T a0baee91f5b44945b595d0e234af018e P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1. Re-sending same reply.
I20240430 20:11:46.842322 20587 raft_consensus.cc:686] T 22fef8bb6b034db098c3372ae992f7a7 P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Becoming Leader. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Running, Role: LEADER
I20240430 20:11:46.842676 20488 raft_consensus.cc:2781] T a0baee91f5b44945b595d0e234af018e P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:46.842608 20587 consensus_queue.cc:237] T 22fef8bb6b034db098c3372ae992f7a7 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.842905 20488 raft_consensus.cc:686] T a0baee91f5b44945b595d0e234af018e P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Becoming Leader. State: Replica: 5e1c6397f09747f091534faf93542710, State: Running, Role: LEADER
I20240430 20:11:46.842088 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 293) took 5307 ms. Trace:
I20240430 20:11:46.843091 19489 rpcz_store.cc:276] 0430 20:11:41.534492 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:41.534556 (+ 64us) service_pool.cc:225] Handling call
0430 20:11:46.783988 (+5249432us) ts_tablet_manager.cc:1120] Deleting tablet 95383d9d5a234a4d9d841a8b114428b0
0430 20:11:46.783999 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:46.828457 (+ 44458us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:46.840834 (+ 12377us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:46.842072 (+ 1238us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2251,"tablet-delete.queue_time_us":5248477}
W20240430 20:11:46.843330 18699 leader_election.cc:341] T 22fef8bb6b034db098c3372ae992f7a7 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:46.843230 20488 consensus_queue.cc:237] T a0baee91f5b44945b595d0e234af018e P 5e1c6397f09747f091534faf93542710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.843539 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 304) took 4628 ms. Trace:
I20240430 20:11:46.843609 19489 rpcz_store.cc:276] 0430 20:11:42.215059 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:42.215108 (+ 49us) service_pool.cc:225] Handling call
0430 20:11:46.843421 (+4628313us) ts_tablet_manager.cc:1120] Deleting tablet b22aaedb0a6c4fd9ad0ea8fa019001e1
0430 20:11:46.843432 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:46.843525 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4628060}
I20240430 20:11:46.843868 19489 tablet_replica.cc:330] T ae87f5e92a934e2182164c231bc76df0 P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:46.844039 19489 raft_consensus.cc:2229] T ae87f5e92a934e2182164c231bc76df0 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:46.844173 19489 raft_consensus.cc:2258] T ae87f5e92a934e2182164c231bc76df0 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Raft consensus is shut down!
W20240430 20:11:46.843353 18697 connection.cc:441] server connection from 127.0.0.1:59830 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 293) could send its response
W20240430 20:11:46.844360 18697 connection.cc:441] server connection from 127.0.0.1:59830 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 304) could send its response
I20240430 20:11:46.844710 19489 ts_tablet_manager.cc:1875] T ae87f5e92a934e2182164c231bc76df0 P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:46.851778 18988 tablet_bootstrap.cc:492] T c6e3d09402574f1e8fe9e4160c70a3ee P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:46.852417 20489 leader_election.cc:288] T f5eaf2413e1d4e69b56fae61f768f0ce P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:46.853098 18988 tablet_bootstrap.cc:654] T c6e3d09402574f1e8fe9e4160c70a3ee P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:46.853210 18473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5eaf2413e1d4e69b56fae61f768f0ce" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
W20240430 20:11:46.853950 18558 leader_election.cc:341] T f5eaf2413e1d4e69b56fae61f768f0ce P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:46.859629 18985 ts_tablet_manager.cc:1398] T 283e5e73cbc04daba0de19f4541f9fd0 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.084s user 0.000s sys 0.002s
I20240430 20:11:46.866123 19489 ts_tablet_manager.cc:1888] T ae87f5e92a934e2182164c231bc76df0 P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:46.866276 19489 log.cc:1196] T ae87f5e92a934e2182164c231bc76df0 P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/ae87f5e92a934e2182164c231bc76df0
I20240430 20:11:46.866626 19489 ts_tablet_manager.cc:1909] T ae87f5e92a934e2182164c231bc76df0 P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:46.867440 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 305) took 4652 ms. Trace:
I20240430 20:11:46.867538 19489 rpcz_store.cc:276] 0430 20:11:42.215168 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:42.215246 (+ 78us) service_pool.cc:225] Handling call
0430 20:11:46.843827 (+4628581us) ts_tablet_manager.cc:1120] Deleting tablet ae87f5e92a934e2182164c231bc76df0
0430 20:11:46.843836 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:46.847713 (+ 3877us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:46.850511 (+ 2798us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:46.867416 (+ 16905us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4628370}
W20240430 20:11:46.867774 18697 connection.cc:441] server connection from 127.0.0.1:59830 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 305) could send its response
I20240430 20:11:46.871995 18988 tablet_bootstrap.cc:492] T c6e3d09402574f1e8fe9e4160c70a3ee P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:46.872170 18988 ts_tablet_manager.cc:1367] T c6e3d09402574f1e8fe9e4160c70a3ee P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.004s
I20240430 20:11:46.872958 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 299) took 4739 ms. Trace:
I20240430 20:11:46.873732 19489 rpcz_store.cc:276] 0430 20:11:42.133722 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:42.133793 (+ 71us) service_pool.cc:225] Handling call
0430 20:11:46.872822 (+4739029us) ts_tablet_manager.cc:1120] Deleting tablet 2aa2a489116940dea3173cad16d94fa5
0430 20:11:46.872832 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:46.872939 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4656133}
I20240430 20:11:46.874577 18983 tablet_bootstrap.cc:492] T a482f4622f774d60b4efd6827b8be7d7 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:46.878275 18985 tablet_bootstrap.cc:492] T 22fef8bb6b034db098c3372ae992f7a7 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:46.880311 18985 tablet_bootstrap.cc:654] T 22fef8bb6b034db098c3372ae992f7a7 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
W20240430 20:11:46.882727 18697 connection.cc:441] server connection from 127.0.0.1:59830 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 299) could send its response
I20240430 20:11:46.882838 19489 tablet_replica.cc:330] T 4f6f9ae722da49e58c9b72107f6da2fa P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:46.882727 18988 raft_consensus.cc:348] T c6e3d09402574f1e8fe9e4160c70a3ee P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.883078 19489 raft_consensus.cc:2229] T 4f6f9ae722da49e58c9b72107f6da2fa P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:46.883136 18988 raft_consensus.cc:374] T c6e3d09402574f1e8fe9e4160c70a3ee P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:46.883196 18988 raft_consensus.cc:729] T c6e3d09402574f1e8fe9e4160c70a3ee P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:46.883260 19489 raft_consensus.cc:2258] T 4f6f9ae722da49e58c9b72107f6da2fa P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:46.883432 18988 consensus_queue.cc:260] T c6e3d09402574f1e8fe9e4160c70a3ee P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.883548 18983 ts_tablet_manager.cc:1367] T a482f4622f774d60b4efd6827b8be7d7 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.086s user 0.003s sys 0.000s
I20240430 20:11:46.884721 19489 ts_tablet_manager.cc:1875] T 4f6f9ae722da49e58c9b72107f6da2fa P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:46.885438 18983 raft_consensus.cc:348] T a482f4622f774d60b4efd6827b8be7d7 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.885618 18983 raft_consensus.cc:374] T a482f4622f774d60b4efd6827b8be7d7 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:46.885668 18983 raft_consensus.cc:729] T a482f4622f774d60b4efd6827b8be7d7 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:46.885867 18983 consensus_queue.cc:260] T a482f4622f774d60b4efd6827b8be7d7 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.888748 18988 ts_tablet_manager.cc:1398] T c6e3d09402574f1e8fe9e4160c70a3ee P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.017s user 0.001s sys 0.001s
I20240430 20:11:46.889642 18983 ts_tablet_manager.cc:1398] T a482f4622f774d60b4efd6827b8be7d7 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240430 20:11:46.895231 18988 tablet_bootstrap.cc:492] T a482f4622f774d60b4efd6827b8be7d7 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:46.896585 18988 tablet_bootstrap.cc:654] T a482f4622f774d60b4efd6827b8be7d7 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:46.898937 19489 ts_tablet_manager.cc:1888] T 4f6f9ae722da49e58c9b72107f6da2fa P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:46.899118 19489 log.cc:1196] T 4f6f9ae722da49e58c9b72107f6da2fa P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/4f6f9ae722da49e58c9b72107f6da2fa
I20240430 20:11:46.901765 19489 ts_tablet_manager.cc:1909] T 4f6f9ae722da49e58c9b72107f6da2fa P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:46.902627 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 302) took 4690 ms. Trace:
I20240430 20:11:46.902724 20630 raft_consensus.cc:482] T 04b3dff86d5c48a09d03da58bd48f360 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:46.902873 20630 raft_consensus.cc:504] T 04b3dff86d5c48a09d03da58bd48f360 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.903463 20630 leader_election.cc:288] T 04b3dff86d5c48a09d03da58bd48f360 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:46.903612 19489 rpcz_store.cc:276] 0430 20:11:42.211988 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:42.212885 (+ 897us) service_pool.cc:225] Handling call
0430 20:11:46.882781 (+4669896us) ts_tablet_manager.cc:1120] Deleting tablet 4f6f9ae722da49e58c9b72107f6da2fa
0430 20:11:46.882793 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:46.895623 (+ 12830us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:46.898826 (+ 3203us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:46.902585 (+ 3759us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4665403}
W20240430 20:11:46.903970 18697 connection.cc:441] server connection from 127.0.0.1:59830 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 302) could send its response
I20240430 20:11:46.907507 18619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04b3dff86d5c48a09d03da58bd48f360" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:46.907711 18619 raft_consensus.cc:2454] T 04b3dff86d5c48a09d03da58bd48f360 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 0.
I20240430 20:11:46.907879 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 298) took 4777 ms. Trace:
I20240430 20:11:46.908154 18756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04b3dff86d5c48a09d03da58bd48f360" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:46.908414 18415 leader_election.cc:302] T 04b3dff86d5c48a09d03da58bd48f360 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:46.908699 20545 consensus_queue.cc:1035] T eb76f60448774fe1967fffe7c6873cfc P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:46.908718 19489 rpcz_store.cc:276] 0430 20:11:42.130122 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:42.130327 (+ 205us) service_pool.cc:225] Handling call
0430 20:11:46.904012 (+4773685us) ts_tablet_manager.cc:1120] Deleting tablet 283e5e73cbc04daba0de19f4541f9fd0
0430 20:11:46.904023 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:46.907861 (+ 3838us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4686540}
I20240430 20:11:46.909008 19489 tablet_replica.cc:330] T f7b0132c17d44a9ba17288e0321a7f11 P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:46.909116 20545 raft_consensus.cc:2781] T 04b3dff86d5c48a09d03da58bd48f360 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:46.909178 20545 raft_consensus.cc:482] T 04b3dff86d5c48a09d03da58bd48f360 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:46.909193 19489 raft_consensus.cc:2229] T f7b0132c17d44a9ba17288e0321a7f11 P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Raft consensus shutting down.
I20240430 20:11:46.909238 20545 raft_consensus.cc:3037] T 04b3dff86d5c48a09d03da58bd48f360 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:46.909512 19489 raft_consensus.cc:2258] T f7b0132c17d44a9ba17288e0321a7f11 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:46.911916 20545 raft_consensus.cc:504] T 04b3dff86d5c48a09d03da58bd48f360 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.912319 19489 ts_tablet_manager.cc:1875] T f7b0132c17d44a9ba17288e0321a7f11 P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:46.913411 20545 leader_election.cc:288] T 04b3dff86d5c48a09d03da58bd48f360 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:46.914129 18756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04b3dff86d5c48a09d03da58bd48f360" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:46.916680 18623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04b3dff86d5c48a09d03da58bd48f360" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:46.916854 18623 raft_consensus.cc:3037] T 04b3dff86d5c48a09d03da58bd48f360 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
W20240430 20:11:46.917505 18697 connection.cc:441] server connection from 127.0.0.1:59830 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 298) could send its response
W20240430 20:11:46.915205 18417 leader_election.cc:341] T 04b3dff86d5c48a09d03da58bd48f360 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
W20240430 20:11:46.918769 18417 leader_election.cc:341] T 04b3dff86d5c48a09d03da58bd48f360 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:46.919732 18623 raft_consensus.cc:2454] T 04b3dff86d5c48a09d03da58bd48f360 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 1.
I20240430 20:11:46.920280 18415 leader_election.cc:302] T 04b3dff86d5c48a09d03da58bd48f360 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters: 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:46.920838 18983 tablet_bootstrap.cc:492] T 331c031d44854d679a27d10d9151d0ea P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:46.920889 20630 raft_consensus.cc:2781] T 04b3dff86d5c48a09d03da58bd48f360 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:46.921010 20630 raft_consensus.cc:686] T 04b3dff86d5c48a09d03da58bd48f360 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Becoming Leader. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Running, Role: LEADER
I20240430 20:11:46.921247 20630 consensus_queue.cc:237] T 04b3dff86d5c48a09d03da58bd48f360 P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.922624 18983 tablet_bootstrap.cc:654] T 331c031d44854d679a27d10d9151d0ea P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:46.927639 19471 ts_tablet_manager.cc:1888] T f7b0132c17d44a9ba17288e0321a7f11 P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:46.927832 19471 log.cc:1196] T f7b0132c17d44a9ba17288e0321a7f11 P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/f7b0132c17d44a9ba17288e0321a7f11
I20240430 20:11:46.928241 19471 ts_tablet_manager.cc:1909] T f7b0132c17d44a9ba17288e0321a7f11 P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:46.937053 19489 ts_tablet_manager.cc:1888] T f7b0132c17d44a9ba17288e0321a7f11 P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:46.937184 19489 log.cc:1196] T f7b0132c17d44a9ba17288e0321a7f11 P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/f7b0132c17d44a9ba17288e0321a7f11
I20240430 20:11:46.937544 19489 ts_tablet_manager.cc:1909] T f7b0132c17d44a9ba17288e0321a7f11 P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:46.937994 20631 consensus_queue.cc:1035] T eb76f60448774fe1967fffe7c6873cfc P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:46.938234 18988 tablet_bootstrap.cc:492] T a482f4622f774d60b4efd6827b8be7d7 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:46.938292 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 301) took 4726 ms. Trace:
I20240430 20:11:46.938350 18988 ts_tablet_manager.cc:1367] T a482f4622f774d60b4efd6827b8be7d7 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.043s user 0.002s sys 0.002s
I20240430 20:11:46.938405 19489 rpcz_store.cc:276] 0430 20:11:42.211896 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:42.211941 (+ 45us) service_pool.cc:225] Handling call
0430 20:11:46.908964 (+4697023us) ts_tablet_manager.cc:1120] Deleting tablet f7b0132c17d44a9ba17288e0321a7f11
0430 20:11:46.908973 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:46.916145 (+ 7172us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:46.936928 (+ 20783us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:46.938270 (+ 1342us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4691385}
W20240430 20:11:46.938738 18697 connection.cc:441] server connection from 127.0.0.1:59830 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 301) could send its response
I20240430 20:11:46.938851 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 300) took 4727 ms. Trace:
I20240430 20:11:46.938911 19489 rpcz_store.cc:276] 0430 20:11:42.211761 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:42.212709 (+ 948us) service_pool.cc:225] Handling call
0430 20:11:46.938728 (+4726019us) ts_tablet_manager.cc:1120] Deleting tablet 9c73d0a6bc414d3b9900570848d19a51
0430 20:11:46.938738 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:46.938838 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4708740}
I20240430 20:11:46.939064 18988 raft_consensus.cc:348] T a482f4622f774d60b4efd6827b8be7d7 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.939218 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 306) took 4723 ms. Trace:
I20240430 20:11:46.939256 18988 raft_consensus.cc:374] T a482f4622f774d60b4efd6827b8be7d7 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:46.939308 18988 raft_consensus.cc:729] T a482f4622f774d60b4efd6827b8be7d7 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:46.939283 19489 rpcz_store.cc:276] 0430 20:11:42.215290 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:42.215725 (+ 435us) service_pool.cc:225] Handling call
0430 20:11:46.939122 (+4723397us) ts_tablet_manager.cc:1120] Deleting tablet fdf53061efd74fa29a9d8208d1fd210e
0430 20:11:46.939133 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:46.939206 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4707817}
I20240430 20:11:46.939545 18988 consensus_queue.cc:260] T a482f4622f774d60b4efd6827b8be7d7 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.939813 19489 tablet_replica.cc:330] T 32c823d1700949ab8b22adcd83b7bfd1 P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:46.940018 19489 raft_consensus.cc:2229] T 32c823d1700949ab8b22adcd83b7bfd1 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:46.940083 18988 ts_tablet_manager.cc:1398] T a482f4622f774d60b4efd6827b8be7d7 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240430 20:11:46.940163 19489 raft_consensus.cc:2258] T 32c823d1700949ab8b22adcd83b7bfd1 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240430 20:11:46.940256 18697 connection.cc:441] server connection from 127.0.0.1:59830 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 300) could send its response
W20240430 20:11:46.940389 18697 connection.cc:441] server connection from 127.0.0.1:59830 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 306) could send its response
I20240430 20:11:46.940364 18988 tablet_bootstrap.cc:492] T 04b3dff86d5c48a09d03da58bd48f360 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:46.946326 18988 tablet_bootstrap.cc:654] T 04b3dff86d5c48a09d03da58bd48f360 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:46.940765 19489 ts_tablet_manager.cc:1875] T 32c823d1700949ab8b22adcd83b7bfd1 P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:46.961681 19489 ts_tablet_manager.cc:1888] T 32c823d1700949ab8b22adcd83b7bfd1 P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:46.961818 19489 log.cc:1196] T 32c823d1700949ab8b22adcd83b7bfd1 P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/32c823d1700949ab8b22adcd83b7bfd1
I20240430 20:11:46.966886 19489 ts_tablet_manager.cc:1909] T 32c823d1700949ab8b22adcd83b7bfd1 P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:46.967707 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 303) took 4752 ms. Trace:
I20240430 20:11:46.967805 19489 rpcz_store.cc:276] 0430 20:11:42.214766 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:42.214837 (+ 71us) service_pool.cc:225] Handling call
0430 20:11:46.939766 (+4724929us) ts_tablet_manager.cc:1120] Deleting tablet 32c823d1700949ab8b22adcd83b7bfd1
0430 20:11:46.939776 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:46.949666 (+ 9890us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:46.961560 (+ 11894us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:46.967685 (+ 6125us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4705092}
I20240430 20:11:46.968202 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 320) took 4288 ms. Trace:
I20240430 20:11:46.968272 19489 rpcz_store.cc:276] 0430 20:11:42.680177 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:42.680240 (+ 63us) service_pool.cc:225] Handling call
0430 20:11:46.968080 (+4287840us) ts_tablet_manager.cc:1120] Deleting tablet dad16ae02b8b4e29a724d8eec67bb4b8
0430 20:11:46.968090 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:46.968188 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4287657}
I20240430 20:11:46.968533 19489 tablet_replica.cc:330] T 7683caba80594627900a0185f772c131 P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:46.968705 19489 raft_consensus.cc:2229] T 7683caba80594627900a0185f772c131 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:46.968863 19489 raft_consensus.cc:2258] T 7683caba80594627900a0185f772c131 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:46.969417 19489 ts_tablet_manager.cc:1875] T 7683caba80594627900a0185f772c131 P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240430 20:11:46.970336 18697 connection.cc:441] server connection from 127.0.0.1:59830 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 303) could send its response
W20240430 20:11:46.970489 18697 connection.cc:441] server connection from 127.0.0.1:59830 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 320) could send its response
I20240430 20:11:46.974547 20587 raft_consensus.cc:482] T a482f4622f774d60b4efd6827b8be7d7 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:46.974700 20587 raft_consensus.cc:504] T a482f4622f774d60b4efd6827b8be7d7 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.975268 20587 leader_election.cc:288] T a482f4622f774d60b4efd6827b8be7d7 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:46.976470 18473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a482f4622f774d60b4efd6827b8be7d7" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:46.976680 18473 raft_consensus.cc:2454] T a482f4622f774d60b4efd6827b8be7d7 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:46.977530 18699 leader_election.cc:302] T a482f4622f774d60b4efd6827b8be7d7 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters:
I20240430 20:11:46.977864 20587 raft_consensus.cc:2781] T a482f4622f774d60b4efd6827b8be7d7 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:46.977943 20587 raft_consensus.cc:482] T a482f4622f774d60b4efd6827b8be7d7 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:46.978003 20587 raft_consensus.cc:3037] T a482f4622f774d60b4efd6827b8be7d7 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:46.980489 20587 raft_consensus.cc:504] T a482f4622f774d60b4efd6827b8be7d7 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.984272 18235 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 89df84ccf5394c9bb60d4e4ed872fc2e [term 6 FOLLOWER]: Refusing update from remote peer 88b3f98793354c7aaa5c106e8ec31ed8: Log matching property violated. Preceding OpId in replica: term: 5 index: 749. Preceding OpId from leader: term: 6 index: 750. (index mismatch)
I20240430 20:11:46.985160 20686 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e" member_type: VOTER last_known_addr { host: "127.17.177.61" port: 34777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 750, Last known committed idx: 748, Time since last communication: 0.000s
I20240430 20:11:46.985210 18473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a482f4622f774d60b4efd6827b8be7d7" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:46.985402 18473 raft_consensus.cc:3037] T a482f4622f774d60b4efd6827b8be7d7 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:46.988147 18473 raft_consensus.cc:2454] T a482f4622f774d60b4efd6827b8be7d7 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:46.990233 18699 leader_election.cc:302] T a482f4622f774d60b4efd6827b8be7d7 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters:
I20240430 20:11:46.990497 20695 raft_consensus.cc:2781] T a482f4622f774d60b4efd6827b8be7d7 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:46.990624 20695 raft_consensus.cc:686] T a482f4622f774d60b4efd6827b8be7d7 P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Becoming Leader. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Running, Role: LEADER
I20240430 20:11:46.990846 20695 consensus_queue.cc:237] T a482f4622f774d60b4efd6827b8be7d7 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.992770 20693 raft_consensus.cc:482] T c6e3d09402574f1e8fe9e4160c70a3ee P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:46.992887 20693 raft_consensus.cc:504] T c6e3d09402574f1e8fe9e4160c70a3ee P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.993412 20693 leader_election.cc:288] T c6e3d09402574f1e8fe9e4160c70a3ee P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:46.995854 19489 ts_tablet_manager.cc:1888] T 7683caba80594627900a0185f772c131 P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:46.996059 19489 log.cc:1196] T 7683caba80594627900a0185f772c131 P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/7683caba80594627900a0185f772c131
I20240430 20:11:46.996510 19489 ts_tablet_manager.cc:1909] T 7683caba80594627900a0185f772c131 P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:46.997088 20693 raft_consensus.cc:482] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 3c51c33d5616482cafb68e26688820f4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 2efb542aa55e439b99ccb98d1c5c884a)
I20240430 20:11:46.997213 20693 raft_consensus.cc:504] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 3c51c33d5616482cafb68e26688820f4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:46.997730 20693 leader_election.cc:288] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
W20240430 20:11:46.999292 18699 leader_election.cc:341] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Not found: Tablet not found: dd6083f12c7e4d1e8a2006b3ddf9aa3a
I20240430 20:11:47.000579 18473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e3d09402574f1e8fe9e4160c70a3ee" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
W20240430 20:11:47.001214 18699 leader_election.cc:341] T c6e3d09402574f1e8fe9e4160c70a3ee P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:47.003119 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 326) took 4258 ms. Trace:
I20240430 20:11:47.003298 19489 rpcz_store.cc:276] 0430 20:11:42.744778 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:42.746714 (+ 1936us) service_pool.cc:225] Handling call
0430 20:11:46.968493 (+4221779us) ts_tablet_manager.cc:1120] Deleting tablet 7683caba80594627900a0185f772c131
0430 20:11:46.968501 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:46.972357 (+ 3856us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:46.995722 (+ 23365us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:47.003093 (+ 7371us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5611,"tablet-delete.queue_time_us":4221536}
W20240430 20:11:47.003744 18697 connection.cc:441] server connection from 127.0.0.1:59830 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 326) could send its response
I20240430 20:11:47.004196 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 327) took 4196 ms. Trace:
I20240430 20:11:47.004345 19489 rpcz_store.cc:276] 0430 20:11:42.807329 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:42.819824 (+ 12495us) service_pool.cc:225] Handling call
0430 20:11:47.004045 (+4184221us) ts_tablet_manager.cc:1120] Deleting tablet 2797eed593fb4b06a3c4d6a28e8075a2
0430 20:11:47.004059 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:47.004178 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4184013}
I20240430 20:11:47.004361 20545 consensus_queue.cc:1035] T 677481aee5fa4eb4b189322d7a3f8f2e P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240430 20:11:47.004698 18697 connection.cc:441] server connection from 127.0.0.1:59830 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 327) could send its response
I20240430 20:11:47.004987 19489 tablet_replica.cc:330] T a05e32a33aa7488eb287cc6314c09c44 P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:47.005273 19489 raft_consensus.cc:2229] T a05e32a33aa7488eb287cc6314c09c44 P 3c51c33d5616482cafb68e26688820f4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:46.935030 18983 tablet_bootstrap.cc:492] T 331c031d44854d679a27d10d9151d0ea P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:46.976045 18623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a482f4622f774d60b4efd6827b8be7d7" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:47.006184 18623 raft_consensus.cc:2454] T a482f4622f774d60b4efd6827b8be7d7 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:47.006292 19489 raft_consensus.cc:2258] T a05e32a33aa7488eb287cc6314c09c44 P 3c51c33d5616482cafb68e26688820f4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:46.981889 18620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a482f4622f774d60b4efd6827b8be7d7" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:47.006511 18620 raft_consensus.cc:3037] T a482f4622f774d60b4efd6827b8be7d7 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:47.006906 19489 ts_tablet_manager.cc:1875] T a05e32a33aa7488eb287cc6314c09c44 P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:47.007143 20488 raft_consensus.cc:482] T 32c823d1700949ab8b22adcd83b7bfd1 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2efb542aa55e439b99ccb98d1c5c884a)
I20240430 20:11:47.007257 20488 raft_consensus.cc:504] T 32c823d1700949ab8b22adcd83b7bfd1 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:47.007786 20488 leader_election.cc:288] T 32c823d1700949ab8b22adcd83b7bfd1 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
W20240430 20:11:47.008844 18557 leader_election.cc:341] T 32c823d1700949ab8b22adcd83b7bfd1 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Not found: Tablet not found: 32c823d1700949ab8b22adcd83b7bfd1
I20240430 20:11:47.009094 18620 raft_consensus.cc:2454] T a482f4622f774d60b4efd6827b8be7d7 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:46.998404 18616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd6083f12c7e4d1e8a2006b3ddf9aa3a" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:47.010607 18696 leader_election.cc:302] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710
I20240430 20:11:47.010876 20693 raft_consensus.cc:2726] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 3c51c33d5616482cafb68e26688820f4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240430 20:11:47.011811 20587 leader_election.cc:288] T a482f4622f774d60b4efd6827b8be7d7 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Requested vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
W20240430 20:11:47.011826 18558 leader_election.cc:341] T 32c823d1700949ab8b22adcd83b7bfd1 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Not found: Tablet not found: 32c823d1700949ab8b22adcd83b7bfd1
I20240430 20:11:47.011962 18558 leader_election.cc:302] T 32c823d1700949ab8b22adcd83b7bfd1 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:47.012198 20488 raft_consensus.cc:2726] T 32c823d1700949ab8b22adcd83b7bfd1 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240430 20:11:47.012521 18983 ts_tablet_manager.cc:1367] T 331c031d44854d679a27d10d9151d0ea P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.092s user 0.002s sys 0.001s
I20240430 20:11:47.013058 19489 ts_tablet_manager.cc:1888] T a05e32a33aa7488eb287cc6314c09c44 P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.3
I20240430 20:11:47.013305 18983 raft_consensus.cc:348] T 331c031d44854d679a27d10d9151d0ea P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:47.013489 18983 raft_consensus.cc:374] T 331c031d44854d679a27d10d9151d0ea P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:47.013566 18983 raft_consensus.cc:729] T 331c031d44854d679a27d10d9151d0ea P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:47.019277 18619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e3d09402574f1e8fe9e4160c70a3ee" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:47.019513 18619 raft_consensus.cc:2454] T c6e3d09402574f1e8fe9e4160c70a3ee P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:47.020862 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 278) took 4827 ms. Trace:
I20240430 20:11:47.020962 19471 rpcz_store.cc:276] 0430 20:11:42.192951 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:42.193136 (+ 185us) service_pool.cc:225] Handling call
0430 20:11:46.774194 (+4581058us) ts_tablet_manager.cc:1120] Deleting tablet f7b0132c17d44a9ba17288e0321a7f11
0430 20:11:46.774203 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:46.862260 (+ 88057us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:46.927527 (+ 65267us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:47.020842 (+ 93315us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4580838}
W20240430 20:11:47.021199 18558 connection.cc:441] server connection from 127.0.0.1:40218 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 278) could send its response
I20240430 20:11:47.023013 19471 tablet_replica.cc:330] T b22aaedb0a6c4fd9ad0ea8fa019001e1 P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:47.023219 19471 raft_consensus.cc:2229] T b22aaedb0a6c4fd9ad0ea8fa019001e1 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:47.023367 19471 raft_consensus.cc:2258] T b22aaedb0a6c4fd9ad0ea8fa019001e1 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:47.024585 18985 tablet_bootstrap.cc:492] T 22fef8bb6b034db098c3372ae992f7a7 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:47.020135 18696 leader_election.cc:302] T c6e3d09402574f1e8fe9e4160c70a3ee P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a
I20240430 20:11:47.024744 18985 ts_tablet_manager.cc:1367] T 22fef8bb6b034db098c3372ae992f7a7 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.147s user 0.003s sys 0.003s
I20240430 20:11:47.023412 18750 raft_consensus.cc:1261] T ca64de16d40f4a44b8b411f5526ba11a P 3c51c33d5616482cafb68e26688820f4 [term 2 FOLLOWER]: Refusing update from remote peer 2efb542aa55e439b99ccb98d1c5c884a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240430 20:11:47.025501 19489 log.cc:1196] T a05e32a33aa7488eb287cc6314c09c44 P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/a05e32a33aa7488eb287cc6314c09c44
I20240430 20:11:47.025436 18985 raft_consensus.cc:348] T 22fef8bb6b034db098c3372ae992f7a7 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:47.025704 18983 consensus_queue.cc:260] T 331c031d44854d679a27d10d9151d0ea P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:47.026196 19489 ts_tablet_manager.cc:1909] T a05e32a33aa7488eb287cc6314c09c44 P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:47.026239 20631 consensus_queue.cc:1035] T ca64de16d40f4a44b8b411f5526ba11a P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:47.027426 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 331) took 4156 ms. Trace:
I20240430 20:11:47.027473 18983 ts_tablet_manager.cc:1398] T 331c031d44854d679a27d10d9151d0ea P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20240430 20:11:47.027748 18983 tablet_bootstrap.cc:492] T 246fb63a5e9842df8263800e28a5935c P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:47.030125 18983 tablet_bootstrap.cc:654] T 246fb63a5e9842df8263800e28a5935c P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:47.025641 18985 raft_consensus.cc:374] T 22fef8bb6b034db098c3372ae992f7a7 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:47.030849 18985 raft_consensus.cc:729] T 22fef8bb6b034db098c3372ae992f7a7 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:47.033838 18985 consensus_queue.cc:260] T 22fef8bb6b034db098c3372ae992f7a7 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:47.027971 20587 raft_consensus.cc:2781] T c6e3d09402574f1e8fe9e4160c70a3ee P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:47.034730 20587 raft_consensus.cc:482] T c6e3d09402574f1e8fe9e4160c70a3ee P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:47.034883 20587 raft_consensus.cc:3037] T c6e3d09402574f1e8fe9e4160c70a3ee P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:47.037389 20587 raft_consensus.cc:504] T c6e3d09402574f1e8fe9e4160c70a3ee P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:47.023737 18988 tablet_bootstrap.cc:492] T 04b3dff86d5c48a09d03da58bd48f360 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:47.038261 18988 ts_tablet_manager.cc:1367] T 04b3dff86d5c48a09d03da58bd48f360 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.098s user 0.006s sys 0.000s
I20240430 20:11:47.038982 18988 raft_consensus.cc:348] T 04b3dff86d5c48a09d03da58bd48f360 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:47.039325 18988 raft_consensus.cc:374] T 04b3dff86d5c48a09d03da58bd48f360 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:47.039438 18988 raft_consensus.cc:729] T 04b3dff86d5c48a09d03da58bd48f360 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:47.039477 20693 raft_consensus.cc:482] T f6cfd089f08b47409fb2a6cf4f6b0385 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:47.039698 18988 consensus_queue.cc:260] T 04b3dff86d5c48a09d03da58bd48f360 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:47.039731 20693 raft_consensus.cc:504] T f6cfd089f08b47409fb2a6cf4f6b0385 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:47.040176 18988 ts_tablet_manager.cc:1398] T 04b3dff86d5c48a09d03da58bd48f360 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240430 20:11:47.040335 20693 leader_election.cc:288] T f6cfd089f08b47409fb2a6cf4f6b0385 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:47.040436 18988 tablet_bootstrap.cc:492] T 47bf64ecc41b445da4ff697e141243dd P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:47.041184 18473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6cfd089f08b47409fb2a6cf4f6b0385" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:47.041754 18988 tablet_bootstrap.cc:654] T 47bf64ecc41b445da4ff697e141243dd P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:47.042505 20587 leader_election.cc:288] T c6e3d09402574f1e8fe9e4160c70a3ee P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Requested vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:47.042666 19489 rpcz_store.cc:276] 0430 20:11:42.871193 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:42.871276 (+ 83us) service_pool.cc:225] Handling call
0430 20:11:47.004943 (+4133667us) ts_tablet_manager.cc:1120] Deleting tablet a05e32a33aa7488eb287cc6314c09c44
0430 20:11:47.004950 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:47.009912 (+ 4962us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:47.012949 (+ 3037us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:47.027408 (+ 14459us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4043457}
W20240430 20:11:47.043021 18697 connection.cc:441] server connection from 127.0.0.1:59830 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 331) could send its response
I20240430 20:11:47.043152 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 332) took 4081 ms. Trace:
I20240430 20:11:47.043246 19489 rpcz_store.cc:276] 0430 20:11:42.961405 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:42.961540 (+ 135us) service_pool.cc:225] Handling call
0430 20:11:47.043052 (+4081512us) ts_tablet_manager.cc:1120] Deleting tablet 7542c7fe6cad452ab718f9ef54dbbbbe
0430 20:11:47.043064 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:47.043136 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4081346}
W20240430 20:11:47.043478 18697 connection.cc:441] server connection from 127.0.0.1:59830 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 332) could send its response
I20240430 20:11:47.043465 18473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e3d09402574f1e8fe9e4160c70a3ee" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:47.043567 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 333) took 4081 ms. Trace:
I20240430 20:11:47.043627 19489 rpcz_store.cc:276] 0430 20:11:42.961661 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:42.961738 (+ 77us) service_pool.cc:225] Handling call
0430 20:11:47.043492 (+4081754us) ts_tablet_manager.cc:1120] Deleting tablet 75ebf051383e42529dcb5f7dd0770a4b
0430 20:11:47.043500 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:47.043554 (+ 54us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4081630}
W20240430 20:11:47.043846 18697 connection.cc:441] server connection from 127.0.0.1:59830 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 333) could send its response
I20240430 20:11:47.043913 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 334) took 4082 ms. Trace:
I20240430 20:11:47.043980 19489 rpcz_store.cc:276] 0430 20:11:42.961764 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:42.961887 (+ 123us) service_pool.cc:225] Handling call
0430 20:11:47.043841 (+4081954us) ts_tablet_manager.cc:1120] Deleting tablet 47c37876694c4a4abb1091c631dab7a5
0430 20:11:47.043849 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:47.043900 (+ 51us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4081834}
I20240430 20:11:47.044209 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 335) took 4082 ms. Trace:
W20240430 20:11:47.044216 18697 connection.cc:441] server connection from 127.0.0.1:59830 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 334) could send its response
I20240430 20:11:47.044281 19489 rpcz_store.cc:276] 0430 20:11:42.961857 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:42.962033 (+ 176us) service_pool.cc:225] Handling call
0430 20:11:47.044149 (+4082116us) ts_tablet_manager.cc:1120] Deleting tablet 255c13ad8fae4ea4aa8e7b7aabb7602e
0430 20:11:47.044157 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:47.044198 (+ 41us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4082007}
W20240430 20:11:47.044478 18697 connection.cc:441] server connection from 127.0.0.1:59830 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 335) could send its response
I20240430 20:11:47.044549 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 336) took 4082 ms. Trace:
I20240430 20:11:47.044610 19489 rpcz_store.cc:276] 0430 20:11:42.961958 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:42.962452 (+ 494us) service_pool.cc:225] Handling call
0430 20:11:47.044459 (+4082007us) ts_tablet_manager.cc:1120] Deleting tablet a663030d37614053a7c0369841403422
0430 20:11:47.044465 (+ 6us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:47.044536 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4076082}
W20240430 20:11:47.044802 18697 connection.cc:441] server connection from 127.0.0.1:59830 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 336) could send its response
I20240430 20:11:47.044910 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 339) took 3917 ms. Trace:
I20240430 20:11:47.044972 19489 rpcz_store.cc:276] 0430 20:11:43.127199 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:43.127247 (+ 48us) service_pool.cc:225] Handling call
0430 20:11:47.044797 (+3917550us) ts_tablet_manager.cc:1120] Deleting tablet 6d1d2193fd424c91b143f5256df5fcd0
0430 20:11:47.044805 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:47.044897 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3917222}
W20240430 20:11:47.045169 18697 connection.cc:441] server connection from 127.0.0.1:59830 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 339) could send its response
I20240430 20:11:47.045254 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 338) took 3918 ms. Trace:
I20240430 20:11:47.045351 19489 rpcz_store.cc:276] 0430 20:11:43.126317 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:43.126618 (+ 301us) service_pool.cc:225] Handling call
0430 20:11:47.045172 (+3918554us) ts_tablet_manager.cc:1120] Deleting tablet aa9998f4e17147aa8f2b35c01980463a
0430 20:11:47.045179 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:47.045243 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3917571}
W20240430 20:11:47.045548 18697 connection.cc:441] server connection from 127.0.0.1:59830 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 338) could send its response
I20240430 20:11:47.045624 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 337) took 3919 ms. Trace:
I20240430 20:11:47.045680 19489 rpcz_store.cc:276] 0430 20:11:43.126193 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:43.126649 (+ 456us) service_pool.cc:225] Handling call
0430 20:11:47.045543 (+3918894us) ts_tablet_manager.cc:1120] Deleting tablet 8901327b3efb4201adce61a9266142ea
0430 20:11:47.045551 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:47.045612 (+ 61us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3916506}
I20240430 20:11:47.045894 19489 tablet_replica.cc:330] T acf3b6fead41483a82c43dc0286fab7c P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
W20240430 20:11:47.046010 18697 connection.cc:441] server connection from 127.0.0.1:59830 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 337) could send its response
I20240430 20:11:47.046062 19489 raft_consensus.cc:2229] T acf3b6fead41483a82c43dc0286fab7c P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:47.046209 19489 raft_consensus.cc:2258] T acf3b6fead41483a82c43dc0286fab7c P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240430 20:11:47.046520 18699 leader_election.cc:341] T c6e3d09402574f1e8fe9e4160c70a3ee P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:47.050832 19489 ts_tablet_manager.cc:1875] T acf3b6fead41483a82c43dc0286fab7c P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:47.052469 20685 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89df84ccf5394c9bb60d4e4ed872fc2e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88b3f98793354c7aaa5c106e8ec31ed8. Latest consensus state: current_term: 6 leader_uuid: "88b3f98793354c7aaa5c106e8ec31ed8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b3f98793354c7aaa5c106e8ec31ed8" member_type: VOTER last_known_addr { host: "127.17.177.62" port: 36441 } } peers { permanent_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e" member_type: VOTER last_known_addr { host: "127.17.177.61" port: 34777 } } peers { permanent_uuid: "b632d6ef4c104a058a6db208024cab33" member_type: VOTER last_known_addr { host: "127.17.177.60" port: 37225 } } }
I20240430 20:11:47.052747 20685 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89df84ccf5394c9bb60d4e4ed872fc2e [sys.catalog]: This master's current role is: FOLLOWER
W20240430 20:11:47.056125 19895 consensus_peers.cc:485] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8 -> Peer b632d6ef4c104a058a6db208024cab33 (127.17.177.60:37225): Couldn't send request to peer b632d6ef4c104a058a6db208024cab33. Status: Network error: Client connection negotiation failed: client connection to 127.17.177.60:37225: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240430 20:11:47.059612 18988 tablet_bootstrap.cc:492] T 47bf64ecc41b445da4ff697e141243dd P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:47.059800 18988 ts_tablet_manager.cc:1367] T 47bf64ecc41b445da4ff697e141243dd P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.000s
I20240430 20:11:47.060524 18988 raft_consensus.cc:348] T 47bf64ecc41b445da4ff697e141243dd P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:47.060721 18988 raft_consensus.cc:374] T 47bf64ecc41b445da4ff697e141243dd P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:47.060807 18988 raft_consensus.cc:729] T 47bf64ecc41b445da4ff697e141243dd P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:47.061022 18988 consensus_queue.cc:260] T 47bf64ecc41b445da4ff697e141243dd P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:47.061857 18988 ts_tablet_manager.cc:1398] T 47bf64ecc41b445da4ff697e141243dd P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240430 20:11:47.062103 18988 tablet_bootstrap.cc:492] T 0a16d98f700c46e29f6dd8568752ef29 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:47.065393 18988 tablet_bootstrap.cc:654] T 0a16d98f700c46e29f6dd8568752ef29 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:47.066138 20631 raft_consensus.cc:482] T 8e33df70a1504dbd852fe0b3c3f61622 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:47.066269 20631 raft_consensus.cc:504] T 8e33df70a1504dbd852fe0b3c3f61622 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:47.066855 20631 leader_election.cc:288] T 8e33df70a1504dbd852fe0b3c3f61622 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:47.069839 20631 consensus_queue.cc:1035] T 677481aee5fa4eb4b189322d7a3f8f2e P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:47.070542 20631 raft_consensus.cc:482] T 283e5e73cbc04daba0de19f4541f9fd0 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:47.070683 20631 raft_consensus.cc:504] T 283e5e73cbc04daba0de19f4541f9fd0 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:47.071252 20631 leader_election.cc:288] T 283e5e73cbc04daba0de19f4541f9fd0 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:47.072937 18758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "283e5e73cbc04daba0de19f4541f9fd0" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:47.073324 18763 raft_consensus.cc:3037] T f6cfd089f08b47409fb2a6cf4f6b0385 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:47.075713 18988 tablet_bootstrap.cc:492] T 0a16d98f700c46e29f6dd8568752ef29 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:47.075865 18988 ts_tablet_manager.cc:1367] T 0a16d98f700c46e29f6dd8568752ef29 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
W20240430 20:11:47.076179 18417 leader_election.cc:341] T 283e5e73cbc04daba0de19f4541f9fd0 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:47.076301 18750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e33df70a1504dbd852fe0b3c3f61622" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:47.076509 18988 raft_consensus.cc:348] T 0a16d98f700c46e29f6dd8568752ef29 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:47.076686 18988 raft_consensus.cc:374] T 0a16d98f700c46e29f6dd8568752ef29 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:47.076757 18988 raft_consensus.cc:729] T 0a16d98f700c46e29f6dd8568752ef29 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:47.076943 18988 consensus_queue.cc:260] T 0a16d98f700c46e29f6dd8568752ef29 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
W20240430 20:11:47.077147 18417 leader_election.cc:341] T 8e33df70a1504dbd852fe0b3c3f61622 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:47.077456 18988 ts_tablet_manager.cc:1398] T 0a16d98f700c46e29f6dd8568752ef29 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240430 20:11:47.077708 18988 tablet_bootstrap.cc:492] T 8e33df70a1504dbd852fe0b3c3f61622 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:47.079249 20545 consensus_queue.cc:1035] T f6cfd089f08b47409fb2a6cf4f6b0385 P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:47.079874 19489 ts_tablet_manager.cc:1888] T acf3b6fead41483a82c43dc0286fab7c P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20240430 20:11:47.082754 18415 leader_election.cc:341] T 283e5e73cbc04daba0de19f4541f9fd0 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Not found: Tablet not found: 283e5e73cbc04daba0de19f4541f9fd0
I20240430 20:11:47.084554 19489 log.cc:1196] T acf3b6fead41483a82c43dc0286fab7c P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/acf3b6fead41483a82c43dc0286fab7c
I20240430 20:11:47.080466 18988 tablet_bootstrap.cc:654] T 8e33df70a1504dbd852fe0b3c3f61622 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:47.084981 18415 leader_election.cc:302] T 283e5e73cbc04daba0de19f4541f9fd0 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a; no voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710
I20240430 20:11:47.085886 19489 ts_tablet_manager.cc:1909] T acf3b6fead41483a82c43dc0286fab7c P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:47.086808 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 341) took 3828 ms. Trace:
I20240430 20:11:47.089947 20631 raft_consensus.cc:2726] T 283e5e73cbc04daba0de19f4541f9fd0 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:47.090643 19489 rpcz_store.cc:276] 0430 20:11:43.258099 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:43.258657 (+ 558us) service_pool.cc:225] Handling call
0430 20:11:47.045858 (+3787201us) ts_tablet_manager.cc:1120] Deleting tablet acf3b6fead41483a82c43dc0286fab7c
0430 20:11:47.045865 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:47.055948 (+ 10083us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:47.079760 (+ 23812us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:47.086772 (+ 7012us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3787019}
W20240430 20:11:47.091032 18697 connection.cc:441] server connection from 127.0.0.1:59830 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 341) could send its response
I20240430 20:11:47.091337 19489 tablet_replica.cc:330] T 3ec3993042f5498b818bdf32d5087928 P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:47.091559 19489 raft_consensus.cc:2229] T 3ec3993042f5498b818bdf32d5087928 P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Raft consensus shutting down.
I20240430 20:11:47.091820 19489 pending_rounds.cc:62] T 3ec3993042f5498b818bdf32d5087928 P 3c51c33d5616482cafb68e26688820f4: Trying to abort 1 pending ops.
I20240430 20:11:47.094650 19489 pending_rounds.cc:68] T 3ec3993042f5498b818bdf32d5087928 P 3c51c33d5616482cafb68e26688820f4: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7022624385665941504 op_type: NO_OP noop_request { }
I20240430 20:11:47.094884 19489 raft_consensus.cc:2866] T 3ec3993042f5498b818bdf32d5087928 P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240430 20:11:47.095037 19489 raft_consensus.cc:2258] T 3ec3993042f5498b818bdf32d5087928 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:47.096025 19489 ts_tablet_manager.cc:1875] T 3ec3993042f5498b818bdf32d5087928 P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:47.099175 18985 ts_tablet_manager.cc:1398] T 22fef8bb6b034db098c3372ae992f7a7 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.074s user 0.002s sys 0.000s
I20240430 20:11:47.103080 18988 tablet_bootstrap.cc:492] T 8e33df70a1504dbd852fe0b3c3f61622 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:47.103292 18988 ts_tablet_manager.cc:1367] T 8e33df70a1504dbd852fe0b3c3f61622 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.026s user 0.004s sys 0.000s
I20240430 20:11:47.104020 18988 raft_consensus.cc:348] T 8e33df70a1504dbd852fe0b3c3f61622 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:47.104296 18988 raft_consensus.cc:374] T 8e33df70a1504dbd852fe0b3c3f61622 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:47.104408 18988 raft_consensus.cc:729] T 8e33df70a1504dbd852fe0b3c3f61622 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:47.104679 18988 consensus_queue.cc:260] T 8e33df70a1504dbd852fe0b3c3f61622 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:47.105618 18988 ts_tablet_manager.cc:1398] T 8e33df70a1504dbd852fe0b3c3f61622 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240430 20:11:47.105952 18988 tablet_bootstrap.cc:492] T 283e5e73cbc04daba0de19f4541f9fd0 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:47.028525 19471 ts_tablet_manager.cc:1875] T b22aaedb0a6c4fd9ad0ea8fa019001e1 P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:47.040930 18616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6cfd089f08b47409fb2a6cf4f6b0385" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:47.107728 18616 raft_consensus.cc:2379] T f6cfd089f08b47409fb2a6cf4f6b0385 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c51c33d5616482cafb68e26688820f4 in current term 1: Already voted for candidate 2efb542aa55e439b99ccb98d1c5c884a in this term.
I20240430 20:11:47.108079 20695 raft_consensus.cc:482] T 246fb63a5e9842df8263800e28a5935c P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:47.108268 20695 raft_consensus.cc:504] T 246fb63a5e9842df8263800e28a5935c P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:47.108966 20695 leader_election.cc:288] T 246fb63a5e9842df8263800e28a5935c P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:47.109459 18616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "246fb63a5e9842df8263800e28a5935c" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:47.110006 18988 tablet_bootstrap.cc:654] T 283e5e73cbc04daba0de19f4541f9fd0 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:47.110145 18696 leader_election.cc:302] T f6cfd089f08b47409fb2a6cf4f6b0385 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710
W20240430 20:11:47.110574 18696 leader_election.cc:341] T 246fb63a5e9842df8263800e28a5935c P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:47.110899 18473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "246fb63a5e9842df8263800e28a5935c" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:47.111160 18473 raft_consensus.cc:2454] T 246fb63a5e9842df8263800e28a5935c P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:47.047142 18619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e3d09402574f1e8fe9e4160c70a3ee" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:47.111552 18699 leader_election.cc:302] T 246fb63a5e9842df8263800e28a5935c P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters: 5e1c6397f09747f091534faf93542710
I20240430 20:11:47.062875 18608 raft_consensus.cc:1261] T ca64de16d40f4a44b8b411f5526ba11a P 5e1c6397f09747f091534faf93542710 [term 2 FOLLOWER]: Refusing update from remote peer 2efb542aa55e439b99ccb98d1c5c884a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240430 20:11:47.111755 18619 raft_consensus.cc:3037] T c6e3d09402574f1e8fe9e4160c70a3ee P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:47.076905 20488 raft_consensus.cc:482] T 0a16d98f700c46e29f6dd8568752ef29 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:47.111922 20587 raft_consensus.cc:2781] T 246fb63a5e9842df8263800e28a5935c P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:47.112015 20587 raft_consensus.cc:482] T 246fb63a5e9842df8263800e28a5935c P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:47.111966 20488 raft_consensus.cc:504] T 0a16d98f700c46e29f6dd8568752ef29 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:47.112102 20587 raft_consensus.cc:3037] T 246fb63a5e9842df8263800e28a5935c P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:47.112615 20488 leader_election.cc:288] T 0a16d98f700c46e29f6dd8568752ef29 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:47.073359 18620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e33df70a1504dbd852fe0b3c3f61622" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:47.112771 18620 raft_consensus.cc:2454] T 8e33df70a1504dbd852fe0b3c3f61622 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 0.
I20240430 20:11:47.113065 18756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a16d98f700c46e29f6dd8568752ef29" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:47.113358 18756 raft_consensus.cc:2454] T 0a16d98f700c46e29f6dd8568752ef29 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 0.
I20240430 20:11:47.114046 18557 leader_election.cc:302] T 0a16d98f700c46e29f6dd8568752ef29 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:47.114447 20489 raft_consensus.cc:482] T ae87f5e92a934e2182164c231bc76df0 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2efb542aa55e439b99ccb98d1c5c884a)
I20240430 20:11:47.114626 20489 raft_consensus.cc:504] T ae87f5e92a934e2182164c231bc76df0 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:47.114917 20587 raft_consensus.cc:504] T 246fb63a5e9842df8263800e28a5935c P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:47.115433 20489 leader_election.cc:288] T ae87f5e92a934e2182164c231bc76df0 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:47.115495 20587 leader_election.cc:288] T 246fb63a5e9842df8263800e28a5935c P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Requested vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:47.115597 20489 raft_consensus.cc:2781] T 0a16d98f700c46e29f6dd8568752ef29 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:47.115667 20489 raft_consensus.cc:482] T 0a16d98f700c46e29f6dd8568752ef29 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:47.115741 20489 raft_consensus.cc:3037] T 0a16d98f700c46e29f6dd8568752ef29 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:47.115873 18473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a16d98f700c46e29f6dd8568752ef29" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:47.116199 18473 raft_consensus.cc:2454] T 0a16d98f700c46e29f6dd8568752ef29 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 0.
I20240430 20:11:47.116750 18478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "246fb63a5e9842df8263800e28a5935c" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:47.116912 18478 raft_consensus.cc:3037] T 246fb63a5e9842df8263800e28a5935c P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:47.119505 18478 raft_consensus.cc:2454] T 246fb63a5e9842df8263800e28a5935c P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
W20240430 20:11:47.119740 18557 leader_election.cc:341] T ae87f5e92a934e2182164c231bc76df0 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Not found: Tablet not found: ae87f5e92a934e2182164c231bc76df0
I20240430 20:11:47.120121 18620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "246fb63a5e9842df8263800e28a5935c" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
W20240430 20:11:47.120903 18558 leader_election.cc:341] T ae87f5e92a934e2182164c231bc76df0 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Not found: Tablet not found: ae87f5e92a934e2182164c231bc76df0
I20240430 20:11:47.120801 20489 raft_consensus.cc:504] T 0a16d98f700c46e29f6dd8568752ef29 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:47.121003 18558 leader_election.cc:302] T ae87f5e92a934e2182164c231bc76df0 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:47.121563 20047 raft_consensus.cc:2726] T ae87f5e92a934e2182164c231bc76df0 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240430 20:11:47.122068 18699 leader_election.cc:302] T 246fb63a5e9842df8263800e28a5935c P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters:
I20240430 20:11:47.122439 19489 ts_tablet_manager.cc:1888] T 3ec3993042f5498b818bdf32d5087928 P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:47.122408 18478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a16d98f700c46e29f6dd8568752ef29" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:47.122700 18478 raft_consensus.cc:3037] T 0a16d98f700c46e29f6dd8568752ef29 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:47.123216 18619 raft_consensus.cc:2454] T c6e3d09402574f1e8fe9e4160c70a3ee P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:47.124402 20545 consensus_queue.cc:1035] T ca64de16d40f4a44b8b411f5526ba11a P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240430 20:11:47.126618 18478 raft_consensus.cc:2454] T 0a16d98f700c46e29f6dd8568752ef29 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
I20240430 20:11:47.127156 18558 leader_election.cc:302] T 0a16d98f700c46e29f6dd8568752ef29 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:47.127513 20047 raft_consensus.cc:2781] T 0a16d98f700c46e29f6dd8568752ef29 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:47.127662 20047 raft_consensus.cc:686] T 0a16d98f700c46e29f6dd8568752ef29 P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Becoming Leader. State: Replica: 5e1c6397f09747f091534faf93542710, State: Running, Role: LEADER
I20240430 20:11:47.128008 18985 tablet_bootstrap.cc:492] T f5eaf2413e1d4e69b56fae61f768f0ce P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:47.128008 20047 consensus_queue.cc:237] T 0a16d98f700c46e29f6dd8568752ef29 P 5e1c6397f09747f091534faf93542710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:47.130393 18415 leader_election.cc:302] T 8e33df70a1504dbd852fe0b3c3f61622 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters: 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:47.130785 20545 raft_consensus.cc:2781] T 8e33df70a1504dbd852fe0b3c3f61622 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:47.130877 20545 raft_consensus.cc:482] T 8e33df70a1504dbd852fe0b3c3f61622 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:47.132027 20545 raft_consensus.cc:3037] T 8e33df70a1504dbd852fe0b3c3f61622 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:47.122491 18756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a16d98f700c46e29f6dd8568752ef29" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:47.132571 18756 raft_consensus.cc:3037] T 0a16d98f700c46e29f6dd8568752ef29 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:47.134145 20489 leader_election.cc:288] T 0a16d98f700c46e29f6dd8568752ef29 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:47.122704 20587 raft_consensus.cc:2781] T 246fb63a5e9842df8263800e28a5935c P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:47.134842 20587 raft_consensus.cc:686] T 246fb63a5e9842df8263800e28a5935c P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Becoming Leader. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Running, Role: LEADER
I20240430 20:11:47.134842 20545 raft_consensus.cc:504] T 8e33df70a1504dbd852fe0b3c3f61622 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:47.135135 20587 consensus_queue.cc:237] T 246fb63a5e9842df8263800e28a5935c P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:47.135911 18763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e33df70a1504dbd852fe0b3c3f61622" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:47.136096 18763 raft_consensus.cc:3037] T 8e33df70a1504dbd852fe0b3c3f61622 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:47.137044 18756 raft_consensus.cc:2454] T 0a16d98f700c46e29f6dd8568752ef29 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
W20240430 20:11:47.123873 18696 leader_election.cc:341] T 246fb63a5e9842df8263800e28a5935c P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:47.137530 18696 leader_election.cc:302] T c6e3d09402574f1e8fe9e4160c70a3ee P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a
I20240430 20:11:47.137786 20693 raft_consensus.cc:2781] T c6e3d09402574f1e8fe9e4160c70a3ee P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:47.137909 20693 raft_consensus.cc:686] T c6e3d09402574f1e8fe9e4160c70a3ee P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Becoming Leader. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Running, Role: LEADER
I20240430 20:11:47.138156 20693 consensus_queue.cc:237] T c6e3d09402574f1e8fe9e4160c70a3ee P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:47.139452 18763 raft_consensus.cc:2454] T 8e33df70a1504dbd852fe0b3c3f61622 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 1.
I20240430 20:11:47.140036 18417 leader_election.cc:302] T 8e33df70a1504dbd852fe0b3c3f61622 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters:
I20240430 20:11:47.140486 20631 raft_consensus.cc:2781] T 8e33df70a1504dbd852fe0b3c3f61622 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:47.140623 20631 raft_consensus.cc:686] T 8e33df70a1504dbd852fe0b3c3f61622 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Becoming Leader. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Running, Role: LEADER
I20240430 20:11:47.140923 20631 consensus_queue.cc:237] T 8e33df70a1504dbd852fe0b3c3f61622 P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:47.145758 20685 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89df84ccf5394c9bb60d4e4ed872fc2e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 6 leader_uuid: "88b3f98793354c7aaa5c106e8ec31ed8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b3f98793354c7aaa5c106e8ec31ed8" member_type: VOTER last_known_addr { host: "127.17.177.62" port: 36441 } } peers { permanent_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e" member_type: VOTER last_known_addr { host: "127.17.177.61" port: 34777 } } peers { permanent_uuid: "b632d6ef4c104a058a6db208024cab33" member_type: VOTER last_known_addr { host: "127.17.177.60" port: 37225 } } }
I20240430 20:11:47.146126 20685 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89df84ccf5394c9bb60d4e4ed872fc2e [sys.catalog]: This master's current role is: FOLLOWER
I20240430 20:11:47.148743 20545 leader_election.cc:288] T 8e33df70a1504dbd852fe0b3c3f61622 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:47.150233 18620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e33df70a1504dbd852fe0b3c3f61622" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:47.150501 18620 raft_consensus.cc:3037] T 8e33df70a1504dbd852fe0b3c3f61622 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:47.150820 19489 log.cc:1196] T 3ec3993042f5498b818bdf32d5087928 P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/3ec3993042f5498b818bdf32d5087928
I20240430 20:11:47.151304 19489 ts_tablet_manager.cc:1909] T 3ec3993042f5498b818bdf32d5087928 P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:47.152215 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 360) took 3731 ms. Trace:
I20240430 20:11:47.154220 18620 raft_consensus.cc:2454] T 8e33df70a1504dbd852fe0b3c3f61622 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 1.
I20240430 20:11:47.154348 20686 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 6 leader_uuid: "88b3f98793354c7aaa5c106e8ec31ed8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b3f98793354c7aaa5c106e8ec31ed8" member_type: VOTER last_known_addr { host: "127.17.177.62" port: 36441 } } peers { permanent_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e" member_type: VOTER last_known_addr { host: "127.17.177.61" port: 34777 } } peers { permanent_uuid: "b632d6ef4c104a058a6db208024cab33" member_type: VOTER last_known_addr { host: "127.17.177.60" port: 37225 } } }
I20240430 20:11:47.154584 20686 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8 [sys.catalog]: This master's current role is: LEADER
I20240430 20:11:47.152419 19489 rpcz_store.cc:276] 0430 20:11:43.420615 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:43.420678 (+ 63us) service_pool.cc:225] Handling call
0430 20:11:47.091291 (+3670613us) ts_tablet_manager.cc:1120] Deleting tablet 3ec3993042f5498b818bdf32d5087928
0430 20:11:47.091300 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:47.119301 (+ 28001us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:47.122342 (+ 3041us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:47.152162 (+ 29820us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3670396}
I20240430 20:11:47.160697 18985 tablet_bootstrap.cc:654] T f5eaf2413e1d4e69b56fae61f768f0ce P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
W20240430 20:11:47.161020 18697 connection.cc:441] server connection from 127.0.0.1:59830 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 360) could send its response
I20240430 20:11:47.161523 19489 tablet_replica.cc:330] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:47.162071 19489 raft_consensus.cc:2229] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 3c51c33d5616482cafb68e26688820f4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:47.162787 19489 raft_consensus.cc:2258] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 3c51c33d5616482cafb68e26688820f4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:47.166805 19489 ts_tablet_manager.cc:1875] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:47.168819 18985 tablet_bootstrap.cc:492] T f5eaf2413e1d4e69b56fae61f768f0ce P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:47.168990 18985 ts_tablet_manager.cc:1367] T f5eaf2413e1d4e69b56fae61f768f0ce P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.041s user 0.003s sys 0.000s
I20240430 20:11:47.178689 18983 tablet_bootstrap.cc:492] T 246fb63a5e9842df8263800e28a5935c P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:47.179041 18983 ts_tablet_manager.cc:1367] T 246fb63a5e9842df8263800e28a5935c P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.151s user 0.003s sys 0.001s
I20240430 20:11:47.179925 18983 raft_consensus.cc:348] T 246fb63a5e9842df8263800e28a5935c P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:47.180258 18983 raft_consensus.cc:374] T 246fb63a5e9842df8263800e28a5935c P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:47.180436 18983 raft_consensus.cc:729] T 246fb63a5e9842df8263800e28a5935c P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:47.180708 18983 consensus_queue.cc:260] T 246fb63a5e9842df8263800e28a5935c P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:47.181270 18983 ts_tablet_manager.cc:1398] T 246fb63a5e9842df8263800e28a5935c P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240430 20:11:47.181790 18983 tablet_bootstrap.cc:492] T 47bf64ecc41b445da4ff697e141243dd P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:47.184356 18985 raft_consensus.cc:348] T f5eaf2413e1d4e69b56fae61f768f0ce P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:47.186298 18985 raft_consensus.cc:374] T f5eaf2413e1d4e69b56fae61f768f0ce P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:47.186443 18985 raft_consensus.cc:729] T f5eaf2413e1d4e69b56fae61f768f0ce P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:47.187711 19471 ts_tablet_manager.cc:1888] T b22aaedb0a6c4fd9ad0ea8fa019001e1 P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:47.187930 19471 log.cc:1196] T b22aaedb0a6c4fd9ad0ea8fa019001e1 P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/b22aaedb0a6c4fd9ad0ea8fa019001e1
I20240430 20:11:47.187501 18985 consensus_queue.cc:260] T f5eaf2413e1d4e69b56fae61f768f0ce P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:47.192385 19489 ts_tablet_manager.cc:1888] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.2
I20240430 20:11:47.193135 18985 ts_tablet_manager.cc:1398] T f5eaf2413e1d4e69b56fae61f768f0ce P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.024s user 0.002s sys 0.000s
I20240430 20:11:47.193372 18983 tablet_bootstrap.cc:654] T 47bf64ecc41b445da4ff697e141243dd P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:47.194077 19471 ts_tablet_manager.cc:1909] T b22aaedb0a6c4fd9ad0ea8fa019001e1 P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:47.195104 19489 log.cc:1196] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/dd6083f12c7e4d1e8a2006b3ddf9aa3a
I20240430 20:11:47.195789 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 281) took 5002 ms. Trace:
I20240430 20:11:47.197145 19489 ts_tablet_manager.cc:1909] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:47.198693 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 362) took 3753 ms. Trace:
I20240430 20:11:47.200778 18985 tablet_bootstrap.cc:492] T 6d1d2193fd424c91b143f5256df5fcd0 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:47.201946 19471 rpcz_store.cc:276] 0430 20:11:42.193174 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:42.193223 (+ 49us) service_pool.cc:225] Handling call
0430 20:11:47.022963 (+4829740us) ts_tablet_manager.cc:1120] Deleting tablet b22aaedb0a6c4fd9ad0ea8fa019001e1
0430 20:11:47.022974 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:47.156248 (+133274us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:47.183728 (+ 27480us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:47.195767 (+ 12039us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4829522}
W20240430 20:11:47.202316 18558 connection.cc:441] server connection from 127.0.0.1:40218 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 281) could send its response
I20240430 20:11:47.202642 19471 tablet_replica.cc:330] T 32c823d1700949ab8b22adcd83b7bfd1 P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:47.202915 19471 raft_consensus.cc:2229] T 32c823d1700949ab8b22adcd83b7bfd1 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:47.202790 19489 rpcz_store.cc:276] 0430 20:11:43.445139 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:43.445204 (+ 65us) service_pool.cc:225] Handling call
0430 20:11:47.161437 (+3716233us) ts_tablet_manager.cc:1120] Deleting tablet dd6083f12c7e4d1e8a2006b3ddf9aa3a
0430 20:11:47.161450 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:47.179603 (+ 18153us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:47.192230 (+ 12627us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:47.198646 (+ 6416us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3715999}
W20240430 20:11:47.203081 18697 connection.cc:441] server connection from 127.0.0.1:59830 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 362) could send its response
I20240430 20:11:47.203811 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 363) took 3704 ms. Trace:
I20240430 20:11:47.204300 19489 rpcz_store.cc:276] 0430 20:11:43.499026 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:43.499086 (+ 60us) service_pool.cc:225] Handling call
0430 20:11:47.203637 (+3704551us) ts_tablet_manager.cc:1120] Deleting tablet 489dc838744a49eba542009c86688aa5
0430 20:11:47.203648 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:47.203765 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3704334}
W20240430 20:11:47.204806 18697 connection.cc:441] server connection from 127.0.0.1:59830 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 363) could send its response
I20240430 20:11:47.205305 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 361) took 3784 ms. Trace:
I20240430 20:11:47.206774 19471 raft_consensus.cc:2258] T 32c823d1700949ab8b22adcd83b7bfd1 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:47.207520 19471 ts_tablet_manager.cc:1875] T 32c823d1700949ab8b22adcd83b7bfd1 P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:47.210161 18985 tablet_bootstrap.cc:654] T 6d1d2193fd424c91b143f5256df5fcd0 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:47.210728 19489 rpcz_store.cc:276] 0430 20:11:43.420721 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:43.539303 (+118582us) service_pool.cc:225] Handling call
0430 20:11:47.205149 (+3665846us) ts_tablet_manager.cc:1120] Deleting tablet b7141e215d6d44bc8406c80508190874
0430 20:11:47.205159 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:47.205253 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3665634}
W20240430 20:11:47.211334 18697 connection.cc:441] server connection from 127.0.0.1:59830 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 361) could send its response
I20240430 20:11:47.211776 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 366) took 3460 ms. Trace:
I20240430 20:11:47.212177 19489 rpcz_store.cc:276] 0430 20:11:43.750893 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:43.750958 (+ 65us) service_pool.cc:225] Handling call
0430 20:11:47.211651 (+3460693us) ts_tablet_manager.cc:1120] Deleting tablet fae9a35a2ac844b4bc2ebdf214aa7171
0430 20:11:47.211663 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:47.211740 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3460394}
W20240430 20:11:47.212648 18697 connection.cc:441] server connection from 127.0.0.1:59830 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 366) could send its response
I20240430 20:11:47.213001 19489 tablet_replica.cc:330] T 73453a7fa30342e29db552b17b62f0c1 P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:47.213436 19489 raft_consensus.cc:2229] T 73453a7fa30342e29db552b17b62f0c1 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:47.213933 19489 raft_consensus.cc:2258] T 73453a7fa30342e29db552b17b62f0c1 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:47.214761 19489 ts_tablet_manager.cc:1875] T 73453a7fa30342e29db552b17b62f0c1 P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:47.215801 18988 tablet_bootstrap.cc:492] T 283e5e73cbc04daba0de19f4541f9fd0 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:47.215945 18988 ts_tablet_manager.cc:1367] T 283e5e73cbc04daba0de19f4541f9fd0 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.110s user 0.006s sys 0.012s
I20240430 20:11:47.216727 18988 raft_consensus.cc:348] T 283e5e73cbc04daba0de19f4541f9fd0 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:47.217748 18988 raft_consensus.cc:374] T 283e5e73cbc04daba0de19f4541f9fd0 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:47.217970 18988 raft_consensus.cc:729] T 283e5e73cbc04daba0de19f4541f9fd0 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:47.218827 18988 consensus_queue.cc:260] T 283e5e73cbc04daba0de19f4541f9fd0 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:47.219833 18988 ts_tablet_manager.cc:1398] T 283e5e73cbc04daba0de19f4541f9fd0 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240430 20:11:47.220135 18988 tablet_bootstrap.cc:492] T 331c031d44854d679a27d10d9151d0ea P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:47.220607 18985 tablet_bootstrap.cc:492] T 6d1d2193fd424c91b143f5256df5fcd0 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:47.220754 18985 ts_tablet_manager.cc:1367] T 6d1d2193fd424c91b143f5256df5fcd0 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.000s
I20240430 20:11:47.221472 18985 raft_consensus.cc:348] T 6d1d2193fd424c91b143f5256df5fcd0 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:47.221683 18985 raft_consensus.cc:374] T 6d1d2193fd424c91b143f5256df5fcd0 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:47.221750 18985 raft_consensus.cc:729] T 6d1d2193fd424c91b143f5256df5fcd0 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:47.224366 20545 consensus_queue.cc:1035] T 66dd767bd3f8494f94061fb89e5e5680 P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:47.225782 18985 consensus_queue.cc:260] T 6d1d2193fd424c91b143f5256df5fcd0 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
W20240430 20:11:47.225894 18557 consensus_peers.cc:485] T 7683caba80594627900a0185f772c131 P 5e1c6397f09747f091534faf93542710 -> Peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Couldn't send request to peer 3c51c33d5616482cafb68e26688820f4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7683caba80594627900a0185f772c131. This is attempt 1: this message will repeat every 5th retry.
I20240430 20:11:47.228583 18985 ts_tablet_manager.cc:1398] T 6d1d2193fd424c91b143f5256df5fcd0 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240430 20:11:47.228891 18985 tablet_bootstrap.cc:492] T 331c031d44854d679a27d10d9151d0ea P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:47.232311 19489 ts_tablet_manager.cc:1888] T 73453a7fa30342e29db552b17b62f0c1 P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:47.234326 20697 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240430 20:11:47.235914 18983 tablet_bootstrap.cc:492] T 47bf64ecc41b445da4ff697e141243dd P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:47.236060 18983 ts_tablet_manager.cc:1367] T 47bf64ecc41b445da4ff697e141243dd P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.054s user 0.003s sys 0.000s
I20240430 20:11:47.236793 18983 raft_consensus.cc:348] T 47bf64ecc41b445da4ff697e141243dd P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:47.236969 18983 raft_consensus.cc:374] T 47bf64ecc41b445da4ff697e141243dd P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:47.237025 18983 raft_consensus.cc:729] T 47bf64ecc41b445da4ff697e141243dd P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:47.237262 18983 consensus_queue.cc:260] T 47bf64ecc41b445da4ff697e141243dd P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:47.237748 18983 ts_tablet_manager.cc:1398] T 47bf64ecc41b445da4ff697e141243dd P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240430 20:11:47.238284 18983 tablet_bootstrap.cc:492] T 78ba48dadef548708153878810cc46f8 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:47.238708 19489 log.cc:1196] T 73453a7fa30342e29db552b17b62f0c1 P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/73453a7fa30342e29db552b17b62f0c1
I20240430 20:11:47.239127 19489 ts_tablet_manager.cc:1909] T 73453a7fa30342e29db552b17b62f0c1 P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:47.239640 19471 ts_tablet_manager.cc:1888] T 32c823d1700949ab8b22adcd83b7bfd1 P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:47.239758 19471 log.cc:1196] T 32c823d1700949ab8b22adcd83b7bfd1 P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/32c823d1700949ab8b22adcd83b7bfd1
I20240430 20:11:47.240074 18983 tablet_bootstrap.cc:654] T 78ba48dadef548708153878810cc46f8 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:47.240123 19471 ts_tablet_manager.cc:1909] T 32c823d1700949ab8b22adcd83b7bfd1 P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:47.240190 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 364) took 3489 ms. Trace:
I20240430 20:11:47.240345 19489 rpcz_store.cc:276] 0430 20:11:43.750650 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:43.751053 (+ 403us) service_pool.cc:225] Handling call
0430 20:11:47.212935 (+3461882us) ts_tablet_manager.cc:1120] Deleting tablet 73453a7fa30342e29db552b17b62f0c1
0430 20:11:47.212944 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:47.224439 (+ 11495us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:47.232167 (+ 7728us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:47.240120 (+ 7953us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3461624}
W20240430 20:11:47.240681 18697 connection.cc:441] server connection from 127.0.0.1:59830 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 364) could send its response
I20240430 20:11:47.241083 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 365) took 3490 ms. Trace:
I20240430 20:11:47.241235 19489 rpcz_store.cc:276] 0430 20:11:43.750795 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:43.750968 (+ 173us) service_pool.cc:225] Handling call
0430 20:11:47.240956 (+3489988us) ts_tablet_manager.cc:1120] Deleting tablet b72fe1f0713b4076a81a3fc1c74f7fa7
0430 20:11:47.240969 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:47.241051 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3489446}
W20240430 20:11:47.241523 18697 connection.cc:441] server connection from 127.0.0.1:59830 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 365) could send its response
I20240430 20:11:47.241604 18985 tablet_bootstrap.cc:654] T 331c031d44854d679a27d10d9151d0ea P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:47.241866 19489 tablet_replica.cc:330] T f940d17b02b541289121463e05b0bb5f P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:47.242125 19489 raft_consensus.cc:2229] T f940d17b02b541289121463e05b0bb5f P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:47.242357 19489 raft_consensus.cc:2258] T f940d17b02b541289121463e05b0bb5f P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:47.243014 19489 ts_tablet_manager.cc:1875] T f940d17b02b541289121463e05b0bb5f P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:47.246182 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 280) took 5053 ms. Trace:
I20240430 20:11:47.246371 19471 rpcz_store.cc:276] 0430 20:11:42.193111 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:42.193479 (+ 368us) service_pool.cc:225] Handling call
0430 20:11:47.202568 (+5009089us) ts_tablet_manager.cc:1120] Deleting tablet 32c823d1700949ab8b22adcd83b7bfd1
0430 20:11:47.202578 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:47.213640 (+ 11062us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:47.239534 (+ 25894us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:47.246155 (+ 6621us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5008943}
I20240430 20:11:47.246946 20545 raft_consensus.cc:482] T 47bf64ecc41b445da4ff697e141243dd P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:47.247098 20545 raft_consensus.cc:504] T 47bf64ecc41b445da4ff697e141243dd P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
W20240430 20:11:47.247373 18558 connection.cc:441] server connection from 127.0.0.1:40218 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 280) could send its response
I20240430 20:11:47.248142 18763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47bf64ecc41b445da4ff697e141243dd" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:47.248370 20545 leader_election.cc:288] T 47bf64ecc41b445da4ff697e141243dd P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:47.248394 18763 raft_consensus.cc:2454] T 47bf64ecc41b445da4ff697e141243dd P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 0.
I20240430 20:11:47.247382 19471 tablet_replica.cc:330] T 4f6f9ae722da49e58c9b72107f6da2fa P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:47.248869 18417 leader_election.cc:302] T 47bf64ecc41b445da4ff697e141243dd P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters:
I20240430 20:11:47.248926 19471 raft_consensus.cc:2229] T 4f6f9ae722da49e58c9b72107f6da2fa P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Raft consensus shutting down.
I20240430 20:11:47.249209 19471 raft_consensus.cc:2258] T 4f6f9ae722da49e58c9b72107f6da2fa P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:47.249281 20545 raft_consensus.cc:2781] T 47bf64ecc41b445da4ff697e141243dd P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:47.249425 20545 raft_consensus.cc:482] T 47bf64ecc41b445da4ff697e141243dd P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:47.249564 20545 raft_consensus.cc:3037] T 47bf64ecc41b445da4ff697e141243dd P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:47.252219 18983 tablet_bootstrap.cc:492] T 78ba48dadef548708153878810cc46f8 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:47.252306 19471 ts_tablet_manager.cc:1875] T 4f6f9ae722da49e58c9b72107f6da2fa P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:47.252348 18983 ts_tablet_manager.cc:1367] T 78ba48dadef548708153878810cc46f8 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.000s
I20240430 20:11:47.252425 20693 consensus_queue.cc:1035] T 2873af6da928463f9809864b8eaf0c4a P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:47.255055 18983 raft_consensus.cc:348] T 78ba48dadef548708153878810cc46f8 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:47.255314 18983 raft_consensus.cc:374] T 78ba48dadef548708153878810cc46f8 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:47.255403 18983 raft_consensus.cc:729] T 78ba48dadef548708153878810cc46f8 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:47.255640 18983 consensus_queue.cc:260] T 78ba48dadef548708153878810cc46f8 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:47.256204 18983 ts_tablet_manager.cc:1398] T 78ba48dadef548708153878810cc46f8 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240430 20:11:47.256961 18983 tablet_bootstrap.cc:492] T fdf53061efd74fa29a9d8208d1fd210e P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:47.258738 19489 ts_tablet_manager.cc:1888] T f940d17b02b541289121463e05b0bb5f P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:47.259203 20545 raft_consensus.cc:504] T 47bf64ecc41b445da4ff697e141243dd P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:47.260486 18983 tablet_bootstrap.cc:654] T fdf53061efd74fa29a9d8208d1fd210e P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:47.261620 20545 leader_election.cc:288] T 47bf64ecc41b445da4ff697e141243dd P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:47.261425 18763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47bf64ecc41b445da4ff697e141243dd" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:47.262797 18763 raft_consensus.cc:3037] T 47bf64ecc41b445da4ff697e141243dd P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:47.263576 18985 tablet_bootstrap.cc:492] T 331c031d44854d679a27d10d9151d0ea P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:47.263700 18985 ts_tablet_manager.cc:1367] T 331c031d44854d679a27d10d9151d0ea P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.035s user 0.001s sys 0.003s
I20240430 20:11:47.264406 18985 raft_consensus.cc:348] T 331c031d44854d679a27d10d9151d0ea P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:47.264585 18985 raft_consensus.cc:374] T 331c031d44854d679a27d10d9151d0ea P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:47.264626 18985 raft_consensus.cc:729] T 331c031d44854d679a27d10d9151d0ea P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:47.264871 18985 consensus_queue.cc:260] T 331c031d44854d679a27d10d9151d0ea P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:47.265419 18763 raft_consensus.cc:2454] T 47bf64ecc41b445da4ff697e141243dd P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 1.
I20240430 20:11:47.265821 18985 ts_tablet_manager.cc:1398] T 331c031d44854d679a27d10d9151d0ea P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240430 20:11:47.266211 19489 log.cc:1196] T f940d17b02b541289121463e05b0bb5f P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/f940d17b02b541289121463e05b0bb5f
I20240430 20:11:47.266459 18417 leader_election.cc:302] T 47bf64ecc41b445da4ff697e141243dd P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters:
I20240430 20:11:47.266700 19489 ts_tablet_manager.cc:1909] T f940d17b02b541289121463e05b0bb5f P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:47.266796 20631 raft_consensus.cc:2781] T 47bf64ecc41b445da4ff697e141243dd P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:47.266906 20631 raft_consensus.cc:686] T 47bf64ecc41b445da4ff697e141243dd P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Becoming Leader. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Running, Role: LEADER
I20240430 20:11:47.267117 20631 consensus_queue.cc:237] T 47bf64ecc41b445da4ff697e141243dd P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:47.267468 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 367) took 3516 ms. Trace:
I20240430 20:11:47.267562 19489 rpcz_store.cc:276] 0430 20:11:43.750998 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:43.757151 (+ 6153us) service_pool.cc:225] Handling call
0430 20:11:47.241782 (+3484631us) ts_tablet_manager.cc:1120] Deleting tablet f940d17b02b541289121463e05b0bb5f
0430 20:11:47.241794 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:47.246082 (+ 4288us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:47.258542 (+ 12460us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:47.267430 (+ 8888us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3484401}
W20240430 20:11:47.267812 18697 connection.cc:441] server connection from 127.0.0.1:59830 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 367) could send its response
I20240430 20:11:47.272049 18985 tablet_bootstrap.cc:492] T c6e3d09402574f1e8fe9e4160c70a3ee P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:47.272147 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 380) took 3422 ms. Trace:
I20240430 20:11:47.272317 19489 rpcz_store.cc:276] 0430 20:11:43.849439 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:43.849577 (+ 138us) service_pool.cc:225] Handling call
0430 20:11:47.271992 (+3422415us) ts_tablet_manager.cc:1120] Deleting tablet 44fa6917a85c46f3aaa363df46ea586b
0430 20:11:47.272003 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:47.272090 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3422140}
W20240430 20:11:47.272693 18697 connection.cc:441] server connection from 127.0.0.1:59830 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 380) could send its response
I20240430 20:11:47.273113 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 379) took 3423 ms. Trace:
I20240430 20:11:47.273245 19489 rpcz_store.cc:276] 0430 20:11:43.849309 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:43.852159 (+ 2850us) service_pool.cc:225] Handling call
0430 20:11:47.272996 (+3420837us) ts_tablet_manager.cc:1120] Deleting tablet 87ac32a0ddb84676adc83038ef664591
0430 20:11:47.273007 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:47.273078 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3420607}
W20240430 20:11:47.273523 18697 connection.cc:441] server connection from 127.0.0.1:59830 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 379) could send its response
I20240430 20:11:47.273867 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 378) took 3424 ms. Trace:
I20240430 20:11:47.273995 19489 rpcz_store.cc:276] 0430 20:11:43.849164 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:43.852424 (+ 3260us) service_pool.cc:225] Handling call
0430 20:11:47.273760 (+3421336us) ts_tablet_manager.cc:1120] Deleting tablet 6e826459c3f4437da81c9e6e09f0c22f
0430 20:11:47.273768 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:47.273823 (+ 55us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3421189}
W20240430 20:11:47.274262 18697 connection.cc:441] server connection from 127.0.0.1:59830 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 378) could send its response
I20240430 20:11:47.274900 19489 tablet_replica.cc:330] T 2873af6da928463f9809864b8eaf0c4a P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:47.275151 19489 raft_consensus.cc:2229] T 2873af6da928463f9809864b8eaf0c4a P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Raft consensus shutting down.
I20240430 20:11:47.275460 19489 pending_rounds.cc:62] T 2873af6da928463f9809864b8eaf0c4a P 3c51c33d5616482cafb68e26688820f4: Trying to abort 1 pending ops.
I20240430 20:11:47.275575 19489 pending_rounds.cc:68] T 2873af6da928463f9809864b8eaf0c4a P 3c51c33d5616482cafb68e26688820f4: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7022624384530063360 op_type: NO_OP noop_request { }
I20240430 20:11:47.275789 19489 raft_consensus.cc:2866] T 2873af6da928463f9809864b8eaf0c4a P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240430 20:11:47.275982 19489 raft_consensus.cc:2258] T 2873af6da928463f9809864b8eaf0c4a P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:47.276809 19489 ts_tablet_manager.cc:1875] T 2873af6da928463f9809864b8eaf0c4a P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:47.278187 18616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47bf64ecc41b445da4ff697e141243dd" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:47.278393 18616 raft_consensus.cc:3037] T 47bf64ecc41b445da4ff697e141243dd P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:47.281119 18985 tablet_bootstrap.cc:654] T c6e3d09402574f1e8fe9e4160c70a3ee P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:47.281198 18616 raft_consensus.cc:2454] T 47bf64ecc41b445da4ff697e141243dd P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 1.
I20240430 20:11:47.278170 18619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47bf64ecc41b445da4ff697e141243dd" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:47.281857 18619 raft_consensus.cc:2362] T 47bf64ecc41b445da4ff697e141243dd P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 1. Re-sending same reply.
I20240430 20:11:47.282763 18983 tablet_bootstrap.cc:492] T fdf53061efd74fa29a9d8208d1fd210e P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:47.283010 18983 ts_tablet_manager.cc:1367] T fdf53061efd74fa29a9d8208d1fd210e P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.026s user 0.002s sys 0.001s
I20240430 20:11:47.283910 18983 raft_consensus.cc:348] T fdf53061efd74fa29a9d8208d1fd210e P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:47.284253 18983 raft_consensus.cc:374] T fdf53061efd74fa29a9d8208d1fd210e P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:47.284348 18983 raft_consensus.cc:729] T fdf53061efd74fa29a9d8208d1fd210e P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:47.284602 18983 consensus_queue.cc:260] T fdf53061efd74fa29a9d8208d1fd210e P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:47.285950 18983 ts_tablet_manager.cc:1398] T fdf53061efd74fa29a9d8208d1fd210e P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240430 20:11:47.286307 18983 tablet_bootstrap.cc:492] T 6d1d2193fd424c91b143f5256df5fcd0 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:47.288410 18983 tablet_bootstrap.cc:654] T 6d1d2193fd424c91b143f5256df5fcd0 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:47.288841 19471 ts_tablet_manager.cc:1888] T 4f6f9ae722da49e58c9b72107f6da2fa P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:47.289131 19471 log.cc:1196] T 4f6f9ae722da49e58c9b72107f6da2fa P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/4f6f9ae722da49e58c9b72107f6da2fa
I20240430 20:11:47.290261 18988 tablet_bootstrap.cc:654] T 331c031d44854d679a27d10d9151d0ea P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:47.291473 19489 ts_tablet_manager.cc:1888] T 2873af6da928463f9809864b8eaf0c4a P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:47.291615 19489 log.cc:1196] T 2873af6da928463f9809864b8eaf0c4a P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/2873af6da928463f9809864b8eaf0c4a
I20240430 20:11:47.292040 19489 ts_tablet_manager.cc:1909] T 2873af6da928463f9809864b8eaf0c4a P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:47.292902 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 383) took 3401 ms. Trace:
I20240430 20:11:47.294679 20545 consensus_queue.cc:1035] T fbdeb4714b8c4b93ab72822c43329aae P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:47.294867 19489 rpcz_store.cc:276] 0430 20:11:43.891736 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:43.893957 (+ 2221us) service_pool.cc:225] Handling call
0430 20:11:47.274827 (+3380870us) ts_tablet_manager.cc:1120] Deleting tablet 2873af6da928463f9809864b8eaf0c4a
0430 20:11:47.274836 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:47.280162 (+ 5326us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:47.291302 (+ 11140us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:47.292841 (+ 1539us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3379914}
W20240430 20:11:47.295629 18697 connection.cc:441] server connection from 127.0.0.1:59830 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 383) could send its response
I20240430 20:11:47.296128 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 384) took 2950 ms. Trace:
I20240430 20:11:47.297022 19471 ts_tablet_manager.cc:1909] T 4f6f9ae722da49e58c9b72107f6da2fa P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:47.297808 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 279) took 5104 ms. Trace:
I20240430 20:11:47.299393 20629 consensus_queue.cc:1035] T fbdeb4714b8c4b93ab72822c43329aae P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:47.298463 19471 rpcz_store.cc:276] 0430 20:11:42.193046 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:42.193380 (+ 334us) service_pool.cc:225] Handling call
0430 20:11:47.247319 (+5053939us) ts_tablet_manager.cc:1120] Deleting tablet 4f6f9ae722da49e58c9b72107f6da2fa
0430 20:11:47.247329 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:47.256786 (+ 9457us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:47.288718 (+ 31932us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:47.297790 (+ 9072us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5051547}
W20240430 20:11:47.307617 18558 connection.cc:441] server connection from 127.0.0.1:40218 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 279) could send its response
I20240430 20:11:47.307724 19471 tablet_replica.cc:330] T ae87f5e92a934e2182164c231bc76df0 P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:47.307991 19471 raft_consensus.cc:2229] T ae87f5e92a934e2182164c231bc76df0 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:47.308185 19471 raft_consensus.cc:2258] T ae87f5e92a934e2182164c231bc76df0 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:47.308846 19471 ts_tablet_manager.cc:1875] T ae87f5e92a934e2182164c231bc76df0 P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:47.316367 18985 tablet_bootstrap.cc:492] T c6e3d09402574f1e8fe9e4160c70a3ee P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:47.316538 18985 ts_tablet_manager.cc:1367] T c6e3d09402574f1e8fe9e4160c70a3ee P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.045s user 0.002s sys 0.002s
I20240430 20:11:47.317268 19489 rpcz_store.cc:276] 0430 20:11:44.345728 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:44.345790 (+ 62us) service_pool.cc:225] Handling call
0430 20:11:47.295970 (+2950180us) ts_tablet_manager.cc:1120] Deleting tablet c1017a50d1d646c486c0b9a75cb88ce4
0430 20:11:47.295982 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:47.296091 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2949859}
W20240430 20:11:47.317557 18697 connection.cc:441] server connection from 127.0.0.1:59830 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 384) could send its response
I20240430 20:11:47.317627 18985 raft_consensus.cc:348] T c6e3d09402574f1e8fe9e4160c70a3ee P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:47.317824 18985 raft_consensus.cc:374] T c6e3d09402574f1e8fe9e4160c70a3ee P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:47.317883 18985 raft_consensus.cc:729] T c6e3d09402574f1e8fe9e4160c70a3ee P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:47.318178 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 386) took 2953 ms. Trace:
I20240430 20:11:47.318369 19489 rpcz_store.cc:276] 0430 20:11:44.365091 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:44.365142 (+ 51us) service_pool.cc:225] Handling call
0430 20:11:47.318019 (+2952877us) ts_tablet_manager.cc:1120] Deleting tablet 024069ab786749028c020abd1c836cf5
0430 20:11:47.318031 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:47.318125 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2952313}
W20240430 20:11:47.318725 18697 connection.cc:441] server connection from 127.0.0.1:59830 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 386) could send its response
I20240430 20:11:47.318725 18985 consensus_queue.cc:260] T c6e3d09402574f1e8fe9e4160c70a3ee P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:47.319072 19489 tablet_replica.cc:330] T f6cfd089f08b47409fb2a6cf4f6b0385 P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:47.322526 20630 consensus_queue.cc:1035] T 66dd767bd3f8494f94061fb89e5e5680 P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:47.326750 18985 ts_tablet_manager.cc:1398] T c6e3d09402574f1e8fe9e4160c70a3ee P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20240430 20:11:47.332695 18985 tablet_bootstrap.cc:492] T 78ba48dadef548708153878810cc46f8 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:47.333254 18988 tablet_bootstrap.cc:492] T 331c031d44854d679a27d10d9151d0ea P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:47.333401 20693 consensus_queue.cc:1035] T 06d28b6f48a34b17b0d7fb3102df2f4c P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:47.335889 18983 tablet_bootstrap.cc:492] T 6d1d2193fd424c91b143f5256df5fcd0 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:47.336123 18983 ts_tablet_manager.cc:1367] T 6d1d2193fd424c91b143f5256df5fcd0 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.050s user 0.006s sys 0.013s
I20240430 20:11:47.336933 18983 raft_consensus.cc:348] T 6d1d2193fd424c91b143f5256df5fcd0 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:47.337267 18983 raft_consensus.cc:374] T 6d1d2193fd424c91b143f5256df5fcd0 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:47.337472 18983 raft_consensus.cc:729] T 6d1d2193fd424c91b143f5256df5fcd0 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:47.337929 18983 consensus_queue.cc:260] T 6d1d2193fd424c91b143f5256df5fcd0 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:47.338706 20695 raft_consensus.cc:2726] T f6cfd089f08b47409fb2a6cf4f6b0385 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:47.339040 19489 raft_consensus.cc:2229] T f6cfd089f08b47409fb2a6cf4f6b0385 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:47.339319 19489 pending_rounds.cc:62] T f6cfd089f08b47409fb2a6cf4f6b0385 P 3c51c33d5616482cafb68e26688820f4: Trying to abort 1 pending ops.
I20240430 20:11:47.333418 18988 ts_tablet_manager.cc:1367] T 331c031d44854d679a27d10d9151d0ea P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.113s user 0.001s sys 0.003s
I20240430 20:11:47.339419 19489 pending_rounds.cc:68] T f6cfd089f08b47409fb2a6cf4f6b0385 P 3c51c33d5616482cafb68e26688820f4: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7022624384250748928 op_type: NO_OP noop_request { }
I20240430 20:11:47.339555 19489 raft_consensus.cc:2866] T f6cfd089f08b47409fb2a6cf4f6b0385 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20240430 20:11:47.339674 19489 raft_consensus.cc:2258] T f6cfd089f08b47409fb2a6cf4f6b0385 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:47.340108 18988 raft_consensus.cc:348] T 331c031d44854d679a27d10d9151d0ea P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:47.340314 18988 raft_consensus.cc:374] T 331c031d44854d679a27d10d9151d0ea P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:47.340433 18988 raft_consensus.cc:729] T 331c031d44854d679a27d10d9151d0ea P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:47.341553 19489 ts_tablet_manager.cc:1875] T f6cfd089f08b47409fb2a6cf4f6b0385 P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:47.334568 18985 tablet_bootstrap.cc:654] T 78ba48dadef548708153878810cc46f8 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:47.346197 18983 ts_tablet_manager.cc:1398] T 6d1d2193fd424c91b143f5256df5fcd0 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20240430 20:11:47.348167 19489 ts_tablet_manager.cc:1888] T f6cfd089f08b47409fb2a6cf4f6b0385 P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:47.348253 18983 tablet_bootstrap.cc:492] T 130007acc8ba4e709c3eb85fabe75903 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:47.348320 19489 log.cc:1196] T f6cfd089f08b47409fb2a6cf4f6b0385 P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/f6cfd089f08b47409fb2a6cf4f6b0385
I20240430 20:11:47.348733 19489 ts_tablet_manager.cc:1909] T f6cfd089f08b47409fb2a6cf4f6b0385 P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:47.349597 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 385) took 2984 ms. Trace:
I20240430 20:11:47.349720 19489 rpcz_store.cc:276] 0430 20:11:44.364961 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:44.365020 (+ 59us) service_pool.cc:225] Handling call
0430 20:11:47.319011 (+2953991us) ts_tablet_manager.cc:1120] Deleting tablet f6cfd089f08b47409fb2a6cf4f6b0385
0430 20:11:47.319020 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:47.339221 (+ 20201us) spinlock_profiling.cc:243] Waited 19.7 ms on lock 0x616000280f54. stack: 00007f30e7d63756 00007f30e7d63510 00007f30e6d9648d 0000000000955804 00000000009557cd 000000000095519a 00007f30f35d1a00 00007f30f48ab853 00007f30f733e4bc 00007f30f7370814 00007f30f7351b00 00007f30f735188d 0000000000a60bcc 00007f30e7da9b74 00007f30e7dad19b 00007f30e7dad018
0430 20:11:47.344872 (+ 5651us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:47.348027 (+ 3155us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:47.349560 (+ 1533us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":26,"spinlock_wait_cycles":45314944,"tablet-delete.queue_time_us":2953316}
I20240430 20:11:47.350035 18983 tablet_bootstrap.cc:654] T 130007acc8ba4e709c3eb85fabe75903 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
W20240430 20:11:47.350075 18697 connection.cc:441] server connection from 127.0.0.1:59830 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 385) could send its response
I20240430 20:11:47.350234 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 387) took 2807 ms. Trace:
I20240430 20:11:47.350368 19489 rpcz_store.cc:276] 0430 20:11:44.543249 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:44.543563 (+ 314us) service_pool.cc:225] Handling call
0430 20:11:47.350086 (+2806523us) ts_tablet_manager.cc:1120] Deleting tablet a37d917bcc45468f9e3ee805640fba98
0430 20:11:47.350095 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:47.350188 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2806306}
W20240430 20:11:47.350580 18417 consensus_peers.cc:485] T f6cfd089f08b47409fb2a6cf4f6b0385 P 2efb542aa55e439b99ccb98d1c5c884a -> Peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Couldn't send request to peer 3c51c33d5616482cafb68e26688820f4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f6cfd089f08b47409fb2a6cf4f6b0385. This is attempt 1: this message will repeat every 5th retry.
W20240430 20:11:47.350651 18697 connection.cc:441] server connection from 127.0.0.1:59830 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 387) could send its response
I20240430 20:11:47.355558 20489 consensus_queue.cc:1035] T de1dfe6c82d34a4d8d02e466c55327ff P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240430 20:11:47.358034 18696 consensus_peers.cc:485] T 06d28b6f48a34b17b0d7fb3102df2f4c P 3c51c33d5616482cafb68e26688820f4 -> Peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Couldn't send request to peer 5e1c6397f09747f091534faf93542710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 06d28b6f48a34b17b0d7fb3102df2f4c. This is attempt 1: this message will repeat every 5th retry.
I20240430 20:11:47.361436 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 388) took 2781 ms. Trace:
I20240430 20:11:47.361579 19489 rpcz_store.cc:276] 0430 20:11:44.584989 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:44.585057 (+ 68us) service_pool.cc:225] Handling call
0430 20:11:47.361279 (+2776222us) ts_tablet_manager.cc:1120] Deleting tablet a445fdfde01c4371a5e6b3daea5b00ab
0430 20:11:47.361291 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:47.361392 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2775939}
W20240430 20:11:47.361832 18697 connection.cc:441] server connection from 127.0.0.1:59830 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 388) could send its response
I20240430 20:11:47.363188 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_5721e7a6b1ef44a780ff57d59fa76f64 [id=01a774990824472a87414d89d98e1863]
I20240430 20:11:47.373430 19471 ts_tablet_manager.cc:1888] T ae87f5e92a934e2182164c231bc76df0 P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:47.380146 18985 tablet_bootstrap.cc:492] T 78ba48dadef548708153878810cc46f8 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:47.380349 18985 ts_tablet_manager.cc:1367] T 78ba48dadef548708153878810cc46f8 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.048s user 0.002s sys 0.002s
I20240430 20:11:47.381109 18985 raft_consensus.cc:348] T 78ba48dadef548708153878810cc46f8 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:47.381325 18985 raft_consensus.cc:374] T 78ba48dadef548708153878810cc46f8 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:47.381381 18985 raft_consensus.cc:729] T 78ba48dadef548708153878810cc46f8 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:47.381657 18985 consensus_queue.cc:260] T 78ba48dadef548708153878810cc46f8 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:47.382112 18985 ts_tablet_manager.cc:1398] T 78ba48dadef548708153878810cc46f8 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240430 20:11:47.382369 18985 tablet_bootstrap.cc:492] T 53da4b3855344b6d92211af767195dc8 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:47.384078 18985 tablet_bootstrap.cc:654] T 53da4b3855344b6d92211af767195dc8 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:47.378918 20384 consensus_queue.cc:1035] T de1dfe6c82d34a4d8d02e466c55327ff P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:47.395817 19471 log.cc:1196] T ae87f5e92a934e2182164c231bc76df0 P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/ae87f5e92a934e2182164c231bc76df0
I20240430 20:11:47.396224 19471 ts_tablet_manager.cc:1909] T ae87f5e92a934e2182164c231bc76df0 P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:47.396241 19489 tablet_replica.cc:330] T e3e74e76ab2b4fe492ffcc1838656fd1 P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:47.395619 18983 tablet_bootstrap.cc:492] T 130007acc8ba4e709c3eb85fabe75903 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:47.396605 18983 ts_tablet_manager.cc:1367] T 130007acc8ba4e709c3eb85fabe75903 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.048s user 0.013s sys 0.026s
I20240430 20:11:47.397373 18983 raft_consensus.cc:348] T 130007acc8ba4e709c3eb85fabe75903 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:47.397570 18983 raft_consensus.cc:374] T 130007acc8ba4e709c3eb85fabe75903 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:47.397632 18983 raft_consensus.cc:729] T 130007acc8ba4e709c3eb85fabe75903 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:47.397830 18983 consensus_queue.cc:260] T 130007acc8ba4e709c3eb85fabe75903 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:47.405594 18983 ts_tablet_manager.cc:1398] T 130007acc8ba4e709c3eb85fabe75903 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20240430 20:11:47.406919 20630 consensus_queue.cc:1035] T f6cfd089f08b47409fb2a6cf4f6b0385 P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:47.409909 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_96aea8c4ff80428da8a0394f1bf7b493 [id=03ccb721140c4e0193e2e48d509430f9]
I20240430 20:11:47.410188 19489 raft_consensus.cc:2229] T e3e74e76ab2b4fe492ffcc1838656fd1 P 3c51c33d5616482cafb68e26688820f4 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:47.410965 19489 raft_consensus.cc:2258] T e3e74e76ab2b4fe492ffcc1838656fd1 P 3c51c33d5616482cafb68e26688820f4 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:47.410564 18988 consensus_queue.cc:260] T 331c031d44854d679a27d10d9151d0ea P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:47.413992 20630 raft_consensus.cc:482] T 6d1d2193fd424c91b143f5256df5fcd0 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:47.414144 20630 raft_consensus.cc:504] T 6d1d2193fd424c91b143f5256df5fcd0 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:47.414783 20630 leader_election.cc:288] T 6d1d2193fd424c91b143f5256df5fcd0 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:47.415861 18985 tablet_bootstrap.cc:492] T 53da4b3855344b6d92211af767195dc8 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:47.415990 18985 ts_tablet_manager.cc:1367] T 53da4b3855344b6d92211af767195dc8 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.034s user 0.003s sys 0.000s
I20240430 20:11:47.416635 18985 raft_consensus.cc:348] T 53da4b3855344b6d92211af767195dc8 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:47.416811 18985 raft_consensus.cc:374] T 53da4b3855344b6d92211af767195dc8 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:47.416855 18985 raft_consensus.cc:729] T 53da4b3855344b6d92211af767195dc8 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:47.417023 18985 consensus_queue.cc:260] T 53da4b3855344b6d92211af767195dc8 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:47.417533 18985 ts_tablet_manager.cc:1398] T 53da4b3855344b6d92211af767195dc8 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240430 20:11:47.417814 18985 tablet_bootstrap.cc:492] T fdf53061efd74fa29a9d8208d1fd210e P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:47.418571 20545 raft_consensus.cc:482] T c6e3d09402574f1e8fe9e4160c70a3ee P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:47.418727 20545 raft_consensus.cc:504] T c6e3d09402574f1e8fe9e4160c70a3ee P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:47.419266 20545 leader_election.cc:288] T c6e3d09402574f1e8fe9e4160c70a3ee P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:47.425796 18985 tablet_bootstrap.cc:654] T fdf53061efd74fa29a9d8208d1fd210e P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:47.427109 18983 tablet_bootstrap.cc:492] T 53da4b3855344b6d92211af767195dc8 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:47.428754 18983 tablet_bootstrap.cc:654] T 53da4b3855344b6d92211af767195dc8 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:47.433495 18608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d1d2193fd424c91b143f5256df5fcd0" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:47.433724 18608 raft_consensus.cc:2454] T 6d1d2193fd424c91b143f5256df5fcd0 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 0.
I20240430 20:11:47.434422 18608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e3d09402574f1e8fe9e4160c70a3ee" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:47.434681 18608 raft_consensus.cc:2379] T c6e3d09402574f1e8fe9e4160c70a3ee P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2efb542aa55e439b99ccb98d1c5c884a in current term 1: Already voted for candidate 3c51c33d5616482cafb68e26688820f4 in this term.
I20240430 20:11:47.435935 20384 raft_consensus.cc:482] T 331c031d44854d679a27d10d9151d0ea P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:47.436085 20384 raft_consensus.cc:504] T 331c031d44854d679a27d10d9151d0ea P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:47.436928 20384 leader_election.cc:288] T 331c031d44854d679a27d10d9151d0ea P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:47.437464 18473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "331c031d44854d679a27d10d9151d0ea" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:47.437675 18473 raft_consensus.cc:2454] T 331c031d44854d679a27d10d9151d0ea P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 0.
I20240430 20:11:47.437853 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 282) took 5232 ms. Trace:
I20240430 20:11:47.437937 19471 rpcz_store.cc:276] 0430 20:11:42.205339 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:42.205700 (+ 361us) service_pool.cc:225] Handling call
0430 20:11:47.307661 (+5101961us) ts_tablet_manager.cc:1120] Deleting tablet ae87f5e92a934e2182164c231bc76df0
0430 20:11:47.307676 (+ 15us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:47.348706 (+ 41030us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:47.373272 (+ 24566us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:47.437839 (+ 64567us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5101728}
I20240430 20:11:47.438093 18558 leader_election.cc:302] T 331c031d44854d679a27d10d9151d0ea P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:47.438287 19471 tablet_replica.cc:330] T fdf53061efd74fa29a9d8208d1fd210e P 5e1c6397f09747f091534faf93542710: stopping tablet replica
W20240430 20:11:47.438313 18558 connection.cc:441] server connection from 127.0.0.1:40218 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 282) could send its response
I20240430 20:11:47.438328 20384 raft_consensus.cc:2781] T 331c031d44854d679a27d10d9151d0ea P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:47.438450 20384 raft_consensus.cc:482] T 331c031d44854d679a27d10d9151d0ea P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:47.438495 19471 raft_consensus.cc:2229] T fdf53061efd74fa29a9d8208d1fd210e P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:47.438529 20384 raft_consensus.cc:3037] T 331c031d44854d679a27d10d9151d0ea P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:47.438639 19471 raft_consensus.cc:2258] T fdf53061efd74fa29a9d8208d1fd210e P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:47.411631 19489 ts_tablet_manager.cc:1875] T e3e74e76ab2b4fe492ffcc1838656fd1 P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:47.441329 20384 raft_consensus.cc:504] T 331c031d44854d679a27d10d9151d0ea P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:47.415468 18750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d1d2193fd424c91b143f5256df5fcd0" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:47.442797 18762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "331c031d44854d679a27d10d9151d0ea" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:47.442927 18762 raft_consensus.cc:3037] T 331c031d44854d679a27d10d9151d0ea P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:47.420905 18763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e3d09402574f1e8fe9e4160c70a3ee" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:47.437711 18758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "331c031d44854d679a27d10d9151d0ea" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
W20240430 20:11:47.444061 18417 leader_election.cc:341] T 6d1d2193fd424c91b143f5256df5fcd0 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:47.412166 18988 ts_tablet_manager.cc:1398] T 331c031d44854d679a27d10d9151d0ea P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.073s user 0.000s sys 0.002s
I20240430 20:11:47.445127 18988 tablet_bootstrap.cc:492] T b22aaedb0a6c4fd9ad0ea8fa019001e1 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:47.447592 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_3a588666409c4a64b13658a42a58f994 [id=053d7bb753464e968b05493053eb229a]
I20240430 20:11:47.449805 20384 leader_election.cc:288] T 331c031d44854d679a27d10d9151d0ea P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:47.445485 18762 raft_consensus.cc:2454] T 331c031d44854d679a27d10d9151d0ea P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
I20240430 20:11:47.450532 18557 leader_election.cc:302] T 331c031d44854d679a27d10d9151d0ea P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:47.450886 20384 raft_consensus.cc:2781] T 331c031d44854d679a27d10d9151d0ea P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:47.450858 18473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "331c031d44854d679a27d10d9151d0ea" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:47.451078 18473 raft_consensus.cc:3037] T 331c031d44854d679a27d10d9151d0ea P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:47.451328 20384 raft_consensus.cc:686] T 331c031d44854d679a27d10d9151d0ea P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Becoming Leader. State: Replica: 5e1c6397f09747f091534faf93542710, State: Running, Role: LEADER
I20240430 20:11:47.451660 20384 consensus_queue.cc:237] T 331c031d44854d679a27d10d9151d0ea P 5e1c6397f09747f091534faf93542710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:47.452137 19471 ts_tablet_manager.cc:1875] T fdf53061efd74fa29a9d8208d1fd210e P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:47.453697 18415 leader_election.cc:302] T 6d1d2193fd424c91b143f5256df5fcd0 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters: 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:47.453866 18473 raft_consensus.cc:2454] T 331c031d44854d679a27d10d9151d0ea P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
I20240430 20:11:47.454160 20630 raft_consensus.cc:2781] T 6d1d2193fd424c91b143f5256df5fcd0 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:47.454229 20630 raft_consensus.cc:482] T 6d1d2193fd424c91b143f5256df5fcd0 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:47.454285 20630 raft_consensus.cc:3037] T 6d1d2193fd424c91b143f5256df5fcd0 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:47.455204 18988 tablet_bootstrap.cc:654] T b22aaedb0a6c4fd9ad0ea8fa019001e1 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:47.458175 20630 raft_consensus.cc:504] T 6d1d2193fd424c91b143f5256df5fcd0 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:47.458887 20630 leader_election.cc:288] T 6d1d2193fd424c91b143f5256df5fcd0 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:47.459501 18985 tablet_bootstrap.cc:492] T fdf53061efd74fa29a9d8208d1fd210e P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:47.459755 18985 ts_tablet_manager.cc:1367] T fdf53061efd74fa29a9d8208d1fd210e P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.042s user 0.005s sys 0.006s
I20240430 20:11:47.460110 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_5f6e0fce999b46de927f9271e5de5071 [id=0baad83a8c0f4aeebc7ddad068625a69]
I20240430 20:11:47.460150 18762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d1d2193fd424c91b143f5256df5fcd0" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
W20240430 20:11:47.460786 18417 leader_election.cc:341] T 6d1d2193fd424c91b143f5256df5fcd0 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:47.461359 18415 leader_election.cc:302] T c6e3d09402574f1e8fe9e4160c70a3ee P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a; no voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710
I20240430 20:11:47.461652 20630 raft_consensus.cc:3037] T c6e3d09402574f1e8fe9e4160c70a3ee P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:47.461478 18985 raft_consensus.cc:348] T fdf53061efd74fa29a9d8208d1fd210e P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:47.461796 18985 raft_consensus.cc:374] T fdf53061efd74fa29a9d8208d1fd210e P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:47.461910 18985 raft_consensus.cc:729] T fdf53061efd74fa29a9d8208d1fd210e P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:47.462239 18985 consensus_queue.cc:260] T fdf53061efd74fa29a9d8208d1fd210e P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:47.464207 20630 raft_consensus.cc:2726] T c6e3d09402574f1e8fe9e4160c70a3ee P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:47.465991 18985 ts_tablet_manager.cc:1398] T fdf53061efd74fa29a9d8208d1fd210e P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240430 20:11:47.466919 18985 tablet_bootstrap.cc:492] T 885d27d4f8df48dd996d5b309b783f3b P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:47.472748 18985 tablet_bootstrap.cc:654] T 885d27d4f8df48dd996d5b309b783f3b P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:47.475042 18988 tablet_bootstrap.cc:492] T b22aaedb0a6c4fd9ad0ea8fa019001e1 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:47.475193 18988 ts_tablet_manager.cc:1367] T b22aaedb0a6c4fd9ad0ea8fa019001e1 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.030s user 0.003s sys 0.000s
I20240430 20:11:47.480830 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_75efd32c9420454f901c61882c8c29bc [id=0d741ba94928463a9929f799e0336fdb]
I20240430 20:11:47.481420 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_9c955414a012419eb8f904a0878a6336 [id=0e5a8ce8c9a440caa87263100895a12e]
I20240430 20:11:47.485325 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_ad4322a6bb564abd80dad138208e420d [id=0f4d7a1eb8834b559e79685fbb9de43b]
I20240430 20:11:47.485973 18983 tablet_bootstrap.cc:492] T 53da4b3855344b6d92211af767195dc8 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:47.486131 18983 ts_tablet_manager.cc:1367] T 53da4b3855344b6d92211af767195dc8 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.059s user 0.003s sys 0.007s
I20240430 20:11:47.487154 18983 raft_consensus.cc:348] T 53da4b3855344b6d92211af767195dc8 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:47.487380 18983 raft_consensus.cc:374] T 53da4b3855344b6d92211af767195dc8 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:47.487435 18983 raft_consensus.cc:729] T 53da4b3855344b6d92211af767195dc8 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:47.487660 18983 consensus_queue.cc:260] T 53da4b3855344b6d92211af767195dc8 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:47.488329 18983 ts_tablet_manager.cc:1398] T 53da4b3855344b6d92211af767195dc8 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240430 20:11:47.488611 18983 tablet_bootstrap.cc:492] T c091701480c4412faabad3d071d85d03 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:47.490132 18983 tablet_bootstrap.cc:654] T c091701480c4412faabad3d071d85d03 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:47.482321 18988 raft_consensus.cc:348] T b22aaedb0a6c4fd9ad0ea8fa019001e1 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:47.490748 18988 raft_consensus.cc:374] T b22aaedb0a6c4fd9ad0ea8fa019001e1 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:47.491289 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_c12ca6938d424fbfa2e1df5cdb00fdbc [id=0fb93f36bc17408ea56bc4cafcfd8580]
I20240430 20:11:47.492637 18988 raft_consensus.cc:729] T b22aaedb0a6c4fd9ad0ea8fa019001e1 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:47.493093 18620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d1d2193fd424c91b143f5256df5fcd0" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:47.493331 18620 raft_consensus.cc:3037] T 6d1d2193fd424c91b143f5256df5fcd0 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:47.496156 18620 raft_consensus.cc:2454] T 6d1d2193fd424c91b143f5256df5fcd0 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 1.
I20240430 20:11:47.497589 18988 consensus_queue.cc:260] T b22aaedb0a6c4fd9ad0ea8fa019001e1 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:47.503050 18988 ts_tablet_manager.cc:1398] T b22aaedb0a6c4fd9ad0ea8fa019001e1 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.028s user 0.003s sys 0.000s
I20240430 20:11:47.505471 18988 tablet_bootstrap.cc:492] T 53da4b3855344b6d92211af767195dc8 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:47.510113 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_70ba80736dca466e9ca9c231a40e7bd4 [id=1399e5a2de60445ab758e25dbc54ba34]
I20240430 20:11:47.510802 18415 leader_election.cc:302] T 6d1d2193fd424c91b143f5256df5fcd0 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters: 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:47.511530 20630 raft_consensus.cc:2781] T 6d1d2193fd424c91b143f5256df5fcd0 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:47.511996 20630 raft_consensus.cc:686] T 6d1d2193fd424c91b143f5256df5fcd0 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Becoming Leader. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Running, Role: LEADER
I20240430 20:11:47.513020 20630 consensus_queue.cc:237] T 6d1d2193fd424c91b143f5256df5fcd0 P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:47.504500 19489 ts_tablet_manager.cc:1888] T e3e74e76ab2b4fe492ffcc1838656fd1 P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 3.2
I20240430 20:11:47.513434 19489 log.cc:1196] T e3e74e76ab2b4fe492ffcc1838656fd1 P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/e3e74e76ab2b4fe492ffcc1838656fd1
I20240430 20:11:47.514179 18988 tablet_bootstrap.cc:654] T 53da4b3855344b6d92211af767195dc8 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:47.515038 19489 ts_tablet_manager.cc:1909] T e3e74e76ab2b4fe492ffcc1838656fd1 P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:47.515945 18985 tablet_bootstrap.cc:492] T 885d27d4f8df48dd996d5b309b783f3b P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:47.517071 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_4adefb6ee63f4301a479e1d042ff1c8b [id=16b11b30a9ab46b5a2c5da5c6a494ac8]
I20240430 20:11:47.517989 18983 tablet_bootstrap.cc:492] T c091701480c4412faabad3d071d85d03 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:47.518185 18983 ts_tablet_manager.cc:1367] T c091701480c4412faabad3d071d85d03 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.030s user 0.003s sys 0.000s
I20240430 20:11:47.519058 18983 raft_consensus.cc:348] T c091701480c4412faabad3d071d85d03 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:47.521821 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_605f23ec0a58457f87356d7f45d3b45c [id=1930f366a0f0456cad9a1c2456eb8fac]
I20240430 20:11:47.522672 18983 raft_consensus.cc:374] T c091701480c4412faabad3d071d85d03 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:47.522815 18983 raft_consensus.cc:729] T c091701480c4412faabad3d071d85d03 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:47.523216 18983 consensus_queue.cc:260] T c091701480c4412faabad3d071d85d03 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:47.524266 18983 ts_tablet_manager.cc:1398] T c091701480c4412faabad3d071d85d03 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240430 20:11:47.519461 19471 ts_tablet_manager.cc:1888] T fdf53061efd74fa29a9d8208d1fd210e P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:47.524508 19471 log.cc:1196] T fdf53061efd74fa29a9d8208d1fd210e P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/fdf53061efd74fa29a9d8208d1fd210e
I20240430 20:11:47.525135 19471 ts_tablet_manager.cc:1909] T fdf53061efd74fa29a9d8208d1fd210e P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:47.528414 20384 raft_consensus.cc:482] T c091701480c4412faabad3d071d85d03 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:47.528740 20384 raft_consensus.cc:504] T c091701480c4412faabad3d071d85d03 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:47.530346 18756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c091701480c4412faabad3d071d85d03" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
W20240430 20:11:47.530885 18557 leader_election.cc:341] T c091701480c4412faabad3d071d85d03 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:47.531545 18473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c091701480c4412faabad3d071d85d03" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
W20240430 20:11:47.532095 18558 leader_election.cc:341] T c091701480c4412faabad3d071d85d03 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:47.532193 18558 leader_election.cc:302] T c091701480c4412faabad3d071d85d03 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:47.532416 20047 raft_consensus.cc:2726] T c091701480c4412faabad3d071d85d03 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:47.532644 20384 leader_election.cc:288] T c091701480c4412faabad3d071d85d03 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:47.534135 20631 raft_consensus.cc:482] T 22fef8bb6b034db098c3372ae992f7a7 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:47.534282 20631 raft_consensus.cc:504] T 22fef8bb6b034db098c3372ae992f7a7 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:47.534932 20631 leader_election.cc:288] T 22fef8bb6b034db098c3372ae992f7a7 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:47.538815 18116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNG3HqZ/build/asan/bin/kudu
/tmp/dist-test-taskNG3HqZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.177.60:37225
--webserver_interface=127.17.177.60
--webserver_port=34267
--builtin_ntp_servers=127.17.177.20:42079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.177.62:36441,127.17.177.61:34777,127.17.177.60:37225
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20240430 20:11:47.538771 18985 ts_tablet_manager.cc:1367] T 885d27d4f8df48dd996d5b309b783f3b P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.072s user 0.001s sys 0.008s
I20240430 20:11:47.542387 18985 raft_consensus.cc:348] T 885d27d4f8df48dd996d5b309b783f3b P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:47.543161 18985 raft_consensus.cc:374] T 885d27d4f8df48dd996d5b309b783f3b P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:47.543428 18985 raft_consensus.cc:729] T 885d27d4f8df48dd996d5b309b783f3b P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:47.544202 18985 consensus_queue.cc:260] T 885d27d4f8df48dd996d5b309b783f3b P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:47.544773 18985 ts_tablet_manager.cc:1398] T 885d27d4f8df48dd996d5b309b783f3b P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240430 20:11:47.545043 18985 tablet_bootstrap.cc:492] T e4cffa295d8c401aa8415ce7e8f56526 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:47.546705 18985 tablet_bootstrap.cc:654] T e4cffa295d8c401aa8415ce7e8f56526 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:47.550030 18756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22fef8bb6b034db098c3372ae992f7a7" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:47.551499 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 390) took 2630 ms. Trace:
I20240430 20:11:47.551656 19489 rpcz_store.cc:276] 0430 20:11:44.920897 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:44.920950 (+ 53us) service_pool.cc:225] Handling call
0430 20:11:47.396148 (+2475198us) ts_tablet_manager.cc:1120] Deleting tablet e3e74e76ab2b4fe492ffcc1838656fd1
0430 20:11:47.396162 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:47.477637 (+ 81475us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:47.504386 (+ 26749us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:47.551475 (+ 47089us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":896,"tablet-delete.queue_time_us":2474942}
W20240430 20:11:47.552132 18697 connection.cc:441] server connection from 127.0.0.1:59830 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 390) could send its response
I20240430 20:11:47.552560 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 389) took 2631 ms. Trace:
I20240430 20:11:47.552700 19489 rpcz_store.cc:276] 0430 20:11:44.920748 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:44.920957 (+ 209us) service_pool.cc:225] Handling call
0430 20:11:47.552432 (+2631475us) ts_tablet_manager.cc:1120] Deleting tablet 9f0b341a31224bf1be525b84b9500304
0430 20:11:47.552443 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:47.552544 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":24,"tablet-delete.queue_time_us":2631227}
W20240430 20:11:47.556604 18697 connection.cc:441] server connection from 127.0.0.1:59830 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 389) could send its response
I20240430 20:11:47.556970 19489 tablet_replica.cc:330] T 76d6e9d547ca4a848eae852944b1d117 P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:47.557219 19489 raft_consensus.cc:2229] T 76d6e9d547ca4a848eae852944b1d117 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:47.557467 19489 raft_consensus.cc:2258] T 76d6e9d547ca4a848eae852944b1d117 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:47.558163 19489 ts_tablet_manager.cc:1875] T 76d6e9d547ca4a848eae852944b1d117 P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:47.562811 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_df19cb8a6f4d46699350ec5b69c14e5a [id=197923633fc54a4582207942b51530e2]
I20240430 20:11:47.563660 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_a9dc8fc6a4aa46dc914d0e327a4de554 [id=19f4e6decedf4ee59fccc0a24a05cedd]
I20240430 20:11:47.564705 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_2eed103638364cde94cd8af68d3cdf9b [id=1a382ad8009b4522a50d55c307db8f96]
I20240430 20:11:47.565472 18983 tablet_bootstrap.cc:492] T 885d27d4f8df48dd996d5b309b783f3b P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:47.565629 18619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22fef8bb6b034db098c3372ae992f7a7" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:47.565827 18619 raft_consensus.cc:2379] T 22fef8bb6b034db098c3372ae992f7a7 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2efb542aa55e439b99ccb98d1c5c884a in current term 1: Already voted for candidate 3c51c33d5616482cafb68e26688820f4 in this term.
I20240430 20:11:47.566056 20198 consensus_queue.cc:1035] T 06e82c706b9c4f35bf53faddf57b3fd7 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:47.567350 18983 tablet_bootstrap.cc:654] T 885d27d4f8df48dd996d5b309b783f3b P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:47.568822 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 283) took 5362 ms. Trace:
I20240430 20:11:47.568904 19471 rpcz_store.cc:276] 0430 20:11:42.206852 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:42.206908 (+ 56us) service_pool.cc:225] Handling call
0430 20:11:47.438238 (+5231330us) ts_tablet_manager.cc:1120] Deleting tablet fdf53061efd74fa29a9d8208d1fd210e
0430 20:11:47.438249 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:47.485678 (+ 47429us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:47.519333 (+ 33655us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:47.568805 (+ 49472us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5231156}
W20240430 20:11:47.569172 18558 connection.cc:441] server connection from 127.0.0.1:40218 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 283) could send its response
I20240430 20:11:47.569669 18415 leader_election.cc:302] T 22fef8bb6b034db098c3372ae992f7a7 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a; no voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710
I20240430 20:11:47.569770 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 298) took 4889 ms. Trace:
I20240430 20:11:47.570168 20631 raft_consensus.cc:3037] T 22fef8bb6b034db098c3372ae992f7a7 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:47.572832 20631 raft_consensus.cc:2726] T 22fef8bb6b034db098c3372ae992f7a7 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:47.569837 19471 rpcz_store.cc:276] 0430 20:11:42.680658 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:42.680712 (+ 54us) service_pool.cc:225] Handling call
0430 20:11:47.569350 (+4888638us) ts_tablet_manager.cc:1120] Deleting tablet dad16ae02b8b4e29a724d8eec67bb4b8
0430 20:11:47.569361 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:47.569754 (+ 393us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4888426}
W20240430 20:11:47.573557 18558 connection.cc:441] server connection from 127.0.0.1:40218 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 298) could send its response
I20240430 20:11:47.573876 19471 tablet_replica.cc:330] T 7683caba80594627900a0185f772c131 P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:47.574170 19471 raft_consensus.cc:2229] T 7683caba80594627900a0185f772c131 P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Raft consensus shutting down.
I20240430 20:11:47.574560 19471 raft_consensus.cc:2258] T 7683caba80594627900a0185f772c131 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:47.575340 19471 ts_tablet_manager.cc:1875] T 7683caba80594627900a0185f772c131 P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:47.582616 20198 consensus_queue.cc:1035] T 06e82c706b9c4f35bf53faddf57b3fd7 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:47.590061 18762 raft_consensus.cc:3037] T b22aaedb0a6c4fd9ad0ea8fa019001e1 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:47.594445 20630 consensus_queue.cc:1035] T b22aaedb0a6c4fd9ad0ea8fa019001e1 P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:47.600955 18988 tablet_bootstrap.cc:492] T 53da4b3855344b6d92211af767195dc8 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:47.601208 18988 ts_tablet_manager.cc:1367] T 53da4b3855344b6d92211af767195dc8 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.096s user 0.003s sys 0.000s
I20240430 20:11:47.602077 18988 raft_consensus.cc:348] T 53da4b3855344b6d92211af767195dc8 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:47.602361 18985 tablet_bootstrap.cc:492] T e4cffa295d8c401aa8415ce7e8f56526 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:47.602397 18988 raft_consensus.cc:374] T 53da4b3855344b6d92211af767195dc8 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:47.602506 18985 ts_tablet_manager.cc:1367] T e4cffa295d8c401aa8415ce7e8f56526 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.058s user 0.002s sys 0.004s
I20240430 20:11:47.602555 18988 raft_consensus.cc:729] T 53da4b3855344b6d92211af767195dc8 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:47.602952 18988 consensus_queue.cc:260] T 53da4b3855344b6d92211af767195dc8 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:47.603120 18985 raft_consensus.cc:348] T e4cffa295d8c401aa8415ce7e8f56526 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:47.603293 18985 raft_consensus.cc:374] T e4cffa295d8c401aa8415ce7e8f56526 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:47.603334 18985 raft_consensus.cc:729] T e4cffa295d8c401aa8415ce7e8f56526 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:47.603499 18985 consensus_queue.cc:260] T e4cffa295d8c401aa8415ce7e8f56526 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:47.604665 18988 ts_tablet_manager.cc:1398] T 53da4b3855344b6d92211af767195dc8 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240430 20:11:47.605196 18988 tablet_bootstrap.cc:492] T 130007acc8ba4e709c3eb85fabe75903 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:47.606189 18985 ts_tablet_manager.cc:1398] T e4cffa295d8c401aa8415ce7e8f56526 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240430 20:11:47.606464 18985 tablet_bootstrap.cc:492] T 9f0b341a31224bf1be525b84b9500304 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:47.607775 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_8db024fb9ef241ed86cb98d1c3939607 [id=1ad13ddf3cfd402a84d2d4a5a55030fa]
I20240430 20:11:47.608585 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_b2d07520b9944a89ad62fca70396174e [id=1c8e0a3a44ec4a798770c0165ed3dffa]
I20240430 20:11:47.608655 18985 tablet_bootstrap.cc:654] T 9f0b341a31224bf1be525b84b9500304 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:47.586310 20488 raft_consensus.cc:482] T 130007acc8ba4e709c3eb85fabe75903 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:47.609297 20488 raft_consensus.cc:504] T 130007acc8ba4e709c3eb85fabe75903 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:47.609952 20488 leader_election.cc:288] T 130007acc8ba4e709c3eb85fabe75903 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:47.599421 20047 consensus_queue.cc:1035] T a0baee91f5b44945b595d0e234af018e P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:47.610508 18474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "130007acc8ba4e709c3eb85fabe75903" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
W20240430 20:11:47.603955 18557 consensus_peers.cc:485] T f940d17b02b541289121463e05b0bb5f P 5e1c6397f09747f091534faf93542710 -> Peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Couldn't send request to peer 3c51c33d5616482cafb68e26688820f4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f940d17b02b541289121463e05b0bb5f. This is attempt 1: this message will repeat every 5th retry.
W20240430 20:11:47.611253 18558 leader_election.cc:341] T 130007acc8ba4e709c3eb85fabe75903 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:47.616338 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_938c56d9ae2f4ff6b3264a53b65d0977 [id=1df4e95702cd46f59ab21b101049a9ed]
I20240430 20:11:47.616405 18762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "130007acc8ba4e709c3eb85fabe75903" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
W20240430 20:11:47.617169 18557 leader_election.cc:341] T 130007acc8ba4e709c3eb85fabe75903 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:47.617288 18557 leader_election.cc:302] T 130007acc8ba4e709c3eb85fabe75903 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:47.617694 20047 raft_consensus.cc:2726] T 130007acc8ba4e709c3eb85fabe75903 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:47.618038 18988 tablet_bootstrap.cc:654] T 130007acc8ba4e709c3eb85fabe75903 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:47.630395 20198 raft_consensus.cc:482] T 283e5e73cbc04daba0de19f4541f9fd0 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:47.630559 20198 raft_consensus.cc:504] T 283e5e73cbc04daba0de19f4541f9fd0 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:47.631191 20198 leader_election.cc:288] T 283e5e73cbc04daba0de19f4541f9fd0 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:47.631376 20489 consensus_queue.cc:1035] T a0baee91f5b44945b595d0e234af018e P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:47.633472 18478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "283e5e73cbc04daba0de19f4541f9fd0" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:47.633677 18478 raft_consensus.cc:2454] T 283e5e73cbc04daba0de19f4541f9fd0 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:47.636749 18699 leader_election.cc:302] T 283e5e73cbc04daba0de19f4541f9fd0 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters:
I20240430 20:11:47.637171 20587 raft_consensus.cc:2781] T 283e5e73cbc04daba0de19f4541f9fd0 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:47.637266 20587 raft_consensus.cc:482] T 283e5e73cbc04daba0de19f4541f9fd0 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:47.637324 20587 raft_consensus.cc:3037] T 283e5e73cbc04daba0de19f4541f9fd0 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:47.640056 20587 raft_consensus.cc:504] T 283e5e73cbc04daba0de19f4541f9fd0 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:47.641422 20587 leader_election.cc:288] T 283e5e73cbc04daba0de19f4541f9fd0 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Requested vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:47.642460 18478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "283e5e73cbc04daba0de19f4541f9fd0" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:47.642675 18478 raft_consensus.cc:3037] T 283e5e73cbc04daba0de19f4541f9fd0 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:47.645318 18478 raft_consensus.cc:2454] T 283e5e73cbc04daba0de19f4541f9fd0 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
W20240430 20:11:47.633877 19895 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.177.60:37225, user_credentials={real_user=slave}} blocked reactor thread for 45601.9us
I20240430 20:11:47.647425 18699 leader_election.cc:302] T 283e5e73cbc04daba0de19f4541f9fd0 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters:
I20240430 20:11:47.647742 20693 raft_consensus.cc:2781] T 283e5e73cbc04daba0de19f4541f9fd0 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:47.648105 20693 raft_consensus.cc:686] T 283e5e73cbc04daba0de19f4541f9fd0 P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Becoming Leader. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Running, Role: LEADER
I20240430 20:11:47.648428 20693 consensus_queue.cc:237] T 283e5e73cbc04daba0de19f4541f9fd0 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:47.652410 18988 tablet_bootstrap.cc:492] T 130007acc8ba4e709c3eb85fabe75903 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
W20240430 20:11:47.652729 18415 consensus_peers.cc:485] T b22aaedb0a6c4fd9ad0ea8fa019001e1 P 2efb542aa55e439b99ccb98d1c5c884a -> Peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Couldn't send request to peer 5e1c6397f09747f091534faf93542710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b22aaedb0a6c4fd9ad0ea8fa019001e1. This is attempt 1: this message will repeat every 5th retry.
W20240430 20:11:47.652966 18696 leader_election.cc:341] T 283e5e73cbc04daba0de19f4541f9fd0 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Not found: Tablet not found: 283e5e73cbc04daba0de19f4541f9fd0
W20240430 20:11:47.654209 18696 leader_election.cc:341] T 283e5e73cbc04daba0de19f4541f9fd0 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Not found: Tablet not found: 283e5e73cbc04daba0de19f4541f9fd0
I20240430 20:11:47.655174 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_9579e986b42043f18f8e4aa8d0f6f754 [id=21e6703c9d844a37977f0f161c0b8c08]
I20240430 20:11:47.660817 19489 ts_tablet_manager.cc:1888] T 76d6e9d547ca4a848eae852944b1d117 P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:47.662773 18985 tablet_bootstrap.cc:492] T 9f0b341a31224bf1be525b84b9500304 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:47.663028 18985 ts_tablet_manager.cc:1367] T 9f0b341a31224bf1be525b84b9500304 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.057s user 0.003s sys 0.001s
I20240430 20:11:47.663795 18985 raft_consensus.cc:348] T 9f0b341a31224bf1be525b84b9500304 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:47.664103 18985 raft_consensus.cc:374] T 9f0b341a31224bf1be525b84b9500304 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:47.664207 18985 raft_consensus.cc:729] T 9f0b341a31224bf1be525b84b9500304 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:47.664469 18985 consensus_queue.cc:260] T 9f0b341a31224bf1be525b84b9500304 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:47.665086 18985 ts_tablet_manager.cc:1398] T 9f0b341a31224bf1be525b84b9500304 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240430 20:11:47.665429 18985 tablet_bootstrap.cc:492] T d7c1fc7e81364fb890bbc13d7fb6848b P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:47.667163 18985 tablet_bootstrap.cc:654] T d7c1fc7e81364fb890bbc13d7fb6848b P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:47.671665 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_68528229d5a94981a7fc0677d48003d1 [id=25ac2d71caf74e5ba14e3e84b84c9c60]
I20240430 20:11:47.672772 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_acddc4cbf6334546b5fb3e4752fef2a5 [id=274cb15fd751410e83327ac4e8dc4629]
W20240430 20:11:47.673432 18557 consensus_peers.cc:485] T acf3b6fead41483a82c43dc0286fab7c P 5e1c6397f09747f091534faf93542710 -> Peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Couldn't send request to peer 3c51c33d5616482cafb68e26688820f4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: acf3b6fead41483a82c43dc0286fab7c. This is attempt 1: this message will repeat every 5th retry.
I20240430 20:11:47.679112 19489 log.cc:1196] T 76d6e9d547ca4a848eae852944b1d117 P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/76d6e9d547ca4a848eae852944b1d117
I20240430 20:11:47.679571 19489 ts_tablet_manager.cc:1909] T 76d6e9d547ca4a848eae852944b1d117 P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:47.680424 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 391) took 2746 ms. Trace:
I20240430 20:11:47.680529 19489 rpcz_store.cc:276] 0430 20:11:44.934177 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:44.934235 (+ 58us) service_pool.cc:225] Handling call
0430 20:11:47.556912 (+2622677us) ts_tablet_manager.cc:1120] Deleting tablet 76d6e9d547ca4a848eae852944b1d117
0430 20:11:47.556925 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:47.561548 (+ 4623us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:47.660692 (+ 99144us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:47.680398 (+ 19706us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2622510}
W20240430 20:11:47.680795 18697 connection.cc:441] server connection from 127.0.0.1:59830 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 391) could send its response
I20240430 20:11:47.680882 19489 tablet_replica.cc:330] T c6378f1495144c458af49b3c19a2b67c P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:47.681056 19489 raft_consensus.cc:2229] T c6378f1495144c458af49b3c19a2b67c P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Raft consensus shutting down.
I20240430 20:11:47.681282 19489 pending_rounds.cc:62] T c6378f1495144c458af49b3c19a2b67c P 3c51c33d5616482cafb68e26688820f4: Trying to abort 1 pending ops.
I20240430 20:11:47.681339 19489 pending_rounds.cc:68] T c6378f1495144c458af49b3c19a2b67c P 3c51c33d5616482cafb68e26688820f4: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7022624385851461632 op_type: NO_OP noop_request { }
I20240430 20:11:47.681463 19489 raft_consensus.cc:2866] T c6378f1495144c458af49b3c19a2b67c P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240430 20:11:47.681551 19489 raft_consensus.cc:2258] T c6378f1495144c458af49b3c19a2b67c P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:47.652972 18988 ts_tablet_manager.cc:1367] T 130007acc8ba4e709c3eb85fabe75903 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.048s user 0.003s sys 0.000s
I20240430 20:11:47.682444 18988 raft_consensus.cc:348] T 130007acc8ba4e709c3eb85fabe75903 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:47.682665 18988 raft_consensus.cc:374] T 130007acc8ba4e709c3eb85fabe75903 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:47.682714 18988 raft_consensus.cc:729] T 130007acc8ba4e709c3eb85fabe75903 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:47.682894 18988 consensus_queue.cc:260] T 130007acc8ba4e709c3eb85fabe75903 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:47.683408 18988 ts_tablet_manager.cc:1398] T 130007acc8ba4e709c3eb85fabe75903 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240430 20:11:47.683729 18988 tablet_bootstrap.cc:492] T 885d27d4f8df48dd996d5b309b783f3b P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:47.685307 18988 tablet_bootstrap.cc:654] T 885d27d4f8df48dd996d5b309b783f3b P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:47.688392 18985 tablet_bootstrap.cc:492] T d7c1fc7e81364fb890bbc13d7fb6848b P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:47.688555 18985 ts_tablet_manager.cc:1367] T d7c1fc7e81364fb890bbc13d7fb6848b P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.023s user 0.005s sys 0.000s
I20240430 20:11:47.689445 18985 raft_consensus.cc:348] T d7c1fc7e81364fb890bbc13d7fb6848b P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:47.689698 18985 raft_consensus.cc:374] T d7c1fc7e81364fb890bbc13d7fb6848b P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:47.689766 18985 raft_consensus.cc:729] T d7c1fc7e81364fb890bbc13d7fb6848b P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:47.689992 18985 consensus_queue.cc:260] T d7c1fc7e81364fb890bbc13d7fb6848b P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:47.690572 18985 ts_tablet_manager.cc:1398] T d7c1fc7e81364fb890bbc13d7fb6848b P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240430 20:11:47.691107 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_8d32c8a8f05e4b599dd6a713133ed82e [id=295c03e97c87495f8d71b255b2ec9065]
I20240430 20:11:47.691766 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_b9983e101a6547649eb44ee04b5e1147 [id=29769be2b9ec4ecba9de5c0a65692f85]
I20240430 20:11:47.692183 20630 raft_consensus.cc:482] T 9f0b341a31224bf1be525b84b9500304 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:47.692312 20630 raft_consensus.cc:504] T 9f0b341a31224bf1be525b84b9500304 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:47.692821 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_fd33fc89e9d84f5c87a6572acdfe2c94 [id=2ab17149d0444085b85e5f0959ebb2ef]
I20240430 20:11:47.692906 20630 leader_election.cc:288] T 9f0b341a31224bf1be525b84b9500304 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:47.693967 18756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f0b341a31224bf1be525b84b9500304" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
W20240430 20:11:47.695143 18417 leader_election.cc:341] T 9f0b341a31224bf1be525b84b9500304 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:47.696905 18608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f0b341a31224bf1be525b84b9500304" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
W20240430 20:11:47.697481 18415 leader_election.cc:341] T 9f0b341a31224bf1be525b84b9500304 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:47.697598 18415 leader_election.cc:302] T 9f0b341a31224bf1be525b84b9500304 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a; no voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710
I20240430 20:11:47.698088 20631 raft_consensus.cc:2726] T 9f0b341a31224bf1be525b84b9500304 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:47.703768 20488 raft_consensus.cc:482] T e3e74e76ab2b4fe492ffcc1838656fd1 P 5e1c6397f09747f091534faf93542710 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 2efb542aa55e439b99ccb98d1c5c884a)
I20240430 20:11:47.703929 20488 raft_consensus.cc:504] T e3e74e76ab2b4fe492ffcc1838656fd1 P 5e1c6397f09747f091534faf93542710 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:47.704567 20488 leader_election.cc:288] T e3e74e76ab2b4fe492ffcc1838656fd1 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
W20240430 20:11:47.705835 18557 leader_election.cc:341] T e3e74e76ab2b4fe492ffcc1838656fd1 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 4 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Not found: Tablet not found: e3e74e76ab2b4fe492ffcc1838656fd1
I20240430 20:11:47.707145 19471 ts_tablet_manager.cc:1888] T 7683caba80594627900a0185f772c131 P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:47.707340 19471 log.cc:1196] T 7683caba80594627900a0185f772c131 P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/7683caba80594627900a0185f772c131
I20240430 20:11:47.707798 19471 ts_tablet_manager.cc:1909] T 7683caba80594627900a0185f772c131 P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:47.708707 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 305) took 4946 ms. Trace:
I20240430 20:11:47.708851 19471 rpcz_store.cc:276] 0430 20:11:42.762414 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:42.764045 (+ 1631us) service_pool.cc:225] Handling call
0430 20:11:47.573823 (+4809778us) ts_tablet_manager.cc:1120] Deleting tablet 7683caba80594627900a0185f772c131
0430 20:11:47.573834 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:47.611867 (+ 38033us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:47.707015 (+ 95148us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:47.708685 (+ 1670us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4809601}
W20240430 20:11:47.709213 18558 connection.cc:441] server connection from 127.0.0.1:40218 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 305) could send its response
I20240430 20:11:47.709313 18985 tablet_bootstrap.cc:492] T c091701480c4412faabad3d071d85d03 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:47.709650 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 306) took 4899 ms. Trace:
I20240430 20:11:47.709791 19471 rpcz_store.cc:276] 0430 20:11:42.810357 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:42.810809 (+ 452us) service_pool.cc:225] Handling call
0430 20:11:47.709511 (+4898702us) ts_tablet_manager.cc:1120] Deleting tablet 2797eed593fb4b06a3c4d6a28e8075a2
0430 20:11:47.709524 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:47.709631 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4898482}
W20240430 20:11:47.710124 18558 connection.cc:441] server connection from 127.0.0.1:40218 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 306) could send its response
W20240430 20:11:47.710340 18558 leader_election.cc:341] T e3e74e76ab2b4fe492ffcc1838656fd1 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 4 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Not found: Tablet not found: e3e74e76ab2b4fe492ffcc1838656fd1
I20240430 20:11:47.710446 18558 leader_election.cc:302] T e3e74e76ab2b4fe492ffcc1838656fd1 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:47.710943 20488 raft_consensus.cc:2726] T e3e74e76ab2b4fe492ffcc1838656fd1 P 5e1c6397f09747f091534faf93542710 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240430 20:11:47.711272 19471 tablet_replica.cc:330] T a05e32a33aa7488eb287cc6314c09c44 P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:47.711524 19471 raft_consensus.cc:2229] T a05e32a33aa7488eb287cc6314c09c44 P 5e1c6397f09747f091534faf93542710 [term 2 LEADER]: Raft consensus shutting down.
I20240430 20:11:47.711939 18985 tablet_bootstrap.cc:654] T c091701480c4412faabad3d071d85d03 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:47.715595 18988 tablet_bootstrap.cc:492] T 885d27d4f8df48dd996d5b309b783f3b P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:47.715808 18988 ts_tablet_manager.cc:1367] T 885d27d4f8df48dd996d5b309b783f3b P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.032s user 0.003s sys 0.000s
I20240430 20:11:47.716537 18988 raft_consensus.cc:348] T 885d27d4f8df48dd996d5b309b783f3b P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:47.723641 18985 tablet_bootstrap.cc:492] T c091701480c4412faabad3d071d85d03 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:47.723798 18985 ts_tablet_manager.cc:1367] T c091701480c4412faabad3d071d85d03 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.004s
I20240430 20:11:47.724761 19471 raft_consensus.cc:2258] T a05e32a33aa7488eb287cc6314c09c44 P 5e1c6397f09747f091534faf93542710 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:47.724720 18985 raft_consensus.cc:348] T c091701480c4412faabad3d071d85d03 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:47.723342 19489 ts_tablet_manager.cc:1875] T c6378f1495144c458af49b3c19a2b67c P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:47.725029 18985 raft_consensus.cc:374] T c091701480c4412faabad3d071d85d03 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:47.725095 18985 raft_consensus.cc:729] T c091701480c4412faabad3d071d85d03 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:47.725392 18985 consensus_queue.cc:260] T c091701480c4412faabad3d071d85d03 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:47.724975 18988 raft_consensus.cc:374] T 885d27d4f8df48dd996d5b309b783f3b P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:47.727237 18985 ts_tablet_manager.cc:1398] T c091701480c4412faabad3d071d85d03 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240430 20:11:47.724406 18983 tablet_bootstrap.cc:492] T 885d27d4f8df48dd996d5b309b783f3b P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:47.728464 18983 ts_tablet_manager.cc:1367] T 885d27d4f8df48dd996d5b309b783f3b P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.163s user 0.002s sys 0.006s
I20240430 20:11:47.729318 18983 raft_consensus.cc:348] T 885d27d4f8df48dd996d5b309b783f3b P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:47.729615 18983 raft_consensus.cc:374] T 885d27d4f8df48dd996d5b309b783f3b P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:47.729715 18983 raft_consensus.cc:729] T 885d27d4f8df48dd996d5b309b783f3b P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:47.730023 18983 consensus_queue.cc:260] T 885d27d4f8df48dd996d5b309b783f3b P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:47.731209 18985 tablet_bootstrap.cc:492] T 3e5225f1f5f544ecbf8dfa470af3daca P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:47.732870 18985 tablet_bootstrap.cc:654] T 3e5225f1f5f544ecbf8dfa470af3daca P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:47.735394 18985 tablet_bootstrap.cc:492] T 3e5225f1f5f544ecbf8dfa470af3daca P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:47.736423 18983 ts_tablet_manager.cc:1398] T 885d27d4f8df48dd996d5b309b783f3b P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20240430 20:11:47.736969 18983 tablet_bootstrap.cc:492] T 488d0e609b224674932f890939664df2 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:47.738807 18983 tablet_bootstrap.cc:654] T 488d0e609b224674932f890939664df2 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:47.744558 18985 ts_tablet_manager.cc:1367] T 3e5225f1f5f544ecbf8dfa470af3daca P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.002s
I20240430 20:11:47.745465 18985 raft_consensus.cc:348] T 3e5225f1f5f544ecbf8dfa470af3daca P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:47.745718 18985 raft_consensus.cc:374] T 3e5225f1f5f544ecbf8dfa470af3daca P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:47.745805 18985 raft_consensus.cc:729] T 3e5225f1f5f544ecbf8dfa470af3daca P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:47.746043 18985 consensus_queue.cc:260] T 3e5225f1f5f544ecbf8dfa470af3daca P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:47.746218 20630 raft_consensus.cc:482] T e4cffa295d8c401aa8415ce7e8f56526 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:47.746330 20630 raft_consensus.cc:504] T e4cffa295d8c401aa8415ce7e8f56526 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:47.746876 20630 leader_election.cc:288] T e4cffa295d8c401aa8415ce7e8f56526 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:47.747804 18985 ts_tablet_manager.cc:1398] T 3e5225f1f5f544ecbf8dfa470af3daca P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240430 20:11:47.748272 18985 tablet_bootstrap.cc:492] T 12795670f21447a8964fe1efc6e92398 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:47.749356 18608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4cffa295d8c401aa8415ce7e8f56526" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
W20240430 20:11:47.749994 18415 leader_election.cc:341] T e4cffa295d8c401aa8415ce7e8f56526 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:47.750136 18985 tablet_bootstrap.cc:654] T 12795670f21447a8964fe1efc6e92398 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:47.728134 20693 consensus_queue.cc:1035] T 22fef8bb6b034db098c3372ae992f7a7 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:47.751160 18988 raft_consensus.cc:729] T 885d27d4f8df48dd996d5b309b783f3b P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:47.751433 18988 consensus_queue.cc:260] T 885d27d4f8df48dd996d5b309b783f3b P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:47.752182 18988 ts_tablet_manager.cc:1398] T 885d27d4f8df48dd996d5b309b783f3b P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.036s user 0.002s sys 0.000s
I20240430 20:11:47.747426 18756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4cffa295d8c401aa8415ce7e8f56526" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
W20240430 20:11:47.754096 18417 leader_election.cc:341] T e4cffa295d8c401aa8415ce7e8f56526 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:47.754215 18417 leader_election.cc:302] T e4cffa295d8c401aa8415ce7e8f56526 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a; no voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710
I20240430 20:11:47.754659 20631 raft_consensus.cc:2726] T e4cffa295d8c401aa8415ce7e8f56526 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:47.758770 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_e3160f5ced1b4718b479401ca4df675c [id=2d44ec94131148048021b2ee6366ba30]
I20240430 20:11:47.759717 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_f30868d759f64cceb682278f0dbb51ae [id=3291bc6bb4f146ee8f90d89029a70aef]
I20240430 20:11:47.760351 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_e3165b7ce52a44bf9d2bed43f9c5d6c7 [id=3546514024eb4b8c9ad58c7af8d63aff]
I20240430 20:11:47.760864 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_6b67c0c03ceb4b9491ad5828022b357f [id=368fcf43e863494ca260862d12331c0d]
I20240430 20:11:47.767287 18985 tablet_bootstrap.cc:492] T 12795670f21447a8964fe1efc6e92398 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:47.769192 20198 consensus_queue.cc:1035] T 22fef8bb6b034db098c3372ae992f7a7 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:47.771378 19471 ts_tablet_manager.cc:1875] T a05e32a33aa7488eb287cc6314c09c44 P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:47.767633 18985 ts_tablet_manager.cc:1367] T 12795670f21447a8964fe1efc6e92398 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.005s
I20240430 20:11:47.774680 18985 raft_consensus.cc:348] T 12795670f21447a8964fe1efc6e92398 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:47.775879 18985 raft_consensus.cc:374] T 12795670f21447a8964fe1efc6e92398 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:47.776125 18985 raft_consensus.cc:729] T 12795670f21447a8964fe1efc6e92398 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:47.776365 20631 raft_consensus.cc:482] T 78ba48dadef548708153878810cc46f8 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:47.776583 20631 raft_consensus.cc:504] T 78ba48dadef548708153878810cc46f8 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:47.777367 20631 leader_election.cc:288] T 78ba48dadef548708153878810cc46f8 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:47.777932 18608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ba48dadef548708153878810cc46f8" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:47.778170 18608 raft_consensus.cc:2454] T 78ba48dadef548708153878810cc46f8 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 0.
I20240430 20:11:47.778429 18985 consensus_queue.cc:260] T 12795670f21447a8964fe1efc6e92398 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:47.778635 18415 leader_election.cc:302] T 78ba48dadef548708153878810cc46f8 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:47.779230 20631 raft_consensus.cc:2781] T 78ba48dadef548708153878810cc46f8 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:47.779299 20631 raft_consensus.cc:482] T 78ba48dadef548708153878810cc46f8 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:47.779354 20631 raft_consensus.cc:3037] T 78ba48dadef548708153878810cc46f8 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:47.780320 18988 tablet_bootstrap.cc:492] T fdf53061efd74fa29a9d8208d1fd210e P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:47.781977 20631 raft_consensus.cc:504] T 78ba48dadef548708153878810cc46f8 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:47.782961 18758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ba48dadef548708153878810cc46f8" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
W20240430 20:11:47.784499 18417 leader_election.cc:341] T 78ba48dadef548708153878810cc46f8 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:47.787726 20488 raft_consensus.cc:482] T 246fb63a5e9842df8263800e28a5935c P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:47.787883 20488 raft_consensus.cc:504] T 246fb63a5e9842df8263800e28a5935c P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:47.788538 20488 leader_election.cc:288] T 246fb63a5e9842df8263800e28a5935c P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:47.788967 18758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "246fb63a5e9842df8263800e28a5935c" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:47.789606 18473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "246fb63a5e9842df8263800e28a5935c" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:47.789866 18473 raft_consensus.cc:2379] T 246fb63a5e9842df8263800e28a5935c P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e1c6397f09747f091534faf93542710 in current term 1: Already voted for candidate 3c51c33d5616482cafb68e26688820f4 in this term.
I20240430 20:11:47.790975 18557 leader_election.cc:302] T 246fb63a5e9842df8263800e28a5935c P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:47.791499 20488 raft_consensus.cc:3037] T 246fb63a5e9842df8263800e28a5935c P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:47.795300 18988 tablet_bootstrap.cc:654] T fdf53061efd74fa29a9d8208d1fd210e P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:47.798130 18983 tablet_bootstrap.cc:492] T 488d0e609b224674932f890939664df2 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:47.798475 18983 ts_tablet_manager.cc:1367] T 488d0e609b224674932f890939664df2 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.062s user 0.003s sys 0.000s
I20240430 20:11:47.799362 18758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ba48dadef548708153878810cc46f8" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:47.799706 18983 raft_consensus.cc:348] T 488d0e609b224674932f890939664df2 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:47.800026 18983 raft_consensus.cc:374] T 488d0e609b224674932f890939664df2 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:47.800153 18983 raft_consensus.cc:729] T 488d0e609b224674932f890939664df2 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
W20240430 20:11:47.800148 18417 leader_election.cc:341] T 78ba48dadef548708153878810cc46f8 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:47.800534 18983 consensus_queue.cc:260] T 488d0e609b224674932f890939664df2 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:47.801227 20631 leader_election.cc:288] T 78ba48dadef548708153878810cc46f8 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:47.801246 18608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ba48dadef548708153878810cc46f8" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:47.794538 20488 raft_consensus.cc:2726] T 246fb63a5e9842df8263800e28a5935c P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:47.801429 18608 raft_consensus.cc:3037] T 78ba48dadef548708153878810cc46f8 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:47.803575 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_78987875650940c293247bc43db345ab [id=36f478a20a1f4a53a52fc29ade62b7f6]
I20240430 20:11:47.804198 18608 raft_consensus.cc:2454] T 78ba48dadef548708153878810cc46f8 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 1.
I20240430 20:11:47.805568 20198 raft_consensus.cc:482] T 885d27d4f8df48dd996d5b309b783f3b P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:47.805707 20198 raft_consensus.cc:504] T 885d27d4f8df48dd996d5b309b783f3b P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:47.806445 20198 leader_election.cc:288] T 885d27d4f8df48dd996d5b309b783f3b P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:47.807495 18473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "885d27d4f8df48dd996d5b309b783f3b" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:47.807720 18473 raft_consensus.cc:2454] T 885d27d4f8df48dd996d5b309b783f3b P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:47.808225 18699 leader_election.cc:302] T 885d27d4f8df48dd996d5b309b783f3b P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters:
I20240430 20:11:47.808488 20198 raft_consensus.cc:2781] T 885d27d4f8df48dd996d5b309b783f3b P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:47.808552 20198 raft_consensus.cc:482] T 885d27d4f8df48dd996d5b309b783f3b P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:47.808606 20198 raft_consensus.cc:3037] T 885d27d4f8df48dd996d5b309b783f3b P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:47.809615 18415 leader_election.cc:302] T 78ba48dadef548708153878810cc46f8 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters: 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:47.811220 20198 raft_consensus.cc:504] T 885d27d4f8df48dd996d5b309b783f3b P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:47.812458 18619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "885d27d4f8df48dd996d5b309b783f3b" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:47.812651 18619 raft_consensus.cc:3037] T 885d27d4f8df48dd996d5b309b783f3b P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:47.812877 18608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "885d27d4f8df48dd996d5b309b783f3b" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:47.815291 18619 raft_consensus.cc:2454] T 885d27d4f8df48dd996d5b309b783f3b P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:47.815913 18473 raft_consensus.cc:3037] T f5eaf2413e1d4e69b56fae61f768f0ce P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:47.816618 19489 ts_tablet_manager.cc:1888] T c6378f1495144c458af49b3c19a2b67c P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:47.816797 19489 log.cc:1196] T c6378f1495144c458af49b3c19a2b67c P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/c6378f1495144c458af49b3c19a2b67c
I20240430 20:11:47.817344 19489 ts_tablet_manager.cc:1909] T c6378f1495144c458af49b3c19a2b67c P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:47.819736 18696 leader_election.cc:302] T 885d27d4f8df48dd996d5b309b783f3b P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:47.820147 20693 raft_consensus.cc:2781] T 885d27d4f8df48dd996d5b309b783f3b P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:47.820279 20693 raft_consensus.cc:686] T 885d27d4f8df48dd996d5b309b783f3b P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Becoming Leader. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Running, Role: LEADER
I20240430 20:11:47.820529 20693 consensus_queue.cc:237] T 885d27d4f8df48dd996d5b309b783f3b P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:47.820484 18983 ts_tablet_manager.cc:1398] T 488d0e609b224674932f890939664df2 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.022s user 0.002s sys 0.001s
I20240430 20:11:47.821409 20629 raft_consensus.cc:2781] T 78ba48dadef548708153878810cc46f8 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:47.821548 20629 raft_consensus.cc:686] T 78ba48dadef548708153878810cc46f8 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Becoming Leader. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Running, Role: LEADER
I20240430 20:11:47.821767 20629 consensus_queue.cc:237] T 78ba48dadef548708153878810cc46f8 P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:47.827009 20488 consensus_queue.cc:1035] T f5eaf2413e1d4e69b56fae61f768f0ce P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:47.827970 20384 raft_consensus.cc:482] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 5e1c6397f09747f091534faf93542710 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 2efb542aa55e439b99ccb98d1c5c884a)
I20240430 20:11:47.828153 20384 raft_consensus.cc:504] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 5e1c6397f09747f091534faf93542710 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:47.828938 20384 leader_election.cc:288] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:47.830394 18983 tablet_bootstrap.cc:492] T 50e1c55a1a0a484984ad5cbb78eec2ed P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
W20240430 20:11:47.830787 18558 leader_election.cc:341] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Not found: Tablet not found: dd6083f12c7e4d1e8a2006b3ddf9aa3a
W20240430 20:11:47.834213 18557 leader_election.cc:341] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Not found: Tablet not found: dd6083f12c7e4d1e8a2006b3ddf9aa3a
I20240430 20:11:47.834354 18557 leader_election.cc:302] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:47.835074 20488 raft_consensus.cc:2726] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 5e1c6397f09747f091534faf93542710 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240430 20:11:47.837616 18988 tablet_bootstrap.cc:492] T fdf53061efd74fa29a9d8208d1fd210e P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:47.837713 20198 leader_election.cc:288] T 885d27d4f8df48dd996d5b309b783f3b P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Requested vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:47.838212 18988 ts_tablet_manager.cc:1367] T fdf53061efd74fa29a9d8208d1fd210e P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.058s user 0.004s sys 0.000s
I20240430 20:11:47.838387 18475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "885d27d4f8df48dd996d5b309b783f3b" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:47.838644 18475 raft_consensus.cc:3037] T 885d27d4f8df48dd996d5b309b783f3b P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:47.839498 18988 raft_consensus.cc:348] T fdf53061efd74fa29a9d8208d1fd210e P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:47.839876 18988 raft_consensus.cc:374] T fdf53061efd74fa29a9d8208d1fd210e P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:47.840020 18988 raft_consensus.cc:729] T fdf53061efd74fa29a9d8208d1fd210e P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:47.840427 18988 consensus_queue.cc:260] T fdf53061efd74fa29a9d8208d1fd210e P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:47.841854 18475 raft_consensus.cc:2454] T 885d27d4f8df48dd996d5b309b783f3b P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:47.843227 18988 ts_tablet_manager.cc:1398] T fdf53061efd74fa29a9d8208d1fd210e P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240430 20:11:47.847033 18988 tablet_bootstrap.cc:492] T 78ba48dadef548708153878810cc46f8 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:47.847707 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 401) took 2037 ms. Trace:
I20240430 20:11:47.849933 18983 tablet_bootstrap.cc:654] T 50e1c55a1a0a484984ad5cbb78eec2ed P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:47.849022 19489 rpcz_store.cc:276] 0430 20:11:45.810045 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:45.810123 (+ 78us) service_pool.cc:225] Handling call
0430 20:11:47.680832 (+1870709us) ts_tablet_manager.cc:1120] Deleting tablet c6378f1495144c458af49b3c19a2b67c
0430 20:11:47.680844 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:47.765716 (+ 84872us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:47.816433 (+ 50717us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:47.847686 (+ 31253us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1847831}
W20240430 20:11:47.851632 18697 connection.cc:441] server connection from 127.0.0.1:59830 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 401) could send its response
I20240430 20:11:47.853657 18985 ts_tablet_manager.cc:1398] T 12795670f21447a8964fe1efc6e92398 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.080s user 0.001s sys 0.001s
I20240430 20:11:47.855209 18985 tablet_bootstrap.cc:492] T dad16ae02b8b4e29a724d8eec67bb4b8 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:47.854635 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 402) took 2028 ms. Trace:
I20240430 20:11:47.859565 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_673d1448679f4b15a5ee2a1369c8f8ea [id=3813b5e9778c4bd683a06598cef0d86f]
I20240430 20:11:47.855466 19489 rpcz_store.cc:276] 0430 20:11:45.826033 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:45.826675 (+ 642us) service_pool.cc:225] Handling call
0430 20:11:47.852998 (+2026323us) ts_tablet_manager.cc:1120] Deleting tablet df92eb85261d47d88cbf16d29d9ad30e
0430 20:11:47.853012 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:47.854229 (+ 1217us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2019597}
W20240430 20:11:47.860538 18697 connection.cc:441] server connection from 127.0.0.1:59830 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 402) could send its response
I20240430 20:11:47.860188 19471 ts_tablet_manager.cc:1888] T a05e32a33aa7488eb287cc6314c09c44 P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.3
I20240430 20:11:47.861709 19471 log.cc:1196] T a05e32a33aa7488eb287cc6314c09c44 P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/a05e32a33aa7488eb287cc6314c09c44
I20240430 20:11:47.863417 18985 tablet_bootstrap.cc:654] T dad16ae02b8b4e29a724d8eec67bb4b8 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:47.862907 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 404) took 2002 ms. Trace:
I20240430 20:11:47.864074 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_18385e32857b4c8bb1e56a85c590153a [id=38a7d7d4f9e64c07b784d9393192ac94]
I20240430 20:11:47.863586 19489 rpcz_store.cc:276] 0430 20:11:45.860244 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:45.860317 (+ 73us) service_pool.cc:225] Handling call
0430 20:11:47.861485 (+2001168us) ts_tablet_manager.cc:1120] Deleting tablet fcc16db4d6be42edb7bd1ce18fe0190d
0430 20:11:47.861500 (+ 15us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:47.862886 (+ 1386us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2000620}
W20240430 20:11:47.865662 18697 connection.cc:441] server connection from 127.0.0.1:59830 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 404) could send its response
I20240430 20:11:47.866127 19471 ts_tablet_manager.cc:1909] T a05e32a33aa7488eb287cc6314c09c44 P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:47.868049 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 403) took 2007 ms. Trace:
I20240430 20:11:47.869295 18988 tablet_bootstrap.cc:654] T 78ba48dadef548708153878810cc46f8 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:47.869088 19489 rpcz_store.cc:276] 0430 20:11:45.860134 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:45.860455 (+ 321us) service_pool.cc:225] Handling call
0430 20:11:47.866565 (+2006110us) ts_tablet_manager.cc:1120] Deleting tablet 47e98935eb21449abd3d4d24327778d5
0430 20:11:47.866579 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:47.868028 (+ 1449us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2005660}
W20240430 20:11:47.870831 18697 connection.cc:441] server connection from 127.0.0.1:59830 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 403) could send its response
I20240430 20:11:47.873087 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 405) took 2012 ms. Trace:
I20240430 20:11:47.874614 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_08dc70f541b842c78a3a946d2d02e207 [id=38f199a7538f4271b3a59fd93f29efa2]
I20240430 20:11:47.873853 19489 rpcz_store.cc:276] 0430 20:11:45.860358 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:45.860602 (+ 244us) service_pool.cc:225] Handling call
0430 20:11:47.872156 (+2011554us) ts_tablet_manager.cc:1120] Deleting tablet dbb427aefa6b47b0805d1751fea4efcb
0430 20:11:47.872396 (+ 240us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:47.873073 (+ 677us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7040,"tablet-delete.queue_time_us":2011076}
W20240430 20:11:47.875416 18697 connection.cc:441] server connection from 127.0.0.1:59830 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 405) could send its response
I20240430 20:11:47.875757 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 309) took 5000 ms. Trace:
I20240430 20:11:47.876945 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 406) took 1999 ms. Trace:
I20240430 20:11:47.878041 19489 rpcz_store.cc:276] 0430 20:11:45.877878 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:45.878173 (+ 295us) service_pool.cc:225] Handling call
0430 20:11:47.876305 (+1998132us) ts_tablet_manager.cc:1120] Deleting tablet 84568f90b9c54cf292099808d4cbf5dc
0430 20:11:47.876316 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:47.876627 (+ 311us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1997679}
W20240430 20:11:47.879206 18697 connection.cc:441] server connection from 127.0.0.1:59830 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 406) could send its response
I20240430 20:11:47.880561 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 408) took 2001 ms. Trace:
I20240430 20:11:47.877614 18983 tablet_bootstrap.cc:492] T 50e1c55a1a0a484984ad5cbb78eec2ed P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:47.882292 19471 rpcz_store.cc:276] 0430 20:11:42.875185 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:42.875239 (+ 54us) service_pool.cc:225] Handling call
0430 20:11:47.711221 (+4835982us) ts_tablet_manager.cc:1120] Deleting tablet a05e32a33aa7488eb287cc6314c09c44
0430 20:11:47.711232 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:47.805837 (+ 94605us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:47.857170 (+ 51333us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:47.875731 (+ 18561us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3419,"tablet-delete.queue_time_us":4835817}
I20240430 20:11:47.881054 19489 rpcz_store.cc:276] 0430 20:11:45.879253 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:45.879314 (+ 61us) service_pool.cc:225] Handling call
0430 20:11:47.879681 (+2000367us) ts_tablet_manager.cc:1120] Deleting tablet 1f4e860484784aa9a2d28af2e9fb611c
0430 20:11:47.879930 (+ 249us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:47.880542 (+ 612us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2000171}
W20240430 20:11:47.883183 18697 connection.cc:441] server connection from 127.0.0.1:59830 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 408) could send its response
I20240430 20:11:47.884462 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 407) took 2006 ms. Trace:
I20240430 20:11:47.885097 19489 rpcz_store.cc:276] 0430 20:11:45.878466 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:45.880268 (+ 1802us) service_pool.cc:225] Handling call
0430 20:11:47.883565 (+2003297us) ts_tablet_manager.cc:1120] Deleting tablet 6db55bb510b24b0cae897936211e15f0
0430 20:11:47.883757 (+ 192us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:47.884442 (+ 685us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2003122}
W20240430 20:11:47.886670 18697 connection.cc:441] server connection from 127.0.0.1:59830 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 407) could send its response
I20240430 20:11:47.891744 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 409) took 2010 ms. Trace:
I20240430 20:11:47.892496 19489 rpcz_store.cc:276] 0430 20:11:45.880885 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:45.880944 (+ 59us) service_pool.cc:225] Handling call
0430 20:11:47.890787 (+2009843us) ts_tablet_manager.cc:1120] Deleting tablet 558e71ec370d46ca8fbdcec5b57eb2ec
0430 20:11:47.890958 (+ 171us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:47.891564 (+ 606us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2006211}
W20240430 20:11:47.894088 18697 connection.cc:441] server connection from 127.0.0.1:59830 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 409) could send its response
I20240430 20:11:47.896693 18988 tablet_bootstrap.cc:492] T 78ba48dadef548708153878810cc46f8 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:47.896924 18988 ts_tablet_manager.cc:1367] T 78ba48dadef548708153878810cc46f8 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.051s user 0.003s sys 0.002s
I20240430 20:11:47.897660 18988 raft_consensus.cc:348] T 78ba48dadef548708153878810cc46f8 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:47.897943 18988 raft_consensus.cc:374] T 78ba48dadef548708153878810cc46f8 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:47.898088 18988 raft_consensus.cc:729] T 78ba48dadef548708153878810cc46f8 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:47.898517 18988 consensus_queue.cc:260] T 78ba48dadef548708153878810cc46f8 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:47.901813 18988 ts_tablet_manager.cc:1398] T 78ba48dadef548708153878810cc46f8 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240430 20:11:47.905084 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_47217f7ee5ef4a50a2e9f146412b198a [id=3b5dcd10bb0d4f2493dcfaae7a2b9d19]
I20240430 20:11:47.905889 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_088b70cec77744aaacf26aa6d4adebbd [id=3e0705b442cc45109c5750e006e0ff04]
I20240430 20:11:47.906491 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_cb2a4b9ceaf4431facfff2c9bcf94c5d [id=3f8690a437ae49a3beb009fa7899d5d0]
I20240430 20:11:47.907691 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_404b186c544e4b30bb36cb77d1a14120 [id=4330626e342a48699ea939add5c91b11]
I20240430 20:11:47.882532 18983 ts_tablet_manager.cc:1367] T 50e1c55a1a0a484984ad5cbb78eec2ed P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.052s user 0.002s sys 0.002s
W20240430 20:11:47.907864 18558 connection.cc:441] server connection from 127.0.0.1:40218 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 309) could send its response
I20240430 20:11:47.908172 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 312) took 5032 ms. Trace:
I20240430 20:11:47.908731 18983 raft_consensus.cc:348] T 50e1c55a1a0a484984ad5cbb78eec2ed P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:47.909011 18983 raft_consensus.cc:374] T 50e1c55a1a0a484984ad5cbb78eec2ed P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:47.909140 18983 raft_consensus.cc:729] T 50e1c55a1a0a484984ad5cbb78eec2ed P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:47.909166 20488 consensus_queue.cc:1035] T 0a16d98f700c46e29f6dd8568752ef29 P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:47.909479 18983 consensus_queue.cc:260] T 50e1c55a1a0a484984ad5cbb78eec2ed P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:47.910032 18983 ts_tablet_manager.cc:1398] T 50e1c55a1a0a484984ad5cbb78eec2ed P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240430 20:11:47.910485 18983 tablet_bootstrap.cc:492] T df92eb85261d47d88cbf16d29d9ad30e P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:47.912039 18983 tablet_bootstrap.cc:654] T df92eb85261d47d88cbf16d29d9ad30e P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:47.916922 20545 consensus_queue.cc:1035] T 8e33df70a1504dbd852fe0b3c3f61622 P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:47.904440 20384 consensus_queue.cc:1035] T f5eaf2413e1d4e69b56fae61f768f0ce P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:47.919121 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 410) took 2014 ms. Trace:
I20240430 20:11:47.919348 19489 rpcz_store.cc:276] 0430 20:11:45.881488 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:45.882021 (+ 533us) service_pool.cc:225] Handling call
0430 20:11:47.894814 (+2012793us) ts_tablet_manager.cc:1120] Deleting tablet 90a694bd5db649afa50f3c4e752939c1
0430 20:11:47.894829 (+ 15us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:47.895447 (+ 618us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2012483}
W20240430 20:11:47.920038 18697 connection.cc:441] server connection from 127.0.0.1:59830 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 410) could send its response
I20240430 20:11:47.920935 18988 tablet_bootstrap.cc:492] T 6d1d2193fd424c91b143f5256df5fcd0 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:47.923907 18988 tablet_bootstrap.cc:654] T 6d1d2193fd424c91b143f5256df5fcd0 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:47.932356 20198 consensus_queue.cc:1035] T c6e3d09402574f1e8fe9e4160c70a3ee P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:47.941872 19489 tablet_replica.cc:330] T 04b3dff86d5c48a09d03da58bd48f360 P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:47.942230 19489 raft_consensus.cc:2229] T 04b3dff86d5c48a09d03da58bd48f360 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:47.942567 19489 raft_consensus.cc:2258] T 04b3dff86d5c48a09d03da58bd48f360 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:47.943948 19489 ts_tablet_manager.cc:1875] T 04b3dff86d5c48a09d03da58bd48f360 P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:47.945041 20489 consensus_queue.cc:1035] T 0a16d98f700c46e29f6dd8568752ef29 P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:47.946575 18983 tablet_bootstrap.cc:492] T df92eb85261d47d88cbf16d29d9ad30e P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:47.947144 18983 ts_tablet_manager.cc:1367] T df92eb85261d47d88cbf16d29d9ad30e P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.037s user 0.003s sys 0.000s
I20240430 20:11:47.947825 18983 raft_consensus.cc:348] T df92eb85261d47d88cbf16d29d9ad30e P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:47.948064 18983 raft_consensus.cc:374] T df92eb85261d47d88cbf16d29d9ad30e P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:47.948158 18983 raft_consensus.cc:729] T df92eb85261d47d88cbf16d29d9ad30e P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:47.948527 18983 consensus_queue.cc:260] T df92eb85261d47d88cbf16d29d9ad30e P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:47.949550 20198 consensus_queue.cc:1035] T a482f4622f774d60b4efd6827b8be7d7 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:47.950524 18983 ts_tablet_manager.cc:1398] T df92eb85261d47d88cbf16d29d9ad30e P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240430 20:11:47.951239 18983 tablet_bootstrap.cc:492] T b285e3eb16d148779c74e057458d8461 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
W20240430 20:11:47.953894 20828 log.cc:924] Time spent T f5eaf2413e1d4e69b56fae61f768f0ce P 5e1c6397f09747f091534faf93542710: Append to log took a long time: real 0.056s user 0.000s sys 0.000s
I20240430 20:11:47.960048 20587 raft_consensus.cc:482] T 53da4b3855344b6d92211af767195dc8 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:47.960271 20587 raft_consensus.cc:504] T 53da4b3855344b6d92211af767195dc8 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:47.960786 20695 consensus_queue.cc:1035] T a482f4622f774d60b4efd6827b8be7d7 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:47.908288 19471 rpcz_store.cc:276] 0430 20:11:42.875992 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:42.876149 (+ 157us) service_pool.cc:225] Handling call
0430 20:11:47.908076 (+5031927us) ts_tablet_manager.cc:1120] Deleting tablet 2f61e3f88a6448dfa323204bd8d14cc5
0430 20:11:47.908085 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:47.908156 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5031765}
W20240430 20:11:47.961211 18558 connection.cc:441] server connection from 127.0.0.1:40218 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 312) could send its response
I20240430 20:11:47.961647 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 311) took 5085 ms. Trace:
I20240430 20:11:47.961711 20587 leader_election.cc:288] T 53da4b3855344b6d92211af767195dc8 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:47.961771 19471 rpcz_store.cc:276] 0430 20:11:42.875874 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:42.876180 (+ 306us) service_pool.cc:225] Handling call
0430 20:11:47.961521 (+5085341us) ts_tablet_manager.cc:1120] Deleting tablet 7542c7fe6cad452ab718f9ef54dbbbbe
0430 20:11:47.961536 (+ 15us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:47.961629 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5085145}
W20240430 20:11:47.962064 18558 connection.cc:441] server connection from 127.0.0.1:40218 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 311) could send its response
I20240430 20:11:47.963450 18620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53da4b3855344b6d92211af767195dc8" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:47.963675 18620 raft_consensus.cc:2454] T 53da4b3855344b6d92211af767195dc8 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:47.964560 18696 leader_election.cc:302] T 53da4b3855344b6d92211af767195dc8 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:47.965164 20693 raft_consensus.cc:2781] T 53da4b3855344b6d92211af767195dc8 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:47.965240 20693 raft_consensus.cc:482] T 53da4b3855344b6d92211af767195dc8 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:47.965297 20693 raft_consensus.cc:3037] T 53da4b3855344b6d92211af767195dc8 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:47.968056 20693 raft_consensus.cc:504] T 53da4b3855344b6d92211af767195dc8 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:47.968942 20693 leader_election.cc:288] T 53da4b3855344b6d92211af767195dc8 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Requested vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:47.969605 18620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53da4b3855344b6d92211af767195dc8" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:47.969826 18620 raft_consensus.cc:3037] T 53da4b3855344b6d92211af767195dc8 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:47.973084 18620 raft_consensus.cc:2454] T 53da4b3855344b6d92211af767195dc8 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:47.973762 18696 leader_election.cc:302] T 53da4b3855344b6d92211af767195dc8 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:47.976068 18983 tablet_bootstrap.cc:654] T b285e3eb16d148779c74e057458d8461 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:47.976168 20693 raft_consensus.cc:2781] T 53da4b3855344b6d92211af767195dc8 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:47.976311 20693 raft_consensus.cc:686] T 53da4b3855344b6d92211af767195dc8 P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Becoming Leader. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Running, Role: LEADER
I20240430 20:11:47.976573 20693 consensus_queue.cc:237] T 53da4b3855344b6d92211af767195dc8 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:47.980830 18473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53da4b3855344b6d92211af767195dc8" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:47.981223 18473 raft_consensus.cc:2454] T 53da4b3855344b6d92211af767195dc8 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:47.982378 18478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53da4b3855344b6d92211af767195dc8" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:47.982728 18478 raft_consensus.cc:3037] T 53da4b3855344b6d92211af767195dc8 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:47.986310 18983 tablet_bootstrap.cc:492] T b285e3eb16d148779c74e057458d8461 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:47.987377 20545 consensus_queue.cc:1035] T 04b3dff86d5c48a09d03da58bd48f360 P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:47.988251 18983 ts_tablet_manager.cc:1367] T b285e3eb16d148779c74e057458d8461 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.037s user 0.004s sys 0.000s
I20240430 20:11:47.997917 18983 raft_consensus.cc:348] T b285e3eb16d148779c74e057458d8461 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:48.000798 18983 raft_consensus.cc:374] T b285e3eb16d148779c74e057458d8461 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:48.002076 18983 raft_consensus.cc:729] T b285e3eb16d148779c74e057458d8461 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:48.007372 20587 consensus_queue.cc:1035] T c6e3d09402574f1e8fe9e4160c70a3ee P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:48.006471 18983 consensus_queue.cc:260] T b285e3eb16d148779c74e057458d8461 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:48.009999 19489 ts_tablet_manager.cc:1888] T 04b3dff86d5c48a09d03da58bd48f360 P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:48.015893 18983 ts_tablet_manager.cc:1398] T b285e3eb16d148779c74e057458d8461 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.027s user 0.003s sys 0.000s
I20240430 20:11:48.018689 18983 tablet_bootstrap.cc:492] T 6db55bb510b24b0cae897936211e15f0 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:48.019444 19489 log.cc:1196] T 04b3dff86d5c48a09d03da58bd48f360 P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/04b3dff86d5c48a09d03da58bd48f360
I20240430 20:11:48.024602 18983 tablet_bootstrap.cc:654] T 6db55bb510b24b0cae897936211e15f0 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:48.030536 19489 ts_tablet_manager.cc:1909] T 04b3dff86d5c48a09d03da58bd48f360 P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:48.035861 19489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 411) took 2153 ms. Trace:
I20240430 20:11:48.036563 18985 tablet_bootstrap.cc:492] T dad16ae02b8b4e29a724d8eec67bb4b8 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:48.036777 18985 ts_tablet_manager.cc:1367] T dad16ae02b8b4e29a724d8eec67bb4b8 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.182s user 0.002s sys 0.002s
I20240430 20:11:48.037737 18985 raft_consensus.cc:348] T dad16ae02b8b4e29a724d8eec67bb4b8 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:48.037981 18985 raft_consensus.cc:374] T dad16ae02b8b4e29a724d8eec67bb4b8 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:48.038067 18985 raft_consensus.cc:729] T dad16ae02b8b4e29a724d8eec67bb4b8 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:48.038321 18985 consensus_queue.cc:260] T dad16ae02b8b4e29a724d8eec67bb4b8 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:48.039394 20587 consensus_queue.cc:1035] T 246fb63a5e9842df8263800e28a5935c P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:48.040370 18985 ts_tablet_manager.cc:1398] T dad16ae02b8b4e29a724d8eec67bb4b8 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240430 20:11:48.042323 18985 tablet_bootstrap.cc:492] T ed0c37a71ede4bfc9eb5ad1a4365f25e P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:48.046257 18478 raft_consensus.cc:2454] T 53da4b3855344b6d92211af767195dc8 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:48.052423 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 310) took 5176 ms. Trace:
I20240430 20:11:48.052592 19471 rpcz_store.cc:276] 0430 20:11:42.875754 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:42.876430 (+ 676us) service_pool.cc:225] Handling call
0430 20:11:48.052260 (+5175830us) ts_tablet_manager.cc:1120] Deleting tablet 5d09baed576e4136bf6d756ca39d08b2
0430 20:11:48.052278 (+ 18us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:48.052395 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5175679}
W20240430 20:11:48.052856 18558 connection.cc:441] server connection from 127.0.0.1:40218 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 310) could send its response
I20240430 20:11:48.052904 20545 consensus_queue.cc:1035] T 47bf64ecc41b445da4ff697e141243dd P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:48.056213 20545 raft_consensus.cc:482] T c091701480c4412faabad3d071d85d03 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:48.056370 20545 raft_consensus.cc:504] T c091701480c4412faabad3d071d85d03 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:48.056994 20545 leader_election.cc:288] T c091701480c4412faabad3d071d85d03 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:48.057399 18756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c091701480c4412faabad3d071d85d03" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:48.057786 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 313) took 5181 ms. Trace:
W20240430 20:11:48.057998 18417 leader_election.cc:341] T c091701480c4412faabad3d071d85d03 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:48.058038 18617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c091701480c4412faabad3d071d85d03" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:48.058302 18617 raft_consensus.cc:2454] T c091701480c4412faabad3d071d85d03 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 0.
I20240430 20:11:48.057883 19471 rpcz_store.cc:276] 0430 20:11:42.876098 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:42.876650 (+ 552us) service_pool.cc:225] Handling call
0430 20:11:48.057673 (+5181023us) ts_tablet_manager.cc:1120] Deleting tablet 0a9066d895744c5bb7d9532b2940603c
0430 20:11:48.057684 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:48.057763 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5180896}
I20240430 20:11:48.058710 18415 leader_election.cc:302] T c091701480c4412faabad3d071d85d03 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters: 3c51c33d5616482cafb68e26688820f4
W20240430 20:11:48.058770 18558 connection.cc:441] server connection from 127.0.0.1:40218 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 313) could send its response
I20240430 20:11:48.059068 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 317) took 5097 ms. Trace:
I20240430 20:11:48.059139 19471 rpcz_store.cc:276] 0430 20:11:42.961795 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:42.962480 (+ 685us) service_pool.cc:225] Handling call
0430 20:11:48.058985 (+5096505us) ts_tablet_manager.cc:1120] Deleting tablet 73453a7fa30342e29db552b17b62f0c1
0430 20:11:48.058994 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:48.059055 (+ 61us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5096179}
I20240430 20:11:48.059204 20545 raft_consensus.cc:2781] T c091701480c4412faabad3d071d85d03 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:48.059309 20545 raft_consensus.cc:482] T c091701480c4412faabad3d071d85d03 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240430 20:11:48.059324 18558 connection.cc:441] server connection from 127.0.0.1:40218 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 317) could send its response
I20240430 20:11:48.059408 20545 raft_consensus.cc:3037] T c091701480c4412faabad3d071d85d03 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:48.059988 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 318) took 5096 ms. Trace:
I20240430 20:11:48.060099 19471 rpcz_store.cc:276] 0430 20:11:42.963958 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:42.964069 (+ 111us) service_pool.cc:225] Handling call
0430 20:11:48.059896 (+5095827us) ts_tablet_manager.cc:1120] Deleting tablet 87ac32a0ddb84676adc83038ef664591
0430 20:11:48.059908 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:48.059970 (+ 62us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5095500}
W20240430 20:11:48.060326 18558 connection.cc:441] server connection from 127.0.0.1:40218 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 318) could send its response
I20240430 20:11:48.060698 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 319) took 4989 ms. Trace:
I20240430 20:11:48.060789 19471 rpcz_store.cc:276] 0430 20:11:43.071468 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:43.072835 (+ 1367us) service_pool.cc:225] Handling call
0430 20:11:48.060575 (+4987740us) ts_tablet_manager.cc:1120] Deleting tablet 8901327b3efb4201adce61a9266142ea
0430 20:11:48.060586 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:48.060683 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4982054}
W20240430 20:11:48.060987 18558 connection.cc:441] server connection from 127.0.0.1:40218 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 319) could send its response
I20240430 20:11:48.061291 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 320) took 4982 ms. Trace:
I20240430 20:11:48.061374 19471 rpcz_store.cc:276] 0430 20:11:43.079296 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:43.079349 (+ 53us) service_pool.cc:225] Handling call
0430 20:11:48.061187 (+4981838us) ts_tablet_manager.cc:1120] Deleting tablet aa9998f4e17147aa8f2b35c01980463a
0430 20:11:48.061196 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:48.061277 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4981613}
W20240430 20:11:48.061590 18558 connection.cc:441] server connection from 127.0.0.1:40218 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 320) could send its response
I20240430 20:11:48.062109 20545 raft_consensus.cc:504] T c091701480c4412faabad3d071d85d03 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:48.063297 20545 leader_election.cc:288] T c091701480c4412faabad3d071d85d03 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:48.063349 18617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c091701480c4412faabad3d071d85d03" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:48.063385 18756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c091701480c4412faabad3d071d85d03" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:48.063575 18617 raft_consensus.cc:3037] T c091701480c4412faabad3d071d85d03 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
W20240430 20:11:48.064124 18417 leader_election.cc:341] T c091701480c4412faabad3d071d85d03 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:48.067387 18617 raft_consensus.cc:2454] T c091701480c4412faabad3d071d85d03 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 1.
I20240430 20:11:48.067886 18415 leader_election.cc:302] T c091701480c4412faabad3d071d85d03 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters: 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:48.068787 20545 raft_consensus.cc:2781] T c091701480c4412faabad3d071d85d03 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:48.068972 20545 raft_consensus.cc:686] T c091701480c4412faabad3d071d85d03 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Becoming Leader. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Running, Role: LEADER
I20240430 20:11:48.069630 20545 consensus_queue.cc:237] T c091701480c4412faabad3d071d85d03 P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:48.077740 20841 raft_consensus.cc:482] T 00000000000000000000000000000000 P 89df84ccf5394c9bb60d4e4ed872fc2e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 88b3f98793354c7aaa5c106e8ec31ed8)
I20240430 20:11:48.078037 20841 raft_consensus.cc:504] T 00000000000000000000000000000000 P 89df84ccf5394c9bb60d4e4ed872fc2e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b3f98793354c7aaa5c106e8ec31ed8" member_type: VOTER last_known_addr { host: "127.17.177.62" port: 36441 } } peers { permanent_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e" member_type: VOTER last_known_addr { host: "127.17.177.61" port: 34777 } } peers { permanent_uuid: "b632d6ef4c104a058a6db208024cab33" member_type: VOTER last_known_addr { host: "127.17.177.60" port: 37225 } }
I20240430 20:11:48.036924 19489 rpcz_store.cc:276] 0430 20:11:45.882015 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:45.883447 (+ 1432us) service_pool.cc:225] Handling call
0430 20:11:47.920352 (+2036905us) ts_tablet_manager.cc:1120] Deleting tablet 04b3dff86d5c48a09d03da58bd48f360
0430 20:11:47.941792 (+ 21440us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:47.982625 (+ 40833us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:48.009877 (+ 27252us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:48.035600 (+ 25723us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2036752}
W20240430 20:11:48.080161 18697 connection.cc:441] server connection from 127.0.0.1:59830 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59830 (request call id 411) could send its response
I20240430 20:11:48.081646 18988 tablet_bootstrap.cc:492] T 6d1d2193fd424c91b143f5256df5fcd0 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:48.081845 18988 ts_tablet_manager.cc:1367] T 6d1d2193fd424c91b143f5256df5fcd0 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.161s user 0.001s sys 0.013s
I20240430 20:11:48.082645 18988 raft_consensus.cc:348] T 6d1d2193fd424c91b143f5256df5fcd0 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:48.082978 18988 raft_consensus.cc:374] T 6d1d2193fd424c91b143f5256df5fcd0 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:48.083135 18988 raft_consensus.cc:729] T 6d1d2193fd424c91b143f5256df5fcd0 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:48.097090 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_27f3d711beeb4387a9088ab291168bc8 [id=434aba8e131342a3a9a1aca3278f9c0d]
I20240430 20:11:48.097668 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_0b4b99c1c03a4e779046063159ce6861 [id=43d7bdc84bab4c54bac58dd1dd0f74ac]
I20240430 20:11:48.098367 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_69622a6c03f14409a34ad4d98f24c5a1 [id=43f5d7056f5d4ef98f6861d8f40643f3]
I20240430 20:11:48.099143 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_027891085c5747d58ddb32ad51a453ba [id=45042e2e54a14f558c9968969d3618c5]
I20240430 20:11:48.101986 18985 tablet_bootstrap.cc:654] T ed0c37a71ede4bfc9eb5ad1a4365f25e P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:48.102293 19471 tablet_replica.cc:330] T 6d1d2193fd424c91b143f5256df5fcd0 P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:48.102535 19471 raft_consensus.cc:2229] T 6d1d2193fd424c91b143f5256df5fcd0 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:48.102742 19471 raft_consensus.cc:2258] T 6d1d2193fd424c91b143f5256df5fcd0 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:48.104650 19471 ts_tablet_manager.cc:1875] T 6d1d2193fd424c91b143f5256df5fcd0 P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:48.106781 20587 consensus_queue.cc:1035] T 246fb63a5e9842df8263800e28a5935c P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:48.108237 18988 consensus_queue.cc:260] T 6d1d2193fd424c91b143f5256df5fcd0 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:48.110055 18988 ts_tablet_manager.cc:1398] T 6d1d2193fd424c91b143f5256df5fcd0 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.028s user 0.002s sys 0.000s
I20240430 20:11:48.110502 20630 consensus_queue.cc:1035] T 47bf64ecc41b445da4ff697e141243dd P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:48.112268 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_656db97355f64d7c8aa29b1352dfe9c8 [id=450f5d1c7e084a1bb88d035c0ccfcf82]
I20240430 20:11:48.112717 20489 raft_consensus.cc:482] T b285e3eb16d148779c74e057458d8461 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:48.112846 20489 raft_consensus.cc:504] T b285e3eb16d148779c74e057458d8461 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:48.113451 20489 leader_election.cc:288] T b285e3eb16d148779c74e057458d8461 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:48.114164 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_401b6bfce5ed4b1bb5643b8a53269b60 [id=4a9083e60fa248c28b61b9a6ee49e9d8]
I20240430 20:11:48.114470 18473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b285e3eb16d148779c74e057458d8461" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
W20240430 20:11:48.115260 18558 leader_election.cc:341] T b285e3eb16d148779c74e057458d8461 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:48.116039 18983 tablet_bootstrap.cc:492] T 6db55bb510b24b0cae897936211e15f0 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:48.116211 18983 ts_tablet_manager.cc:1367] T 6db55bb510b24b0cae897936211e15f0 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.098s user 0.004s sys 0.001s
I20240430 20:11:48.116209 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_b6ff64dbfc324fb199b8e6eebd51fd50 [id=4b13edffc2a4446dba14079d2b0f2ca2]
I20240430 20:11:48.116869 18983 raft_consensus.cc:348] T 6db55bb510b24b0cae897936211e15f0 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:48.113960 18750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b285e3eb16d148779c74e057458d8461" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:48.118880 18988 tablet_bootstrap.cc:492] T b1792c160af346ffb4f047c5d4435c10 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
W20240430 20:11:48.118050 18557 leader_election.cc:341] T b285e3eb16d148779c74e057458d8461 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:48.123838 18557 leader_election.cc:302] T b285e3eb16d148779c74e057458d8461 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:48.124472 20488 raft_consensus.cc:2726] T b285e3eb16d148779c74e057458d8461 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:48.124848 20629 consensus_queue.cc:1035] T 8e33df70a1504dbd852fe0b3c3f61622 P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:48.125077 18983 raft_consensus.cc:374] T 6db55bb510b24b0cae897936211e15f0 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:48.125182 18983 raft_consensus.cc:729] T 6db55bb510b24b0cae897936211e15f0 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:48.126356 18983 consensus_queue.cc:260] T 6db55bb510b24b0cae897936211e15f0 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
W20240430 20:11:48.128675 18417 consensus_peers.cc:485] T 04b3dff86d5c48a09d03da58bd48f360 P 2efb542aa55e439b99ccb98d1c5c884a -> Peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Couldn't send request to peer 3c51c33d5616482cafb68e26688820f4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 04b3dff86d5c48a09d03da58bd48f360. This is attempt 1: this message will repeat every 5th retry.
I20240430 20:11:48.135267 18983 ts_tablet_manager.cc:1398] T 6db55bb510b24b0cae897936211e15f0 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.019s user 0.001s sys 0.001s
I20240430 20:11:48.139261 18983 tablet_bootstrap.cc:492] T 9f0b341a31224bf1be525b84b9500304 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:48.140344 18988 tablet_bootstrap.cc:654] T b1792c160af346ffb4f047c5d4435c10 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:48.150943 18983 tablet_bootstrap.cc:654] T 9f0b341a31224bf1be525b84b9500304 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:48.159344 20587 raft_consensus.cc:482] T 130007acc8ba4e709c3eb85fabe75903 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:48.159495 20587 raft_consensus.cc:504] T 130007acc8ba4e709c3eb85fabe75903 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:48.159719 18985 tablet_bootstrap.cc:492] T ed0c37a71ede4bfc9eb5ad1a4365f25e P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:48.159981 18985 ts_tablet_manager.cc:1367] T ed0c37a71ede4bfc9eb5ad1a4365f25e P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.118s user 0.003s sys 0.000s
I20240430 20:11:48.160056 20587 leader_election.cc:288] T 130007acc8ba4e709c3eb85fabe75903 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:48.160754 18608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "130007acc8ba4e709c3eb85fabe75903" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:48.161011 18608 raft_consensus.cc:2454] T 130007acc8ba4e709c3eb85fabe75903 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:48.161584 18696 leader_election.cc:302] T 130007acc8ba4e709c3eb85fabe75903 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:48.161908 20587 raft_consensus.cc:2781] T 130007acc8ba4e709c3eb85fabe75903 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:48.161988 20587 raft_consensus.cc:482] T 130007acc8ba4e709c3eb85fabe75903 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:48.162062 20587 raft_consensus.cc:3037] T 130007acc8ba4e709c3eb85fabe75903 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:48.164659 20587 raft_consensus.cc:504] T 130007acc8ba4e709c3eb85fabe75903 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:48.165019 20841 leader_election.cc:288] T 00000000000000000000000000000000 P 89df84ccf5394c9bb60d4e4ed872fc2e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 88b3f98793354c7aaa5c106e8ec31ed8 (127.17.177.62:36441), b632d6ef4c104a058a6db208024cab33 (127.17.177.60:37225)
I20240430 20:11:48.164798 19471 ts_tablet_manager.cc:1888] T 6d1d2193fd424c91b143f5256df5fcd0 P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:48.165912 18473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "130007acc8ba4e709c3eb85fabe75903" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:48.166090 19471 log.cc:1196] T 6d1d2193fd424c91b143f5256df5fcd0 P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/6d1d2193fd424c91b143f5256df5fcd0
I20240430 20:11:48.166440 19471 ts_tablet_manager.cc:1909] T 6d1d2193fd424c91b143f5256df5fcd0 P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
W20240430 20:11:48.166468 18699 leader_election.cc:341] T 130007acc8ba4e709c3eb85fabe75903 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:48.166638 20587 leader_election.cc:288] T 130007acc8ba4e709c3eb85fabe75903 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Requested vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:48.167243 18608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "130007acc8ba4e709c3eb85fabe75903" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:48.167440 18608 raft_consensus.cc:3037] T 130007acc8ba4e709c3eb85fabe75903 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:48.167415 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 321) took 5069 ms. Trace:
I20240430 20:11:48.167568 19471 rpcz_store.cc:276] 0430 20:11:43.097832 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:43.097891 (+ 59us) service_pool.cc:225] Handling call
0430 20:11:48.102231 (+5004340us) ts_tablet_manager.cc:1120] Deleting tablet 6d1d2193fd424c91b143f5256df5fcd0
0430 20:11:48.102245 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:48.117587 (+ 15342us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:48.164610 (+ 47023us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:48.167355 (+ 2745us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5004108}
W20240430 20:11:48.167858 18558 connection.cc:441] server connection from 127.0.0.1:40218 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 321) could send its response
I20240430 20:11:48.168130 19471 tablet_replica.cc:330] T acf3b6fead41483a82c43dc0286fab7c P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:48.168315 19471 raft_consensus.cc:2229] T acf3b6fead41483a82c43dc0286fab7c P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Raft consensus shutting down.
I20240430 20:11:48.168625 19471 raft_consensus.cc:2258] T acf3b6fead41483a82c43dc0286fab7c P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:48.170235 18608 raft_consensus.cc:2454] T 130007acc8ba4e709c3eb85fabe75903 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:48.171653 18696 leader_election.cc:302] T 130007acc8ba4e709c3eb85fabe75903 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a
I20240430 20:11:48.171805 18983 tablet_bootstrap.cc:492] T 9f0b341a31224bf1be525b84b9500304 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:48.171936 18983 ts_tablet_manager.cc:1367] T 9f0b341a31224bf1be525b84b9500304 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.033s user 0.003s sys 0.002s
I20240430 20:11:48.171975 20695 raft_consensus.cc:2781] T 130007acc8ba4e709c3eb85fabe75903 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:48.172091 20695 raft_consensus.cc:686] T 130007acc8ba4e709c3eb85fabe75903 P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Becoming Leader. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Running, Role: LEADER
I20240430 20:11:48.172343 20695 consensus_queue.cc:237] T 130007acc8ba4e709c3eb85fabe75903 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:48.172559 18983 raft_consensus.cc:348] T 9f0b341a31224bf1be525b84b9500304 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:48.172729 18983 raft_consensus.cc:374] T 9f0b341a31224bf1be525b84b9500304 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:48.172771 18983 raft_consensus.cc:729] T 9f0b341a31224bf1be525b84b9500304 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:48.172942 18983 consensus_queue.cc:260] T 9f0b341a31224bf1be525b84b9500304 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:48.175702 19471 ts_tablet_manager.cc:1875] T acf3b6fead41483a82c43dc0286fab7c P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:48.176024 18988 tablet_bootstrap.cc:492] T b1792c160af346ffb4f047c5d4435c10 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:48.176102 18475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "130007acc8ba4e709c3eb85fabe75903" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:48.176303 18983 ts_tablet_manager.cc:1398] T 9f0b341a31224bf1be525b84b9500304 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240430 20:11:48.176899 18983 tablet_bootstrap.cc:492] T e4cffa295d8c401aa8415ce7e8f56526 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
W20240430 20:11:48.177690 18699 leader_election.cc:341] T 130007acc8ba4e709c3eb85fabe75903 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:48.177040 18985 raft_consensus.cc:348] T ed0c37a71ede4bfc9eb5ad1a4365f25e P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:48.178783 18983 tablet_bootstrap.cc:654] T e4cffa295d8c401aa8415ce7e8f56526 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:48.178793 18985 raft_consensus.cc:374] T ed0c37a71ede4bfc9eb5ad1a4365f25e P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:48.179116 18985 raft_consensus.cc:729] T ed0c37a71ede4bfc9eb5ad1a4365f25e P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:48.179841 18985 consensus_queue.cc:260] T ed0c37a71ede4bfc9eb5ad1a4365f25e P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:48.180809 18985 ts_tablet_manager.cc:1398] T ed0c37a71ede4bfc9eb5ad1a4365f25e P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.021s user 0.001s sys 0.001s
I20240430 20:11:48.181736 18985 tablet_bootstrap.cc:492] T df92eb85261d47d88cbf16d29d9ad30e P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
W20240430 20:11:48.184652 18211 leader_election.cc:334] T 00000000000000000000000000000000 P 89df84ccf5394c9bb60d4e4ed872fc2e [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer b632d6ef4c104a058a6db208024cab33 (127.17.177.60:37225): Network error: Client connection negotiation failed: client connection to 127.17.177.60:37225: connect: Connection refused (error 111)
I20240430 20:11:48.185521 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_5dd3db8b666a491e8b33cbb5a0b54ff5 [id=4b7dfcb0dd2b4e8ba3a9a4a2bce81287]
I20240430 20:11:48.195360 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_d8df23edb8ce480bb59a2ab0f1fc7521 [id=4c9193fcff2f41768d578803372434e0]
I20240430 20:11:48.195824 19471 ts_tablet_manager.cc:1888] T acf3b6fead41483a82c43dc0286fab7c P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:48.195921 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_50a4836df84543a888fcdb06ccc5ec41 [id=4cc347c303a04ae2864738e6b2cdc519]
I20240430 20:11:48.196012 19471 log.cc:1196] T acf3b6fead41483a82c43dc0286fab7c P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/acf3b6fead41483a82c43dc0286fab7c
I20240430 20:11:48.196444 19471 ts_tablet_manager.cc:1909] T acf3b6fead41483a82c43dc0286fab7c P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:48.197005 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_19df33614a6c4f94b6419a5b441e3ef8 [id=4dce8a767f0541e1beb5464b2ed95d06]
I20240430 20:11:48.197403 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 322) took 4938 ms. Trace:
I20240430 20:11:48.197490 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_25ddc3adef6b4ccf8b9c00cc6772031c [id=4fa623ffdcf347639150a355806038af]
I20240430 20:11:48.197525 19471 rpcz_store.cc:276] 0430 20:11:43.258953 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:43.259014 (+ 61us) service_pool.cc:225] Handling call
0430 20:11:48.168085 (+4909071us) ts_tablet_manager.cc:1120] Deleting tablet acf3b6fead41483a82c43dc0286fab7c
0430 20:11:48.168095 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:48.182427 (+ 14332us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:48.195694 (+ 13267us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:48.197380 (+ 1686us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4908866}
W20240430 20:11:48.197890 18558 connection.cc:441] server connection from 127.0.0.1:40218 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 322) could send its response
I20240430 20:11:48.198230 19471 tablet_replica.cc:330] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:48.198494 19471 raft_consensus.cc:2229] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 5e1c6397f09747f091534faf93542710 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:48.198741 19471 raft_consensus.cc:2258] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 5e1c6397f09747f091534faf93542710 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:48.199327 19471 ts_tablet_manager.cc:1875] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:48.199424 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_685a26737bb046fb85ced15df7ae3423 [id=51d177e8498848b38c43e3f8d11f092c]
I20240430 20:11:48.199846 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_7390a5615e234bf38ddf41eefe70bdf3 [id=52259eea51074e14b472d8532e08a990]
I20240430 20:11:48.199959 18988 ts_tablet_manager.cc:1367] T b1792c160af346ffb4f047c5d4435c10 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.081s user 0.001s sys 0.006s
I20240430 20:11:48.202047 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_b064d5659f7243f5b0ec167fb6c3f404 [id=5227206f44d04fa89b4842b899f33afb]
I20240430 20:11:48.202383 18988 raft_consensus.cc:348] T b1792c160af346ffb4f047c5d4435c10 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:48.202980 18988 raft_consensus.cc:374] T b1792c160af346ffb4f047c5d4435c10 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:48.203155 18988 raft_consensus.cc:729] T b1792c160af346ffb4f047c5d4435c10 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:48.204007 18988 consensus_queue.cc:260] T b1792c160af346ffb4f047c5d4435c10 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:48.207505 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_5d3cb1a25c884804b05b07531e1b6965 [id=55e60456c89d441a8b5874a2063d52ef]
I20240430 20:11:48.208745 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_925990f1c73245818b39aafeb0de55e3 [id=5979225bccda4e35bea4bec2c67b22df]
I20240430 20:11:48.209357 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_69892a8264a4477db48ee6248758e046 [id=5a1c4a7be2fc465fac89a50854eb3322]
I20240430 20:11:48.209597 18988 ts_tablet_manager.cc:1398] T b1792c160af346ffb4f047c5d4435c10 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.009s user 0.002s sys 0.001s
I20240430 20:11:48.209861 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_b99234d57c5c42d58e4a6516fc1482d4 [id=5b3ce682abaf4acc8b62f9d34ff63c64]
I20240430 20:11:48.210579 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_766e3df58895427b9e5179eb555fe7e4 [id=5c11311047b34a3eb06cc06e8cc10223]
I20240430 20:11:48.211529 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_c5c549bbf87d46c7886dad5f93086cd5 [id=5db65c3521504fffbe2599dbb5337a6a]
I20240430 20:11:48.212656 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_204dce4e6128482583d8869c83413f25 [id=618fcf2891c84a238fcde6f9b62bba3f]
I20240430 20:11:48.216377 18983 tablet_bootstrap.cc:492] T e4cffa295d8c401aa8415ce7e8f56526 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:48.216657 18983 ts_tablet_manager.cc:1367] T e4cffa295d8c401aa8415ce7e8f56526 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.040s user 0.005s sys 0.005s
I20240430 20:11:48.218021 18983 raft_consensus.cc:348] T e4cffa295d8c401aa8415ce7e8f56526 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:48.218434 18983 raft_consensus.cc:374] T e4cffa295d8c401aa8415ce7e8f56526 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:48.218572 18983 raft_consensus.cc:729] T e4cffa295d8c401aa8415ce7e8f56526 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:48.218986 18983 consensus_queue.cc:260] T e4cffa295d8c401aa8415ce7e8f56526 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:48.220317 18983 ts_tablet_manager.cc:1398] T e4cffa295d8c401aa8415ce7e8f56526 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240430 20:11:48.220979 18983 tablet_bootstrap.cc:492] T d7c1fc7e81364fb890bbc13d7fb6848b P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:48.221343 20629 raft_consensus.cc:482] T 3e5225f1f5f544ecbf8dfa470af3daca P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:48.221489 20629 raft_consensus.cc:504] T 3e5225f1f5f544ecbf8dfa470af3daca P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:48.222110 20629 leader_election.cc:288] T 3e5225f1f5f544ecbf8dfa470af3daca P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:48.222837 18608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e5225f1f5f544ecbf8dfa470af3daca" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
W20240430 20:11:48.223459 18415 leader_election.cc:341] T 3e5225f1f5f544ecbf8dfa470af3daca P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:48.229511 18983 tablet_bootstrap.cc:654] T d7c1fc7e81364fb890bbc13d7fb6848b P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:48.231464 18750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e5225f1f5f544ecbf8dfa470af3daca" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
W20240430 20:11:48.232178 18417 leader_election.cc:341] T 3e5225f1f5f544ecbf8dfa470af3daca P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:48.232321 18417 leader_election.cc:302] T 3e5225f1f5f544ecbf8dfa470af3daca P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a; no voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710
I20240430 20:11:48.232729 20629 raft_consensus.cc:2726] T 3e5225f1f5f544ecbf8dfa470af3daca P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:48.233840 18988 tablet_bootstrap.cc:492] T c091701480c4412faabad3d071d85d03 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:48.237251 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_9049c9eeb10540d3815bab49caa1e82e [id=619746ec6b3244ee9ba511745c38bc25]
I20240430 20:11:48.238153 18988 tablet_bootstrap.cc:654] T c091701480c4412faabad3d071d85d03 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:48.238564 19471 ts_tablet_manager.cc:1888] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.2
I20240430 20:11:48.238695 19471 log.cc:1196] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/dd6083f12c7e4d1e8a2006b3ddf9aa3a
I20240430 20:11:48.239076 19471 ts_tablet_manager.cc:1909] T dd6083f12c7e4d1e8a2006b3ddf9aa3a P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:48.240846 18985 tablet_bootstrap.cc:654] T df92eb85261d47d88cbf16d29d9ad30e P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:48.241647 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_763f0800e8bb4fa69976886416250f42 [id=61c8a5b99ef34a0baeeb343fef7cfd05]
I20240430 20:11:48.245036 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_b49a7942aa8c4f58acb94c35d1cdbb70 [id=6257559b0e9545c58848a76af0c5a5f7]
I20240430 20:11:48.247102 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 343) took 4794 ms. Trace:
I20240430 20:11:48.247247 19471 rpcz_store.cc:276] 0430 20:11:43.452916 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:43.453080 (+ 164us) service_pool.cc:225] Handling call
0430 20:11:48.198165 (+4745085us) ts_tablet_manager.cc:1120] Deleting tablet dd6083f12c7e4d1e8a2006b3ddf9aa3a
0430 20:11:48.198179 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:48.234419 (+ 36240us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:48.238458 (+ 4039us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:48.247077 (+ 8619us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4744841}
W20240430 20:11:48.247543 18558 connection.cc:441] server connection from 127.0.0.1:40218 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 343) could send its response
I20240430 20:11:48.247829 19471 tablet_replica.cc:330] T 3ec3993042f5498b818bdf32d5087928 P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:48.248052 19471 raft_consensus.cc:2229] T 3ec3993042f5498b818bdf32d5087928 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:48.248231 19471 raft_consensus.cc:2258] T 3ec3993042f5498b818bdf32d5087928 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:48.255332 19471 ts_tablet_manager.cc:1875] T 3ec3993042f5498b818bdf32d5087928 P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:48.255934 20489 raft_consensus.cc:482] T 9f0b341a31224bf1be525b84b9500304 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:48.256060 20489 raft_consensus.cc:504] T 9f0b341a31224bf1be525b84b9500304 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:48.256740 20489 leader_election.cc:288] T 9f0b341a31224bf1be525b84b9500304 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:48.257428 18475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f0b341a31224bf1be525b84b9500304" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:48.257719 18475 raft_consensus.cc:2454] T 9f0b341a31224bf1be525b84b9500304 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 0.
I20240430 20:11:48.258097 18750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f0b341a31224bf1be525b84b9500304" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:48.258204 18558 leader_election.cc:302] T 9f0b341a31224bf1be525b84b9500304 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:48.258481 20489 raft_consensus.cc:2781] T 9f0b341a31224bf1be525b84b9500304 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:48.258564 20489 raft_consensus.cc:482] T 9f0b341a31224bf1be525b84b9500304 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:48.258651 20489 raft_consensus.cc:3037] T 9f0b341a31224bf1be525b84b9500304 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
W20240430 20:11:48.258731 18557 leader_election.cc:341] T 9f0b341a31224bf1be525b84b9500304 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:48.260282 18988 tablet_bootstrap.cc:492] T c091701480c4412faabad3d071d85d03 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:48.260583 18988 ts_tablet_manager.cc:1367] T c091701480c4412faabad3d071d85d03 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.027s user 0.009s sys 0.000s
I20240430 20:11:48.261205 20489 raft_consensus.cc:504] T 9f0b341a31224bf1be525b84b9500304 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:48.261821 20489 leader_election.cc:288] T 9f0b341a31224bf1be525b84b9500304 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:48.262250 18750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f0b341a31224bf1be525b84b9500304" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:48.262770 18475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f0b341a31224bf1be525b84b9500304" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
W20240430 20:11:48.262923 18557 leader_election.cc:341] T 9f0b341a31224bf1be525b84b9500304 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:48.263022 18475 raft_consensus.cc:3037] T 9f0b341a31224bf1be525b84b9500304 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:48.263207 18988 raft_consensus.cc:348] T c091701480c4412faabad3d071d85d03 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:48.263566 18988 raft_consensus.cc:374] T c091701480c4412faabad3d071d85d03 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:48.263693 18988 raft_consensus.cc:729] T c091701480c4412faabad3d071d85d03 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:48.264478 18988 consensus_queue.cc:260] T c091701480c4412faabad3d071d85d03 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:48.266209 18475 raft_consensus.cc:2454] T 9f0b341a31224bf1be525b84b9500304 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
I20240430 20:11:48.266788 18558 leader_election.cc:302] T 9f0b341a31224bf1be525b84b9500304 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters: 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:48.267248 20489 raft_consensus.cc:2781] T 9f0b341a31224bf1be525b84b9500304 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:48.267379 20489 raft_consensus.cc:686] T 9f0b341a31224bf1be525b84b9500304 P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Becoming Leader. State: Replica: 5e1c6397f09747f091534faf93542710, State: Running, Role: LEADER
I20240430 20:11:48.267629 20489 consensus_queue.cc:237] T 9f0b341a31224bf1be525b84b9500304 P 5e1c6397f09747f091534faf93542710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:48.268079 18988 ts_tablet_manager.cc:1398] T c091701480c4412faabad3d071d85d03 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20240430 20:11:48.271939 18983 tablet_bootstrap.cc:492] T d7c1fc7e81364fb890bbc13d7fb6848b P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:48.272223 18983 ts_tablet_manager.cc:1367] T d7c1fc7e81364fb890bbc13d7fb6848b P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.052s user 0.003s sys 0.005s
I20240430 20:11:48.272848 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_2c1b15dc75504997924b1f36ca1f69a8 [id=634df0651c254f1c9e8db138ad908f31]
I20240430 20:11:48.273164 18985 tablet_bootstrap.cc:492] T df92eb85261d47d88cbf16d29d9ad30e P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:48.273806 18983 raft_consensus.cc:348] T d7c1fc7e81364fb890bbc13d7fb6848b P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:48.274171 18983 raft_consensus.cc:374] T d7c1fc7e81364fb890bbc13d7fb6848b P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:48.274315 18983 raft_consensus.cc:729] T d7c1fc7e81364fb890bbc13d7fb6848b P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:48.274789 18983 consensus_queue.cc:260] T d7c1fc7e81364fb890bbc13d7fb6848b P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:48.276881 18983 ts_tablet_manager.cc:1398] T d7c1fc7e81364fb890bbc13d7fb6848b P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240430 20:11:48.277244 18985 ts_tablet_manager.cc:1367] T df92eb85261d47d88cbf16d29d9ad30e P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.096s user 0.004s sys 0.000s
I20240430 20:11:48.278771 18988 tablet_bootstrap.cc:492] T d7c1fc7e81364fb890bbc13d7fb6848b P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:48.279541 20695 raft_consensus.cc:482] T fdf53061efd74fa29a9d8208d1fd210e P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:48.279706 20695 raft_consensus.cc:504] T fdf53061efd74fa29a9d8208d1fd210e P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:48.280827 20695 leader_election.cc:288] T fdf53061efd74fa29a9d8208d1fd210e P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:48.281673 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_e311a584368b4233b6e5beb24587ce00 [id=65018c93b8e2409f984c52764e82dcb3]
W20240430 20:11:48.281827 18696 leader_election.cc:341] T fdf53061efd74fa29a9d8208d1fd210e P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Not found: Tablet not found: fdf53061efd74fa29a9d8208d1fd210e
I20240430 20:11:48.282219 19471 ts_tablet_manager.cc:1888] T 3ec3993042f5498b818bdf32d5087928 P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:48.283106 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_19a91a7f17ff456ebb145e8dd89c0940 [id=67283dd3995a44c6a7a7bc8722e4f955]
I20240430 20:11:48.282915 18983 tablet_bootstrap.cc:492] T b1792c160af346ffb4f047c5d4435c10 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:48.284124 18988 tablet_bootstrap.cc:654] T d7c1fc7e81364fb890bbc13d7fb6848b P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:48.285600 18475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdf53061efd74fa29a9d8208d1fd210e" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:48.285354 18983 tablet_bootstrap.cc:654] T b1792c160af346ffb4f047c5d4435c10 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:48.283838 19471 log.cc:1196] T 3ec3993042f5498b818bdf32d5087928 P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/3ec3993042f5498b818bdf32d5087928
I20240430 20:11:48.286972 18475 raft_consensus.cc:2454] T fdf53061efd74fa29a9d8208d1fd210e P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:48.287477 18699 leader_election.cc:302] T fdf53061efd74fa29a9d8208d1fd210e P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters: 5e1c6397f09747f091534faf93542710
I20240430 20:11:48.287551 19471 ts_tablet_manager.cc:1909] T 3ec3993042f5498b818bdf32d5087928 P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:48.287705 20695 raft_consensus.cc:2781] T fdf53061efd74fa29a9d8208d1fd210e P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:48.287765 20695 raft_consensus.cc:482] T fdf53061efd74fa29a9d8208d1fd210e P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:48.287818 20695 raft_consensus.cc:3037] T fdf53061efd74fa29a9d8208d1fd210e P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:48.289126 18985 raft_consensus.cc:348] T df92eb85261d47d88cbf16d29d9ad30e P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:48.290434 20695 raft_consensus.cc:504] T fdf53061efd74fa29a9d8208d1fd210e P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:48.290720 18985 raft_consensus.cc:374] T df92eb85261d47d88cbf16d29d9ad30e P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:48.290843 18985 raft_consensus.cc:729] T df92eb85261d47d88cbf16d29d9ad30e P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:48.291139 20695 leader_election.cc:288] T fdf53061efd74fa29a9d8208d1fd210e P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Requested vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:48.291705 18475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdf53061efd74fa29a9d8208d1fd210e" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
W20240430 20:11:48.291829 18696 leader_election.cc:341] T fdf53061efd74fa29a9d8208d1fd210e P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Not found: Tablet not found: fdf53061efd74fa29a9d8208d1fd210e
I20240430 20:11:48.291983 18475 raft_consensus.cc:3037] T fdf53061efd74fa29a9d8208d1fd210e P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:48.295810 18475 raft_consensus.cc:2454] T fdf53061efd74fa29a9d8208d1fd210e P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:48.296177 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 341) took 4843 ms. Trace:
I20240430 20:11:48.296327 18699 leader_election.cc:302] T fdf53061efd74fa29a9d8208d1fd210e P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters: 5e1c6397f09747f091534faf93542710
I20240430 20:11:48.296281 19471 rpcz_store.cc:276] 0430 20:11:43.452749 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:43.457253 (+ 4504us) service_pool.cc:225] Handling call
0430 20:11:48.247775 (+4790522us) ts_tablet_manager.cc:1120] Deleting tablet 3ec3993042f5498b818bdf32d5087928
0430 20:11:48.247786 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:48.264455 (+ 16669us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:48.282108 (+ 17653us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:48.296154 (+ 14046us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4790301}
W20240430 20:11:48.296542 18558 connection.cc:441] server connection from 127.0.0.1:40218 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 341) could send its response
I20240430 20:11:48.296691 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 342) took 4843 ms. Trace:
I20240430 20:11:48.296767 19471 rpcz_store.cc:276] 0430 20:11:43.452833 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:43.523973 (+ 71140us) service_pool.cc:225] Handling call
0430 20:11:48.296565 (+4772592us) ts_tablet_manager.cc:1120] Deleting tablet b7141e215d6d44bc8406c80508190874
0430 20:11:48.296575 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:48.296674 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4757479}
W20240430 20:11:48.296970 18558 connection.cc:441] server connection from 127.0.0.1:40218 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 342) could send its response
I20240430 20:11:48.297080 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 344) took 4791 ms. Trace:
I20240430 20:11:48.297147 19471 rpcz_store.cc:276] 0430 20:11:43.505652 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:43.539137 (+ 33485us) service_pool.cc:225] Handling call
0430 20:11:48.296994 (+4757857us) ts_tablet_manager.cc:1120] Deleting tablet 489dc838744a49eba542009c86688aa5
0430 20:11:48.297003 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:48.297067 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4757658}
I20240430 20:11:48.297240 20693 raft_consensus.cc:2781] T fdf53061efd74fa29a9d8208d1fd210e P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:48.297345 20693 raft_consensus.cc:686] T fdf53061efd74fa29a9d8208d1fd210e P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Becoming Leader. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Running, Role: LEADER
I20240430 20:11:48.297412 19471 tablet_replica.cc:330] T 115c4a12440647a4bb46eb6824f7ea98 P 5e1c6397f09747f091534faf93542710: stopping tablet replica
W20240430 20:11:48.297533 18558 connection.cc:441] server connection from 127.0.0.1:40218 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 344) could send its response
I20240430 20:11:48.297585 19471 raft_consensus.cc:2229] T 115c4a12440647a4bb46eb6824f7ea98 P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Raft consensus shutting down.
I20240430 20:11:48.297586 20693 consensus_queue.cc:237] T fdf53061efd74fa29a9d8208d1fd210e P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:48.297892 19471 raft_consensus.cc:2258] T 115c4a12440647a4bb46eb6824f7ea98 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:48.304219 19471 ts_tablet_manager.cc:1875] T 115c4a12440647a4bb46eb6824f7ea98 P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:48.307025 18985 consensus_queue.cc:260] T df92eb85261d47d88cbf16d29d9ad30e P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:48.312690 18988 tablet_bootstrap.cc:492] T d7c1fc7e81364fb890bbc13d7fb6848b P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:48.312999 18988 ts_tablet_manager.cc:1367] T d7c1fc7e81364fb890bbc13d7fb6848b P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.034s user 0.008s sys 0.000s
I20240430 20:11:48.315253 18988 raft_consensus.cc:348] T d7c1fc7e81364fb890bbc13d7fb6848b P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:48.315860 18988 raft_consensus.cc:374] T d7c1fc7e81364fb890bbc13d7fb6848b P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:48.316025 18988 raft_consensus.cc:729] T d7c1fc7e81364fb890bbc13d7fb6848b P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:48.316738 20488 raft_consensus.cc:482] T e4cffa295d8c401aa8415ce7e8f56526 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:48.316496 18988 consensus_queue.cc:260] T d7c1fc7e81364fb890bbc13d7fb6848b P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:48.316884 20488 raft_consensus.cc:504] T e4cffa295d8c401aa8415ce7e8f56526 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:48.317472 20488 leader_election.cc:288] T e4cffa295d8c401aa8415ce7e8f56526 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:48.318308 18988 ts_tablet_manager.cc:1398] T d7c1fc7e81364fb890bbc13d7fb6848b P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240430 20:11:48.320192 18988 tablet_bootstrap.cc:492] T e4cffa295d8c401aa8415ce7e8f56526 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:48.320883 18750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4cffa295d8c401aa8415ce7e8f56526" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
W20240430 20:11:48.321404 18557 leader_election.cc:341] T e4cffa295d8c401aa8415ce7e8f56526 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:48.318308 18473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4cffa295d8c401aa8415ce7e8f56526" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:48.327121 18473 raft_consensus.cc:2454] T e4cffa295d8c401aa8415ce7e8f56526 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 0.
I20240430 20:11:48.329566 19471 ts_tablet_manager.cc:1888] T 115c4a12440647a4bb46eb6824f7ea98 P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:48.329666 19471 log.cc:1196] T 115c4a12440647a4bb46eb6824f7ea98 P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/115c4a12440647a4bb46eb6824f7ea98
I20240430 20:11:48.330001 19471 ts_tablet_manager.cc:1909] T 115c4a12440647a4bb46eb6824f7ea98 P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:48.329958 18988 tablet_bootstrap.cc:654] T e4cffa295d8c401aa8415ce7e8f56526 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:48.330828 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 345) took 4763 ms. Trace:
I20240430 20:11:48.330925 19471 rpcz_store.cc:276] 0430 20:11:43.567169 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:43.584724 (+ 17555us) service_pool.cc:225] Handling call
0430 20:11:48.297359 (+4712635us) ts_tablet_manager.cc:1120] Deleting tablet 115c4a12440647a4bb46eb6824f7ea98
0430 20:11:48.297368 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:48.322808 (+ 25440us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:48.329461 (+ 6653us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:48.330805 (+ 1344us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4676255}
W20240430 20:11:48.335608 18558 connection.cc:441] server connection from 127.0.0.1:40218 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 345) could send its response
I20240430 20:11:48.336076 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 360) took 3990 ms. Trace:
I20240430 20:11:48.336145 19471 rpcz_store.cc:276] 0430 20:11:44.345923 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:44.346013 (+ 90us) service_pool.cc:225] Handling call
0430 20:11:48.335942 (+3989929us) ts_tablet_manager.cc:1120] Deleting tablet c1017a50d1d646c486c0b9a75cb88ce4
0430 20:11:48.335952 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:48.336059 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3985856}
W20240430 20:11:48.336337 18558 connection.cc:441] server connection from 127.0.0.1:40218 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 360) could send its response
I20240430 20:11:48.337005 19471 tablet_replica.cc:330] T f6cfd089f08b47409fb2a6cf4f6b0385 P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:48.337191 19471 raft_consensus.cc:2229] T f6cfd089f08b47409fb2a6cf4f6b0385 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:48.337355 19471 raft_consensus.cc:2258] T f6cfd089f08b47409fb2a6cf4f6b0385 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:48.338835 19471 ts_tablet_manager.cc:1875] T f6cfd089f08b47409fb2a6cf4f6b0385 P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:48.345167 20545 raft_consensus.cc:482] T 12795670f21447a8964fe1efc6e92398 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:48.345326 20545 raft_consensus.cc:504] T 12795670f21447a8964fe1efc6e92398 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:48.345937 20545 leader_election.cc:288] T 12795670f21447a8964fe1efc6e92398 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:48.346586 18608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12795670f21447a8964fe1efc6e92398" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
W20240430 20:11:48.347357 18415 leader_election.cc:341] T 12795670f21447a8964fe1efc6e92398 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:48.348034 18750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12795670f21447a8964fe1efc6e92398" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:48.348555 18988 tablet_bootstrap.cc:492] T e4cffa295d8c401aa8415ce7e8f56526 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
W20240430 20:11:48.348695 18417 leader_election.cc:341] T 12795670f21447a8964fe1efc6e92398 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:48.348805 18417 leader_election.cc:302] T 12795670f21447a8964fe1efc6e92398 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a; no voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710
I20240430 20:11:48.348831 18988 ts_tablet_manager.cc:1367] T e4cffa295d8c401aa8415ce7e8f56526 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.029s user 0.004s sys 0.002s
I20240430 20:11:48.349265 20631 raft_consensus.cc:2726] T 12795670f21447a8964fe1efc6e92398 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:48.350478 18988 raft_consensus.cc:348] T e4cffa295d8c401aa8415ce7e8f56526 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:48.350919 18988 raft_consensus.cc:374] T e4cffa295d8c401aa8415ce7e8f56526 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:48.351063 18988 raft_consensus.cc:729] T e4cffa295d8c401aa8415ce7e8f56526 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:48.351786 18988 consensus_queue.cc:260] T e4cffa295d8c401aa8415ce7e8f56526 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:48.355849 18988 ts_tablet_manager.cc:1398] T e4cffa295d8c401aa8415ce7e8f56526 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.007s user 0.001s sys 0.002s
I20240430 20:11:48.362123 19471 ts_tablet_manager.cc:1888] T f6cfd089f08b47409fb2a6cf4f6b0385 P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:48.362663 19471 log.cc:1196] T f6cfd089f08b47409fb2a6cf4f6b0385 P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/f6cfd089f08b47409fb2a6cf4f6b0385
I20240430 20:11:48.363394 19471 ts_tablet_manager.cc:1909] T f6cfd089f08b47409fb2a6cf4f6b0385 P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:48.364729 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 361) took 3996 ms. Trace:
I20240430 20:11:48.365837 19471 rpcz_store.cc:276] 0430 20:11:44.368755 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:44.368808 (+ 53us) service_pool.cc:225] Handling call
0430 20:11:48.336963 (+3968155us) ts_tablet_manager.cc:1120] Deleting tablet f6cfd089f08b47409fb2a6cf4f6b0385
0430 20:11:48.336971 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:48.353788 (+ 16817us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:48.361981 (+ 8193us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:48.364700 (+ 2719us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3967941}
W20240430 20:11:48.371244 18558 connection.cc:441] server connection from 127.0.0.1:40218 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 361) could send its response
I20240430 20:11:48.371760 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 362) took 4002 ms. Trace:
I20240430 20:11:48.372210 19471 rpcz_store.cc:276] 0430 20:11:44.368868 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:44.368916 (+ 48us) service_pool.cc:225] Handling call
0430 20:11:48.371611 (+4002695us) ts_tablet_manager.cc:1120] Deleting tablet 024069ab786749028c020abd1c836cf5
0430 20:11:48.371623 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:48.371741 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3,"tablet-delete.queue_time_us":4002574}
W20240430 20:11:48.372685 18558 connection.cc:441] server connection from 127.0.0.1:40218 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 362) could send its response
I20240430 20:11:48.373050 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 366) took 3825 ms. Trace:
I20240430 20:11:48.373404 19471 rpcz_store.cc:276] 0430 20:11:44.547142 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:44.547201 (+ 59us) service_pool.cc:225] Handling call
0430 20:11:48.372941 (+3825740us) ts_tablet_manager.cc:1120] Deleting tablet a37d917bcc45468f9e3ee805640fba98
0430 20:11:48.372952 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:48.373033 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3825534}
W20240430 20:11:48.373806 18558 connection.cc:441] server connection from 127.0.0.1:40218 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 366) could send its response
I20240430 20:11:48.374146 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 367) took 3802 ms. Trace:
I20240430 20:11:48.374504 19471 rpcz_store.cc:276] 0430 20:11:44.571996 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:44.572046 (+ 50us) service_pool.cc:225] Handling call
0430 20:11:48.374053 (+3802007us) ts_tablet_manager.cc:1120] Deleting tablet a445fdfde01c4371a5e6b3daea5b00ab
0430 20:11:48.374061 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:48.374132 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3801820}
W20240430 20:11:48.374984 18558 connection.cc:441] server connection from 127.0.0.1:40218 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 367) could send its response
I20240430 20:11:48.375267 19471 tablet_replica.cc:330] T e3e74e76ab2b4fe492ffcc1838656fd1 P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:48.375617 19471 raft_consensus.cc:2229] T e3e74e76ab2b4fe492ffcc1838656fd1 P 5e1c6397f09747f091534faf93542710 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:48.376014 19471 raft_consensus.cc:2258] T e3e74e76ab2b4fe492ffcc1838656fd1 P 5e1c6397f09747f091534faf93542710 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:48.377285 19471 ts_tablet_manager.cc:1875] T e3e74e76ab2b4fe492ffcc1838656fd1 P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:48.378746 18985 ts_tablet_manager.cc:1398] T df92eb85261d47d88cbf16d29d9ad30e P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.101s user 0.002s sys 0.000s
I20240430 20:11:48.380153 18988 tablet_bootstrap.cc:492] T 90a694bd5db649afa50f3c4e752939c1 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:48.382035 18988 tablet_bootstrap.cc:654] T 90a694bd5db649afa50f3c4e752939c1 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:48.385102 20587 raft_consensus.cc:482] T e4cffa295d8c401aa8415ce7e8f56526 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:48.385248 20587 raft_consensus.cc:504] T e4cffa295d8c401aa8415ce7e8f56526 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:48.386040 20587 leader_election.cc:288] T e4cffa295d8c401aa8415ce7e8f56526 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:48.386802 18608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4cffa295d8c401aa8415ce7e8f56526" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:48.387002 18608 raft_consensus.cc:2454] T e4cffa295d8c401aa8415ce7e8f56526 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:48.387082 18474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4cffa295d8c401aa8415ce7e8f56526" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:48.387533 18696 leader_election.cc:302] T e4cffa295d8c401aa8415ce7e8f56526 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:48.387850 20587 raft_consensus.cc:2781] T e4cffa295d8c401aa8415ce7e8f56526 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:48.387917 20587 raft_consensus.cc:482] T e4cffa295d8c401aa8415ce7e8f56526 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:48.387970 20587 raft_consensus.cc:3037] T e4cffa295d8c401aa8415ce7e8f56526 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:48.389747 20489 consensus_queue.cc:1035] T 331c031d44854d679a27d10d9151d0ea P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:48.390504 20587 raft_consensus.cc:504] T e4cffa295d8c401aa8415ce7e8f56526 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:48.392475 18608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4cffa295d8c401aa8415ce7e8f56526" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:48.392647 18608 raft_consensus.cc:3037] T e4cffa295d8c401aa8415ce7e8f56526 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:48.394991 18608 raft_consensus.cc:2454] T e4cffa295d8c401aa8415ce7e8f56526 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:48.398872 18696 leader_election.cc:302] T e4cffa295d8c401aa8415ce7e8f56526 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:48.398267 19471 ts_tablet_manager.cc:1888] T e3e74e76ab2b4fe492ffcc1838656fd1 P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 3.2
I20240430 20:11:48.399122 19471 log.cc:1196] T e3e74e76ab2b4fe492ffcc1838656fd1 P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/e3e74e76ab2b4fe492ffcc1838656fd1
I20240430 20:11:48.399169 20695 raft_consensus.cc:2781] T e4cffa295d8c401aa8415ce7e8f56526 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:48.399309 20695 raft_consensus.cc:686] T e4cffa295d8c401aa8415ce7e8f56526 P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Becoming Leader. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Running, Role: LEADER
I20240430 20:11:48.399626 20695 consensus_queue.cc:237] T e4cffa295d8c401aa8415ce7e8f56526 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:48.400142 18474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4cffa295d8c401aa8415ce7e8f56526" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:48.400095 20587 leader_election.cc:288] T e4cffa295d8c401aa8415ce7e8f56526 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Requested vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:48.401767 18756 raft_consensus.cc:3037] T 6d1d2193fd424c91b143f5256df5fcd0 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:48.406162 20630 consensus_queue.cc:1035] T 6d1d2193fd424c91b143f5256df5fcd0 P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:48.420966 18985 tablet_bootstrap.cc:492] T 130007acc8ba4e709c3eb85fabe75903 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:48.424002 20630 raft_consensus.cc:482] T d7c1fc7e81364fb890bbc13d7fb6848b P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:48.426717 20630 raft_consensus.cc:504] T d7c1fc7e81364fb890bbc13d7fb6848b P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:48.426923 19471 ts_tablet_manager.cc:1909] T e3e74e76ab2b4fe492ffcc1838656fd1 P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:48.427459 20630 leader_election.cc:288] T d7c1fc7e81364fb890bbc13d7fb6848b P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:48.427866 18608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c1fc7e81364fb890bbc13d7fb6848b" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:48.428134 18608 raft_consensus.cc:2454] T d7c1fc7e81364fb890bbc13d7fb6848b P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 0.
I20240430 20:11:48.428519 18415 leader_election.cc:302] T d7c1fc7e81364fb890bbc13d7fb6848b P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:48.429874 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 371) took 3466 ms. Trace:
I20240430 20:11:48.430678 20630 raft_consensus.cc:2781] T d7c1fc7e81364fb890bbc13d7fb6848b P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:48.430761 20630 raft_consensus.cc:482] T d7c1fc7e81364fb890bbc13d7fb6848b P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:48.430837 20630 raft_consensus.cc:3037] T d7c1fc7e81364fb890bbc13d7fb6848b P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:48.430665 18756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c1fc7e81364fb890bbc13d7fb6848b" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:48.431061 18756 raft_consensus.cc:2454] T d7c1fc7e81364fb890bbc13d7fb6848b P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 0.
I20240430 20:11:48.433022 18988 tablet_bootstrap.cc:492] T 90a694bd5db649afa50f3c4e752939c1 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:48.433452 18988 ts_tablet_manager.cc:1367] T 90a694bd5db649afa50f3c4e752939c1 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.053s user 0.000s sys 0.012s
W20240430 20:11:48.433790 18415 consensus_peers.cc:485] T f6cfd089f08b47409fb2a6cf4f6b0385 P 2efb542aa55e439b99ccb98d1c5c884a -> Peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Couldn't send request to peer 5e1c6397f09747f091534faf93542710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f6cfd089f08b47409fb2a6cf4f6b0385. This is attempt 1: this message will repeat every 5th retry.
I20240430 20:11:48.434235 20630 raft_consensus.cc:504] T d7c1fc7e81364fb890bbc13d7fb6848b P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:48.435657 20630 leader_election.cc:288] T d7c1fc7e81364fb890bbc13d7fb6848b P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:48.435602 18756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c1fc7e81364fb890bbc13d7fb6848b" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:48.435890 18756 raft_consensus.cc:3037] T d7c1fc7e81364fb890bbc13d7fb6848b P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:48.436301 18988 raft_consensus.cc:348] T 90a694bd5db649afa50f3c4e752939c1 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:48.438463 18608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c1fc7e81364fb890bbc13d7fb6848b" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:48.437431 18988 raft_consensus.cc:374] T 90a694bd5db649afa50f3c4e752939c1 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:48.438779 18608 raft_consensus.cc:3037] T d7c1fc7e81364fb890bbc13d7fb6848b P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:48.438835 18988 raft_consensus.cc:729] T 90a694bd5db649afa50f3c4e752939c1 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:48.440213 18985 tablet_bootstrap.cc:654] T 130007acc8ba4e709c3eb85fabe75903 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:48.439801 18988 consensus_queue.cc:260] T 90a694bd5db649afa50f3c4e752939c1 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
W20240430 20:11:48.441303 18415 consensus_peers.cc:485] T 6d1d2193fd424c91b143f5256df5fcd0 P 2efb542aa55e439b99ccb98d1c5c884a -> Peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Couldn't send request to peer 5e1c6397f09747f091534faf93542710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6d1d2193fd424c91b143f5256df5fcd0. This is attempt 1: this message will repeat every 5th retry.
I20240430 20:11:48.442996 18988 ts_tablet_manager.cc:1398] T 90a694bd5db649afa50f3c4e752939c1 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20240430 20:11:48.443542 18756 raft_consensus.cc:2454] T d7c1fc7e81364fb890bbc13d7fb6848b P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 1.
I20240430 20:11:48.444168 18417 leader_election.cc:302] T d7c1fc7e81364fb890bbc13d7fb6848b P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters:
I20240430 20:11:48.444613 20630 raft_consensus.cc:2781] T d7c1fc7e81364fb890bbc13d7fb6848b P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:48.444756 20630 raft_consensus.cc:686] T d7c1fc7e81364fb890bbc13d7fb6848b P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Becoming Leader. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Running, Role: LEADER
I20240430 20:11:48.445051 20630 consensus_queue.cc:237] T d7c1fc7e81364fb890bbc13d7fb6848b P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:48.445873 18608 raft_consensus.cc:2454] T d7c1fc7e81364fb890bbc13d7fb6848b P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 1.
I20240430 20:11:48.447316 18988 tablet_bootstrap.cc:492] T b285e3eb16d148779c74e057458d8461 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:48.435310 19471 rpcz_store.cc:276] 0430 20:11:44.963908 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:44.964108 (+ 200us) service_pool.cc:225] Handling call
0430 20:11:48.375217 (+3411109us) ts_tablet_manager.cc:1120] Deleting tablet e3e74e76ab2b4fe492ffcc1838656fd1
0430 20:11:48.375227 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:48.385005 (+ 9778us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:48.398140 (+ 13135us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:48.429853 (+ 31713us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3410873}
W20240430 20:11:48.449244 18558 connection.cc:441] server connection from 127.0.0.1:40218 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 371) could send its response
I20240430 20:11:48.449646 19471 tablet_replica.cc:330] T 76d6e9d547ca4a848eae852944b1d117 P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:48.450105 19471 raft_consensus.cc:2229] T 76d6e9d547ca4a848eae852944b1d117 P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Raft consensus shutting down.
I20240430 20:11:48.450771 19471 raft_consensus.cc:2258] T 76d6e9d547ca4a848eae852944b1d117 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:48.451849 19471 ts_tablet_manager.cc:1875] T 76d6e9d547ca4a848eae852944b1d117 P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:48.459508 18988 tablet_bootstrap.cc:654] T b285e3eb16d148779c74e057458d8461 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:48.467891 19471 ts_tablet_manager.cc:1888] T 76d6e9d547ca4a848eae852944b1d117 P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:48.469763 19471 log.cc:1196] T 76d6e9d547ca4a848eae852944b1d117 P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/76d6e9d547ca4a848eae852944b1d117
I20240430 20:11:48.470430 19471 ts_tablet_manager.cc:1909] T 76d6e9d547ca4a848eae852944b1d117 P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:48.472693 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 373) took 3493 ms. Trace:
I20240430 20:11:48.473253 19471 rpcz_store.cc:276] 0430 20:11:44.979642 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:44.979708 (+ 66us) service_pool.cc:225] Handling call
0430 20:11:48.449585 (+3469877us) ts_tablet_manager.cc:1120] Deleting tablet 76d6e9d547ca4a848eae852944b1d117
0430 20:11:48.449598 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:48.455621 (+ 6023us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:48.467769 (+ 12148us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:48.472672 (+ 4903us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3469619}
W20240430 20:11:48.473802 18558 connection.cc:441] server connection from 127.0.0.1:40218 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 373) could send its response
I20240430 20:11:48.474159 19471 tablet_replica.cc:330] T 9f0b341a31224bf1be525b84b9500304 P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:48.474552 19471 raft_consensus.cc:2229] T 9f0b341a31224bf1be525b84b9500304 P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Raft consensus shutting down.
I20240430 20:11:48.478058 20489 consensus_queue.cc:1035] T 331c031d44854d679a27d10d9151d0ea P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:48.487577 18988 tablet_bootstrap.cc:492] T b285e3eb16d148779c74e057458d8461 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:48.487911 18988 ts_tablet_manager.cc:1367] T b285e3eb16d148779c74e057458d8461 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.042s user 0.001s sys 0.006s
I20240430 20:11:48.489383 18988 raft_consensus.cc:348] T b285e3eb16d148779c74e057458d8461 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:48.489957 18988 raft_consensus.cc:374] T b285e3eb16d148779c74e057458d8461 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:48.490152 18988 raft_consensus.cc:729] T b285e3eb16d148779c74e057458d8461 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:48.490653 18988 consensus_queue.cc:260] T b285e3eb16d148779c74e057458d8461 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:48.491760 18988 ts_tablet_manager.cc:1398] T b285e3eb16d148779c74e057458d8461 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20240430 20:11:48.494750 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_650b1b52ac7a453a95434b798c408f6e [id=68e19fd08f164090a84319c51b938b03]
I20240430 20:11:48.495487 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_972f13be4dca4cd19b9a04534dc59f51 [id=69f37db4de634a3185613e40013e0ed7]
I20240430 20:11:48.496050 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_2f6a85489f2e49db8974ff86813c85ee [id=6a249076822e47eda013a4d4f23ab8da]
I20240430 20:11:48.495939 18988 tablet_bootstrap.cc:492] T 8e91920e7e36418d8c27d1f0bfd9b981 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:48.496608 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_5d0995eff94c404fa97b08aa190f4a7f [id=6a55f1d4a8094a88bd8dd5a076630b24]
I20240430 20:11:48.498117 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_5fc77d5f60ae44f4bb4ca9f7f183672c [id=6ca223ca6354449c959d85ed2ac6a9c8]
I20240430 20:11:48.504801 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_3776a4b9d4f24aa6b7f882bb4d0116ef [id=7127dbd368864f7ba80a755549fe7ede]
I20240430 20:11:48.475088 19471 pending_rounds.cc:62] T 9f0b341a31224bf1be525b84b9500304 P 5e1c6397f09747f091534faf93542710: Trying to abort 1 pending ops.
I20240430 20:11:48.507537 19471 pending_rounds.cc:68] T 9f0b341a31224bf1be525b84b9500304 P 5e1c6397f09747f091534faf93542710: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7022624392266657792 op_type: NO_OP noop_request { }
I20240430 20:11:48.507710 19471 raft_consensus.cc:2866] T 9f0b341a31224bf1be525b84b9500304 P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240430 20:11:48.507844 19471 raft_consensus.cc:2258] T 9f0b341a31224bf1be525b84b9500304 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:48.510265 18988 tablet_bootstrap.cc:654] T 8e91920e7e36418d8c27d1f0bfd9b981 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:48.519302 20693 raft_consensus.cc:482] T b1792c160af346ffb4f047c5d4435c10 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:48.519505 19471 ts_tablet_manager.cc:1875] T 9f0b341a31224bf1be525b84b9500304 P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:48.520283 20693 raft_consensus.cc:504] T b1792c160af346ffb4f047c5d4435c10 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:48.521792 18608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1792c160af346ffb4f047c5d4435c10" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:48.522091 18474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1792c160af346ffb4f047c5d4435c10" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
W20240430 20:11:48.522409 18696 leader_election.cc:341] T b1792c160af346ffb4f047c5d4435c10 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Illegal state: must be running to vote when last-logged opid is not known
W20240430 20:11:48.523008 18699 leader_election.cc:341] T b1792c160af346ffb4f047c5d4435c10 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:48.523169 18699 leader_election.cc:302] T b1792c160af346ffb4f047c5d4435c10 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710
I20240430 20:11:48.523784 20695 raft_consensus.cc:2726] T b1792c160af346ffb4f047c5d4435c10 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:48.526100 19471 ts_tablet_manager.cc:1888] T 9f0b341a31224bf1be525b84b9500304 P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:48.536183 18988 tablet_bootstrap.cc:492] T 8e91920e7e36418d8c27d1f0bfd9b981 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:48.536553 18988 ts_tablet_manager.cc:1367] T 8e91920e7e36418d8c27d1f0bfd9b981 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.041s user 0.005s sys 0.002s
I20240430 20:11:48.538328 18988 raft_consensus.cc:348] T 8e91920e7e36418d8c27d1f0bfd9b981 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:48.538805 18988 raft_consensus.cc:374] T 8e91920e7e36418d8c27d1f0bfd9b981 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:48.538985 18988 raft_consensus.cc:729] T 8e91920e7e36418d8c27d1f0bfd9b981 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:48.539351 18988 consensus_queue.cc:260] T 8e91920e7e36418d8c27d1f0bfd9b981 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:48.543869 18988 ts_tablet_manager.cc:1398] T 8e91920e7e36418d8c27d1f0bfd9b981 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20240430 20:11:48.544065 18985 tablet_bootstrap.cc:492] T 130007acc8ba4e709c3eb85fabe75903 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:48.544333 18985 ts_tablet_manager.cc:1367] T 130007acc8ba4e709c3eb85fabe75903 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.123s user 0.003s sys 0.000s
I20240430 20:11:48.531020 18983 tablet_bootstrap.cc:492] T b1792c160af346ffb4f047c5d4435c10 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:48.546495 18988 tablet_bootstrap.cc:492] T fda942d50b014f3d9589e1fb909fc2fc P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:48.546784 18983 ts_tablet_manager.cc:1367] T b1792c160af346ffb4f047c5d4435c10 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.264s user 0.029s sys 0.033s
I20240430 20:11:48.541378 20489 raft_consensus.cc:482] T df92eb85261d47d88cbf16d29d9ad30e P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:48.547082 20489 raft_consensus.cc:504] T df92eb85261d47d88cbf16d29d9ad30e P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:48.547595 18983 raft_consensus.cc:348] T b1792c160af346ffb4f047c5d4435c10 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:48.547794 20489 leader_election.cc:288] T df92eb85261d47d88cbf16d29d9ad30e P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:48.547820 18983 raft_consensus.cc:374] T b1792c160af346ffb4f047c5d4435c10 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:48.547973 18983 raft_consensus.cc:729] T b1792c160af346ffb4f047c5d4435c10 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:48.548218 18983 consensus_queue.cc:260] T b1792c160af346ffb4f047c5d4435c10 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:48.548797 18983 ts_tablet_manager.cc:1398] T b1792c160af346ffb4f047c5d4435c10 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240430 20:11:48.549122 18983 tablet_bootstrap.cc:492] T 1f4e860484784aa9a2d28af2e9fb611c P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:48.550902 18983 tablet_bootstrap.cc:654] T 1f4e860484784aa9a2d28af2e9fb611c P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:48.551321 18750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df92eb85261d47d88cbf16d29d9ad30e" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:48.551815 19471 log.cc:1196] T 9f0b341a31224bf1be525b84b9500304 P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/9f0b341a31224bf1be525b84b9500304
W20240430 20:11:48.551829 18557 leader_election.cc:341] T df92eb85261d47d88cbf16d29d9ad30e P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:48.552290 19471 ts_tablet_manager.cc:1909] T 9f0b341a31224bf1be525b84b9500304 P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:48.552291 18475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df92eb85261d47d88cbf16d29d9ad30e" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:48.552593 18475 raft_consensus.cc:2454] T df92eb85261d47d88cbf16d29d9ad30e P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 0.
I20240430 20:11:48.553011 18558 leader_election.cc:302] T df92eb85261d47d88cbf16d29d9ad30e P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters: 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:48.553154 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 372) took 3573 ms. Trace:
I20240430 20:11:48.553285 19471 rpcz_store.cc:276] 0430 20:11:44.979539 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:44.980496 (+ 957us) service_pool.cc:225] Handling call
0430 20:11:48.474108 (+3493612us) ts_tablet_manager.cc:1120] Deleting tablet 9f0b341a31224bf1be525b84b9500304
0430 20:11:48.474120 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:48.522749 (+ 48629us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:48.525947 (+ 3198us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:48.553109 (+ 27162us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3493428}
I20240430 20:11:48.553485 20488 raft_consensus.cc:2781] T df92eb85261d47d88cbf16d29d9ad30e P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:48.553607 20488 raft_consensus.cc:482] T df92eb85261d47d88cbf16d29d9ad30e P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:48.553700 20488 raft_consensus.cc:3037] T df92eb85261d47d88cbf16d29d9ad30e P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
W20240430 20:11:48.558877 18558 connection.cc:441] server connection from 127.0.0.1:40218 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 372) could send its response
I20240430 20:11:48.559459 18988 tablet_bootstrap.cc:654] T fda942d50b014f3d9589e1fb909fc2fc P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:48.566347 20693 leader_election.cc:288] T b1792c160af346ffb4f047c5d4435c10 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:48.578330 18558 leader_election.cc:302] T e4cffa295d8c401aa8415ce7e8f56526 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters: 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:48.578982 20489 raft_consensus.cc:2741] T e4cffa295d8c401aa8415ce7e8f56526 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20240430 20:11:48.553711 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 386) took 2759 ms. Trace:
I20240430 20:11:48.581738 19471 rpcz_store.cc:276] 0430 20:11:45.794485 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:45.794536 (+ 51us) service_pool.cc:225] Handling call
0430 20:11:48.553578 (+2759042us) ts_tablet_manager.cc:1120] Deleting tablet 50486644ec234933b9a0b45ee39a6685
0430 20:11:48.553588 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:48.553666 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2758793}
W20240430 20:11:48.582145 18558 connection.cc:441] server connection from 127.0.0.1:40218 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 386) could send its response
I20240430 20:11:48.586100 18985 raft_consensus.cc:348] T 130007acc8ba4e709c3eb85fabe75903 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:48.586407 18985 raft_consensus.cc:374] T 130007acc8ba4e709c3eb85fabe75903 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:48.586529 18985 raft_consensus.cc:729] T 130007acc8ba4e709c3eb85fabe75903 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:48.586848 18985 consensus_queue.cc:260] T 130007acc8ba4e709c3eb85fabe75903 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:48.587492 18985 ts_tablet_manager.cc:1398] T 130007acc8ba4e709c3eb85fabe75903 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.039s user 0.002s sys 0.000s
I20240430 20:11:48.587843 18985 tablet_bootstrap.cc:492] T 70247a4c910946c0afc24ce5ce0e85d6 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:48.589365 18985 tablet_bootstrap.cc:654] T 70247a4c910946c0afc24ce5ce0e85d6 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:48.593187 18988 tablet_bootstrap.cc:492] T fda942d50b014f3d9589e1fb909fc2fc P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:48.593545 18988 ts_tablet_manager.cc:1367] T fda942d50b014f3d9589e1fb909fc2fc P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.048s user 0.007s sys 0.001s
I20240430 20:11:48.594985 18988 raft_consensus.cc:348] T fda942d50b014f3d9589e1fb909fc2fc P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:48.595405 18988 raft_consensus.cc:374] T fda942d50b014f3d9589e1fb909fc2fc P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:48.595538 18988 raft_consensus.cc:729] T fda942d50b014f3d9589e1fb909fc2fc P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:48.596006 18988 consensus_queue.cc:260] T fda942d50b014f3d9589e1fb909fc2fc P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:48.598258 18988 ts_tablet_manager.cc:1398] T fda942d50b014f3d9589e1fb909fc2fc P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240430 20:11:48.601895 18988 tablet_bootstrap.cc:492] T 70247a4c910946c0afc24ce5ce0e85d6 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:48.606400 19471 tablet_replica.cc:330] T c6378f1495144c458af49b3c19a2b67c P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:48.606757 19471 raft_consensus.cc:2229] T c6378f1495144c458af49b3c19a2b67c P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:48.606992 19471 raft_consensus.cc:2258] T c6378f1495144c458af49b3c19a2b67c P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:48.607579 19471 ts_tablet_manager.cc:1875] T c6378f1495144c458af49b3c19a2b67c P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:48.612711 18988 tablet_bootstrap.cc:654] T 70247a4c910946c0afc24ce5ce0e85d6 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:48.619158 18983 tablet_bootstrap.cc:492] T 1f4e860484784aa9a2d28af2e9fb611c P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:48.619405 18983 ts_tablet_manager.cc:1367] T 1f4e860484784aa9a2d28af2e9fb611c P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.070s user 0.011s sys 0.013s
I20240430 20:11:48.619519 20384 raft_consensus.cc:482] T 50e1c55a1a0a484984ad5cbb78eec2ed P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:48.619652 20384 raft_consensus.cc:504] T 50e1c55a1a0a484984ad5cbb78eec2ed P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:48.620138 18983 raft_consensus.cc:348] T 1f4e860484784aa9a2d28af2e9fb611c P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:48.620271 20384 leader_election.cc:288] T 50e1c55a1a0a484984ad5cbb78eec2ed P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:48.620339 18983 raft_consensus.cc:374] T 1f4e860484784aa9a2d28af2e9fb611c P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:48.620393 18983 raft_consensus.cc:729] T 1f4e860484784aa9a2d28af2e9fb611c P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:48.620602 18983 consensus_queue.cc:260] T 1f4e860484784aa9a2d28af2e9fb611c P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:48.621948 18475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50e1c55a1a0a484984ad5cbb78eec2ed" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:48.622867 18750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50e1c55a1a0a484984ad5cbb78eec2ed" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:48.621165 18983 ts_tablet_manager.cc:1398] T 1f4e860484784aa9a2d28af2e9fb611c P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
W20240430 20:11:48.622498 18558 leader_election.cc:341] T 50e1c55a1a0a484984ad5cbb78eec2ed P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
W20240430 20:11:48.624326 18557 leader_election.cc:341] T 50e1c55a1a0a484984ad5cbb78eec2ed P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:48.624452 18557 leader_election.cc:302] T 50e1c55a1a0a484984ad5cbb78eec2ed P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:48.624908 20384 raft_consensus.cc:2726] T 50e1c55a1a0a484984ad5cbb78eec2ed P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:48.625208 20695 consensus_queue.cc:1035] T 283e5e73cbc04daba0de19f4541f9fd0 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:48.628407 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_6d9a8c8bd13b42eca6834e707653a66b [id=71c1c0c9a8b544a0a9b1b1a414a6f749]
I20240430 20:11:48.630844 18983 tablet_bootstrap.cc:492] T 90a694bd5db649afa50f3c4e752939c1 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:48.632403 18983 tablet_bootstrap.cc:654] T 90a694bd5db649afa50f3c4e752939c1 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:48.635540 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_1844f6b1bda84812825384e14a2fd94e [id=71da9acc3cff4d95bbac4a686b708fd5]
I20240430 20:11:48.644173 18988 tablet_bootstrap.cc:492] T 70247a4c910946c0afc24ce5ce0e85d6 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:48.644469 18988 ts_tablet_manager.cc:1367] T 70247a4c910946c0afc24ce5ce0e85d6 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.043s user 0.005s sys 0.002s
I20240430 20:11:48.645833 18988 raft_consensus.cc:348] T 70247a4c910946c0afc24ce5ce0e85d6 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:48.646234 18988 raft_consensus.cc:374] T 70247a4c910946c0afc24ce5ce0e85d6 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:48.646396 18988 raft_consensus.cc:729] T 70247a4c910946c0afc24ce5ce0e85d6 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:48.646929 18988 consensus_queue.cc:260] T 70247a4c910946c0afc24ce5ce0e85d6 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:48.648063 18983 tablet_bootstrap.cc:492] T 90a694bd5db649afa50f3c4e752939c1 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:48.648193 18983 ts_tablet_manager.cc:1367] T 90a694bd5db649afa50f3c4e752939c1 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.003s
I20240430 20:11:48.648804 18983 raft_consensus.cc:348] T 90a694bd5db649afa50f3c4e752939c1 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:48.648991 18983 raft_consensus.cc:374] T 90a694bd5db649afa50f3c4e752939c1 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:48.649036 18983 raft_consensus.cc:729] T 90a694bd5db649afa50f3c4e752939c1 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:48.649255 18983 consensus_queue.cc:260] T 90a694bd5db649afa50f3c4e752939c1 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:48.649770 18983 ts_tablet_manager.cc:1398] T 90a694bd5db649afa50f3c4e752939c1 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240430 20:11:48.650977 18983 tablet_bootstrap.cc:492] T 8e91920e7e36418d8c27d1f0bfd9b981 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:48.652904 18983 tablet_bootstrap.cc:654] T 8e91920e7e36418d8c27d1f0bfd9b981 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:48.655130 18988 ts_tablet_manager.cc:1398] T 70247a4c910946c0afc24ce5ce0e85d6 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.010s user 0.000s sys 0.003s
I20240430 20:11:48.658003 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_88c44b20f2ce4eeca7ed9583a70c7c65 [id=73caec5fb65e4cb090e7ae8c391501fd]
I20240430 20:11:48.658500 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_94e5f2a77765434e8a76d9a558b23ae2 [id=73e5c9f0470d47849109819c063e1ce4]
W20240430 20:11:48.659423 18696 consensus_peers.cc:485] T 283e5e73cbc04daba0de19f4541f9fd0 P 3c51c33d5616482cafb68e26688820f4 -> Peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Couldn't send request to peer 5e1c6397f09747f091534faf93542710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 283e5e73cbc04daba0de19f4541f9fd0. This is attempt 1: this message will repeat every 5th retry.
I20240430 20:11:48.667155 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_77b10083620349b0bcbb86e5de19f2ce [id=756ff53a80c842c5b62c972f542a7f34]
I20240430 20:11:48.668310 18988 tablet_bootstrap.cc:492] T dad16ae02b8b4e29a724d8eec67bb4b8 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:48.675705 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_edc4c27ed846454288f73f6a0d88cb69 [id=78fd3f516083489ab3b42d90dcef7e4d]
I20240430 20:11:48.677587 18985 tablet_bootstrap.cc:492] T 70247a4c910946c0afc24ce5ce0e85d6 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:48.677760 18985 ts_tablet_manager.cc:1367] T 70247a4c910946c0afc24ce5ce0e85d6 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.090s user 0.001s sys 0.001s
I20240430 20:11:48.678416 18985 raft_consensus.cc:348] T 70247a4c910946c0afc24ce5ce0e85d6 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:48.678643 18985 raft_consensus.cc:374] T 70247a4c910946c0afc24ce5ce0e85d6 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:48.678702 18985 raft_consensus.cc:729] T 70247a4c910946c0afc24ce5ce0e85d6 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:48.678918 18985 consensus_queue.cc:260] T 70247a4c910946c0afc24ce5ce0e85d6 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:48.680543 18988 tablet_bootstrap.cc:654] T dad16ae02b8b4e29a724d8eec67bb4b8 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:48.681136 18983 tablet_bootstrap.cc:492] T 8e91920e7e36418d8c27d1f0bfd9b981 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:48.681288 18983 ts_tablet_manager.cc:1367] T 8e91920e7e36418d8c27d1f0bfd9b981 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.030s user 0.001s sys 0.001s
I20240430 20:11:48.682317 20489 raft_consensus.cc:482] T 488d0e609b224674932f890939664df2 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:48.682507 20489 raft_consensus.cc:504] T 488d0e609b224674932f890939664df2 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:48.683271 20489 leader_election.cc:288] T 488d0e609b224674932f890939664df2 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:48.684901 20587 raft_consensus.cc:482] T 78ba48dadef548708153878810cc46f8 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:48.685133 20587 raft_consensus.cc:504] T 78ba48dadef548708153878810cc46f8 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:48.685946 20587 leader_election.cc:288] T 78ba48dadef548708153878810cc46f8 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:48.686744 18983 raft_consensus.cc:348] T 8e91920e7e36418d8c27d1f0bfd9b981 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:48.686954 18983 raft_consensus.cc:374] T 8e91920e7e36418d8c27d1f0bfd9b981 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:48.686918 18608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ba48dadef548708153878810cc46f8" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:48.687002 18983 raft_consensus.cc:729] T 8e91920e7e36418d8c27d1f0bfd9b981 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:48.687108 18608 raft_consensus.cc:2379] T 78ba48dadef548708153878810cc46f8 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c51c33d5616482cafb68e26688820f4 in current term 1: Already voted for candidate 2efb542aa55e439b99ccb98d1c5c884a in this term.
I20240430 20:11:48.687192 18983 consensus_queue.cc:260] T 8e91920e7e36418d8c27d1f0bfd9b981 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:48.687736 18983 ts_tablet_manager.cc:1398] T 8e91920e7e36418d8c27d1f0bfd9b981 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240430 20:11:48.687803 20488 raft_consensus.cc:504] T df92eb85261d47d88cbf16d29d9ad30e P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:48.684901 18750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "488d0e609b224674932f890939664df2" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:48.690485 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_4991bbcb43c540629cbc285f0484d79d [id=79162f889e6340248053e0814ebb387a]
I20240430 20:11:48.693107 18473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "488d0e609b224674932f890939664df2" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:48.693663 18985 ts_tablet_manager.cc:1398] T 70247a4c910946c0afc24ce5ce0e85d6 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.016s user 0.000s sys 0.002s
I20240430 20:11:48.693727 18474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ba48dadef548708153878810cc46f8" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:48.695014 18699 leader_election.cc:302] T 78ba48dadef548708153878810cc46f8 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710
I20240430 20:11:48.695485 20587 raft_consensus.cc:3037] T 78ba48dadef548708153878810cc46f8 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:48.695694 18985 tablet_bootstrap.cc:492] T 9c73d0a6bc414d3b9900570848d19a51 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:48.698374 20587 raft_consensus.cc:2726] T 78ba48dadef548708153878810cc46f8 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:48.706168 18985 tablet_bootstrap.cc:654] T 9c73d0a6bc414d3b9900570848d19a51 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:48.709805 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_4440df38610340758047115845b42b95 [id=7d186ca20c094cc19aef792fe1c094e9]
I20240430 20:11:48.711943 18985 tablet_bootstrap.cc:492] T 9c73d0a6bc414d3b9900570848d19a51 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:48.712715 18985 ts_tablet_manager.cc:1367] T 9c73d0a6bc414d3b9900570848d19a51 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.002s
I20240430 20:11:48.713740 18985 raft_consensus.cc:348] T 9c73d0a6bc414d3b9900570848d19a51 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:48.714908 18985 raft_consensus.cc:374] T 9c73d0a6bc414d3b9900570848d19a51 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:48.715147 18985 raft_consensus.cc:729] T 9c73d0a6bc414d3b9900570848d19a51 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:48.715613 18985 consensus_queue.cc:260] T 9c73d0a6bc414d3b9900570848d19a51 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:48.717435 18985 ts_tablet_manager.cc:1398] T 9c73d0a6bc414d3b9900570848d19a51 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240430 20:11:48.719058 18985 tablet_bootstrap.cc:492] T 2a08e36c39824abf98b9b6214c8b84cb P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:48.721423 18985 tablet_bootstrap.cc:654] T 2a08e36c39824abf98b9b6214c8b84cb P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:48.731765 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_02d2b1f3815549c481a305c622c809ff [id=7e61baf429e4487fab0ba2761a2af49d]
I20240430 20:11:48.735008 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_0163ce6f5a4c45299106daeffb3c22b1 [id=7f3a15a442984827a7521d5d1c98dc3d]
W20240430 20:11:48.739828 18211 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.177.62:36441, user_credentials={real_user=slave}} blocked reactor thread for 57904.1us
I20240430 20:11:48.688643 18983 tablet_bootstrap.cc:492] T 70247a4c910946c0afc24ce5ce0e85d6 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:48.740593 20545 raft_consensus.cc:482] T 130007acc8ba4e709c3eb85fabe75903 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:48.740741 20545 raft_consensus.cc:504] T 130007acc8ba4e709c3eb85fabe75903 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:48.740917 20488 leader_election.cc:288] T df92eb85261d47d88cbf16d29d9ad30e P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:48.741319 20545 leader_election.cc:288] T 130007acc8ba4e709c3eb85fabe75903 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:48.741914 18608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "130007acc8ba4e709c3eb85fabe75903" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:48.742069 18983 tablet_bootstrap.cc:654] T 70247a4c910946c0afc24ce5ce0e85d6 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:48.742143 18608 raft_consensus.cc:2379] T 130007acc8ba4e709c3eb85fabe75903 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2efb542aa55e439b99ccb98d1c5c884a in current term 1: Already voted for candidate 3c51c33d5616482cafb68e26688820f4 in this term.
I20240430 20:11:48.743419 18750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "130007acc8ba4e709c3eb85fabe75903" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:48.744163 18417 leader_election.cc:302] T 130007acc8ba4e709c3eb85fabe75903 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a; no voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710
I20240430 20:11:48.690340 19471 ts_tablet_manager.cc:1888] T c6378f1495144c458af49b3c19a2b67c P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20240430 20:11:48.691166 18557 leader_election.cc:341] T 488d0e609b224674932f890939664df2 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
W20240430 20:11:48.744421 18557 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.177.3:44867, user_credentials={real_user=slave}} blocked reactor thread for 53261.9us
W20240430 20:11:48.744717 18558 leader_election.cc:341] T 488d0e609b224674932f890939664df2 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:48.744779 20545 raft_consensus.cc:3037] T 130007acc8ba4e709c3eb85fabe75903 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:48.744966 18558 leader_election.cc:302] T 488d0e609b224674932f890939664df2 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4
W20240430 20:11:48.745276 18558 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.177.1:41217, user_credentials={real_user=slave}} blocked reactor thread for 50714.3us
I20240430 20:11:48.745785 20489 raft_consensus.cc:2726] T 488d0e609b224674932f890939664df2 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:48.746313 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_0b4ca4cd0af148399dfa21f2d32114e3 [id=7f602584cef9486eb7d9b4389be3f43e]
I20240430 20:11:48.746580 18750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df92eb85261d47d88cbf16d29d9ad30e" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:48.747133 19471 log.cc:1196] T c6378f1495144c458af49b3c19a2b67c P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/c6378f1495144c458af49b3c19a2b67c
I20240430 20:11:48.747534 20545 raft_consensus.cc:2726] T 130007acc8ba4e709c3eb85fabe75903 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240430 20:11:48.747516 18557 leader_election.cc:341] T df92eb85261d47d88cbf16d29d9ad30e P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:48.748183 18988 tablet_bootstrap.cc:492] T dad16ae02b8b4e29a724d8eec67bb4b8 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:48.748692 18988 ts_tablet_manager.cc:1367] T dad16ae02b8b4e29a724d8eec67bb4b8 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.091s user 0.002s sys 0.005s
I20240430 20:11:48.747530 19471 ts_tablet_manager.cc:1909] T c6378f1495144c458af49b3c19a2b67c P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:48.747185 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_3c8f9f7b22f64fb1bdc3f1550987233a [id=7fd6f8576f4c4cbc8ffcdb4ab95e15dc]
I20240430 20:11:48.753434 18988 raft_consensus.cc:348] T dad16ae02b8b4e29a724d8eec67bb4b8 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:48.754494 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 387) took 2941 ms. Trace:
I20240430 20:11:48.755623 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_f1ad29d5986d4b799cfc67afb2b0dfdf [id=838fc8b61a864dea8877a6f144e4cb4f]
I20240430 20:11:48.755643 18475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df92eb85261d47d88cbf16d29d9ad30e" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:48.755884 18475 raft_consensus.cc:3037] T df92eb85261d47d88cbf16d29d9ad30e P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:48.756438 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_744b5c96ea014bdb8feae740356a790e [id=83b1ac2e9c3d4a10b02f96ebb17faf44]
I20240430 20:11:48.758034 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_1280cfe32d5b41abb4d3eb1bd70641bf [id=8489b872074a42b6abb713e420968141]
I20240430 20:11:48.758328 18985 tablet_bootstrap.cc:492] T 2a08e36c39824abf98b9b6214c8b84cb P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:48.758576 18985 ts_tablet_manager.cc:1367] T 2a08e36c39824abf98b9b6214c8b84cb P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.041s user 0.004s sys 0.012s
I20240430 20:11:48.759004 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_a2e85efc6b454524b0b122102ec07002 [id=86d795a82e674d859104741ebd04f4e8]
I20240430 20:11:48.759449 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_07e40af8f1c546048975c8edf88170d7 [id=86df5d161d8641c89070e7eed9a39360]
I20240430 20:11:48.760015 18985 raft_consensus.cc:348] T 2a08e36c39824abf98b9b6214c8b84cb P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:48.760236 18475 raft_consensus.cc:2454] T df92eb85261d47d88cbf16d29d9ad30e P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
I20240430 20:11:48.760913 18558 leader_election.cc:302] T df92eb85261d47d88cbf16d29d9ad30e P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters: 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:48.761624 18985 raft_consensus.cc:374] T 2a08e36c39824abf98b9b6214c8b84cb P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:48.761777 20488 raft_consensus.cc:2781] T df92eb85261d47d88cbf16d29d9ad30e P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:48.761812 18985 raft_consensus.cc:729] T 2a08e36c39824abf98b9b6214c8b84cb P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:48.762271 20488 raft_consensus.cc:686] T df92eb85261d47d88cbf16d29d9ad30e P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Becoming Leader. State: Replica: 5e1c6397f09747f091534faf93542710, State: Running, Role: LEADER
I20240430 20:11:48.762269 18985 consensus_queue.cc:260] T 2a08e36c39824abf98b9b6214c8b84cb P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:48.762908 20488 consensus_queue.cc:237] T df92eb85261d47d88cbf16d29d9ad30e P 5e1c6397f09747f091534faf93542710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
W20240430 20:11:48.740196 20795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
I20240430 20:11:48.763801 18985 ts_tablet_manager.cc:1398] T 2a08e36c39824abf98b9b6214c8b84cb P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
W20240430 20:11:48.763931 20795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240430 20:11:48.764358 20795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240430 20:11:48.764966 19471 rpcz_store.cc:276] 0430 20:11:45.812671 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:45.812728 (+ 57us) service_pool.cc:225] Handling call
0430 20:11:48.582423 (+2769695us) ts_tablet_manager.cc:1120] Deleting tablet c6378f1495144c458af49b3c19a2b67c
0430 20:11:48.605968 (+ 23545us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:48.661095 (+ 55127us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:48.690222 (+ 29127us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:48.754235 (+ 64013us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2769499}
I20240430 20:11:48.761098 18988 raft_consensus.cc:374] T dad16ae02b8b4e29a724d8eec67bb4b8 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:48.766793 18985 tablet_bootstrap.cc:492] T 6db55bb510b24b0cae897936211e15f0 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:48.766479 18988 raft_consensus.cc:729] T dad16ae02b8b4e29a724d8eec67bb4b8 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:48.765129 18983 tablet_bootstrap.cc:492] T 70247a4c910946c0afc24ce5ce0e85d6 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:48.767645 18983 ts_tablet_manager.cc:1367] T 70247a4c910946c0afc24ce5ce0e85d6 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.079s user 0.008s sys 0.003s
I20240430 20:11:48.769167 18983 raft_consensus.cc:348] T 70247a4c910946c0afc24ce5ce0e85d6 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:48.769582 18983 raft_consensus.cc:374] T 70247a4c910946c0afc24ce5ce0e85d6 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:48.769680 18983 raft_consensus.cc:729] T 70247a4c910946c0afc24ce5ce0e85d6 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:48.770114 18985 tablet_bootstrap.cc:654] T 6db55bb510b24b0cae897936211e15f0 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:48.770099 18983 consensus_queue.cc:260] T 70247a4c910946c0afc24ce5ce0e85d6 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:48.772262 18983 ts_tablet_manager.cc:1398] T 70247a4c910946c0afc24ce5ce0e85d6 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240430 20:11:48.773357 18983 tablet_bootstrap.cc:492] T fda942d50b014f3d9589e1fb909fc2fc P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:48.773473 20587 consensus_queue.cc:1035] T 53da4b3855344b6d92211af767195dc8 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240430 20:11:48.776633 18558 connection.cc:441] server connection from 127.0.0.1:40218 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 387) could send its response
I20240430 20:11:48.776958 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_57208e9f76f14782a806f0a2971a3a58 [id=872c2faa4a464edd98a51df2e1b3f817]
I20240430 20:11:48.777148 19471 tablet_replica.cc:330] T df92eb85261d47d88cbf16d29d9ad30e P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:48.780006 18988 consensus_queue.cc:260] T dad16ae02b8b4e29a724d8eec67bb4b8 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:48.787086 20587 consensus_queue.cc:1035] T 885d27d4f8df48dd996d5b309b783f3b P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240430 20:11:48.789602 20795 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240430 20:11:48.789850 20795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240430 20:11:48.789994 20795 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240430 20:11:48.790160 20795 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240430 20:11:48.797935 18988 ts_tablet_manager.cc:1398] T dad16ae02b8b4e29a724d8eec67bb4b8 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.049s user 0.004s sys 0.000s
I20240430 20:11:48.804034 20693 consensus_queue.cc:1035] T 885d27d4f8df48dd996d5b309b783f3b P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20240430 20:11:48.808257 18988 tablet_bootstrap.cc:492] T 2a08e36c39824abf98b9b6214c8b84cb P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:48.813899 18983 tablet_bootstrap.cc:654] T fda942d50b014f3d9589e1fb909fc2fc P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:48.821476 19471 raft_consensus.cc:2229] T df92eb85261d47d88cbf16d29d9ad30e P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Raft consensus shutting down.
I20240430 20:11:48.822302 19471 pending_rounds.cc:62] T df92eb85261d47d88cbf16d29d9ad30e P 5e1c6397f09747f091534faf93542710: Trying to abort 1 pending ops.
I20240430 20:11:48.823387 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_fac4c5c0ce954200838069be87be2ad9 [id=873474f62b8243e4b9068504b9f189a4]
I20240430 20:11:48.823891 19471 pending_rounds.cc:68] T df92eb85261d47d88cbf16d29d9ad30e P 5e1c6397f09747f091534faf93542710: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7022624394298671104 op_type: NO_OP noop_request { }
I20240430 20:11:48.825567 19471 raft_consensus.cc:2866] T df92eb85261d47d88cbf16d29d9ad30e P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240430 20:11:48.825771 19471 raft_consensus.cc:2258] T df92eb85261d47d88cbf16d29d9ad30e P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:48.826225 20795 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.177.20:42079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/master-2/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.17.177.62:36441,127.17.177.61:34777,127.17.177.60:37225
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.177.60:37225
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.17.177.60
--webserver_port=34267
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision ad3936521af034ffcac637f97cd8c932f6289b4f
build type FASTDEBUG
built by None at 30 Apr 2024 19:43:22 UTC on efd1337a2aa3
build id 63
ASAN enabled
I20240430 20:11:48.827358 19471 ts_tablet_manager.cc:1875] T df92eb85261d47d88cbf16d29d9ad30e P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:48.828071 20795 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240430 20:11:48.828343 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_dfb59f35018441fd9a72ea860cdd01f6 [id=880609dd541c4ce5af4d27b5a2e0c9a7]
I20240430 20:11:48.829244 18988 tablet_bootstrap.cc:654] T 2a08e36c39824abf98b9b6214c8b84cb P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:48.829710 20545 consensus_queue.cc:1035] T 78ba48dadef548708153878810cc46f8 P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:48.832973 20695 raft_consensus.cc:482] T 90a694bd5db649afa50f3c4e752939c1 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:48.833156 20695 raft_consensus.cc:504] T 90a694bd5db649afa50f3c4e752939c1 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:48.833828 20695 leader_election.cc:288] T 90a694bd5db649afa50f3c4e752939c1 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:48.834412 18474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90a694bd5db649afa50f3c4e752939c1" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:48.835161 18623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90a694bd5db649afa50f3c4e752939c1" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:48.835403 18623 raft_consensus.cc:2454] T 90a694bd5db649afa50f3c4e752939c1 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:48.837687 20795 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240430 20:11:48.837706 18699 leader_election.cc:341] T 90a694bd5db649afa50f3c4e752939c1 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:48.837915 18699 leader_election.cc:302] T 90a694bd5db649afa50f3c4e752939c1 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a
I20240430 20:11:48.838460 20695 raft_consensus.cc:2781] T 90a694bd5db649afa50f3c4e752939c1 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:48.838583 20695 raft_consensus.cc:482] T 90a694bd5db649afa50f3c4e752939c1 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:48.838694 20695 raft_consensus.cc:3037] T 90a694bd5db649afa50f3c4e752939c1 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:48.841732 20695 raft_consensus.cc:504] T 90a694bd5db649afa50f3c4e752939c1 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:48.842949 20695 leader_election.cc:288] T 90a694bd5db649afa50f3c4e752939c1 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Requested vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:48.842967 18474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90a694bd5db649afa50f3c4e752939c1" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
W20240430 20:11:48.843705 18699 leader_election.cc:341] T 90a694bd5db649afa50f3c4e752939c1 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:48.844170 20695 raft_consensus.cc:482] T 50486644ec234933b9a0b45ee39a6685 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:48.844111 18623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90a694bd5db649afa50f3c4e752939c1" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:48.844302 18623 raft_consensus.cc:3037] T 90a694bd5db649afa50f3c4e752939c1 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:48.844339 20695 raft_consensus.cc:504] T 50486644ec234933b9a0b45ee39a6685 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:48.845082 20695 leader_election.cc:288] T 50486644ec234933b9a0b45ee39a6685 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
W20240430 20:11:48.849319 18696 leader_election.cc:341] T 50486644ec234933b9a0b45ee39a6685 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Not found: Tablet not found: 50486644ec234933b9a0b45ee39a6685
W20240430 20:11:48.849694 18699 leader_election.cc:341] T 50486644ec234933b9a0b45ee39a6685 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Not found: Tablet not found: 50486644ec234933b9a0b45ee39a6685
I20240430 20:11:48.849812 18699 leader_election.cc:302] T 50486644ec234933b9a0b45ee39a6685 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710
I20240430 20:11:48.850145 20693 raft_consensus.cc:2726] T 50486644ec234933b9a0b45ee39a6685 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240430 20:11:48.850737 18623 raft_consensus.cc:2454] T 90a694bd5db649afa50f3c4e752939c1 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:48.851305 18696 leader_election.cc:302] T 90a694bd5db649afa50f3c4e752939c1 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a
I20240430 20:11:48.851905 20693 raft_consensus.cc:2781] T 90a694bd5db649afa50f3c4e752939c1 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:48.852030 20693 raft_consensus.cc:686] T 90a694bd5db649afa50f3c4e752939c1 P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Becoming Leader. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Running, Role: LEADER
I20240430 20:11:48.852295 20693 consensus_queue.cc:237] T 90a694bd5db649afa50f3c4e752939c1 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:48.853560 20384 raft_consensus.cc:482] T 8e91920e7e36418d8c27d1f0bfd9b981 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:48.853708 20384 raft_consensus.cc:504] T 8e91920e7e36418d8c27d1f0bfd9b981 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:48.853621 18983 tablet_bootstrap.cc:492] T fda942d50b014f3d9589e1fb909fc2fc P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:48.853899 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_47779624a66e46468c8e9e35a8e9487f [id=8ce80df659a146c9b7d5fdb3059902cd]
I20240430 20:11:48.854341 20384 leader_election.cc:288] T 8e91920e7e36418d8c27d1f0bfd9b981 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:48.854377 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_c6c962da746e4fd8a4d7200d09f08e9e [id=8e0f5d8c941d423ea7b30a89513d5170]
I20240430 20:11:48.854267 18983 ts_tablet_manager.cc:1367] T fda942d50b014f3d9589e1fb909fc2fc P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.081s user 0.006s sys 0.000s
I20240430 20:11:48.854952 18750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e91920e7e36418d8c27d1f0bfd9b981" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:48.855171 18750 raft_consensus.cc:2454] T 8e91920e7e36418d8c27d1f0bfd9b981 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 0.
I20240430 20:11:48.855990 18557 leader_election.cc:302] T 8e91920e7e36418d8c27d1f0bfd9b981 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:48.856637 20384 raft_consensus.cc:2781] T 8e91920e7e36418d8c27d1f0bfd9b981 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:48.856740 20384 raft_consensus.cc:482] T 8e91920e7e36418d8c27d1f0bfd9b981 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:48.856814 20384 raft_consensus.cc:3037] T 8e91920e7e36418d8c27d1f0bfd9b981 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:48.858878 18474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e91920e7e36418d8c27d1f0bfd9b981" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
W20240430 20:11:48.859561 18558 leader_election.cc:341] T 8e91920e7e36418d8c27d1f0bfd9b981 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:48.860882 20384 raft_consensus.cc:504] T 8e91920e7e36418d8c27d1f0bfd9b981 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:48.861246 18988 tablet_bootstrap.cc:492] T 2a08e36c39824abf98b9b6214c8b84cb P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:48.860633 18983 raft_consensus.cc:348] T fda942d50b014f3d9589e1fb909fc2fc P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:48.862222 18750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e91920e7e36418d8c27d1f0bfd9b981" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:48.862481 18750 raft_consensus.cc:3037] T 8e91920e7e36418d8c27d1f0bfd9b981 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:48.862459 18983 raft_consensus.cc:374] T fda942d50b014f3d9589e1fb909fc2fc P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:48.863329 18988 ts_tablet_manager.cc:1367] T 2a08e36c39824abf98b9b6214c8b84cb P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.057s user 0.004s sys 0.003s
I20240430 20:11:48.864378 18983 raft_consensus.cc:729] T fda942d50b014f3d9589e1fb909fc2fc P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:48.865862 18750 raft_consensus.cc:2454] T 8e91920e7e36418d8c27d1f0bfd9b981 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
I20240430 20:11:48.866441 18557 leader_election.cc:302] T 8e91920e7e36418d8c27d1f0bfd9b981 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:48.866789 20488 raft_consensus.cc:2781] T 8e91920e7e36418d8c27d1f0bfd9b981 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:48.866904 20488 raft_consensus.cc:686] T 8e91920e7e36418d8c27d1f0bfd9b981 P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Becoming Leader. State: Replica: 5e1c6397f09747f091534faf93542710, State: Running, Role: LEADER
I20240430 20:11:48.867111 20384 leader_election.cc:288] T 8e91920e7e36418d8c27d1f0bfd9b981 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:48.867151 20488 consensus_queue.cc:237] T 8e91920e7e36418d8c27d1f0bfd9b981 P 5e1c6397f09747f091534faf93542710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:48.867614 18474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e91920e7e36418d8c27d1f0bfd9b981" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
W20240430 20:11:48.868441 18558 leader_election.cc:341] T 8e91920e7e36418d8c27d1f0bfd9b981 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:48.869199 18983 consensus_queue.cc:260] T fda942d50b014f3d9589e1fb909fc2fc P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:48.877628 18983 ts_tablet_manager.cc:1398] T fda942d50b014f3d9589e1fb909fc2fc P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.023s user 0.004s sys 0.000s
I20240430 20:11:48.883167 20629 consensus_queue.cc:1035] T 78ba48dadef548708153878810cc46f8 P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:48.906944 20629 raft_consensus.cc:482] T ed0c37a71ede4bfc9eb5ad1a4365f25e P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:48.907150 20629 raft_consensus.cc:504] T ed0c37a71ede4bfc9eb5ad1a4365f25e P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:48.907941 20629 leader_election.cc:288] T ed0c37a71ede4bfc9eb5ad1a4365f25e P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:48.907918 20693 consensus_queue.cc:1035] T 53da4b3855344b6d92211af767195dc8 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:48.913573 18985 tablet_bootstrap.cc:492] T 6db55bb510b24b0cae897936211e15f0 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:48.899187 18983 tablet_bootstrap.cc:492] T 12795670f21447a8964fe1efc6e92398 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:48.909752 18750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed0c37a71ede4bfc9eb5ad1a4365f25e" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
W20240430 20:11:48.914556 18417 leader_election.cc:341] T ed0c37a71ede4bfc9eb5ad1a4365f25e P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:48.910883 18608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed0c37a71ede4bfc9eb5ad1a4365f25e" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:48.880484 18988 raft_consensus.cc:348] T 2a08e36c39824abf98b9b6214c8b84cb P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
W20240430 20:11:48.915266 18415 leader_election.cc:341] T ed0c37a71ede4bfc9eb5ad1a4365f25e P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:48.915426 18415 leader_election.cc:302] T ed0c37a71ede4bfc9eb5ad1a4365f25e P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a; no voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710
I20240430 20:11:48.915752 20630 raft_consensus.cc:2726] T ed0c37a71ede4bfc9eb5ad1a4365f25e P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:48.916095 18988 raft_consensus.cc:374] T 2a08e36c39824abf98b9b6214c8b84cb P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:48.919620 18985 ts_tablet_manager.cc:1367] T 6db55bb510b24b0cae897936211e15f0 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.153s user 0.000s sys 0.011s
I20240430 20:11:48.920297 20630 raft_consensus.cc:482] T dad16ae02b8b4e29a724d8eec67bb4b8 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:48.920635 20630 raft_consensus.cc:504] T dad16ae02b8b4e29a724d8eec67bb4b8 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:48.922097 18750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dad16ae02b8b4e29a724d8eec67bb4b8" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:48.922402 18750 raft_consensus.cc:2454] T dad16ae02b8b4e29a724d8eec67bb4b8 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 0.
I20240430 20:11:48.923032 18417 leader_election.cc:302] T dad16ae02b8b4e29a724d8eec67bb4b8 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters:
I20240430 20:11:48.924031 18619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dad16ae02b8b4e29a724d8eec67bb4b8" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:48.924078 20630 leader_election.cc:288] T dad16ae02b8b4e29a724d8eec67bb4b8 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:48.918936 18988 raft_consensus.cc:729] T 2a08e36c39824abf98b9b6214c8b84cb P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
W20240430 20:11:48.925246 18415 leader_election.cc:341] T dad16ae02b8b4e29a724d8eec67bb4b8 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:48.926648 20630 raft_consensus.cc:2781] T dad16ae02b8b4e29a724d8eec67bb4b8 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:48.926775 20630 raft_consensus.cc:482] T dad16ae02b8b4e29a724d8eec67bb4b8 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:48.926869 20630 raft_consensus.cc:3037] T dad16ae02b8b4e29a724d8eec67bb4b8 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:48.928469 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_cf60b652e9dc49e7a948ce66fd1cbffe [id=9734bb8dd05a400ca4244b3829f01dee]
I20240430 20:11:48.929409 18985 raft_consensus.cc:348] T 6db55bb510b24b0cae897936211e15f0 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:48.931314 18985 raft_consensus.cc:374] T 6db55bb510b24b0cae897936211e15f0 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:48.931917 20384 raft_consensus.cc:482] T 6db55bb510b24b0cae897936211e15f0 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:48.931733 18985 raft_consensus.cc:729] T 6db55bb510b24b0cae897936211e15f0 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:48.932049 20384 raft_consensus.cc:504] T 6db55bb510b24b0cae897936211e15f0 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:48.933795 18475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6db55bb510b24b0cae897936211e15f0" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:48.932950 18985 consensus_queue.cc:260] T 6db55bb510b24b0cae897936211e15f0 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:48.934106 20384 leader_election.cc:288] T 6db55bb510b24b0cae897936211e15f0 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:48.934235 18475 raft_consensus.cc:2454] T 6db55bb510b24b0cae897936211e15f0 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 0.
I20240430 20:11:48.934685 18558 leader_election.cc:302] T 6db55bb510b24b0cae897936211e15f0 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:48.934958 20384 raft_consensus.cc:2781] T 6db55bb510b24b0cae897936211e15f0 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:48.935124 20384 raft_consensus.cc:482] T 6db55bb510b24b0cae897936211e15f0 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:48.935247 20384 raft_consensus.cc:3037] T 6db55bb510b24b0cae897936211e15f0 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:48.936446 18985 ts_tablet_manager.cc:1398] T 6db55bb510b24b0cae897936211e15f0 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.016s user 0.000s sys 0.003s
I20240430 20:11:48.934489 18750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6db55bb510b24b0cae897936211e15f0" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
W20240430 20:11:48.937314 18557 leader_election.cc:341] T 6db55bb510b24b0cae897936211e15f0 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:48.939105 18985 tablet_bootstrap.cc:492] T 8e91920e7e36418d8c27d1f0bfd9b981 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:48.939195 20384 raft_consensus.cc:504] T 6db55bb510b24b0cae897936211e15f0 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:48.933154 18988 consensus_queue.cc:260] T 2a08e36c39824abf98b9b6214c8b84cb P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:48.941673 18985 tablet_bootstrap.cc:654] T 8e91920e7e36418d8c27d1f0bfd9b981 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:48.941982 18750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6db55bb510b24b0cae897936211e15f0" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
W20240430 20:11:48.942560 18557 leader_election.cc:341] T 6db55bb510b24b0cae897936211e15f0 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:48.943389 18475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6db55bb510b24b0cae897936211e15f0" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:48.943608 18475 raft_consensus.cc:3037] T 6db55bb510b24b0cae897936211e15f0 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:48.946287 18475 raft_consensus.cc:2454] T 6db55bb510b24b0cae897936211e15f0 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
I20240430 20:11:48.947302 18558 leader_election.cc:302] T 6db55bb510b24b0cae897936211e15f0 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters: 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:48.947602 20489 raft_consensus.cc:2781] T 6db55bb510b24b0cae897936211e15f0 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:48.947731 20489 raft_consensus.cc:686] T 6db55bb510b24b0cae897936211e15f0 P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Becoming Leader. State: Replica: 5e1c6397f09747f091534faf93542710, State: Running, Role: LEADER
I20240430 20:11:48.947993 20489 consensus_queue.cc:237] T 6db55bb510b24b0cae897936211e15f0 P 5e1c6397f09747f091534faf93542710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:48.930527 20630 raft_consensus.cc:504] T dad16ae02b8b4e29a724d8eec67bb4b8 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:48.953699 20384 leader_election.cc:288] T 6db55bb510b24b0cae897936211e15f0 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:48.956158 18750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dad16ae02b8b4e29a724d8eec67bb4b8" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:48.956449 18750 raft_consensus.cc:3037] T dad16ae02b8b4e29a724d8eec67bb4b8 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:48.960295 18750 raft_consensus.cc:2454] T dad16ae02b8b4e29a724d8eec67bb4b8 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 1.
I20240430 20:11:48.960969 18417 leader_election.cc:302] T dad16ae02b8b4e29a724d8eec67bb4b8 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters:
I20240430 20:11:48.961364 20629 raft_consensus.cc:2781] T dad16ae02b8b4e29a724d8eec67bb4b8 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:48.961555 20629 raft_consensus.cc:686] T dad16ae02b8b4e29a724d8eec67bb4b8 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Becoming Leader. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Running, Role: LEADER
I20240430 20:11:48.961889 20629 consensus_queue.cc:237] T dad16ae02b8b4e29a724d8eec67bb4b8 P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:48.977108 18983 tablet_bootstrap.cc:654] T 12795670f21447a8964fe1efc6e92398 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:48.977313 18988 ts_tablet_manager.cc:1398] T 2a08e36c39824abf98b9b6214c8b84cb P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.089s user 0.003s sys 0.002s
I20240430 20:11:48.988152 18988 tablet_bootstrap.cc:492] T 47e98935eb21449abd3d4d24327778d5 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:48.992229 20630 leader_election.cc:288] T dad16ae02b8b4e29a724d8eec67bb4b8 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:48.998273 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_01d62e066e4942d9aa1f582189d60d49 [id=97861b0b1ce94638b62bbd123d774d7f]
I20240430 20:11:48.986359 18620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dad16ae02b8b4e29a724d8eec67bb4b8" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
W20240430 20:11:49.003713 18415 leader_election.cc:341] T dad16ae02b8b4e29a724d8eec67bb4b8 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:49.018873 18988 tablet_bootstrap.cc:654] T 47e98935eb21449abd3d4d24327778d5 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:49.023989 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_dfbb6c73df814ce49886f7b8f0e0968d [id=9b01fa1f94b24da799aed1617be3c892]
I20240430 20:11:49.033998 19471 ts_tablet_manager.cc:1888] T df92eb85261d47d88cbf16d29d9ad30e P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:49.034261 19471 log.cc:1196] T df92eb85261d47d88cbf16d29d9ad30e P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/df92eb85261d47d88cbf16d29d9ad30e
I20240430 20:11:49.034085 18983 tablet_bootstrap.cc:492] T 12795670f21447a8964fe1efc6e92398 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
W20240430 20:11:49.035665 20910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240430 20:11:49.035898 18983 ts_tablet_manager.cc:1367] T 12795670f21447a8964fe1efc6e92398 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.155s user 0.007s sys 0.000s
I20240430 20:11:49.036859 18983 raft_consensus.cc:348] T 12795670f21447a8964fe1efc6e92398 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.037154 18983 raft_consensus.cc:374] T 12795670f21447a8964fe1efc6e92398 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:49.037225 18983 raft_consensus.cc:729] T 12795670f21447a8964fe1efc6e92398 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:49.037495 18983 consensus_queue.cc:260] T 12795670f21447a8964fe1efc6e92398 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.038233 18983 ts_tablet_manager.cc:1398] T 12795670f21447a8964fe1efc6e92398 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240430 20:11:49.040836 20198 raft_consensus.cc:482] T fda942d50b014f3d9589e1fb909fc2fc P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:49.041062 20198 raft_consensus.cc:504] T fda942d50b014f3d9589e1fb909fc2fc P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.041996 20198 leader_election.cc:288] T fda942d50b014f3d9589e1fb909fc2fc P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:49.042773 18620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fda942d50b014f3d9589e1fb909fc2fc" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:49.042775 18475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fda942d50b014f3d9589e1fb909fc2fc" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:49.043040 18620 raft_consensus.cc:2454] T fda942d50b014f3d9589e1fb909fc2fc P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:49.043607 18696 leader_election.cc:302] T fda942d50b014f3d9589e1fb909fc2fc P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
W20240430 20:11:49.044363 18699 leader_election.cc:341] T fda942d50b014f3d9589e1fb909fc2fc P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:49.044888 20198 raft_consensus.cc:2781] T fda942d50b014f3d9589e1fb909fc2fc P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:49.045044 20198 raft_consensus.cc:482] T fda942d50b014f3d9589e1fb909fc2fc P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:49.045152 20198 raft_consensus.cc:3037] T fda942d50b014f3d9589e1fb909fc2fc P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:49.048530 20198 raft_consensus.cc:504] T fda942d50b014f3d9589e1fb909fc2fc P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.049763 18475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fda942d50b014f3d9589e1fb909fc2fc" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:49.049952 20198 leader_election.cc:288] T fda942d50b014f3d9589e1fb909fc2fc P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Requested vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:49.050282 18620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fda942d50b014f3d9589e1fb909fc2fc" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:49.050571 18620 raft_consensus.cc:3037] T fda942d50b014f3d9589e1fb909fc2fc P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
W20240430 20:11:49.050571 18699 leader_election.cc:341] T fda942d50b014f3d9589e1fb909fc2fc P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:49.054872 18983 tablet_bootstrap.cc:492] T dad16ae02b8b4e29a724d8eec67bb4b8 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:49.055615 18620 raft_consensus.cc:2454] T fda942d50b014f3d9589e1fb909fc2fc P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:49.056293 18696 leader_election.cc:302] T fda942d50b014f3d9589e1fb909fc2fc P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a
I20240430 20:11:49.057369 20198 raft_consensus.cc:2781] T fda942d50b014f3d9589e1fb909fc2fc P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:49.058326 20198 raft_consensus.cc:686] T fda942d50b014f3d9589e1fb909fc2fc P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Becoming Leader. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Running, Role: LEADER
W20240430 20:11:49.065876 20903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240430 20:11:49.066884 19471 ts_tablet_manager.cc:1909] T df92eb85261d47d88cbf16d29d9ad30e P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:49.067394 18988 tablet_bootstrap.cc:492] T 47e98935eb21449abd3d4d24327778d5 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:49.067442 20384 raft_consensus.cc:482] T 70247a4c910946c0afc24ce5ce0e85d6 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:49.067593 18988 ts_tablet_manager.cc:1367] T 47e98935eb21449abd3d4d24327778d5 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.080s user 0.007s sys 0.000s
I20240430 20:11:49.068011 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 388) took 3238 ms. Trace:
I20240430 20:11:49.067847 20384 raft_consensus.cc:504] T 70247a4c910946c0afc24ce5ce0e85d6 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.068130 19471 rpcz_store.cc:276] 0430 20:11:45.829094 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:45.829153 (+ 59us) service_pool.cc:225] Handling call
0430 20:11:48.777059 (+2947906us) ts_tablet_manager.cc:1120] Deleting tablet df92eb85261d47d88cbf16d29d9ad30e
0430 20:11:48.777073 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:48.822047 (+ 44974us) spinlock_profiling.cc:243] Waited 43.9 ms on lock 0x6160000d4254. stack: 00007ffa8b97e756 00007ffa8b97e510 00007ffa8a9b148d 0000000000955804 00000000009557cd 000000000095519a 00007ffa971eca00 00007ffa984c6853 00007ffa9af594bc 00007ffa9af8b814 00007ffa9af6cb00 00007ffa9af6c88d 0000000000a60bcc 00007ffa8b9c4b74 00007ffa8b9c819b 00007ffa8b9c8018
0430 20:11:48.928699 (+106652us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:49.033836 (+105137us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:49.067990 (+ 34154us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":101064448,"tablet-delete.queue_time_us":2947672}
W20240430 20:11:49.068528 18558 connection.cc:441] server connection from 127.0.0.1:40218 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 388) could send its response
I20240430 20:11:49.068528 18988 raft_consensus.cc:348] T 47e98935eb21449abd3d4d24327778d5 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.068727 18988 raft_consensus.cc:374] T 47e98935eb21449abd3d4d24327778d5 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:49.068792 18988 raft_consensus.cc:729] T 47e98935eb21449abd3d4d24327778d5 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
W20240430 20:11:49.069036 20917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240430 20:11:49.069079 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 389) took 3206 ms. Trace:
I20240430 20:11:49.069026 18988 consensus_queue.cc:260] T 47e98935eb21449abd3d4d24327778d5 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.069597 18988 ts_tablet_manager.cc:1398] T 47e98935eb21449abd3d4d24327778d5 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240430 20:11:49.070165 18988 tablet_bootstrap.cc:492] T f3711610610542128c4a84e17bce4a3e P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:49.070132 18758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70247a4c910946c0afc24ce5ce0e85d6" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:49.070366 18758 raft_consensus.cc:2454] T 70247a4c910946c0afc24ce5ce0e85d6 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 0.
I20240430 20:11:49.069193 19471 rpcz_store.cc:276] 0430 20:11:45.862407 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:45.862486 (+ 79us) service_pool.cc:225] Handling call
0430 20:11:49.068881 (+3206395us) ts_tablet_manager.cc:1120] Deleting tablet 47e98935eb21449abd3d4d24327778d5
0430 20:11:49.068892 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:49.069060 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3206120}
W20240430 20:11:49.070521 18558 connection.cc:441] server connection from 127.0.0.1:40218 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 389) could send its response
I20240430 20:11:49.070998 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 390) took 3208 ms. Trace:
I20240430 20:11:49.071012 18557 leader_election.cc:302] T 70247a4c910946c0afc24ce5ce0e85d6 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:49.071111 19471 rpcz_store.cc:276] 0430 20:11:45.862943 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:45.862994 (+ 51us) service_pool.cc:225] Handling call
0430 20:11:49.070756 (+3207762us) ts_tablet_manager.cc:1120] Deleting tablet fcc16db4d6be42edb7bd1ce18fe0190d
0430 20:11:49.070766 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:49.070982 (+ 216us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3207598}
I20240430 20:11:49.071321 20488 raft_consensus.cc:2781] T 70247a4c910946c0afc24ce5ce0e85d6 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:49.071442 20488 raft_consensus.cc:482] T 70247a4c910946c0afc24ce5ce0e85d6 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:49.071499 20488 raft_consensus.cc:3037] T 70247a4c910946c0afc24ce5ce0e85d6 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:49.074045 20488 raft_consensus.cc:504] T 70247a4c910946c0afc24ce5ce0e85d6 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.074287 20384 leader_election.cc:288] T 70247a4c910946c0afc24ce5ce0e85d6 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:49.074734 18475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70247a4c910946c0afc24ce5ce0e85d6" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:49.074952 18475 raft_consensus.cc:2454] T 70247a4c910946c0afc24ce5ce0e85d6 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 0.
I20240430 20:11:49.075003 20488 leader_election.cc:288] T 70247a4c910946c0afc24ce5ce0e85d6 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:49.075081 18758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70247a4c910946c0afc24ce5ce0e85d6" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:49.075263 18758 raft_consensus.cc:3037] T 70247a4c910946c0afc24ce5ce0e85d6 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
W20240430 20:11:49.075508 18558 connection.cc:441] server connection from 127.0.0.1:40218 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 390) could send its response
I20240430 20:11:49.075506 18475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70247a4c910946c0afc24ce5ce0e85d6" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:49.075649 18475 raft_consensus.cc:3037] T 70247a4c910946c0afc24ce5ce0e85d6 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:49.075853 19471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 391) took 3212 ms. Trace:
I20240430 20:11:49.075925 19471 rpcz_store.cc:276] 0430 20:11:45.863640 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:45.863687 (+ 47us) service_pool.cc:225] Handling call
0430 20:11:49.075715 (+3212028us) ts_tablet_manager.cc:1120] Deleting tablet dbb427aefa6b47b0805d1751fea4efcb
0430 20:11:49.075725 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:49.075838 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3211886}
W20240430 20:11:49.076114 18558 connection.cc:441] server connection from 127.0.0.1:40218 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40218 (request call id 391) could send its response
I20240430 20:11:49.076591 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_a8d5ac73c6d047698d87b3d1ba34e66f [id=9c80193cde2346c982cf84ae9612e2f7]
I20240430 20:11:49.077463 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_b869ed2a85d64fcf83a0eb41c1a86446 [id=9d43a387b6884f6dbb32a65cc1f0bb97]
I20240430 20:11:49.077585 18983 tablet_bootstrap.cc:654] T dad16ae02b8b4e29a724d8eec67bb4b8 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:49.077903 18758 raft_consensus.cc:2454] T 70247a4c910946c0afc24ce5ce0e85d6 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
I20240430 20:11:49.078013 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_dd1d113759284183905a803b2733436d [id=9d581bf380f54b4dab56aa34b9c4b1ea]
I20240430 20:11:49.078537 18475 raft_consensus.cc:2454] T 70247a4c910946c0afc24ce5ce0e85d6 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
I20240430 20:11:49.078552 18557 leader_election.cc:302] T 70247a4c910946c0afc24ce5ce0e85d6 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:49.078724 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_2ec9e316bc3347b189a13b7a8bbcf829 [id=9fa22833979f4c2f978de651145acef9]
I20240430 20:11:49.078979 20488 raft_consensus.cc:2781] T 70247a4c910946c0afc24ce5ce0e85d6 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:49.079144 20488 raft_consensus.cc:686] T 70247a4c910946c0afc24ce5ce0e85d6 P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Becoming Leader. State: Replica: 5e1c6397f09747f091534faf93542710, State: Running, Role: LEADER
I20240430 20:11:49.079406 20488 consensus_queue.cc:237] T 70247a4c910946c0afc24ce5ce0e85d6 P 5e1c6397f09747f091534faf93542710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.080051 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_e20b13b4103849b9bc5e5f96f9b911e4 [id=9fd8ccb805024968b9ad8ab060d5db3b]
I20240430 20:11:49.080802 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_83b26c6b5e1942da864b8e60e882f5b4 [id=a09a93469a8f450895fa1756e1074c16]
I20240430 20:11:49.082311 20488 raft_consensus.cc:482] T b1792c160af346ffb4f047c5d4435c10 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:49.082440 20488 raft_consensus.cc:504] T b1792c160af346ffb4f047c5d4435c10 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.083161 20488 leader_election.cc:288] T b1792c160af346ffb4f047c5d4435c10 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:49.083581 18758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1792c160af346ffb4f047c5d4435c10" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:49.083725 18475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1792c160af346ffb4f047c5d4435c10" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:49.083801 18758 raft_consensus.cc:2454] T b1792c160af346ffb4f047c5d4435c10 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 0.
I20240430 20:11:49.084182 18557 leader_election.cc:302] T b1792c160af346ffb4f047c5d4435c10 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:49.084228 20795 server_base.cc:1034] running on GCE node
W20240430 20:11:49.084410 18558 leader_election.cc:341] T b1792c160af346ffb4f047c5d4435c10 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:49.084683 20384 raft_consensus.cc:2781] T b1792c160af346ffb4f047c5d4435c10 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:49.084766 20384 raft_consensus.cc:482] T b1792c160af346ffb4f047c5d4435c10 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:49.084861 20384 raft_consensus.cc:3037] T b1792c160af346ffb4f047c5d4435c10 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:49.085624 20795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240430 20:11:49.087936 20384 raft_consensus.cc:504] T b1792c160af346ffb4f047c5d4435c10 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.089115 18758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1792c160af346ffb4f047c5d4435c10" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:49.089237 18475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1792c160af346ffb4f047c5d4435c10" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:49.089421 18758 raft_consensus.cc:3037] T b1792c160af346ffb4f047c5d4435c10 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
W20240430 20:11:49.089915 18558 leader_election.cc:341] T b1792c160af346ffb4f047c5d4435c10 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:49.090114 20384 leader_election.cc:288] T b1792c160af346ffb4f047c5d4435c10 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:49.092823 18758 raft_consensus.cc:2454] T b1792c160af346ffb4f047c5d4435c10 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
I20240430 20:11:49.093659 18557 leader_election.cc:302] T b1792c160af346ffb4f047c5d4435c10 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a
I20240430 20:11:49.094121 20488 raft_consensus.cc:2781] T b1792c160af346ffb4f047c5d4435c10 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:49.094393 20488 raft_consensus.cc:686] T b1792c160af346ffb4f047c5d4435c10 P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Becoming Leader. State: Replica: 5e1c6397f09747f091534faf93542710, State: Running, Role: LEADER
I20240430 20:11:49.094692 20488 consensus_queue.cc:237] T b1792c160af346ffb4f047c5d4435c10 P 5e1c6397f09747f091534faf93542710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.095849 20795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240430 20:11:49.097152 20795 hybrid_clock.cc:648] HybridClock initialized: now 1714507909097061 us; error 60 us; skew 500 ppm
I20240430 20:11:49.097625 20795 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240430 20:11:49.100554 18983 tablet_bootstrap.cc:492] T dad16ae02b8b4e29a724d8eec67bb4b8 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:49.100708 18983 ts_tablet_manager.cc:1367] T dad16ae02b8b4e29a724d8eec67bb4b8 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.062s user 0.004s sys 0.000s
I20240430 20:11:49.100916 18988 tablet_bootstrap.cc:654] T f3711610610542128c4a84e17bce4a3e P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:49.101469 18983 raft_consensus.cc:348] T dad16ae02b8b4e29a724d8eec67bb4b8 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.101645 18983 raft_consensus.cc:374] T dad16ae02b8b4e29a724d8eec67bb4b8 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:49.101696 18983 raft_consensus.cc:729] T dad16ae02b8b4e29a724d8eec67bb4b8 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:49.101889 18983 consensus_queue.cc:260] T dad16ae02b8b4e29a724d8eec67bb4b8 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.102676 18983 ts_tablet_manager.cc:1398] T dad16ae02b8b4e29a724d8eec67bb4b8 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240430 20:11:49.108927 18983 tablet_bootstrap.cc:492] T 3e5225f1f5f544ecbf8dfa470af3daca P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:49.111196 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_ea4369e78de74045baa10ea2a61bfe38 [id=a0ccb667cf974306a952751800568d98]
I20240430 20:11:49.111837 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_f86df7f6c95e4da48b80ae4733dc6994 [id=a29411bd07ba400c942bedc9e8e19884]
I20240430 20:11:49.111882 18983 tablet_bootstrap.cc:654] T 3e5225f1f5f544ecbf8dfa470af3daca P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:49.112588 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_c1f36f7df5ae4e7c8eb8b4e99b89748b [id=a50abf28790d4284a76ac6ee22c02606]
I20240430 20:11:49.113544 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_211cbad17eba4c48ab02ad45543616ca [id=a693b863f7f34539aa46c3f006aca925]
I20240430 20:11:49.114277 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_71775acb989f4c2f87d670dff67a93b6 [id=a6c9697e1e9949b1a14d62c9f9ba4fed]
I20240430 20:11:49.115149 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_123c6be593f44fc2ac4115d192c43e68 [id=a7adb9bbadef40609bee9bd9e86ad818]
I20240430 20:11:49.116357 18985 tablet_bootstrap.cc:492] T 8e91920e7e36418d8c27d1f0bfd9b981 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:49.116601 18985 ts_tablet_manager.cc:1367] T 8e91920e7e36418d8c27d1f0bfd9b981 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.178s user 0.020s sys 0.044s
I20240430 20:11:49.117729 18985 raft_consensus.cc:348] T 8e91920e7e36418d8c27d1f0bfd9b981 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.118117 18985 raft_consensus.cc:374] T 8e91920e7e36418d8c27d1f0bfd9b981 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:49.118222 18985 raft_consensus.cc:729] T 8e91920e7e36418d8c27d1f0bfd9b981 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:49.118548 18985 consensus_queue.cc:260] T 8e91920e7e36418d8c27d1f0bfd9b981 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.120218 20795 webserver.cc:460] Webserver started at http://127.17.177.60:34267/ using document root <none> and password file <none>
I20240430 20:11:49.122920 18985 ts_tablet_manager.cc:1398] T 8e91920e7e36418d8c27d1f0bfd9b981 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240430 20:11:49.124125 18985 tablet_bootstrap.cc:492] T 47e98935eb21449abd3d4d24327778d5 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:49.125962 20795 fs_manager.cc:359] Metadata directory not provided
I20240430 20:11:49.126456 20795 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240430 20:11:49.126633 18985 tablet_bootstrap.cc:654] T 47e98935eb21449abd3d4d24327778d5 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:49.128278 18983 tablet_bootstrap.cc:492] T 3e5225f1f5f544ecbf8dfa470af3daca P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:49.128418 18983 ts_tablet_manager.cc:1367] T 3e5225f1f5f544ecbf8dfa470af3daca P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.001s
I20240430 20:11:49.129436 18983 raft_consensus.cc:348] T 3e5225f1f5f544ecbf8dfa470af3daca P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.129644 18983 raft_consensus.cc:374] T 3e5225f1f5f544ecbf8dfa470af3daca P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:49.129794 18983 raft_consensus.cc:729] T 3e5225f1f5f544ecbf8dfa470af3daca P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:49.130043 18983 consensus_queue.cc:260] T 3e5225f1f5f544ecbf8dfa470af3daca P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.130568 18983 ts_tablet_manager.cc:1398] T 3e5225f1f5f544ecbf8dfa470af3daca P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240430 20:11:49.131125 18983 tablet_bootstrap.cc:492] T ed0c37a71ede4bfc9eb5ad1a4365f25e P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:49.133759 18983 tablet_bootstrap.cc:654] T ed0c37a71ede4bfc9eb5ad1a4365f25e P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:49.134313 18988 tablet_bootstrap.cc:492] T f3711610610542128c4a84e17bce4a3e P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:49.134502 18988 ts_tablet_manager.cc:1367] T f3711610610542128c4a84e17bce4a3e P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.065s user 0.008s sys 0.002s
I20240430 20:11:49.135375 18988 raft_consensus.cc:348] T f3711610610542128c4a84e17bce4a3e P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.135620 18988 raft_consensus.cc:374] T f3711610610542128c4a84e17bce4a3e P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:49.135707 18988 raft_consensus.cc:729] T f3711610610542128c4a84e17bce4a3e P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:49.136008 18988 consensus_queue.cc:260] T f3711610610542128c4a84e17bce4a3e P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.136570 18988 ts_tablet_manager.cc:1398] T f3711610610542128c4a84e17bce4a3e P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240430 20:11:49.137004 18988 tablet_bootstrap.cc:492] T 12795670f21447a8964fe1efc6e92398 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:49.139427 18988 tablet_bootstrap.cc:654] T 12795670f21447a8964fe1efc6e92398 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:49.142757 20695 raft_consensus.cc:482] T b285e3eb16d148779c74e057458d8461 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:49.142931 20695 raft_consensus.cc:504] T b285e3eb16d148779c74e057458d8461 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.143790 20695 leader_election.cc:288] T b285e3eb16d148779c74e057458d8461 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:49.144361 18475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b285e3eb16d148779c74e057458d8461" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
W20240430 20:11:49.145155 18699 leader_election.cc:341] T b285e3eb16d148779c74e057458d8461 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:49.147049 18620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b285e3eb16d148779c74e057458d8461" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:49.147305 18620 raft_consensus.cc:2454] T b285e3eb16d148779c74e057458d8461 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:49.147816 18696 leader_election.cc:302] T b285e3eb16d148779c74e057458d8461 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a
I20240430 20:11:49.148391 20693 raft_consensus.cc:2781] T b285e3eb16d148779c74e057458d8461 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:49.148509 20693 raft_consensus.cc:482] T b285e3eb16d148779c74e057458d8461 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:49.148597 20693 raft_consensus.cc:3037] T b285e3eb16d148779c74e057458d8461 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:49.150777 20695 consensus_queue.cc:1035] T 130007acc8ba4e709c3eb85fabe75903 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:49.152237 20693 raft_consensus.cc:504] T b285e3eb16d148779c74e057458d8461 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.152910 20693 leader_election.cc:288] T b285e3eb16d148779c74e057458d8461 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Requested vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:49.153555 18475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b285e3eb16d148779c74e057458d8461" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
W20240430 20:11:49.154269 18699 leader_election.cc:341] T b285e3eb16d148779c74e057458d8461 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:49.154290 18623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b285e3eb16d148779c74e057458d8461" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:49.154469 18623 raft_consensus.cc:3037] T b285e3eb16d148779c74e057458d8461 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:49.157989 18623 raft_consensus.cc:2454] T b285e3eb16d148779c74e057458d8461 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:49.159382 18696 leader_election.cc:302] T b285e3eb16d148779c74e057458d8461 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a
I20240430 20:11:49.161294 20693 raft_consensus.cc:2781] T b285e3eb16d148779c74e057458d8461 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:49.161653 20693 raft_consensus.cc:686] T b285e3eb16d148779c74e057458d8461 P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Becoming Leader. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Running, Role: LEADER
I20240430 20:11:49.161952 18988 tablet_bootstrap.cc:492] T 12795670f21447a8964fe1efc6e92398 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:49.162876 18988 ts_tablet_manager.cc:1367] T 12795670f21447a8964fe1efc6e92398 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.026s user 0.003s sys 0.000s
I20240430 20:11:49.163199 20693 consensus_queue.cc:237] T b285e3eb16d148779c74e057458d8461 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.163929 18988 raft_consensus.cc:348] T 12795670f21447a8964fe1efc6e92398 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.164120 18988 raft_consensus.cc:374] T 12795670f21447a8964fe1efc6e92398 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:49.164243 18988 raft_consensus.cc:729] T 12795670f21447a8964fe1efc6e92398 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:49.164798 18988 consensus_queue.cc:260] T 12795670f21447a8964fe1efc6e92398 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.165416 18988 ts_tablet_manager.cc:1398] T 12795670f21447a8964fe1efc6e92398 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240430 20:11:49.165776 18988 tablet_bootstrap.cc:492] T f90a5c5818c54390a2ae323fed7ebe57 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:49.167454 18988 tablet_bootstrap.cc:654] T f90a5c5818c54390a2ae323fed7ebe57 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:49.178520 20795 fs_manager.cc:709] Time spent opening directory manager: real 0.030s user 0.004s sys 0.000s
I20240430 20:11:49.184005 18988 tablet_bootstrap.cc:492] T f90a5c5818c54390a2ae323fed7ebe57 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:49.184849 18988 ts_tablet_manager.cc:1367] T f90a5c5818c54390a2ae323fed7ebe57 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.019s user 0.007s sys 0.001s
I20240430 20:11:49.186060 18988 raft_consensus.cc:348] T f90a5c5818c54390a2ae323fed7ebe57 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.186318 18988 raft_consensus.cc:374] T f90a5c5818c54390a2ae323fed7ebe57 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:49.187258 20693 raft_consensus.cc:482] T c091701480c4412faabad3d071d85d03 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:49.187585 18988 raft_consensus.cc:729] T f90a5c5818c54390a2ae323fed7ebe57 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:49.187633 20693 raft_consensus.cc:504] T c091701480c4412faabad3d071d85d03 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.188145 18988 consensus_queue.cc:260] T f90a5c5818c54390a2ae323fed7ebe57 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.188606 20693 leader_election.cc:288] T c091701480c4412faabad3d071d85d03 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:49.189321 18623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c091701480c4412faabad3d071d85d03" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:49.189409 18988 ts_tablet_manager.cc:1398] T f90a5c5818c54390a2ae323fed7ebe57 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240430 20:11:49.189363 18475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c091701480c4412faabad3d071d85d03" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:49.189661 18623 raft_consensus.cc:2379] T c091701480c4412faabad3d071d85d03 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c51c33d5616482cafb68e26688820f4 in current term 1: Already voted for candidate 2efb542aa55e439b99ccb98d1c5c884a in this term.
I20240430 20:11:49.190213 18988 tablet_bootstrap.cc:492] T 0368d37dd9e2452982fb81f539c6805b P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:49.190382 18699 leader_election.cc:302] T c091701480c4412faabad3d071d85d03 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710
I20240430 20:11:49.190766 20695 raft_consensus.cc:3037] T c091701480c4412faabad3d071d85d03 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:49.192488 18988 tablet_bootstrap.cc:654] T 0368d37dd9e2452982fb81f539c6805b P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:49.193809 20695 raft_consensus.cc:2726] T c091701480c4412faabad3d071d85d03 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:49.194059 20587 raft_consensus.cc:482] T 2cdccf4be7094ca5b863dd5c3432d55d P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:49.194200 20587 raft_consensus.cc:504] T 2cdccf4be7094ca5b863dd5c3432d55d P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.194787 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_9110c0cb4db441dbb8bdce422eadc189 [id=a826c08c7eef44ec83feae3a6cb384e5]
I20240430 20:11:49.194897 20587 leader_election.cc:288] T 2cdccf4be7094ca5b863dd5c3432d55d P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
W20240430 20:11:49.196038 18696 leader_election.cc:341] T 2cdccf4be7094ca5b863dd5c3432d55d P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Not found: Tablet not found: 2cdccf4be7094ca5b863dd5c3432d55d
W20240430 20:11:49.196213 18699 leader_election.cc:341] T 2cdccf4be7094ca5b863dd5c3432d55d P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Not found: Tablet not found: 2cdccf4be7094ca5b863dd5c3432d55d
I20240430 20:11:49.196329 18699 leader_election.cc:302] T 2cdccf4be7094ca5b863dd5c3432d55d P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710
I20240430 20:11:49.196765 20693 raft_consensus.cc:2726] T 2cdccf4be7094ca5b863dd5c3432d55d P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:49.200366 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_1177828cad1f4a24bd397619dad96170 [id=a95fed1bc3fa4865bf9a35dfc51b659c]
I20240430 20:11:49.200903 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_1ebe0eb1a1984ee9bbb16d634ff2cedb [id=aa91ac92289a48b784e97eea9967192f]
I20240430 20:11:49.201547 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_c7c87a8e019944cab3ec99e9563ac01a [id=abf3765074eb4d7d9d2bcc4ab721e6bd]
I20240430 20:11:49.202013 20587 raft_consensus.cc:482] T 2a08e36c39824abf98b9b6214c8b84cb P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:49.202137 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_80306c0ee72842209ca8feac52c0f247 [id=ac6acb1d504243e292723b12e0ad2b33]
I20240430 20:11:49.202142 20587 raft_consensus.cc:504] T 2a08e36c39824abf98b9b6214c8b84cb P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.202852 20587 leader_election.cc:288] T 2a08e36c39824abf98b9b6214c8b84cb P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:49.203514 18475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a08e36c39824abf98b9b6214c8b84cb" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:49.203708 18475 raft_consensus.cc:2454] T 2a08e36c39824abf98b9b6214c8b84cb P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:49.204351 18699 leader_election.cc:302] T 2a08e36c39824abf98b9b6214c8b84cb P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters:
I20240430 20:11:49.204033 20935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240430 20:11:49.204650 20587 raft_consensus.cc:2781] T 2a08e36c39824abf98b9b6214c8b84cb P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:49.204715 20587 raft_consensus.cc:482] T 2a08e36c39824abf98b9b6214c8b84cb P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:49.204774 20587 raft_consensus.cc:3037] T 2a08e36c39824abf98b9b6214c8b84cb P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:49.207134 20630 consensus_queue.cc:1035] T c091701480c4412faabad3d071d85d03 P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:49.208214 20587 raft_consensus.cc:504] T 2a08e36c39824abf98b9b6214c8b84cb P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.209924 18474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a08e36c39824abf98b9b6214c8b84cb" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:49.210135 18474 raft_consensus.cc:3037] T 2a08e36c39824abf98b9b6214c8b84cb P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:49.213284 18474 raft_consensus.cc:2454] T 2a08e36c39824abf98b9b6214c8b84cb P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:49.213339 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_2e9c7c683df84a41942d31600a265641 [id=aca6fb02776a40bbac312191adbdc136]
I20240430 20:11:49.214094 18699 leader_election.cc:302] T 2a08e36c39824abf98b9b6214c8b84cb P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters:
I20240430 20:11:49.214725 20695 raft_consensus.cc:2781] T 2a08e36c39824abf98b9b6214c8b84cb P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:49.214876 20695 raft_consensus.cc:686] T 2a08e36c39824abf98b9b6214c8b84cb P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Becoming Leader. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Running, Role: LEADER
I20240430 20:11:49.215183 20695 consensus_queue.cc:237] T 2a08e36c39824abf98b9b6214c8b84cb P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.216529 18988 tablet_bootstrap.cc:492] T 0368d37dd9e2452982fb81f539c6805b P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:49.217429 18985 tablet_bootstrap.cc:492] T 47e98935eb21449abd3d4d24327778d5 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:49.217623 18985 ts_tablet_manager.cc:1367] T 47e98935eb21449abd3d4d24327778d5 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.094s user 0.015s sys 0.009s
I20240430 20:11:49.218185 20587 leader_election.cc:288] T 2a08e36c39824abf98b9b6214c8b84cb P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Requested vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:49.218508 18985 raft_consensus.cc:348] T 47e98935eb21449abd3d4d24327778d5 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.218830 18985 raft_consensus.cc:374] T 47e98935eb21449abd3d4d24327778d5 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:49.218840 18620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a08e36c39824abf98b9b6214c8b84cb" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:49.218940 18985 raft_consensus.cc:729] T 47e98935eb21449abd3d4d24327778d5 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:49.219164 18985 consensus_queue.cc:260] T 47e98935eb21449abd3d4d24327778d5 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.224017 18620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a08e36c39824abf98b9b6214c8b84cb" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:49.228582 20795 fs_manager.cc:725] Time spent opening block manager: real 0.048s user 0.009s sys 0.009s
I20240430 20:11:49.228837 18985 ts_tablet_manager.cc:1398] T 47e98935eb21449abd3d4d24327778d5 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240430 20:11:49.228813 20795 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/master-2/data,/tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/master-2/wal
uuid: "b632d6ef4c104a058a6db208024cab33"
format_stamp: "Formatted at 2024-04-30 20:11:18 on dist-test-slave-54l7"
I20240430 20:11:49.230126 20795 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240430 20:11:49.231611 20693 consensus_queue.cc:1035] T 130007acc8ba4e709c3eb85fabe75903 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:49.235415 18985 tablet_bootstrap.cc:492] T a445fdfde01c4371a5e6b3daea5b00ab P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:49.237454 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_ba8b218e936b4bfcb073f8182f474cca [id=aedf11fda8914a3b843a57f8a7b69a53]
I20240430 20:11:49.241528 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_4bac430d51a7461e919b79d3a2f92a8e [id=afaa8b1a82404015ba960f018dd7550b]
I20240430 20:11:49.241848 18988 ts_tablet_manager.cc:1367] T 0368d37dd9e2452982fb81f539c6805b P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.052s user 0.003s sys 0.003s
I20240430 20:11:49.242928 18988 raft_consensus.cc:348] T 0368d37dd9e2452982fb81f539c6805b P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.244241 18985 tablet_bootstrap.cc:654] T a445fdfde01c4371a5e6b3daea5b00ab P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:49.245054 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_2e4e483bcd6c4dd6bdfcc938cb0e2da6 [id=afdab68d2df0433f9102de19b55c60b5]
I20240430 20:11:49.245267 18983 tablet_bootstrap.cc:492] T ed0c37a71ede4bfc9eb5ad1a4365f25e P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:49.245486 18983 ts_tablet_manager.cc:1367] T ed0c37a71ede4bfc9eb5ad1a4365f25e P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.114s user 0.003s sys 0.007s
I20240430 20:11:49.250921 18988 raft_consensus.cc:374] T 0368d37dd9e2452982fb81f539c6805b P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:49.251664 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_59297b42e8cf4ab3b9e95e2e70d3c1fe [id=b1312a6cd1a54003b1db727a2a9d125d]
I20240430 20:11:49.252705 18988 raft_consensus.cc:729] T 0368d37dd9e2452982fb81f539c6805b P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:49.253132 18988 consensus_queue.cc:260] T 0368d37dd9e2452982fb81f539c6805b P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.246510 18983 raft_consensus.cc:348] T ed0c37a71ede4bfc9eb5ad1a4365f25e P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.253873 18983 raft_consensus.cc:374] T ed0c37a71ede4bfc9eb5ad1a4365f25e P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:49.254788 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_5ed3ea587cf843db9a1ff1ffab40b678 [id=b3d1c2f97c4648518dbea83a46cadd85]
I20240430 20:11:49.254933 18983 raft_consensus.cc:729] T ed0c37a71ede4bfc9eb5ad1a4365f25e P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:49.255411 18988 ts_tablet_manager.cc:1398] T 0368d37dd9e2452982fb81f539c6805b P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20240430 20:11:49.255717 18983 consensus_queue.cc:260] T ed0c37a71ede4bfc9eb5ad1a4365f25e P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.256636 18983 ts_tablet_manager.cc:1398] T ed0c37a71ede4bfc9eb5ad1a4365f25e P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.011s user 0.001s sys 0.001s
I20240430 20:11:49.256822 18988 tablet_bootstrap.cc:492] T 488d0e609b224674932f890939664df2 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:49.259387 18988 tablet_bootstrap.cc:654] T 488d0e609b224674932f890939664df2 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:49.259757 18983 tablet_bootstrap.cc:492] T f90a5c5818c54390a2ae323fed7ebe57 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:49.261083 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_91c01cce23fe4f4182952019c19875c0 [id=b7251a2a9c91439db19f9b910ebb5a3b]
I20240430 20:11:49.262718 18983 tablet_bootstrap.cc:654] T f90a5c5818c54390a2ae323fed7ebe57 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:49.263327 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_1407edea18d440e7adacadcfc8d89054 [id=b74e2a754da140628893bab105373983]
I20240430 20:11:49.267354 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_fe491607e9e7480b9d9d2c8eb89ccac9 [id=ba205c2256d24f998fb16e6590b74cf4]
I20240430 20:11:49.260936 20693 consensus_queue.cc:1035] T fdf53061efd74fa29a9d8208d1fd210e P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:49.273020 18988 tablet_bootstrap.cc:492] T 488d0e609b224674932f890939664df2 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:49.280516 18988 ts_tablet_manager.cc:1367] T 488d0e609b224674932f890939664df2 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.024s user 0.004s sys 0.000s
I20240430 20:11:49.274271 18983 tablet_bootstrap.cc:492] T f90a5c5818c54390a2ae323fed7ebe57 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:49.280968 18983 ts_tablet_manager.cc:1367] T f90a5c5818c54390a2ae323fed7ebe57 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.021s user 0.000s sys 0.003s
I20240430 20:11:49.281213 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_5d51defcfdfd4d7f8e7d341c5daf0cdf [id=bdab45e393374761bea47d965b81f043]
I20240430 20:11:49.284546 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_75336e7a0e5f4c2d909f5c5c385bc379 [id=be4876d319a24a958d50e3d0c508e95f]
I20240430 20:11:49.285322 18988 raft_consensus.cc:348] T 488d0e609b224674932f890939664df2 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.286419 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_5f50cb33cf774b86b2380ee4a569dac9 [id=bf4dcbec74f946398c7fc3f6eec6d068]
I20240430 20:11:49.282513 18983 raft_consensus.cc:348] T f90a5c5818c54390a2ae323fed7ebe57 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.286960 18988 raft_consensus.cc:374] T 488d0e609b224674932f890939664df2 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:49.287158 18983 raft_consensus.cc:374] T f90a5c5818c54390a2ae323fed7ebe57 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:49.287329 18988 raft_consensus.cc:729] T 488d0e609b224674932f890939664df2 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:49.287956 18988 consensus_queue.cc:260] T 488d0e609b224674932f890939664df2 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.287528 18983 raft_consensus.cc:729] T f90a5c5818c54390a2ae323fed7ebe57 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:49.288506 18983 consensus_queue.cc:260] T f90a5c5818c54390a2ae323fed7ebe57 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.290238 18983 ts_tablet_manager.cc:1398] T f90a5c5818c54390a2ae323fed7ebe57 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.008s user 0.000s sys 0.001s
I20240430 20:11:49.292151 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_1a641ff0ab5944a495a73189a6e8499a [id=c2bcc99b99d14bd480881a9c8dc37dbb]
I20240430 20:11:49.292640 18988 ts_tablet_manager.cc:1398] T 488d0e609b224674932f890939664df2 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240430 20:11:49.293685 18983 tablet_bootstrap.cc:492] T 1701a15b115741aea13f8327e3d26b34 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:49.294525 18988 tablet_bootstrap.cc:492] T a445fdfde01c4371a5e6b3daea5b00ab P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
W20240430 20:11:49.299468 18211 leader_election.cc:334] T 00000000000000000000000000000000 P 89df84ccf5394c9bb60d4e4ed872fc2e [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 88b3f98793354c7aaa5c106e8ec31ed8 (127.17.177.62:36441): Timed out: RequestConsensusVote RPC to 127.17.177.62:36441 timed out after 0.546s (SENT)
I20240430 20:11:49.299638 18211 leader_election.cc:302] T 00000000000000000000000000000000 P 89df84ccf5394c9bb60d4e4ed872fc2e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89df84ccf5394c9bb60d4e4ed872fc2e; no voters: 88b3f98793354c7aaa5c106e8ec31ed8, b632d6ef4c104a058a6db208024cab33
I20240430 20:11:49.308848 20630 consensus_queue.cc:1035] T c091701480c4412faabad3d071d85d03 P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:49.335258 20947 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 89df84ccf5394c9bb60d4e4ed872fc2e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240430 20:11:49.337251 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_e686cdfcbcc046749fa5b51578a64794 [id=c3dc9f05316d46d6a3ef4d03f40877ec]
I20240430 20:11:49.338181 18983 tablet_bootstrap.cc:654] T 1701a15b115741aea13f8327e3d26b34 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:49.343808 18988 tablet_bootstrap.cc:654] T a445fdfde01c4371a5e6b3daea5b00ab P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:49.346321 18985 tablet_bootstrap.cc:492] T a445fdfde01c4371a5e6b3daea5b00ab P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:49.346539 18985 ts_tablet_manager.cc:1367] T a445fdfde01c4371a5e6b3daea5b00ab P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.111s user 0.006s sys 0.004s
I20240430 20:11:49.346840 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_de88cafd92b149f9b545369bff71033c [id=c4bf4806f3d84579b16a3038d8d982c3]
I20240430 20:11:49.346746 20198 consensus_queue.cc:237] T fda942d50b014f3d9589e1fb909fc2fc P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.347388 18985 raft_consensus.cc:348] T a445fdfde01c4371a5e6b3daea5b00ab P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.347360 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_43043172400c49598042d752412c3a28 [id=c4cfc76234794e428b432eb2f5c74bb9]
I20240430 20:11:49.347611 18985 raft_consensus.cc:374] T a445fdfde01c4371a5e6b3daea5b00ab P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:49.347724 18985 raft_consensus.cc:729] T a445fdfde01c4371a5e6b3daea5b00ab P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:49.348008 18985 consensus_queue.cc:260] T a445fdfde01c4371a5e6b3daea5b00ab P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.348153 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_304521bb46d9434481340ff5ce7ca994 [id=c51917e099fa4676a092373e1ad201f7]
I20240430 20:11:49.353103 18985 ts_tablet_manager.cc:1398] T a445fdfde01c4371a5e6b3daea5b00ab P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.006s user 0.000s sys 0.001s
I20240430 20:11:49.353950 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_ae525cb900474e458017ab35d825fc1c [id=c8d3ef652a764c3a9d658fbfc9f41ede]
I20240430 20:11:49.354336 18985 tablet_bootstrap.cc:492] T 50e1c55a1a0a484984ad5cbb78eec2ed P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:49.355327 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_cfb6af12a3114f96837868eaa33be09b [id=cdf3c4662ece4b0ba1c88057affd26e6]
I20240430 20:11:49.355835 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_c9603ec75ad0491592883980e5efc388 [id=ce2d781a514e4c10a2fde5b9d4551ff5]
I20240430 20:11:49.356887 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_4d68c38d9ba7452a829f8318b2b267b8 [id=d06e3e9045b1429aaeebd1fd67d841de]
I20240430 20:11:49.357426 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_5572f0acadaa432e8570b159fd9b76bc [id=d0f3b1709ac94aed8ec568c0327eab34]
I20240430 20:11:49.358039 20940 raft_consensus.cc:482] T 488d0e609b224674932f890939664df2 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:49.358660 20940 raft_consensus.cc:504] T 488d0e609b224674932f890939664df2 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.361203 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_7fb60e4b56f444009ab5da219238eec7 [id=d1655c507b434f129ad7ae0b04365e51]
I20240430 20:11:49.361512 18985 tablet_bootstrap.cc:654] T 50e1c55a1a0a484984ad5cbb78eec2ed P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:49.361439 18474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "488d0e609b224674932f890939664df2" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:49.362126 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_1ce204e15fce4c469816cf4e7f80cac7 [id=d268c31d82df4871bc1a78aea5871b33]
W20240430 20:11:49.362445 18699 leader_election.cc:341] T 488d0e609b224674932f890939664df2 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:49.362804 20940 leader_election.cc:288] T 488d0e609b224674932f890939664df2 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:49.362946 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_326878a52ab041609f3c339cd65b7013 [id=d4763eaeacb546ae94b77b5a78fef986]
I20240430 20:11:49.363526 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_896bbae1f6944005b0272ca8ebc97218 [id=d5cea90cbc2c48df9d8b80da1eccc949]
W20240430 20:11:49.363976 18696 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.177.2:46027, user_credentials={real_user=slave}} blocked reactor thread for 138182us
I20240430 20:11:49.364042 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_fc68663ab2d74c34bedefcbad2d98a69 [id=d5cfd1c086e543b5a220e614a5241457]
W20240430 20:11:49.364203 18696 leader_election.cc:341] T 2a08e36c39824abf98b9b6214c8b84cb P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Illegal state: must be running to vote when last-logged opid is not known
W20240430 20:11:49.364482 18696 leader_election.cc:341] T 2a08e36c39824abf98b9b6214c8b84cb P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:49.365020 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_d82824c897a94dd0ba45955f9473b626 [id=d794640d21a843a185038673f7eb2b00]
I20240430 20:11:49.365790 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_d770977de70d4cac8511d8d125c1b160 [id=d80cad6e0e734b8fa6d2011a30f9657b]
I20240430 20:11:49.366530 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_21208fd7cdca4c73a4857bc4692187f6 [id=da22cb339ab647e8afda7aa92f3548a6]
I20240430 20:11:49.367041 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_41ccec912bfa439c888281ac81447a38 [id=da57ee424e3f481f8e0c00399bb21ffa]
I20240430 20:11:49.367765 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_b7c7025fbf2e4039aadd143d021bfef5 [id=db696ce66005401fa73746e55c071f44]
I20240430 20:11:49.368258 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_da5216d025844afda5a83ada555bc120 [id=dc55b25c3ca14863a21042e805e3bf9a]
I20240430 20:11:49.368126 18983 tablet_bootstrap.cc:492] T 1701a15b115741aea13f8327e3d26b34 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:49.368796 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_8039a17f6e4b4e9d90222f99d53d90a1 [id=e05d019ab2ad44c8af71fb04b4a27fdd]
I20240430 20:11:49.369256 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_81547db3074240c98cdd95ed3b2dd813 [id=e0ea901452334b36a8493baed478cc1a]
I20240430 20:11:49.370026 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_dd819f96762949c786839dda00487cc2 [id=e1e2e2a5b0b740cba09110991dd28193]
I20240430 20:11:49.370669 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_2e5bcbca76d345fea6d7b7dac08a7afb [id=e2122b91429d4acc939e280d8261c452]
I20240430 20:11:49.371452 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_3da5b4c8668c470483ae2a0c7773eaaf [id=e292d818ee3c45df90522f08eb2d6227]
I20240430 20:11:49.371438 20587 raft_consensus.cc:482] T 47e98935eb21449abd3d4d24327778d5 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:49.371564 20587 raft_consensus.cc:504] T 47e98935eb21449abd3d4d24327778d5 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.372059 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_8e9e6e39df854972804dfa7c260af4e4 [id=e36e9fabccf14feeb0c643cb41b4a4f6]
I20240430 20:11:49.372311 20587 leader_election.cc:288] T 47e98935eb21449abd3d4d24327778d5 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:49.372676 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_3127eaa465c84dcab3eb9935d15667ce [id=e4cf2d34f4b747afa5b8a512f211c01f]
I20240430 20:11:49.373172 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_4839b72b4f2143b2b23e8f3865d3e203 [id=e5126f46127442f1ad6ea61d40f983bb]
I20240430 20:11:49.373908 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_7d5f6f18cbd64c869739872a7dff9be2 [id=e67d619e36c0485f8f22b4a81cb4cb6c]
W20240430 20:11:49.374284 18696 consensus_peers.cc:485] T fdf53061efd74fa29a9d8208d1fd210e P 3c51c33d5616482cafb68e26688820f4 -> Peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Couldn't send request to peer 5e1c6397f09747f091534faf93542710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fdf53061efd74fa29a9d8208d1fd210e. This is attempt 1: this message will repeat every 5th retry.
I20240430 20:11:49.374531 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_f45eee4d4b6a4baf83111f12dfd25a9f [id=ea6b9f7d86424c84a8ea05a5877afd06]
I20240430 20:11:49.375291 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_2dbff4cecdff4d2dbbb0004c88377ce7 [id=ebf546ade5f140f19fff7c5841bfee30]
I20240430 20:11:49.376081 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_8236b26459a348f99005b9ef3033a30f [id=ec659350534140e89c40849e2a7dac27]
I20240430 20:11:49.376751 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_0366454b829949928c7b5e79bbf3a649 [id=edc38e64373242f7bb394a526bb2a4f8]
I20240430 20:11:49.377220 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_4bb81ccd60a148d197b8a4ebd9c9cc49 [id=ee0542402cc1426681e9583fbb7340e1]
I20240430 20:11:49.377439 18474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47e98935eb21449abd3d4d24327778d5" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:49.378245 18474 raft_consensus.cc:2454] T 47e98935eb21449abd3d4d24327778d5 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:49.378687 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_276bbdc7cfe542d1a2936485a2aad1ec [id=f146a15e6e8a4acbb4c734c1c863da57]
I20240430 20:11:49.379132 18699 leader_election.cc:302] T 47e98935eb21449abd3d4d24327778d5 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters:
I20240430 20:11:49.379216 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_457e03bc2d524accaa76649790c42fc8 [id=f26e395466f24585b9524a1ec2f77051]
I20240430 20:11:49.379422 20695 raft_consensus.cc:2781] T 47e98935eb21449abd3d4d24327778d5 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:49.379554 20695 raft_consensus.cc:482] T 47e98935eb21449abd3d4d24327778d5 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:49.379674 20695 raft_consensus.cc:3037] T 47e98935eb21449abd3d4d24327778d5 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:49.379853 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_07993f6164104b888058e1b35c9f4ed7 [id=f2e6cac20f6a4261bb60f2e40f210491]
I20240430 20:11:49.380610 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_df40d2a666d94cada72ebdb96b7d7f9e [id=f33b27d82b54476eb1eab57b39b91108]
I20240430 20:11:49.381273 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_e860191f401c434e8cf0085ef41952b9 [id=f393bcc9669c4ea985a2d40deb161f0c]
I20240430 20:11:49.381332 20629 consensus_queue.cc:1035] T d7c1fc7e81364fb890bbc13d7fb6848b P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:49.381909 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_3a35534b6ef2416b968740d28b156dd0 [id=f3c08d1b91be4bf58242ca535ff28897]
I20240430 20:11:49.382402 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_fc0a4540f98b4eea95099b8ab46ea235 [id=f3c53ccffe64410495cc8c379a05cfae]
I20240430 20:11:49.382860 20695 raft_consensus.cc:504] T 47e98935eb21449abd3d4d24327778d5 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.383307 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_1cf914acf7fe4e98b8dbc297b6372f21 [id=f4ba9536e9c141deb68f4f62b7e8fa70]
I20240430 20:11:49.384130 20695 leader_election.cc:288] T 47e98935eb21449abd3d4d24327778d5 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Requested vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:49.384177 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_7199d7c36d3f45f79cf9e4e2b4145592 [id=f6aef8f6a9164dd78f49e03f723b45a9]
I20240430 20:11:49.384790 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_2582fc0f97c1470d8b6b21cd1240697e [id=fc059935f9c943389180fab316c33a05]
I20240430 20:11:49.385407 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_f2a19db5098448f79bf95cccb1abd368 [id=fcaf7a6481644ae8990aa4e3d473bd50]
I20240430 20:11:49.385890 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_6c7df3aa7d8741ae9ac67cf4af868788 [id=fd542088d62a4234980d673ec2f0c9bf]
I20240430 20:11:49.386072 20629 raft_consensus.cc:482] T 9c73d0a6bc414d3b9900570848d19a51 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:49.386220 20629 raft_consensus.cc:504] T 9c73d0a6bc414d3b9900570848d19a51 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.386883 20697 catalog_manager.cc:669] Loaded metadata for table default.Table_58f35a2120874472978ca0e5f816e86c [id=ff18001c6d814afc8c1768d0c84a6bac]
I20240430 20:11:49.386888 20629 leader_election.cc:288] T 9c73d0a6bc414d3b9900570848d19a51 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:49.386849 18475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47e98935eb21449abd3d4d24327778d5" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:49.387010 18475 raft_consensus.cc:3037] T 47e98935eb21449abd3d4d24327778d5 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:49.388715 20695 consensus_queue.cc:1035] T e4cffa295d8c401aa8415ce7e8f56526 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:49.389564 18750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c73d0a6bc414d3b9900570848d19a51" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:49.389647 18475 raft_consensus.cc:2454] T 47e98935eb21449abd3d4d24327778d5 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
W20240430 20:11:49.389971 18417 leader_election.cc:341] T 9c73d0a6bc414d3b9900570848d19a51 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:49.390424 18699 leader_election.cc:302] T 47e98935eb21449abd3d4d24327778d5 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters:
I20240430 20:11:49.390933 18985 tablet_bootstrap.cc:492] T 50e1c55a1a0a484984ad5cbb78eec2ed P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:49.391113 18985 ts_tablet_manager.cc:1367] T 50e1c55a1a0a484984ad5cbb78eec2ed P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.037s user 0.000s sys 0.004s
I20240430 20:11:49.391896 18985 raft_consensus.cc:348] T 50e1c55a1a0a484984ad5cbb78eec2ed P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.392136 18985 raft_consensus.cc:374] T 50e1c55a1a0a484984ad5cbb78eec2ed P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:49.392247 18985 raft_consensus.cc:729] T 50e1c55a1a0a484984ad5cbb78eec2ed P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:49.392506 20587 raft_consensus.cc:2781] T 47e98935eb21449abd3d4d24327778d5 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:49.392638 20587 raft_consensus.cc:686] T 47e98935eb21449abd3d4d24327778d5 P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Becoming Leader. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Running, Role: LEADER
I20240430 20:11:49.392539 18985 consensus_queue.cc:260] T 50e1c55a1a0a484984ad5cbb78eec2ed P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.392884 20587 consensus_queue.cc:237] T 47e98935eb21449abd3d4d24327778d5 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.394088 18985 ts_tablet_manager.cc:1398] T 50e1c55a1a0a484984ad5cbb78eec2ed P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240430 20:11:49.395200 18985 tablet_bootstrap.cc:492] T 488d0e609b224674932f890939664df2 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:49.399832 18985 tablet_bootstrap.cc:654] T 488d0e609b224674932f890939664df2 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:49.369730 18983 ts_tablet_manager.cc:1367] T 1701a15b115741aea13f8327e3d26b34 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.076s user 0.005s sys 0.001s
I20240430 20:11:49.370365 18623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "488d0e609b224674932f890939664df2" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:49.404460 18623 raft_consensus.cc:2454] T 488d0e609b224674932f890939664df2 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:49.374045 18619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47e98935eb21449abd3d4d24327778d5" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:49.388823 18608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47e98935eb21449abd3d4d24327778d5" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:49.404449 18983 raft_consensus.cc:348] T 1701a15b115741aea13f8327e3d26b34 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.404958 18983 raft_consensus.cc:374] T 1701a15b115741aea13f8327e3d26b34 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:49.405175 18983 raft_consensus.cc:729] T 1701a15b115741aea13f8327e3d26b34 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:49.405833 18983 consensus_queue.cc:260] T 1701a15b115741aea13f8327e3d26b34 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.401382 20489 raft_consensus.cc:482] T 1f4e860484784aa9a2d28af2e9fb611c P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:49.413215 20489 raft_consensus.cc:504] T 1f4e860484784aa9a2d28af2e9fb611c P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.413878 20489 leader_election.cc:288] T 1f4e860484784aa9a2d28af2e9fb611c P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:49.388283 18620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c73d0a6bc414d3b9900570848d19a51" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:49.415791 18474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f4e860484784aa9a2d28af2e9fb611c" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
W20240430 20:11:49.416630 18558 leader_election.cc:341] T 1f4e860484784aa9a2d28af2e9fb611c P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:49.417533 18750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f4e860484784aa9a2d28af2e9fb611c" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
W20240430 20:11:49.418596 18557 leader_election.cc:341] T 1f4e860484784aa9a2d28af2e9fb611c P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:49.418723 18557 leader_election.cc:302] T 1f4e860484784aa9a2d28af2e9fb611c P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:49.419096 20489 raft_consensus.cc:2726] T 1f4e860484784aa9a2d28af2e9fb611c P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:49.421628 18988 tablet_bootstrap.cc:492] T a445fdfde01c4371a5e6b3daea5b00ab P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:49.421958 18988 ts_tablet_manager.cc:1367] T a445fdfde01c4371a5e6b3daea5b00ab P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.128s user 0.002s sys 0.002s
W20240430 20:11:49.423235 18415 leader_election.cc:341] T 9c73d0a6bc414d3b9900570848d19a51 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:49.423310 20697 tablet_loader.cc:96] loaded metadata for tablet 00be6ddd91e84865a48743fb4d077eec (table default.Table_c5c549bbf87d46c7886dad5f93086cd5 [id=5db65c3521504fffbe2599dbb5337a6a])
I20240430 20:11:49.423357 18415 leader_election.cc:302] T 9c73d0a6bc414d3b9900570848d19a51 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a; no voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710
I20240430 20:11:49.423862 20629 raft_consensus.cc:2726] T 9c73d0a6bc414d3b9900570848d19a51 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:49.423630 18988 raft_consensus.cc:348] T a445fdfde01c4371a5e6b3daea5b00ab P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.423986 20697 tablet_loader.cc:96] loaded metadata for tablet 01436e5b6232494babbb414342f03090 (table default.Table_a8d5ac73c6d047698d87b3d1ba34e66f [id=9c80193cde2346c982cf84ae9612e2f7])
I20240430 20:11:49.424050 18988 raft_consensus.cc:374] T a445fdfde01c4371a5e6b3daea5b00ab P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:49.424198 18988 raft_consensus.cc:729] T a445fdfde01c4371a5e6b3daea5b00ab P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:49.424474 20629 consensus_queue.cc:1035] T d7c1fc7e81364fb890bbc13d7fb6848b P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:49.426551 18983 ts_tablet_manager.cc:1398] T 1701a15b115741aea13f8327e3d26b34 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.024s user 0.001s sys 0.003s
I20240430 20:11:49.427358 18983 tablet_bootstrap.cc:492] T f3711610610542128c4a84e17bce4a3e P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:49.430243 20697 tablet_loader.cc:96] loaded metadata for tablet 0368d37dd9e2452982fb81f539c6805b (table default.Table_404b186c544e4b30bb36cb77d1a14120 [id=4330626e342a48699ea939add5c91b11])
I20240430 20:11:49.431047 18696 leader_election.cc:302] T 488d0e609b224674932f890939664df2 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a
W20240430 20:11:49.431970 18696 leader_election.cc:341] T 47e98935eb21449abd3d4d24327778d5 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:49.432096 20697 tablet_loader.cc:96] loaded metadata for tablet 06e82c706b9c4f35bf53faddf57b3fd7 (table default.Table_4bac430d51a7461e919b79d3a2f92a8e [id=afaa8b1a82404015ba960f018dd7550b])
I20240430 20:11:49.435215 20697 tablet_loader.cc:96] loaded metadata for tablet 086b3dfa84c045a18c8c4f177dce606a (table default.Table_df19cb8a6f4d46699350ec5b69c14e5a [id=197923633fc54a4582207942b51530e2])
I20240430 20:11:49.432193 20695 raft_consensus.cc:2781] T 488d0e609b224674932f890939664df2 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:49.435961 20695 raft_consensus.cc:482] T 488d0e609b224674932f890939664df2 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:49.436143 20695 raft_consensus.cc:3037] T 488d0e609b224674932f890939664df2 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:49.436208 20697 tablet_loader.cc:96] loaded metadata for tablet 0a16d98f700c46e29f6dd8568752ef29 (table default.Table_925990f1c73245818b39aafeb0de55e3 [id=5979225bccda4e35bea4bec2c67b22df])
I20240430 20:11:49.436669 20697 tablet_loader.cc:96] loaded metadata for tablet 0a558d18cdfa4b33b4f72dc9041bfd53 (table default.Table_50a4836df84543a888fcdb06ccc5ec41 [id=4cc347c303a04ae2864738e6b2cdc519])
I20240430 20:11:49.425169 18988 consensus_queue.cc:260] T a445fdfde01c4371a5e6b3daea5b00ab P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.440829 20697 tablet_loader.cc:96] loaded metadata for tablet 0c42e66f0e4f4291ad906b66f212165d (table default.Table_43043172400c49598042d752412c3a28 [id=c4cfc76234794e428b432eb2f5c74bb9])
I20240430 20:11:49.441454 18988 ts_tablet_manager.cc:1398] T a445fdfde01c4371a5e6b3daea5b00ab P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.019s user 0.001s sys 0.001s
I20240430 20:11:49.441941 20697 tablet_loader.cc:96] loaded metadata for tablet 0eb4f3e7ece6496293e01b63620d644b (table default.Table_b6ff64dbfc324fb199b8e6eebd51fd50 [id=4b13edffc2a4446dba14079d2b0f2ca2])
I20240430 20:11:49.442257 18988 tablet_bootstrap.cc:492] T 50e1c55a1a0a484984ad5cbb78eec2ed P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:49.447573 18988 tablet_bootstrap.cc:654] T 50e1c55a1a0a484984ad5cbb78eec2ed P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:49.448405 20697 tablet_loader.cc:96] loaded metadata for tablet 1050ab528c7d41559831652a1437d358 (table default.Table_acddc4cbf6334546b5fb3e4752fef2a5 [id=274cb15fd751410e83327ac4e8dc4629])
I20240430 20:11:49.449215 20697 tablet_loader.cc:96] loaded metadata for tablet 107638df57ed43adb6e13d47997df541 (table default.Table_650b1b52ac7a453a95434b798c408f6e [id=68e19fd08f164090a84319c51b938b03])
I20240430 20:11:49.449648 20697 tablet_loader.cc:96] loaded metadata for tablet 108e88d8caf6449092b3df372cdb4f56 (table default.Table_70ba80736dca466e9ca9c231a40e7bd4 [id=1399e5a2de60445ab758e25dbc54ba34])
I20240430 20:11:49.450115 20697 tablet_loader.cc:96] loaded metadata for tablet 115c4a12440647a4bb46eb6824f7ea98 (table default.Table_cd719a43207f4bf983e1b8e6d236bb16 [id=47e0b04b17834deb833a8f31c123edc5])
I20240430 20:11:49.450572 20697 tablet_loader.cc:96] loaded metadata for tablet 118eb508b3294b55abbda27e98495353 (table default.Table_d770977de70d4cac8511d8d125c1b160 [id=d80cad6e0e734b8fa6d2011a30f9657b])
I20240430 20:11:49.451277 20697 tablet_loader.cc:96] loaded metadata for tablet 12795670f21447a8964fe1efc6e92398 (table default.Table_896bbae1f6944005b0272ca8ebc97218 [id=d5cea90cbc2c48df9d8b80da1eccc949])
I20240430 20:11:49.452042 20697 tablet_loader.cc:96] loaded metadata for tablet 12c6e131fa904d23865b8673b58a7e33 (table default.Table_a9dc8fc6a4aa46dc914d0e327a4de554 [id=19f4e6decedf4ee59fccc0a24a05cedd])
I20240430 20:11:49.452561 20697 tablet_loader.cc:96] loaded metadata for tablet 130007acc8ba4e709c3eb85fabe75903 (table default.Table_78987875650940c293247bc43db345ab [id=36f478a20a1f4a53a52fc29ade62b7f6])
I20240430 20:11:49.453454 20697 tablet_loader.cc:96] loaded metadata for tablet 13c2e69ae91f4957887fd64b90f6e435 (table default.Table_6b67c0c03ceb4b9491ad5828022b357f [id=368fcf43e863494ca260862d12331c0d])
I20240430 20:11:49.453891 20697 tablet_loader.cc:96] loaded metadata for tablet 13f0a0a87da242c8acb9892bc2897616 (table default.Table_2e4e483bcd6c4dd6bdfcc938cb0e2da6 [id=afdab68d2df0433f9102de19b55c60b5])
I20240430 20:11:49.454059 20695 raft_consensus.cc:504] T 488d0e609b224674932f890939664df2 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.454434 20697 tablet_loader.cc:96] loaded metadata for tablet 1462ecf1176c4e66be42abddbe36b00c (table default.Table_b49a7942aa8c4f58acb94c35d1cdbb70 [id=6257559b0e9545c58848a76af0c5a5f7])
I20240430 20:11:49.456702 20545 raft_consensus.cc:482] T 7683caba80594627900a0185f772c131 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5e1c6397f09747f091534faf93542710)
I20240430 20:11:49.456877 20545 raft_consensus.cc:504] T 7683caba80594627900a0185f772c131 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.457577 20545 leader_election.cc:288] T 7683caba80594627900a0185f772c131 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:49.457669 20695 leader_election.cc:288] T 488d0e609b224674932f890939664df2 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Requested vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
W20240430 20:11:49.435729 18696 leader_election.cc:341] T 47e98935eb21449abd3d4d24327778d5 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:49.458079 18983 tablet_bootstrap.cc:654] T f3711610610542128c4a84e17bce4a3e P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:49.458667 18608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "488d0e609b224674932f890939664df2" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:49.458703 18474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "488d0e609b224674932f890939664df2" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:49.458845 18608 raft_consensus.cc:3037] T 488d0e609b224674932f890939664df2 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
W20240430 20:11:49.459694 18417 leader_election.cc:341] T 7683caba80594627900a0185f772c131 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Not found: Tablet not found: 7683caba80594627900a0185f772c131
W20240430 20:11:49.460479 18699 leader_election.cc:341] T 488d0e609b224674932f890939664df2 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:49.461450 18608 raft_consensus.cc:2454] T 488d0e609b224674932f890939664df2 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:49.465214 20489 raft_consensus.cc:482] T 3e5225f1f5f544ecbf8dfa470af3daca P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:49.465342 20489 raft_consensus.cc:504] T 3e5225f1f5f544ecbf8dfa470af3daca P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.465924 20489 leader_election.cc:288] T 3e5225f1f5f544ecbf8dfa470af3daca P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:49.467121 18474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e5225f1f5f544ecbf8dfa470af3daca" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:49.467336 18474 raft_consensus.cc:2454] T 3e5225f1f5f544ecbf8dfa470af3daca P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 0.
I20240430 20:11:49.467638 18758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e5225f1f5f544ecbf8dfa470af3daca" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
W20240430 20:11:49.468235 18557 leader_election.cc:341] T 3e5225f1f5f544ecbf8dfa470af3daca P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:49.468796 18558 leader_election.cc:302] T 3e5225f1f5f544ecbf8dfa470af3daca P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters: 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:49.469805 20489 raft_consensus.cc:2781] T 3e5225f1f5f544ecbf8dfa470af3daca P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:49.469890 20489 raft_consensus.cc:482] T 3e5225f1f5f544ecbf8dfa470af3daca P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:49.469969 20489 raft_consensus.cc:3037] T 3e5225f1f5f544ecbf8dfa470af3daca P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:49.470386 20697 tablet_loader.cc:96] loaded metadata for tablet 1701a15b115741aea13f8327e3d26b34 (table default.Table_3776a4b9d4f24aa6b7f882bb4d0116ef [id=7127dbd368864f7ba80a755549fe7ede])
I20240430 20:11:49.473397 20489 raft_consensus.cc:504] T 3e5225f1f5f544ecbf8dfa470af3daca P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.474750 18474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e5225f1f5f544ecbf8dfa470af3daca" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:49.474972 18474 raft_consensus.cc:3037] T 3e5225f1f5f544ecbf8dfa470af3daca P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:49.477895 18474 raft_consensus.cc:2454] T 3e5225f1f5f544ecbf8dfa470af3daca P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
I20240430 20:11:49.478299 18558 leader_election.cc:302] T 3e5225f1f5f544ecbf8dfa470af3daca P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:49.478406 20489 leader_election.cc:288] T 3e5225f1f5f544ecbf8dfa470af3daca P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:49.478664 20384 raft_consensus.cc:2781] T 3e5225f1f5f544ecbf8dfa470af3daca P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:49.478623 18758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e5225f1f5f544ecbf8dfa470af3daca" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:49.478801 20384 raft_consensus.cc:686] T 3e5225f1f5f544ecbf8dfa470af3daca P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Becoming Leader. State: Replica: 5e1c6397f09747f091534faf93542710, State: Running, Role: LEADER
I20240430 20:11:49.478479 18985 tablet_bootstrap.cc:492] T 488d0e609b224674932f890939664df2 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:49.479085 20384 consensus_queue.cc:237] T 3e5225f1f5f544ecbf8dfa470af3daca P 5e1c6397f09747f091534faf93542710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
W20240430 20:11:49.479326 18415 leader_election.cc:341] T 7683caba80594627900a0185f772c131 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Not found: Tablet not found: 7683caba80594627900a0185f772c131
I20240430 20:11:49.479460 18415 leader_election.cc:302] T 7683caba80594627900a0185f772c131 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a; no voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710
I20240430 20:11:49.479876 20545 raft_consensus.cc:2726] T 7683caba80594627900a0185f772c131 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240430 20:11:49.480230 18557 leader_election.cc:341] T 3e5225f1f5f544ecbf8dfa470af3daca P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:49.481308 18985 ts_tablet_manager.cc:1367] T 488d0e609b224674932f890939664df2 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.087s user 0.003s sys 0.005s
I20240430 20:11:49.486498 20697 tablet_loader.cc:96] loaded metadata for tablet 17ddcaf971154a52b606bbf29a539bf0 (table default.Table_57208e9f76f14782a806f0a2971a3a58 [id=872c2faa4a464edd98a51df2e1b3f817])
I20240430 20:11:49.481105 18696 leader_election.cc:302] T 488d0e609b224674932f890939664df2 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a
I20240430 20:11:49.481715 20587 raft_consensus.cc:482] T f3711610610542128c4a84e17bce4a3e P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:49.487075 20587 raft_consensus.cc:504] T f3711610610542128c4a84e17bce4a3e P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.487821 20587 leader_election.cc:288] T f3711610610542128c4a84e17bce4a3e P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:49.487987 20587 raft_consensus.cc:2781] T 488d0e609b224674932f890939664df2 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:49.488098 20587 raft_consensus.cc:686] T 488d0e609b224674932f890939664df2 P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Becoming Leader. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Running, Role: LEADER
I20240430 20:11:49.488375 20587 consensus_queue.cc:237] T 488d0e609b224674932f890939664df2 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.489169 20545 raft_consensus.cc:482] T a445fdfde01c4371a5e6b3daea5b00ab P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:49.489311 20545 raft_consensus.cc:504] T a445fdfde01c4371a5e6b3daea5b00ab P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.489965 20697 tablet_loader.cc:96] loaded metadata for tablet 1ba8b96864214bb3bc71ab99026632f3 (table default.Table_fac4c5c0ce954200838069be87be2ad9 [id=873474f62b8243e4b9068504b9f189a4])
I20240430 20:11:49.489971 20545 leader_election.cc:288] T a445fdfde01c4371a5e6b3daea5b00ab P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:49.490510 18474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3711610610542128c4a84e17bce4a3e" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:49.491029 18620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a445fdfde01c4371a5e6b3daea5b00ab" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
W20240430 20:11:49.491876 18415 leader_election.cc:341] T a445fdfde01c4371a5e6b3daea5b00ab P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Illegal state: must be running to vote when last-logged opid is not known
W20240430 20:11:49.492209 18699 leader_election.cc:341] T f3711610610542128c4a84e17bce4a3e P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:49.492529 20697 tablet_loader.cc:96] loaded metadata for tablet 1faa2f2c84184b129ba6e4d2857822f0 (table default.Table_2eed103638364cde94cd8af68d3cdf9b [id=1a382ad8009b4522a50d55c307db8f96])
I20240430 20:11:49.493134 20697 tablet_loader.cc:96] loaded metadata for tablet 1fe7c6f1ab54499facd4d01573d3e7bc (table default.Table_0366454b829949928c7b5e79bbf3a649 [id=edc38e64373242f7bb394a526bb2a4f8])
I20240430 20:11:49.494174 18758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a445fdfde01c4371a5e6b3daea5b00ab" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:49.494393 18758 raft_consensus.cc:2454] T a445fdfde01c4371a5e6b3daea5b00ab P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 0.
W20240430 20:11:49.494818 19895 consensus_peers.cc:485] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8 -> Peer b632d6ef4c104a058a6db208024cab33 (127.17.177.60:37225): Couldn't send request to peer b632d6ef4c104a058a6db208024cab33. Status: Network error: Client connection negotiation failed: client connection to 127.17.177.60:37225: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240430 20:11:49.495481 18417 leader_election.cc:302] T a445fdfde01c4371a5e6b3daea5b00ab P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters: 5e1c6397f09747f091534faf93542710
I20240430 20:11:49.495982 20545 raft_consensus.cc:2781] T a445fdfde01c4371a5e6b3daea5b00ab P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:49.496058 20545 raft_consensus.cc:482] T a445fdfde01c4371a5e6b3daea5b00ab P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:49.496120 20545 raft_consensus.cc:3037] T a445fdfde01c4371a5e6b3daea5b00ab P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:49.493445 18985 raft_consensus.cc:348] T 488d0e609b224674932f890939664df2 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.496632 18985 raft_consensus.cc:374] T 488d0e609b224674932f890939664df2 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:49.497092 18988 tablet_bootstrap.cc:492] T 50e1c55a1a0a484984ad5cbb78eec2ed P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:49.497236 18985 raft_consensus.cc:729] T 488d0e609b224674932f890939664df2 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:49.497437 20697 tablet_loader.cc:96] loaded metadata for tablet 22fef8bb6b034db098c3372ae992f7a7 (table default.Table_4440df38610340758047115845b42b95 [id=7d186ca20c094cc19aef792fe1c094e9])
I20240430 20:11:49.498631 18988 ts_tablet_manager.cc:1367] T 50e1c55a1a0a484984ad5cbb78eec2ed P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.057s user 0.007s sys 0.001s
I20240430 20:11:49.499389 20545 raft_consensus.cc:504] T a445fdfde01c4371a5e6b3daea5b00ab P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.500886 20697 tablet_loader.cc:96] loaded metadata for tablet 266fd23786bc435db5083370f3330ed0 (table default.Table_cf60b652e9dc49e7a948ce66fd1cbffe [id=9734bb8dd05a400ca4244b3829f01dee])
I20240430 20:11:49.500927 20545 leader_election.cc:288] T a445fdfde01c4371a5e6b3daea5b00ab P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:49.501017 18616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a445fdfde01c4371a5e6b3daea5b00ab" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:49.499797 18985 consensus_queue.cc:260] T 488d0e609b224674932f890939664df2 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.501776 18758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a445fdfde01c4371a5e6b3daea5b00ab" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:49.501986 18758 raft_consensus.cc:3037] T a445fdfde01c4371a5e6b3daea5b00ab P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:49.502025 20697 tablet_loader.cc:96] loaded metadata for tablet 2736c1920d264d7b9d7fd5da930543a5 (table default.Table_123c6be593f44fc2ac4115d192c43e68 [id=a7adb9bbadef40609bee9bd9e86ad818])
W20240430 20:11:49.502835 18415 leader_election.cc:341] T a445fdfde01c4371a5e6b3daea5b00ab P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:49.504261 20697 tablet_loader.cc:96] loaded metadata for tablet 2a08e36c39824abf98b9b6214c8b84cb (table default.Table_027891085c5747d58ddb32ad51a453ba [id=45042e2e54a14f558c9968969d3618c5])
I20240430 20:11:49.505625 18758 raft_consensus.cc:2454] T a445fdfde01c4371a5e6b3daea5b00ab P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 1.
I20240430 20:11:49.507385 18417 leader_election.cc:302] T a445fdfde01c4371a5e6b3daea5b00ab P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters: 5e1c6397f09747f091534faf93542710
I20240430 20:11:49.507836 20630 raft_consensus.cc:2781] T a445fdfde01c4371a5e6b3daea5b00ab P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:49.508008 20630 raft_consensus.cc:686] T a445fdfde01c4371a5e6b3daea5b00ab P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Becoming Leader. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Running, Role: LEADER
I20240430 20:11:49.508303 20630 consensus_queue.cc:237] T a445fdfde01c4371a5e6b3daea5b00ab P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.511157 18985 ts_tablet_manager.cc:1398] T 488d0e609b224674932f890939664df2 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.029s user 0.003s sys 0.001s
I20240430 20:11:49.514832 18620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3711610610542128c4a84e17bce4a3e" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:49.512209 18988 raft_consensus.cc:348] T 50e1c55a1a0a484984ad5cbb78eec2ed P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.516176 18985 tablet_bootstrap.cc:492] T fda942d50b014f3d9589e1fb909fc2fc P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:49.516041 18988 raft_consensus.cc:374] T 50e1c55a1a0a484984ad5cbb78eec2ed P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20240430 20:11:49.518095 18696 leader_election.cc:341] T f3711610610542128c4a84e17bce4a3e P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:49.523849 20697 tablet_loader.cc:96] loaded metadata for tablet 2cd41b407d6b444d844c4c7c0ebf82e5 (table default.Table_673d1448679f4b15a5ee2a1369c8f8ea [id=3813b5e9778c4bd683a06598cef0d86f])
I20240430 20:11:49.525210 20697 tablet_loader.cc:96] loaded metadata for tablet 2cd6744ae3354f31ae8fa68f751c5483 (table default.Table_f86df7f6c95e4da48b80ae4733dc6994 [id=a29411bd07ba400c942bedc9e8e19884])
I20240430 20:11:49.517556 18988 raft_consensus.cc:729] T 50e1c55a1a0a484984ad5cbb78eec2ed P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:49.526436 18696 leader_election.cc:302] T f3711610610542128c4a84e17bce4a3e P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710
I20240430 20:11:49.526471 20697 tablet_loader.cc:96] loaded metadata for tablet 2da900b6bbfa4fa99fe8494fd2e68da8 (table default.Table_19a91a7f17ff456ebb145e8dd89c0940 [id=67283dd3995a44c6a7a7bc8722e4f955])
I20240430 20:11:49.527107 18988 consensus_queue.cc:260] T 50e1c55a1a0a484984ad5cbb78eec2ed P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.528542 20697 tablet_loader.cc:96] loaded metadata for tablet 2e1cac64cefe4336bd3207b5ca58b631 (table default.Table_ad4322a6bb564abd80dad138208e420d [id=0f4d7a1eb8834b559e79685fbb9de43b])
I20240430 20:11:49.529172 20587 raft_consensus.cc:2726] T f3711610610542128c4a84e17bce4a3e P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:49.529724 20697 tablet_loader.cc:96] loaded metadata for tablet 2e54ea9db8584db8abc36c1d3fd02c76 (table default.Table_75efd32c9420454f901c61882c8c29bc [id=0d741ba94928463a9929f799e0336fdb])
I20240430 20:11:49.531055 18988 ts_tablet_manager.cc:1398] T 50e1c55a1a0a484984ad5cbb78eec2ed P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.031s user 0.004s sys 0.000s
I20240430 20:11:49.532722 20697 tablet_loader.cc:96] loaded metadata for tablet 2e79df8744e74b5991a79ce978ff0202 (table default.Table_276bbdc7cfe542d1a2936485a2aad1ec [id=f146a15e6e8a4acbb4c734c1c863da57])
I20240430 20:11:49.533860 20697 tablet_loader.cc:96] loaded metadata for tablet 2fc6908ecf954020a676b9d10a2b00e7 (table default.Table_766e3df58895427b9e5179eb555fe7e4 [id=5c11311047b34a3eb06cc06e8cc10223])
I20240430 20:11:49.535063 20697 tablet_loader.cc:96] loaded metadata for tablet 2fe548ac307d4b159c30659aa83fd76b (table default.Table_1407edea18d440e7adacadcfc8d89054 [id=b74e2a754da140628893bab105373983])
I20240430 20:11:49.535351 18988 tablet_bootstrap.cc:492] T 6db55bb510b24b0cae897936211e15f0 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:49.537343 20697 tablet_loader.cc:96] loaded metadata for tablet 30a521916bda46678be20a066bd1d3f3 (table default.Table_9579e986b42043f18f8e4aa8d0f6f754 [id=21e6703c9d844a37977f0f161c0b8c08])
I20240430 20:11:49.537930 20697 tablet_loader.cc:96] loaded metadata for tablet 3191aa54a9644783a2403991c3be27c8 (table default.Table_75336e7a0e5f4c2d909f5c5c385bc379 [id=be4876d319a24a958d50e3d0c508e95f])
I20240430 20:11:49.541328 20697 tablet_loader.cc:96] loaded metadata for tablet 330453ddeea6456e8ef93fa92608b0d3 (table default.Table_c1f36f7df5ae4e7c8eb8b4e99b89748b [id=a50abf28790d4284a76ac6ee22c02606])
I20240430 20:11:49.542177 20697 tablet_loader.cc:96] loaded metadata for tablet 331c031d44854d679a27d10d9151d0ea (table default.Table_763f0800e8bb4fa69976886416250f42 [id=61c8a5b99ef34a0baeeb343fef7cfd05])
I20240430 20:11:49.544307 18985 tablet_bootstrap.cc:654] T fda942d50b014f3d9589e1fb909fc2fc P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:49.545701 20697 tablet_loader.cc:96] loaded metadata for tablet 347a8224114b49e5b7dcb75d9b726e0a (table default.Table_dd1d113759284183905a803b2733436d [id=9d581bf380f54b4dab56aa34b9c4b1ea])
I20240430 20:11:49.547439 20697 tablet_loader.cc:96] loaded metadata for tablet 35875133118541ea909b9e644251a01e (table default.Table_6c7df3aa7d8741ae9ac67cf4af868788 [id=fd542088d62a4234980d673ec2f0c9bf])
I20240430 20:11:49.548828 20697 tablet_loader.cc:96] loaded metadata for tablet 35fdb052135541cfb42a8aeb03b64dc8 (table default.Table_088b70cec77744aaacf26aa6d4adebbd [id=3e0705b442cc45109c5750e006e0ff04])
I20240430 20:11:49.549605 20697 tablet_loader.cc:96] loaded metadata for tablet 36a07d27145a46e48c3f10e8211a389e (table default.Table_326878a52ab041609f3c339cd65b7013 [id=d4763eaeacb546ae94b77b5a78fef986])
I20240430 20:11:49.553809 18474 raft_consensus.cc:3037] T e4cffa295d8c401aa8415ce7e8f56526 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:49.555171 20697 tablet_loader.cc:96] loaded metadata for tablet 372b135dd01f44f98cd713ca8857ba61 (table default.Table_0163ce6f5a4c45299106daeffb3c22b1 [id=7f3a15a442984827a7521d5d1c98dc3d])
I20240430 20:11:49.555354 18988 tablet_bootstrap.cc:654] T 6db55bb510b24b0cae897936211e15f0 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:49.555619 20697 tablet_loader.cc:96] loaded metadata for tablet 37b2ea4be7494f0985808ade788dbe2b (table default.Table_5f6e0fce999b46de927f9271e5de5071 [id=0baad83a8c0f4aeebc7ddad068625a69])
I20240430 20:11:49.555992 20697 tablet_loader.cc:96] loaded metadata for tablet 3a03b6997921408cb5a65f4cae223e9c (table default.Table_744b5c96ea014bdb8feae740356a790e [id=83b1ac2e9c3d4a10b02f96ebb17faf44])
I20240430 20:11:49.556771 20697 tablet_loader.cc:96] loaded metadata for tablet 3e5225f1f5f544ecbf8dfa470af3daca (table default.Table_c7c87a8e019944cab3ec99e9563ac01a [id=abf3765074eb4d7d9d2bcc4ab721e6bd])
I20240430 20:11:49.557648 20697 tablet_loader.cc:96] loaded metadata for tablet 3fa6c1078c074ef4a7a408efa8c19467 (table default.Table_e311a584368b4233b6e5beb24587ce00 [id=65018c93b8e2409f984c52764e82dcb3])
I20240430 20:11:49.557940 20587 consensus_queue.cc:1035] T e4cffa295d8c401aa8415ce7e8f56526 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:49.558274 20697 tablet_loader.cc:96] loaded metadata for tablet 3fd0e8c360734dcbb75dc2958b70c2b8 (table default.Table_02d2b1f3815549c481a305c622c809ff [id=7e61baf429e4487fab0ba2761a2af49d])
I20240430 20:11:49.558687 20697 tablet_loader.cc:96] loaded metadata for tablet 40b4a8641a9b4d5195c9f9857019acd7 (table default.Table_8db024fb9ef241ed86cb98d1c3939607 [id=1ad13ddf3cfd402a84d2d4a5a55030fa])
I20240430 20:11:49.559190 20697 tablet_loader.cc:96] loaded metadata for tablet 417351cbc9d54f9d84bb7155f39da119 (table default.Table_5d0995eff94c404fa97b08aa190f4a7f [id=6a55f1d4a8094a88bd8dd5a076630b24])
I20240430 20:11:49.560889 20489 raft_consensus.cc:482] T 1701a15b115741aea13f8327e3d26b34 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:49.561076 20489 raft_consensus.cc:504] T 1701a15b115741aea13f8327e3d26b34 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.561623 20697 tablet_loader.cc:96] loaded metadata for tablet 4391b5227fb64ef481542a962a5e21a5 (table default.Table_d82824c897a94dd0ba45955f9473b626 [id=d794640d21a843a185038673f7eb2b00])
I20240430 20:11:49.561844 20489 leader_election.cc:288] T 1701a15b115741aea13f8327e3d26b34 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:49.562337 20697 tablet_loader.cc:96] loaded metadata for tablet 43b5e2c005a34df08cc4335c6e014cb5 (table default.Table_08dc70f541b842c78a3a946d2d02e207 [id=38f199a7538f4271b3a59fd93f29efa2])
I20240430 20:11:49.562851 18475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1701a15b115741aea13f8327e3d26b34" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:49.562997 20697 tablet_loader.cc:96] loaded metadata for tablet 43ec03fd93da43569bb9d3be01de43bc (table default.Table_cfb6af12a3114f96837868eaa33be09b [id=cdf3c4662ece4b0ba1c88057affd26e6])
I20240430 20:11:49.563444 20697 tablet_loader.cc:96] loaded metadata for tablet 4432d2c97b634a36b44e1d61d8bf2458 (table default.Table_fc68663ab2d74c34bedefcbad2d98a69 [id=d5cfd1c086e543b5a220e614a5241457])
W20240430 20:11:49.563575 18558 leader_election.cc:341] T 1701a15b115741aea13f8327e3d26b34 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:49.564139 18758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1701a15b115741aea13f8327e3d26b34" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
W20240430 20:11:49.564625 18557 leader_election.cc:341] T 1701a15b115741aea13f8327e3d26b34 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:49.564759 18557 leader_election.cc:302] T 1701a15b115741aea13f8327e3d26b34 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:49.565189 20489 raft_consensus.cc:2726] T 1701a15b115741aea13f8327e3d26b34 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:49.568604 20697 tablet_loader.cc:96] loaded metadata for tablet 46251c08277e498da04a41d009310c62 (table default.Table_59297b42e8cf4ab3b9e95e2e70d3c1fe [id=b1312a6cd1a54003b1db727a2a9d125d])
I20240430 20:11:49.570037 20697 tablet_loader.cc:96] loaded metadata for tablet 467cafc47106474e96f619b4043db233 (table default.Table_f45eee4d4b6a4baf83111f12dfd25a9f [id=ea6b9f7d86424c84a8ea05a5877afd06])
I20240430 20:11:49.570499 20697 tablet_loader.cc:96] loaded metadata for tablet 47bf64ecc41b445da4ff697e141243dd (table default.Table_f1ad29d5986d4b799cfc67afb2b0dfdf [id=838fc8b61a864dea8877a6f144e4cb4f])
I20240430 20:11:49.573307 20795 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240430 20:11:49.574054 20697 tablet_loader.cc:96] loaded metadata for tablet 488d0e609b224674932f890939664df2 (table default.Table_58f35a2120874472978ca0e5f816e86c [id=ff18001c6d814afc8c1768d0c84a6bac])
I20240430 20:11:49.574493 20795 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240430 20:11:49.578771 18988 tablet_bootstrap.cc:492] T 6db55bb510b24b0cae897936211e15f0 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:49.579200 20697 tablet_loader.cc:96] loaded metadata for tablet 4fa9575d96a84bef876467e21dada803 (table default.Table_27f3d711beeb4387a9088ab291168bc8 [id=434aba8e131342a3a9a1aca3278f9c0d])
I20240430 20:11:49.579823 18988 ts_tablet_manager.cc:1367] T 6db55bb510b24b0cae897936211e15f0 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.045s user 0.007s sys 0.000s
I20240430 20:11:49.580207 20697 tablet_loader.cc:96] loaded metadata for tablet 50e1c55a1a0a484984ad5cbb78eec2ed (table default.Table_211cbad17eba4c48ab02ad45543616ca [id=a693b863f7f34539aa46c3f006aca925])
I20240430 20:11:49.580938 20697 tablet_loader.cc:96] loaded metadata for tablet 53da4b3855344b6d92211af767195dc8 (table default.Table_07993f6164104b888058e1b35c9f4ed7 [id=f2e6cac20f6a4261bb60f2e40f210491])
I20240430 20:11:49.581961 18985 tablet_bootstrap.cc:492] T fda942d50b014f3d9589e1fb909fc2fc P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:49.582798 18983 tablet_bootstrap.cc:492] T f3711610610542128c4a84e17bce4a3e P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:49.582962 18983 ts_tablet_manager.cc:1367] T f3711610610542128c4a84e17bce4a3e P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.156s user 0.009s sys 0.004s
I20240430 20:11:49.583748 18983 raft_consensus.cc:348] T f3711610610542128c4a84e17bce4a3e P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.583971 18983 raft_consensus.cc:374] T f3711610610542128c4a84e17bce4a3e P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:49.584072 18983 raft_consensus.cc:729] T f3711610610542128c4a84e17bce4a3e P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:49.584306 18983 consensus_queue.cc:260] T f3711610610542128c4a84e17bce4a3e P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.584919 18983 ts_tablet_manager.cc:1398] T f3711610610542128c4a84e17bce4a3e P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240430 20:11:49.585202 18983 tablet_bootstrap.cc:492] T a445fdfde01c4371a5e6b3daea5b00ab P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:49.587239 18983 tablet_bootstrap.cc:654] T a445fdfde01c4371a5e6b3daea5b00ab P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:49.583154 20795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240430 20:11:49.590509 20697 tablet_loader.cc:96] loaded metadata for tablet 58c5ac58847e4e769e8e88e7e27494ae (table default.Table_71775acb989f4c2f87d670dff67a93b6 [id=a6c9697e1e9949b1a14d62c9f9ba4fed])
I20240430 20:11:49.590559 18985 ts_tablet_manager.cc:1367] T fda942d50b014f3d9589e1fb909fc2fc P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.076s user 0.003s sys 0.005s
I20240430 20:11:49.591405 20697 tablet_loader.cc:96] loaded metadata for tablet 5b26319488644d6a8b6be60460a5e7af (table default.Table_69892a8264a4477db48ee6248758e046 [id=5a1c4a7be2fc465fac89a50854eb3322])
I20240430 20:11:49.592707 20697 tablet_loader.cc:96] loaded metadata for tablet 5df364628561475391c9a5e10ff5a05f (table default.Table_3c8f9f7b22f64fb1bdc3f1550987233a [id=7fd6f8576f4c4cbc8ffcdb4ab95e15dc])
I20240430 20:11:49.592218 18985 raft_consensus.cc:348] T fda942d50b014f3d9589e1fb909fc2fc P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.593026 18985 raft_consensus.cc:374] T fda942d50b014f3d9589e1fb909fc2fc P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:49.593353 20697 tablet_loader.cc:96] loaded metadata for tablet 5f02d8862e1f4ccfa9dbda73dbe18615 (table default.Table_0b4b99c1c03a4e779046063159ce6861 [id=43d7bdc84bab4c54bac58dd1dd0f74ac])
I20240430 20:11:49.593327 18985 raft_consensus.cc:729] T fda942d50b014f3d9589e1fb909fc2fc P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:49.594622 20697 tablet_loader.cc:96] loaded metadata for tablet 62732af95c6a4c749c5b726f0460dc72 (table default.Table_e3160f5ced1b4718b479401ca4df675c [id=2d44ec94131148048021b2ee6366ba30])
I20240430 20:11:49.595111 20697 tablet_loader.cc:96] loaded metadata for tablet 66dd767bd3f8494f94061fb89e5e5680 (table default.Table_b064d5659f7243f5b0ec167fb6c3f404 [id=5227206f44d04fa89b4842b899f33afb])
I20240430 20:11:49.595995 20697 tablet_loader.cc:96] loaded metadata for tablet 677481aee5fa4eb4b189322d7a3f8f2e (table default.Table_1ce204e15fce4c469816cf4e7f80cac7 [id=d268c31d82df4871bc1a78aea5871b33])
I20240430 20:11:49.596701 20697 tablet_loader.cc:96] loaded metadata for tablet 681cdd85ac6847a0a1aaded310456582 (table default.Table_8e9e6e39df854972804dfa7c260af4e4 [id=e36e9fabccf14feeb0c643cb41b4a4f6])
I20240430 20:11:49.597569 20697 tablet_loader.cc:96] loaded metadata for tablet 69f88532952a4591bf1577541c3430e6 (table default.Table_1177828cad1f4a24bd397619dad96170 [id=a95fed1bc3fa4865bf9a35dfc51b659c])
I20240430 20:11:49.595993 18985 consensus_queue.cc:260] T fda942d50b014f3d9589e1fb909fc2fc P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.598114 20697 tablet_loader.cc:96] loaded metadata for tablet 6b51cbc731794083a88fe4bed1ddeebf (table default.Table_c6c962da746e4fd8a4d7200d09f08e9e [id=8e0f5d8c941d423ea7b30a89513d5170])
I20240430 20:11:49.599256 20697 tablet_loader.cc:96] loaded metadata for tablet 6e754af64cb64fd2a082adae7d5f24da (table default.Table_938c56d9ae2f4ff6b3264a53b65d0977 [id=1df4e95702cd46f59ab21b101049a9ed])
I20240430 20:11:49.600293 20697 tablet_loader.cc:96] loaded metadata for tablet 6f3996b7e29941e391ea141dd6cc44b2 (table default.Table_da5216d025844afda5a83ada555bc120 [id=dc55b25c3ca14863a21042e805e3bf9a])
I20240430 20:11:49.601027 20697 tablet_loader.cc:96] loaded metadata for tablet 70247a4c910946c0afc24ce5ce0e85d6 (table default.Table_e20b13b4103849b9bc5e5f96f9b911e4 [id=9fd8ccb805024968b9ad8ab060d5db3b])
I20240430 20:11:49.601506 20697 tablet_loader.cc:96] loaded metadata for tablet 716c62f14ca74da9af3282128441b44f (table default.Table_2c1b15dc75504997924b1f36ca1f69a8 [id=634df0651c254f1c9e8db138ad908f31])
I20240430 20:11:49.602013 20697 tablet_loader.cc:96] loaded metadata for tablet 72205028d0654e69a6fa20f105604e1c (table default.Table_83b26c6b5e1942da864b8e60e882f5b4 [id=a09a93469a8f450895fa1756e1074c16])
I20240430 20:11:49.602480 20697 tablet_loader.cc:96] loaded metadata for tablet 724e51ff05ec4baebc2bb55e05c8eddc (table default.Table_19df33614a6c4f94b6419a5b441e3ef8 [id=4dce8a767f0541e1beb5464b2ed95d06])
I20240430 20:11:49.603332 20697 tablet_loader.cc:96] loaded metadata for tablet 72d316f424214792ac1d5029ffb1b04a (table default.Table_69622a6c03f14409a34ad4d98f24c5a1 [id=43f5d7056f5d4ef98f6861d8f40643f3])
I20240430 20:11:49.601830 18988 raft_consensus.cc:348] T 6db55bb510b24b0cae897936211e15f0 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.605392 20697 tablet_loader.cc:96] loaded metadata for tablet 772eba5f24c549c791cad75565858285 (table default.Table_5fc77d5f60ae44f4bb4ca9f7f183672c [id=6ca223ca6354449c959d85ed2ac6a9c8])
I20240430 20:11:49.605970 20697 tablet_loader.cc:96] loaded metadata for tablet 78ba48dadef548708153878810cc46f8 (table default.Table_b9983e101a6547649eb44ee04b5e1147 [id=29769be2b9ec4ecba9de5c0a65692f85])
I20240430 20:11:49.606423 20697 tablet_loader.cc:96] loaded metadata for tablet 798a13f545fd4d8f8c654eaa2452eda1 (table default.Table_fd33fc89e9d84f5c87a6572acdfe2c94 [id=2ab17149d0444085b85e5f0959ebb2ef])
I20240430 20:11:49.605693 18988 raft_consensus.cc:374] T 6db55bb510b24b0cae897936211e15f0 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:49.607110 20697 tablet_loader.cc:96] loaded metadata for tablet 7a0176f8a74c47efbbe3d3343b3dc896 (table default.Table_94e5f2a77765434e8a76d9a558b23ae2 [id=73e5c9f0470d47849109819c063e1ce4])
I20240430 20:11:49.606127 18985 ts_tablet_manager.cc:1398] T fda942d50b014f3d9589e1fb909fc2fc P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.015s user 0.000s sys 0.004s
I20240430 20:11:49.607632 20697 tablet_loader.cc:96] loaded metadata for tablet 7bde1d6934784c9aa5911181e8df9eeb (table default.Table_7199d7c36d3f45f79cf9e4e2b4145592 [id=f6aef8f6a9164dd78f49e03f723b45a9])
I20240430 20:11:49.608065 20697 tablet_loader.cc:96] loaded metadata for tablet 7c61dc93df2541f09e9e525181a02b22 (table default.Table_9c955414a012419eb8f904a0878a6336 [id=0e5a8ce8c9a440caa87263100895a12e])
I20240430 20:11:49.608175 20545 raft_consensus.cc:482] T 50e1c55a1a0a484984ad5cbb78eec2ed P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:49.607266 18988 raft_consensus.cc:729] T 6db55bb510b24b0cae897936211e15f0 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:49.608728 20697 tablet_loader.cc:96] loaded metadata for tablet 7ce09fcbe1ac457a9c05efaf000c3263 (table default.Table_edc4c27ed846454288f73f6a0d88cb69 [id=78fd3f516083489ab3b42d90dcef7e4d])
I20240430 20:11:49.608870 20545 raft_consensus.cc:504] T 50e1c55a1a0a484984ad5cbb78eec2ed P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.609628 20697 tablet_loader.cc:96] loaded metadata for tablet 7eae3a1a8d6c4b438d3424eb909a1fe6 (table default.Table_304521bb46d9434481340ff5ce7ca994 [id=c51917e099fa4676a092373e1ad201f7])
I20240430 20:11:49.610038 20697 tablet_loader.cc:96] loaded metadata for tablet 7fceb89e8ee148999f747b08a15fd274 (table default.Table_5d3cb1a25c884804b05b07531e1b6965 [id=55e60456c89d441a8b5874a2063d52ef])
I20240430 20:11:49.610602 18758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50e1c55a1a0a484984ad5cbb78eec2ed" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:49.610848 18758 raft_consensus.cc:2454] T 50e1c55a1a0a484984ad5cbb78eec2ed P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 0.
I20240430 20:11:49.611196 20697 tablet_loader.cc:96] loaded metadata for tablet 81787eccca6c42c1b421377a3bf9ebe6 (table default.Table_0b4ca4cd0af148399dfa21f2d32114e3 [id=7f602584cef9486eb7d9b4389be3f43e])
I20240430 20:11:49.611292 18417 leader_election.cc:302] T 50e1c55a1a0a484984ad5cbb78eec2ed P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters:
I20240430 20:11:49.611652 20629 raft_consensus.cc:2781] T 50e1c55a1a0a484984ad5cbb78eec2ed P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:49.611757 20629 raft_consensus.cc:482] T 50e1c55a1a0a484984ad5cbb78eec2ed P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:49.611853 20629 raft_consensus.cc:3037] T 50e1c55a1a0a484984ad5cbb78eec2ed P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:49.611702 20697 tablet_loader.cc:96] loaded metadata for tablet 821c55edb3414d72b1489eea97d3f73a (table default.Table_4d68c38d9ba7452a829f8318b2b267b8 [id=d06e3e9045b1429aaeebd1fd67d841de])
I20240430 20:11:49.612371 20697 tablet_loader.cc:96] loaded metadata for tablet 828ca913e2aa4102a6277bdbc6686ef5 (table default.Table_5d51defcfdfd4d7f8e7d341c5daf0cdf [id=bdab45e393374761bea47d965b81f043])
I20240430 20:11:49.612061 18985 tablet_bootstrap.cc:492] T b285e3eb16d148779c74e057458d8461 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:49.613210 20697 tablet_loader.cc:96] loaded metadata for tablet 8668f7a285de43f48d83abecb4b2e96c (table default.Table_6d9a8c8bd13b42eca6834e707653a66b [id=71c1c0c9a8b544a0a9b1b1a414a6f749])
I20240430 20:11:49.613943 20697 tablet_loader.cc:96] loaded metadata for tablet 885d27d4f8df48dd996d5b309b783f3b (table default.Table_47779624a66e46468c8e9e35a8e9487f [id=8ce80df659a146c9b7d5fdb3059902cd])
I20240430 20:11:49.611186 18988 consensus_queue.cc:260] T 6db55bb510b24b0cae897936211e15f0 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.615052 20697 tablet_loader.cc:96] loaded metadata for tablet 89b774967624498fa2be499878b71124 (table default.Table_2ec9e316bc3347b189a13b7a8bbcf829 [id=9fa22833979f4c2f978de651145acef9])
I20240430 20:11:49.615154 20587 raft_consensus.cc:482] T 50e1c55a1a0a484984ad5cbb78eec2ed P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:49.615279 20587 raft_consensus.cc:504] T 50e1c55a1a0a484984ad5cbb78eec2ed P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.615594 20697 tablet_loader.cc:96] loaded metadata for tablet 8db2038dbc5845489f414441898288f2 (table default.Table_b99234d57c5c42d58e4a6516fc1482d4 [id=5b3ce682abaf4acc8b62f9d34ff63c64])
I20240430 20:11:49.616003 20697 tablet_loader.cc:96] loaded metadata for tablet 8e33df70a1504dbd852fe0b3c3f61622 (table default.Table_21208fd7cdca4c73a4857bc4692187f6 [id=da22cb339ab647e8afda7aa92f3548a6])
I20240430 20:11:49.616075 20587 leader_election.cc:288] T 50e1c55a1a0a484984ad5cbb78eec2ed P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:49.616473 18616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50e1c55a1a0a484984ad5cbb78eec2ed" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:49.616585 20697 tablet_loader.cc:96] loaded metadata for tablet 8e7f08e9ac274bfa86763eec895bbfda (table default.Table_5721e7a6b1ef44a780ff57d59fa76f64 [id=01a774990824472a87414d89d98e1863])
I20240430 20:11:49.616708 18616 raft_consensus.cc:2454] T 50e1c55a1a0a484984ad5cbb78eec2ed P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:49.616690 20629 raft_consensus.cc:504] T 50e1c55a1a0a484984ad5cbb78eec2ed P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.616952 20697 tablet_loader.cc:96] loaded metadata for tablet 8e91920e7e36418d8c27d1f0bfd9b981 (table default.Table_e860191f401c434e8cf0085ef41952b9 [id=f393bcc9669c4ea985a2d40deb161f0c])
I20240430 20:11:49.617245 18983 tablet_bootstrap.cc:492] T a445fdfde01c4371a5e6b3daea5b00ab P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:49.617624 18983 ts_tablet_manager.cc:1367] T a445fdfde01c4371a5e6b3daea5b00ab P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.032s user 0.006s sys 0.000s
I20240430 20:11:49.617954 18696 leader_election.cc:302] T 50e1c55a1a0a484984ad5cbb78eec2ed P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:49.618168 18474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50e1c55a1a0a484984ad5cbb78eec2ed" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:49.618265 20587 raft_consensus.cc:2781] T 50e1c55a1a0a484984ad5cbb78eec2ed P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:49.618337 20587 raft_consensus.cc:482] T 50e1c55a1a0a484984ad5cbb78eec2ed P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:49.618400 20587 raft_consensus.cc:3037] T 50e1c55a1a0a484984ad5cbb78eec2ed P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:49.618418 18474 raft_consensus.cc:2379] T 50e1c55a1a0a484984ad5cbb78eec2ed P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c51c33d5616482cafb68e26688820f4 in current term 1: Already voted for candidate 2efb542aa55e439b99ccb98d1c5c884a in this term.
I20240430 20:11:49.618513 18758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50e1c55a1a0a484984ad5cbb78eec2ed" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:49.619063 20629 leader_election.cc:288] T 50e1c55a1a0a484984ad5cbb78eec2ed P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:49.619565 20697 tablet_loader.cc:96] loaded metadata for tablet 8e9711d0a382424f9f758bfa883866a0 (table default.Table_b7c7025fbf2e4039aadd143d021bfef5 [id=db696ce66005401fa73746e55c071f44])
I20240430 20:11:49.619532 18983 raft_consensus.cc:348] T a445fdfde01c4371a5e6b3daea5b00ab P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.619958 18983 raft_consensus.cc:374] T a445fdfde01c4371a5e6b3daea5b00ab P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:49.620126 18983 raft_consensus.cc:729] T a445fdfde01c4371a5e6b3daea5b00ab P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:49.620987 20587 raft_consensus.cc:504] T 50e1c55a1a0a484984ad5cbb78eec2ed P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.620636 18983 consensus_queue.cc:260] T a445fdfde01c4371a5e6b3daea5b00ab P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.621670 18758 raft_consensus.cc:2379] T 50e1c55a1a0a484984ad5cbb78eec2ed P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 2efb542aa55e439b99ccb98d1c5c884a in current term 1: Already voted for candidate 3c51c33d5616482cafb68e26688820f4 in this term.
I20240430 20:11:49.621948 20697 tablet_loader.cc:96] loaded metadata for tablet 94fbd4a57a1e438891fcb456f6b40be3 (table default.Table_25ddc3adef6b4ccf8b9c00cc6772031c [id=4fa623ffdcf347639150a355806038af])
I20240430 20:11:49.621344 18616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50e1c55a1a0a484984ad5cbb78eec2ed" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:49.622607 18616 raft_consensus.cc:3037] T 50e1c55a1a0a484984ad5cbb78eec2ed P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:49.623157 18620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50e1c55a1a0a484984ad5cbb78eec2ed" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:49.623745 20545 leader_election.cc:288] T 50e1c55a1a0a484984ad5cbb78eec2ed P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:49.625377 18616 raft_consensus.cc:2454] T 50e1c55a1a0a484984ad5cbb78eec2ed P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 1.
I20240430 20:11:49.625747 18415 leader_election.cc:302] T 50e1c55a1a0a484984ad5cbb78eec2ed P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters: 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:49.625849 18988 ts_tablet_manager.cc:1398] T 6db55bb510b24b0cae897936211e15f0 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.046s user 0.003s sys 0.000s
I20240430 20:11:49.626801 18474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50e1c55a1a0a484984ad5cbb78eec2ed" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:49.627090 18474 raft_consensus.cc:2379] T 50e1c55a1a0a484984ad5cbb78eec2ed P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 3c51c33d5616482cafb68e26688820f4 in current term 1: Already voted for candidate 2efb542aa55e439b99ccb98d1c5c884a in this term.
I20240430 20:11:49.627686 20587 leader_election.cc:288] T 50e1c55a1a0a484984ad5cbb78eec2ed P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Requested vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:49.628538 18616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50e1c55a1a0a484984ad5cbb78eec2ed" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:49.628398 18983 ts_tablet_manager.cc:1398] T a445fdfde01c4371a5e6b3daea5b00ab P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.010s user 0.003s sys 0.000s
I20240430 20:11:49.629180 18616 raft_consensus.cc:2379] T 50e1c55a1a0a484984ad5cbb78eec2ed P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 3c51c33d5616482cafb68e26688820f4 in current term 1: Already voted for candidate 2efb542aa55e439b99ccb98d1c5c884a in this term.
I20240430 20:11:49.629633 18696 leader_election.cc:302] T 50e1c55a1a0a484984ad5cbb78eec2ed P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710
I20240430 20:11:49.630029 20697 tablet_loader.cc:96] loaded metadata for tablet 975d145de1794f05b1dd844b4b86ce40 (table default.Table_2e5bcbca76d345fea6d7b7dac08a7afb [id=e2122b91429d4acc939e280d8261c452])
I20240430 20:11:49.630048 20587 raft_consensus.cc:2726] T 50e1c55a1a0a484984ad5cbb78eec2ed P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240430 20:11:49.630261 18988 tablet_bootstrap.cc:492] T 9c73d0a6bc414d3b9900570848d19a51 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:49.630712 20697 tablet_loader.cc:96] loaded metadata for tablet 99203cc522de424184c6f97453f166fd (table default.Table_cb2a4b9ceaf4431facfff2c9bcf94c5d [id=3f8690a437ae49a3beb009fa7899d5d0])
I20240430 20:11:49.631187 20697 tablet_loader.cc:96] loaded metadata for tablet 9a76db2c7ff146758c54908e428a0e02 (table default.Table_4991bbcb43c540629cbc285f0484d79d [id=79162f889e6340248053e0814ebb387a])
I20240430 20:11:49.631651 20697 tablet_loader.cc:96] loaded metadata for tablet 9b95098d3af04dd5a9ae3972e44b3a6c (table default.Table_80306c0ee72842209ca8feac52c0f247 [id=ac6acb1d504243e292723b12e0ad2b33])
I20240430 20:11:49.632136 20629 raft_consensus.cc:2781] T 50e1c55a1a0a484984ad5cbb78eec2ed P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:49.632660 20697 tablet_loader.cc:96] loaded metadata for tablet 9cc9251abd4e40d59a2bd2d0fada6165 (table default.Table_77b10083620349b0bcbb86e5de19f2ce [id=756ff53a80c842c5b62c972f542a7f34])
I20240430 20:11:49.632819 20629 raft_consensus.cc:686] T 50e1c55a1a0a484984ad5cbb78eec2ed P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Becoming Leader. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Running, Role: LEADER
I20240430 20:11:49.632685 18983 tablet_bootstrap.cc:492] T e900f62a5aa143e791c7539660b60cd9 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:49.633271 20697 tablet_loader.cc:96] loaded metadata for tablet 9e741b00c74d43b5b6243ba7003f8804 (table default.Table_f2a19db5098448f79bf95cccb1abd368 [id=fcaf7a6481644ae8990aa4e3d473bd50])
I20240430 20:11:49.633262 20629 consensus_queue.cc:237] T 50e1c55a1a0a484984ad5cbb78eec2ed P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.633770 20697 tablet_loader.cc:96] loaded metadata for tablet 9e98e20abc924873853d176455f3678e (table default.Table_4839b72b4f2143b2b23e8f3865d3e203 [id=e5126f46127442f1ad6ea61d40f983bb])
I20240430 20:11:49.634137 20697 tablet_loader.cc:96] loaded metadata for tablet 9ea773945f294938a0352d28f8d9f93e (table default.Table_ea4369e78de74045baa10ea2a61bfe38 [id=a0ccb667cf974306a952751800568d98])
I20240430 20:11:49.635421 20697 tablet_loader.cc:96] loaded metadata for tablet a0baee91f5b44945b595d0e234af018e (table default.Table_5ed3ea587cf843db9a1ff1ffab40b678 [id=b3d1c2f97c4648518dbea83a46cadd85])
I20240430 20:11:49.635890 20697 tablet_loader.cc:96] loaded metadata for tablet a0d3c4ba64024bccad95ecc909d03ae0 (table default.Table_5f50cb33cf774b86b2380ee4a569dac9 [id=bf4dcbec74f946398c7fc3f6eec6d068])
I20240430 20:11:49.637154 20697 tablet_loader.cc:96] loaded metadata for tablet a3b6e2ed50184e97914a703e2d73a285 (table default.Table_8d32c8a8f05e4b599dd6a713133ed82e [id=295c03e97c87495f8d71b255b2ec9065])
I20240430 20:11:49.637846 20697 tablet_loader.cc:96] loaded metadata for tablet a482f4622f774d60b4efd6827b8be7d7 (table default.Table_e686cdfcbcc046749fa5b51578a64794 [id=c3dc9f05316d46d6a3ef4d03f40877ec])
I20240430 20:11:49.638327 20697 tablet_loader.cc:96] loaded metadata for tablet a59ef6c545dd41a28265a4c1b8ce10e0 (table default.Table_07e40af8f1c546048975c8edf88170d7 [id=86df5d161d8641c89070e7eed9a39360])
I20240430 20:11:49.638942 20697 tablet_loader.cc:96] loaded metadata for tablet a663030d37614053a7c0369841403422 (table default.Table_e34613952dcb46618c3bc7cecb31fdaf [id=8f1ad1f8d0224b3daa5d84dc6916227b])
I20240430 20:11:49.639436 20697 tablet_loader.cc:96] loaded metadata for tablet a692b253db914531b8fc6ffaf3e350b5 (table default.Table_96aea8c4ff80428da8a0394f1bf7b493 [id=03ccb721140c4e0193e2e48d509430f9])
I20240430 20:11:49.641525 20697 tablet_loader.cc:96] loaded metadata for tablet aaef85eea87f4443943fd3692bb2a38f (table default.Table_3da5b4c8668c470483ae2a0c7773eaaf [id=e292d818ee3c45df90522f08eb2d6227])
I20240430 20:11:49.642174 20697 tablet_loader.cc:96] loaded metadata for tablet abdaf243e5be4ad58036b23151b14fe4 (table default.Table_1a641ff0ab5944a495a73189a6e8499a [id=c2bcc99b99d14bd480881a9c8dc37dbb])
I20240430 20:11:49.642940 20697 tablet_loader.cc:96] loaded metadata for tablet acd23f0a63d5448aaa7ed78422e67bad (table default.Table_972f13be4dca4cd19b9a04534dc59f51 [id=69f37db4de634a3185613e40013e0ed7])
I20240430 20:11:49.644898 18985 tablet_bootstrap.cc:654] T b285e3eb16d148779c74e057458d8461 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:49.647356 20697 tablet_loader.cc:96] loaded metadata for tablet ad01c979b2f7439bab83137f87f1405c (table default.Table_e3165b7ce52a44bf9d2bed43f9c5d6c7 [id=3546514024eb4b8c9ad58c7af8d63aff])
I20240430 20:11:49.647929 20697 tablet_loader.cc:96] loaded metadata for tablet ae20f44df50343289b0a512a4ea53692 (table default.Table_1844f6b1bda84812825384e14a2fd94e [id=71da9acc3cff4d95bbac4a686b708fd5])
I20240430 20:11:49.651643 20697 tablet_loader.cc:96] loaded metadata for tablet ae24e75a0d42441ebc97400ffa94145e (table default.Table_c9603ec75ad0491592883980e5efc388 [id=ce2d781a514e4c10a2fde5b9d4551ff5])
I20240430 20:11:49.653973 20697 tablet_loader.cc:96] loaded metadata for tablet aed9a3eefe3f4136af1b9469c9e2e23c (table default.Table_457e03bc2d524accaa76649790c42fc8 [id=f26e395466f24585b9524a1ec2f77051])
I20240430 20:11:49.654837 20697 tablet_loader.cc:96] loaded metadata for tablet b1277899f0c040cea7dc2b12f3fd68de (table default.Table_2582fc0f97c1470d8b6b21cd1240697e [id=fc059935f9c943389180fab316c33a05])
I20240430 20:11:49.655349 20697 tablet_loader.cc:96] loaded metadata for tablet b1792c160af346ffb4f047c5d4435c10 (table default.Table_3127eaa465c84dcab3eb9935d15667ce [id=e4cf2d34f4b747afa5b8a512f211c01f])
I20240430 20:11:49.656067 20697 tablet_loader.cc:96] loaded metadata for tablet b285e3eb16d148779c74e057458d8461 (table default.Table_7fb60e4b56f444009ab5da219238eec7 [id=d1655c507b434f129ad7ae0b04365e51])
I20240430 20:11:49.657864 20697 tablet_loader.cc:96] loaded metadata for tablet b7d8da6e2fe8499d9d53af84a25f2faf (table default.Table_5dd3db8b666a491e8b33cbb5a0b54ff5 [id=4b7dfcb0dd2b4e8ba3a9a4a2bce81287])
I20240430 20:11:49.658313 20697 tablet_loader.cc:96] loaded metadata for tablet bb471d33b0784c9c9de1e5425cdd1fd7 (table default.Table_41ccec912bfa439c888281ac81447a38 [id=da57ee424e3f481f8e0c00399bb21ffa])
I20240430 20:11:49.658798 20697 tablet_loader.cc:96] loaded metadata for tablet bbdbb8a3a4b444f58a2761466577a88f (table default.Table_dd819f96762949c786839dda00487cc2 [id=e1e2e2a5b0b740cba09110991dd28193])
I20240430 20:11:49.659497 20697 tablet_loader.cc:96] loaded metadata for tablet bd11c0335a58448da56533c3f2e785a9 (table default.Table_88c44b20f2ce4eeca7ed9583a70c7c65 [id=73caec5fb65e4cb090e7ae8c391501fd])
I20240430 20:11:49.660130 20697 tablet_loader.cc:96] loaded metadata for tablet bdc9e34851a9418ba7576163a266d74c (table default.Table_ae525cb900474e458017ab35d825fc1c [id=c8d3ef652a764c3a9d658fbfc9f41ede])
I20240430 20:11:49.660828 20697 tablet_loader.cc:96] loaded metadata for tablet bea2ca3d09164afd9941cb7e35f5ae24 (table default.Table_01d62e066e4942d9aa1f582189d60d49 [id=97861b0b1ce94638b62bbd123d774d7f])
I20240430 20:11:49.661550 20697 tablet_loader.cc:96] loaded metadata for tablet c091701480c4412faabad3d071d85d03 (table default.Table_3a35534b6ef2416b968740d28b156dd0 [id=f3c08d1b91be4bf58242ca535ff28897])
I20240430 20:11:49.662869 20384 raft_consensus.cc:482] T f3711610610542128c4a84e17bce4a3e P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:49.663048 20384 raft_consensus.cc:504] T f3711610610542128c4a84e17bce4a3e P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.663336 20697 tablet_loader.cc:96] loaded metadata for tablet c4953578de5f45f3a6ba092a13357460 (table default.Table_9110c0cb4db441dbb8bdce422eadc189 [id=a826c08c7eef44ec83feae3a6cb384e5])
I20240430 20:11:49.663673 20384 leader_election.cc:288] T f3711610610542128c4a84e17bce4a3e P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:49.664470 20697 tablet_loader.cc:96] loaded metadata for tablet c6e3d09402574f1e8fe9e4160c70a3ee (table default.Table_656db97355f64d7c8aa29b1352dfe9c8 [id=450f5d1c7e084a1bb88d035c0ccfcf82])
I20240430 20:11:49.664476 18474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3711610610542128c4a84e17bce4a3e" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:49.665009 20697 tablet_loader.cc:96] loaded metadata for tablet c85a6d11d8af486492d8b843c75632e6 (table default.Table_dfb59f35018441fd9a72ea860cdd01f6 [id=880609dd541c4ce5af4d27b5a2e0c9a7])
I20240430 20:11:49.665388 18758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3711610610542128c4a84e17bce4a3e" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:49.665594 18758 raft_consensus.cc:2454] T f3711610610542128c4a84e17bce4a3e P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 0.
I20240430 20:11:49.666038 20697 tablet_loader.cc:96] loaded metadata for tablet c9e7660eccc448dbb99d135f82bfd261 (table default.Table_685a26737bb046fb85ced15df7ae3423 [id=51d177e8498848b38c43e3f8d11f092c])
W20240430 20:11:49.666066 18558 leader_election.cc:341] T f3711610610542128c4a84e17bce4a3e P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:49.666908 20697 tablet_loader.cc:96] loaded metadata for tablet ca64de16d40f4a44b8b411f5526ba11a (table default.Table_1cf914acf7fe4e98b8dbc297b6372f21 [id=f4ba9536e9c141deb68f4f62b7e8fa70])
I20240430 20:11:49.667008 18557 leader_election.cc:302] T f3711610610542128c4a84e17bce4a3e P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a
I20240430 20:11:49.667524 20384 raft_consensus.cc:2781] T f3711610610542128c4a84e17bce4a3e P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:49.667613 20384 raft_consensus.cc:482] T f3711610610542128c4a84e17bce4a3e P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:49.667698 20384 raft_consensus.cc:3037] T f3711610610542128c4a84e17bce4a3e P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:49.667850 20697 tablet_loader.cc:96] loaded metadata for tablet cc267ce92ffd415fbe3377afc0503690 (table default.Table_2f6a85489f2e49db8974ff86813c85ee [id=6a249076822e47eda013a4d4f23ab8da])
I20240430 20:11:49.668191 20697 tablet_loader.cc:96] loaded metadata for tablet cdc07bb607374c6e8eb6ca35a86d0204 (table default.Table_fe491607e9e7480b9d9d2c8eb89ccac9 [id=ba205c2256d24f998fb16e6590b74cf4])
I20240430 20:11:49.668556 20697 tablet_loader.cc:96] loaded metadata for tablet cdeef95b54ca42d38d25b57738d84ef1 (table default.Table_8039a17f6e4b4e9d90222f99d53d90a1 [id=e05d019ab2ad44c8af71fb04b4a27fdd])
I20240430 20:11:49.668905 20697 tablet_loader.cc:96] loaded metadata for tablet cefce2f2a392495bb71399ed75db938c (table default.Table_47217f7ee5ef4a50a2e9f146412b198a [id=3b5dcd10bb0d4f2493dcfaae7a2b9d19])
I20240430 20:11:49.669255 20697 tablet_loader.cc:96] loaded metadata for tablet d2381047f6dc4bdf962f9383e6c41548 (table default.Table_3a588666409c4a64b13658a42a58f994 [id=053d7bb753464e968b05493053eb229a])
I20240430 20:11:49.669639 20697 tablet_loader.cc:96] loaded metadata for tablet d34a450a91d542c98261456d1f635e9c (table default.Table_f30868d759f64cceb682278f0dbb51ae [id=3291bc6bb4f146ee8f90d89029a70aef])
I20240430 20:11:49.670277 20384 raft_consensus.cc:504] T f3711610610542128c4a84e17bce4a3e P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.670979 20697 tablet_loader.cc:96] loaded metadata for tablet d78dd72c3a254fa5a98382a2d24b41f8 (table default.Table_df40d2a666d94cada72ebdb96b7d7f9e [id=f33b27d82b54476eb1eab57b39b91108])
I20240430 20:11:49.672782 18983 tablet_bootstrap.cc:654] T e900f62a5aa143e791c7539660b60cd9 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:49.673379 18758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3711610610542128c4a84e17bce4a3e" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:49.673585 18758 raft_consensus.cc:3037] T f3711610610542128c4a84e17bce4a3e P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:49.674414 20697 tablet_loader.cc:96] loaded metadata for tablet da3e038a135846f38e98de970ece273d (table default.Table_b869ed2a85d64fcf83a0eb41c1a86446 [id=9d43a387b6884f6dbb32a65cc1f0bb97])
I20240430 20:11:49.674791 20697 tablet_loader.cc:96] loaded metadata for tablet dac141d1b70c48d785caddd7182811ee (table default.Table_fc0a4540f98b4eea95099b8ab46ea235 [id=f3c53ccffe64410495cc8c379a05cfae])
I20240430 20:11:49.676095 20697 tablet_loader.cc:96] loaded metadata for tablet ddae8bc160b34f5590e5b0f09d36749d (table default.Table_8236b26459a348f99005b9ef3033a30f [id=ec659350534140e89c40849e2a7dac27])
I20240430 20:11:49.676241 18758 raft_consensus.cc:2454] T f3711610610542128c4a84e17bce4a3e P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
I20240430 20:11:49.676769 20697 tablet_loader.cc:96] loaded metadata for tablet de546e2d6a5142159f23bd50d78ff735 (table default.Table_401b6bfce5ed4b1bb5643b8a53269b60 [id=4a9083e60fa248c28b61b9a6ee49e9d8])
I20240430 20:11:49.676944 18557 leader_election.cc:302] T f3711610610542128c4a84e17bce4a3e P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:49.677258 20489 raft_consensus.cc:2781] T f3711610610542128c4a84e17bce4a3e P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:49.677381 20489 raft_consensus.cc:686] T f3711610610542128c4a84e17bce4a3e P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Becoming Leader. State: Replica: 5e1c6397f09747f091534faf93542710, State: Running, Role: LEADER
I20240430 20:11:49.677723 20697 tablet_loader.cc:96] loaded metadata for tablet e041756c962b4a9f97c2cb314e7d8c05 (table default.Table_de88cafd92b149f9b545369bff71033c [id=c4bf4806f3d84579b16a3038d8d982c3])
I20240430 20:11:49.677670 20489 consensus_queue.cc:237] T f3711610610542128c4a84e17bce4a3e P 5e1c6397f09747f091534faf93542710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.681264 20697 tablet_loader.cc:96] loaded metadata for tablet e28529e1c2ae4c7c9b9b0e4bd709d132 (table default.Table_d8df23edb8ce480bb59a2ab0f1fc7521 [id=4c9193fcff2f41768d578803372434e0])
I20240430 20:11:49.681883 20697 tablet_loader.cc:96] loaded metadata for tablet e2f65b284e794ac28a18670f152a82b2 (table default.Table_4adefb6ee63f4301a479e1d042ff1c8b [id=16b11b30a9ab46b5a2c5da5c6a494ac8])
I20240430 20:11:49.682291 18988 tablet_bootstrap.cc:654] T 9c73d0a6bc414d3b9900570848d19a51 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:49.684273 18474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3711610610542128c4a84e17bce4a3e" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:49.684751 20697 tablet_loader.cc:96] loaded metadata for tablet e5b85a066cdc46d48ccf8201e5423c29 (table default.Table_81547db3074240c98cdd95ed3b2dd813 [id=e0ea901452334b36a8493baed478cc1a])
W20240430 20:11:49.684962 18558 leader_election.cc:341] T f3711610610542128c4a84e17bce4a3e P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:49.687038 20697 tablet_loader.cc:96] loaded metadata for tablet e848539446a64a968b37045d367bb3ce (table default.Table_605f23ec0a58457f87356d7f45d3b45c [id=1930f366a0f0456cad9a1c2456eb8fac])
I20240430 20:11:49.687687 20697 tablet_loader.cc:96] loaded metadata for tablet e900f62a5aa143e791c7539660b60cd9 (table default.Table_2dbff4cecdff4d2dbbb0004c88377ce7 [id=ebf546ade5f140f19fff7c5841bfee30])
I20240430 20:11:49.688351 20384 leader_election.cc:288] T f3711610610542128c4a84e17bce4a3e P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:49.695886 20697 tablet_loader.cc:96] loaded metadata for tablet eb6c96047b114ebba73db7a371f2da8e (table default.Table_7d5f6f18cbd64c869739872a7dff9be2 [id=e67d619e36c0485f8f22b4a81cb4cb6c])
I20240430 20:11:49.696518 20697 tablet_loader.cc:96] loaded metadata for tablet eb7554e115d84e6ca4f05509c5f33e2a (table default.Table_4bb81ccd60a148d197b8a4ebd9c9cc49 [id=ee0542402cc1426681e9583fbb7340e1])
I20240430 20:11:49.697258 20697 tablet_loader.cc:96] loaded metadata for tablet ebf74809b52f47f3b0ed21b6ac0f541a (table default.Table_b2d07520b9944a89ad62fca70396174e [id=1c8e0a3a44ec4a798770c0165ed3dffa])
I20240430 20:11:49.697974 20697 tablet_loader.cc:96] loaded metadata for tablet ec060e847f024f9fa369cff320e71558 (table default.Table_2e9c7c683df84a41942d31600a265641 [id=aca6fb02776a40bbac312191adbdc136])
I20240430 20:11:49.698480 20697 tablet_loader.cc:96] loaded metadata for tablet ed0c37a71ede4bfc9eb5ad1a4365f25e (table default.Table_a2e85efc6b454524b0b122102ec07002 [id=86d795a82e674d859104741ebd04f4e8])
I20240430 20:11:49.699195 20697 tablet_loader.cc:96] loaded metadata for tablet ef40bfdb5bf84f54a0068f40c7d1617a (table default.Table_91c01cce23fe4f4182952019c19875c0 [id=b7251a2a9c91439db19f9b910ebb5a3b])
I20240430 20:11:49.700603 20697 tablet_loader.cc:96] loaded metadata for tablet f1073fc792954c5795a05aa7d0ec24df (table default.Table_ba8b218e936b4bfcb073f8182f474cca [id=aedf11fda8914a3b843a57f8a7b69a53])
I20240430 20:11:49.701015 20697 tablet_loader.cc:96] loaded metadata for tablet f133adb974b94afd8ab4626f9b26949e (table default.Table_dfbb6c73df814ce49886f7b8f0e0968d [id=9b01fa1f94b24da799aed1617be3c892])
I20240430 20:11:49.702019 18985 tablet_bootstrap.cc:492] T b285e3eb16d148779c74e057458d8461 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:49.702740 18985 ts_tablet_manager.cc:1367] T b285e3eb16d148779c74e057458d8461 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.093s user 0.004s sys 0.002s
I20240430 20:11:49.703649 20384 raft_consensus.cc:482] T f90a5c5818c54390a2ae323fed7ebe57 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:49.703732 18988 tablet_bootstrap.cc:492] T 9c73d0a6bc414d3b9900570848d19a51 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:49.703891 18988 ts_tablet_manager.cc:1367] T 9c73d0a6bc414d3b9900570848d19a51 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.075s user 0.005s sys 0.000s
I20240430 20:11:49.703855 20384 raft_consensus.cc:504] T f90a5c5818c54390a2ae323fed7ebe57 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.704044 20697 tablet_loader.cc:96] loaded metadata for tablet f5eaf2413e1d4e69b56fae61f768f0ce (table default.Table_1280cfe32d5b41abb4d3eb1bd70641bf [id=8489b872074a42b6abb713e420968141])
I20240430 20:11:49.704675 18988 raft_consensus.cc:348] T 9c73d0a6bc414d3b9900570848d19a51 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.704932 18988 raft_consensus.cc:374] T 9c73d0a6bc414d3b9900570848d19a51 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:49.704990 18988 raft_consensus.cc:729] T 9c73d0a6bc414d3b9900570848d19a51 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:49.705876 18983 tablet_bootstrap.cc:492] T e900f62a5aa143e791c7539660b60cd9 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:49.706004 20697 tablet_loader.cc:96] loaded metadata for tablet f8bfbf673ea346529d1b2b376a3da496 (table default.Table_1ebe0eb1a1984ee9bbb16d634ff2cedb [id=aa91ac92289a48b784e97eea9967192f])
I20240430 20:11:49.706066 18983 ts_tablet_manager.cc:1367] T e900f62a5aa143e791c7539660b60cd9 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.075s user 0.003s sys 0.002s
I20240430 20:11:49.706182 18758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f90a5c5818c54390a2ae323fed7ebe57" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:49.706372 18758 raft_consensus.cc:2454] T f90a5c5818c54390a2ae323fed7ebe57 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 0.
I20240430 20:11:49.706413 20697 tablet_loader.cc:96] loaded metadata for tablet f90a5c5818c54390a2ae323fed7ebe57 (table default.Table_18385e32857b4c8bb1e56a85c590153a [id=38a7d7d4f9e64c07b784d9393192ac94])
I20240430 20:11:49.706851 18557 leader_election.cc:302] T f90a5c5818c54390a2ae323fed7ebe57 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:49.706943 18983 raft_consensus.cc:348] T e900f62a5aa143e791c7539660b60cd9 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.707126 18983 raft_consensus.cc:374] T e900f62a5aa143e791c7539660b60cd9 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:49.707170 20489 raft_consensus.cc:2781] T f90a5c5818c54390a2ae323fed7ebe57 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:49.707211 18983 raft_consensus.cc:729] T e900f62a5aa143e791c7539660b60cd9 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:49.707299 20489 raft_consensus.cc:482] T f90a5c5818c54390a2ae323fed7ebe57 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:49.707391 20489 raft_consensus.cc:3037] T f90a5c5818c54390a2ae323fed7ebe57 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:49.705912 20384 leader_election.cc:288] T f90a5c5818c54390a2ae323fed7ebe57 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:49.705241 18474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f90a5c5818c54390a2ae323fed7ebe57" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:49.708026 18988 consensus_queue.cc:260] T 9c73d0a6bc414d3b9900570848d19a51 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
W20240430 20:11:49.708397 18558 leader_election.cc:341] T f90a5c5818c54390a2ae323fed7ebe57 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:49.708561 18988 ts_tablet_manager.cc:1398] T 9c73d0a6bc414d3b9900570848d19a51 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240430 20:11:49.708868 18985 raft_consensus.cc:348] T b285e3eb16d148779c74e057458d8461 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.709062 18985 raft_consensus.cc:374] T b285e3eb16d148779c74e057458d8461 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:49.709129 18985 raft_consensus.cc:729] T b285e3eb16d148779c74e057458d8461 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:49.709328 18985 consensus_queue.cc:260] T b285e3eb16d148779c74e057458d8461 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.710045 20489 raft_consensus.cc:504] T f90a5c5818c54390a2ae323fed7ebe57 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.712080 18758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f90a5c5818c54390a2ae323fed7ebe57" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:49.712275 18758 raft_consensus.cc:3037] T f90a5c5818c54390a2ae323fed7ebe57 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:49.713534 18985 ts_tablet_manager.cc:1398] T b285e3eb16d148779c74e057458d8461 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240430 20:11:49.714124 18985 tablet_bootstrap.cc:492] T f90a5c5818c54390a2ae323fed7ebe57 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:49.715983 18758 raft_consensus.cc:2454] T f90a5c5818c54390a2ae323fed7ebe57 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
I20240430 20:11:49.716399 20489 leader_election.cc:288] T f90a5c5818c54390a2ae323fed7ebe57 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:49.716764 20697 tablet_loader.cc:96] loaded metadata for tablet f9bf604e6c234349a9f72f45a4c4aef6 (table default.Table_9049c9eeb10540d3815bab49caa1e82e [id=619746ec6b3244ee9ba511745c38bc25])
I20240430 20:11:49.717113 18985 tablet_bootstrap.cc:654] T f90a5c5818c54390a2ae323fed7ebe57 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:49.717012 18474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f90a5c5818c54390a2ae323fed7ebe57" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:49.717350 18557 leader_election.cc:302] T f90a5c5818c54390a2ae323fed7ebe57 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:49.719278 20697 tablet_loader.cc:96] loaded metadata for tablet fbdeb4714b8c4b93ab72822c43329aae (table default.Table_c12ca6938d424fbfa2e1df5cdb00fdbc [id=0fb93f36bc17408ea56bc4cafcfd8580])
W20240430 20:11:49.720214 18558 leader_election.cc:341] T f90a5c5818c54390a2ae323fed7ebe57 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:49.720655 20697 tablet_loader.cc:96] loaded metadata for tablet fd1b8407071e4e4287fc172009777414 (table default.Table_5572f0acadaa432e8570b159fd9b76bc [id=d0f3b1709ac94aed8ec568c0327eab34])
I20240430 20:11:49.720662 20384 raft_consensus.cc:2781] T f90a5c5818c54390a2ae323fed7ebe57 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:49.720809 20384 raft_consensus.cc:686] T f90a5c5818c54390a2ae323fed7ebe57 P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Becoming Leader. State: Replica: 5e1c6397f09747f091534faf93542710, State: Running, Role: LEADER
I20240430 20:11:49.721266 20384 consensus_queue.cc:237] T f90a5c5818c54390a2ae323fed7ebe57 P 5e1c6397f09747f091534faf93542710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.722541 20697 tablet_loader.cc:96] loaded metadata for tablet fd423a49fb694695a8de7ecd54330127 (table default.Table_204dce4e6128482583d8869c83413f25 [id=618fcf2891c84a238fcde6f9b62bba3f])
I20240430 20:11:49.724269 20697 tablet_loader.cc:96] loaded metadata for tablet fda942d50b014f3d9589e1fb909fc2fc (table default.Table_68528229d5a94981a7fc0677d48003d1 [id=25ac2d71caf74e5ba14e3e84b84c9c60])
I20240430 20:11:49.725862 20697 tablet_loader.cc:96] loaded metadata for tablet ffd6e420e6d44c039832de38de001c1b (table default.Table_7390a5615e234bf38ddf41eefe70bdf3 [id=52259eea51074e14b472d8532e08a990])
I20240430 20:11:49.736992 20489 consensus_queue.cc:1035] T 8e91920e7e36418d8c27d1f0bfd9b981 P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240430 20:11:49.737900 20697 catalog_manager.cc:1476] Time spent T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: Loading table and tablet metadata into memory: real 2.503s user 0.218s sys 0.075s
I20240430 20:11:49.738334 19942 rpcz_store.cc:275] Call kudu.master.MasterService.ConnectToMaster from 127.0.0.1:50746 (request call id 17) took 2484 ms. Trace:
I20240430 20:11:49.738461 19942 rpcz_store.cc:276] 0430 20:11:47.254104 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:47.254184 (+ 80us) service_pool.cc:225] Handling call
0430 20:11:49.738267 (+2484083us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240430 20:11:49.738466 20697 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240430 20:11:49.739043 19979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e" candidate_term: 7 candidate_status { last_received { term: 6 index: 750 } } ignore_live_leader: false dest_uuid: "88b3f98793354c7aaa5c106e8ec31ed8" is_pre_election: true
W20240430 20:11:49.739326 19979 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33774 (request call id 3) took 988 ms (client timeout 546 ms). Trace:
W20240430 20:11:49.739408 19979 rpcz_store.cc:269] 0430 20:11:48.750854 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:48.750943 (+ 89us) service_pool.cc:225] Handling call
0430 20:11:49.739273 (+988330us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240430 20:11:49.737982 19980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e" candidate_term: 7 candidate_status { last_received { term: 6 index: 750 } } ignore_live_leader: false dest_uuid: "88b3f98793354c7aaa5c106e8ec31ed8" is_pre_election: true
W20240430 20:11:49.739837 19980 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33774 (request call id 2) took 1554 ms (client timeout 546 ms). Trace:
W20240430 20:11:49.739928 19980 rpcz_store.cc:269] 0430 20:11:48.185017 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:48.186767 (+ 1750us) service_pool.cc:225] Handling call
0430 20:11:49.739787 (+1553020us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240430 20:11:49.742501 18988 tablet_bootstrap.cc:492] T 2fc6908ecf954020a676b9d10a2b00e7 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:49.707428 18983 consensus_queue.cc:260] T e900f62a5aa143e791c7539660b60cd9 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.743481 20545 raft_consensus.cc:482] T 4f6f9ae722da49e58c9b72107f6da2fa P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5e1c6397f09747f091534faf93542710)
I20240430 20:11:49.743721 18983 ts_tablet_manager.cc:1398] T e900f62a5aa143e791c7539660b60cd9 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.038s user 0.002s sys 0.000s
I20240430 20:11:49.743660 20545 raft_consensus.cc:504] T 4f6f9ae722da49e58c9b72107f6da2fa P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.744072 18983 tablet_bootstrap.cc:492] T 558e71ec370d46ca8fbdcec5b57eb2ec P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:49.744333 20545 leader_election.cc:288] T 4f6f9ae722da49e58c9b72107f6da2fa P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
W20240430 20:11:49.745004 18417 leader_election.cc:341] T 4f6f9ae722da49e58c9b72107f6da2fa P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Not found: Tablet not found: 4f6f9ae722da49e58c9b72107f6da2fa
W20240430 20:11:49.745512 18415 leader_election.cc:341] T 4f6f9ae722da49e58c9b72107f6da2fa P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Not found: Tablet not found: 4f6f9ae722da49e58c9b72107f6da2fa
I20240430 20:11:49.745859 18415 leader_election.cc:302] T 4f6f9ae722da49e58c9b72107f6da2fa P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a; no voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710
I20240430 20:11:49.747118 18983 tablet_bootstrap.cc:654] T 558e71ec370d46ca8fbdcec5b57eb2ec P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:49.749015 18985 tablet_bootstrap.cc:492] T f90a5c5818c54390a2ae323fed7ebe57 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:49.749192 18985 ts_tablet_manager.cc:1367] T f90a5c5818c54390a2ae323fed7ebe57 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.035s user 0.005s sys 0.001s
I20240430 20:11:49.749399 20629 raft_consensus.cc:2726] T 4f6f9ae722da49e58c9b72107f6da2fa P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240430 20:11:49.750087 18985 raft_consensus.cc:348] T f90a5c5818c54390a2ae323fed7ebe57 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.750360 18985 raft_consensus.cc:374] T f90a5c5818c54390a2ae323fed7ebe57 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:49.750437 18985 raft_consensus.cc:729] T f90a5c5818c54390a2ae323fed7ebe57 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:49.751600 18985 consensus_queue.cc:260] T f90a5c5818c54390a2ae323fed7ebe57 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.754691 18983 tablet_bootstrap.cc:492] T 558e71ec370d46ca8fbdcec5b57eb2ec P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:49.754947 18983 ts_tablet_manager.cc:1367] T 558e71ec370d46ca8fbdcec5b57eb2ec P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.003s
I20240430 20:11:49.756572 18988 tablet_bootstrap.cc:654] T 2fc6908ecf954020a676b9d10a2b00e7 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:49.760201 18983 raft_consensus.cc:348] T 558e71ec370d46ca8fbdcec5b57eb2ec P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.760946 18983 raft_consensus.cc:374] T 558e71ec370d46ca8fbdcec5b57eb2ec P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:49.761267 18983 raft_consensus.cc:729] T 558e71ec370d46ca8fbdcec5b57eb2ec P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:49.761669 18983 consensus_queue.cc:260] T 558e71ec370d46ca8fbdcec5b57eb2ec P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.762974 18983 ts_tablet_manager.cc:1398] T 558e71ec370d46ca8fbdcec5b57eb2ec P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240430 20:11:49.763909 18983 tablet_bootstrap.cc:492] T 0368d37dd9e2452982fb81f539c6805b P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:49.766396 18983 tablet_bootstrap.cc:654] T 0368d37dd9e2452982fb81f539c6805b P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:49.771749 18988 tablet_bootstrap.cc:492] T 2fc6908ecf954020a676b9d10a2b00e7 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:49.771987 18988 ts_tablet_manager.cc:1367] T 2fc6908ecf954020a676b9d10a2b00e7 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.030s user 0.005s sys 0.005s
I20240430 20:11:49.772701 18988 raft_consensus.cc:348] T 2fc6908ecf954020a676b9d10a2b00e7 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.772991 18988 raft_consensus.cc:374] T 2fc6908ecf954020a676b9d10a2b00e7 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:49.773083 18988 raft_consensus.cc:729] T 2fc6908ecf954020a676b9d10a2b00e7 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:49.773326 18988 consensus_queue.cc:260] T 2fc6908ecf954020a676b9d10a2b00e7 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.774124 18988 ts_tablet_manager.cc:1398] T 2fc6908ecf954020a676b9d10a2b00e7 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240430 20:11:49.774673 18988 tablet_bootstrap.cc:492] T 1f4e860484784aa9a2d28af2e9fb611c P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:49.776185 18988 tablet_bootstrap.cc:654] T 1f4e860484784aa9a2d28af2e9fb611c P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:49.781625 18985 ts_tablet_manager.cc:1398] T f90a5c5818c54390a2ae323fed7ebe57 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240430 20:11:49.782768 20697 catalog_manager.cc:1259] Loaded cluster ID: 889e0faede1f41d1ab74633d6826c0d4
I20240430 20:11:49.782992 20697 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240430 20:11:49.787185 18985 tablet_bootstrap.cc:492] T e900f62a5aa143e791c7539660b60cd9 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:49.788959 18985 tablet_bootstrap.cc:654] T e900f62a5aa143e791c7539660b60cd9 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:49.791496 18983 tablet_bootstrap.cc:492] T 0368d37dd9e2452982fb81f539c6805b P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:49.791671 18983 ts_tablet_manager.cc:1367] T 0368d37dd9e2452982fb81f539c6805b P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.028s user 0.002s sys 0.004s
I20240430 20:11:49.792503 18983 raft_consensus.cc:348] T 0368d37dd9e2452982fb81f539c6805b P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.792722 18983 raft_consensus.cc:374] T 0368d37dd9e2452982fb81f539c6805b P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:49.792814 18983 raft_consensus.cc:729] T 0368d37dd9e2452982fb81f539c6805b P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:49.793040 18983 consensus_queue.cc:260] T 0368d37dd9e2452982fb81f539c6805b P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.793591 18983 ts_tablet_manager.cc:1398] T 0368d37dd9e2452982fb81f539c6805b P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240430 20:11:49.793895 18983 tablet_bootstrap.cc:492] T 2a08e36c39824abf98b9b6214c8b84cb P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:49.794049 20695 raft_consensus.cc:482] T 0368d37dd9e2452982fb81f539c6805b P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:49.794189 20695 raft_consensus.cc:504] T 0368d37dd9e2452982fb81f539c6805b P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.794850 20695 leader_election.cc:288] T 0368d37dd9e2452982fb81f539c6805b P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:49.795406 18474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0368d37dd9e2452982fb81f539c6805b" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:49.795639 18983 tablet_bootstrap.cc:654] T 2a08e36c39824abf98b9b6214c8b84cb P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
W20240430 20:11:49.795979 18699 leader_election.cc:341] T 0368d37dd9e2452982fb81f539c6805b P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:49.796594 18620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0368d37dd9e2452982fb81f539c6805b" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:49.796825 18620 raft_consensus.cc:2454] T 0368d37dd9e2452982fb81f539c6805b P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:49.797350 18696 leader_election.cc:302] T 0368d37dd9e2452982fb81f539c6805b P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a
I20240430 20:11:49.797863 20695 raft_consensus.cc:2781] T 0368d37dd9e2452982fb81f539c6805b P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:49.797967 20695 raft_consensus.cc:482] T 0368d37dd9e2452982fb81f539c6805b P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:49.798065 20695 raft_consensus.cc:3037] T 0368d37dd9e2452982fb81f539c6805b P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:49.801151 20695 raft_consensus.cc:504] T 0368d37dd9e2452982fb81f539c6805b P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.802100 18988 tablet_bootstrap.cc:492] T 1f4e860484784aa9a2d28af2e9fb611c P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:49.802251 18988 ts_tablet_manager.cc:1367] T 1f4e860484784aa9a2d28af2e9fb611c P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.028s user 0.003s sys 0.003s
I20240430 20:11:49.802361 18474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0368d37dd9e2452982fb81f539c6805b" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
W20240430 20:11:49.802999 18699 leader_election.cc:341] T 0368d37dd9e2452982fb81f539c6805b P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:49.802964 18988 raft_consensus.cc:348] T 1f4e860484784aa9a2d28af2e9fb611c P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.803156 18988 raft_consensus.cc:374] T 1f4e860484784aa9a2d28af2e9fb611c P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:49.803251 18988 raft_consensus.cc:729] T 1f4e860484784aa9a2d28af2e9fb611c P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:49.803498 18988 consensus_queue.cc:260] T 1f4e860484784aa9a2d28af2e9fb611c P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.804234 18620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0368d37dd9e2452982fb81f539c6805b" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:49.804553 18620 raft_consensus.cc:3037] T 0368d37dd9e2452982fb81f539c6805b P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:49.807919 18758 raft_consensus.cc:3037] T 6db55bb510b24b0cae897936211e15f0 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:49.810932 18985 tablet_bootstrap.cc:492] T e900f62a5aa143e791c7539660b60cd9 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:49.811153 18985 ts_tablet_manager.cc:1367] T e900f62a5aa143e791c7539660b60cd9 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.029s user 0.002s sys 0.008s
I20240430 20:11:49.812080 18985 raft_consensus.cc:348] T e900f62a5aa143e791c7539660b60cd9 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.812309 18985 raft_consensus.cc:374] T e900f62a5aa143e791c7539660b60cd9 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:49.812211 18620 raft_consensus.cc:2454] T 0368d37dd9e2452982fb81f539c6805b P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:49.812414 18985 raft_consensus.cc:729] T e900f62a5aa143e791c7539660b60cd9 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:49.812566 18983 tablet_bootstrap.cc:492] T 2a08e36c39824abf98b9b6214c8b84cb P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:49.812768 18983 ts_tablet_manager.cc:1367] T 2a08e36c39824abf98b9b6214c8b84cb P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.000s
I20240430 20:11:49.812669 18985 consensus_queue.cc:260] T e900f62a5aa143e791c7539660b60cd9 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.813534 18983 raft_consensus.cc:348] T 2a08e36c39824abf98b9b6214c8b84cb P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.813756 18983 raft_consensus.cc:374] T 2a08e36c39824abf98b9b6214c8b84cb P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:49.813846 18983 raft_consensus.cc:729] T 2a08e36c39824abf98b9b6214c8b84cb P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:49.814129 18983 consensus_queue.cc:260] T 2a08e36c39824abf98b9b6214c8b84cb P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.814721 18983 ts_tablet_manager.cc:1398] T 2a08e36c39824abf98b9b6214c8b84cb P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240430 20:11:49.814953 20489 consensus_queue.cc:1035] T 6db55bb510b24b0cae897936211e15f0 P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:49.816587 18985 ts_tablet_manager.cc:1398] T e900f62a5aa143e791c7539660b60cd9 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240430 20:11:49.816903 18985 tablet_bootstrap.cc:492] T 1f4e860484784aa9a2d28af2e9fb611c P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:49.817260 18696 leader_election.cc:302] T 0368d37dd9e2452982fb81f539c6805b P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a
I20240430 20:11:49.820811 18983 tablet_bootstrap.cc:492] T 9c73d0a6bc414d3b9900570848d19a51 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:49.822711 20695 leader_election.cc:288] T 0368d37dd9e2452982fb81f539c6805b P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Requested vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:49.823227 18985 tablet_bootstrap.cc:654] T 1f4e860484784aa9a2d28af2e9fb611c P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:49.824450 20489 consensus_queue.cc:1035] T 6db55bb510b24b0cae897936211e15f0 P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:49.828444 18475 raft_consensus.cc:3037] T 8e91920e7e36418d8c27d1f0bfd9b981 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:49.829916 20697 catalog_manager.cc:1504] Loading token signing keys...
I20240430 20:11:49.831442 18988 ts_tablet_manager.cc:1398] T 1f4e860484784aa9a2d28af2e9fb611c P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.029s user 0.002s sys 0.000s
I20240430 20:11:49.831862 18988 tablet_bootstrap.cc:492] T 9f0b341a31224bf1be525b84b9500304 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:49.832226 20695 raft_consensus.cc:2781] T 0368d37dd9e2452982fb81f539c6805b P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:49.832556 20695 raft_consensus.cc:686] T 0368d37dd9e2452982fb81f539c6805b P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Becoming Leader. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Running, Role: LEADER
I20240430 20:11:49.832801 20488 consensus_queue.cc:1035] T 8e91920e7e36418d8c27d1f0bfd9b981 P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:49.833333 18988 tablet_bootstrap.cc:654] T 9f0b341a31224bf1be525b84b9500304 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:49.833869 20695 consensus_queue.cc:237] T 0368d37dd9e2452982fb81f539c6805b P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.838661 18985 tablet_bootstrap.cc:492] T 1f4e860484784aa9a2d28af2e9fb611c P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:49.838819 18985 ts_tablet_manager.cc:1367] T 1f4e860484784aa9a2d28af2e9fb611c P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.003s
I20240430 20:11:49.839673 18985 raft_consensus.cc:348] T 1f4e860484784aa9a2d28af2e9fb611c P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.839964 18985 raft_consensus.cc:374] T 1f4e860484784aa9a2d28af2e9fb611c P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:49.840049 18985 raft_consensus.cc:729] T 1f4e860484784aa9a2d28af2e9fb611c P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:49.840320 18985 consensus_queue.cc:260] T 1f4e860484784aa9a2d28af2e9fb611c P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.841105 18985 ts_tablet_manager.cc:1398] T 1f4e860484784aa9a2d28af2e9fb611c P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240430 20:11:49.844117 18983 tablet_bootstrap.cc:654] T 9c73d0a6bc414d3b9900570848d19a51 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:49.849772 20697 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: Loaded TSK: 0
I20240430 20:11:49.851375 20697 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240430 20:11:49.857069 18985 tablet_bootstrap.cc:492] T b1792c160af346ffb4f047c5d4435c10 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:49.858959 18988 tablet_bootstrap.cc:492] T 9f0b341a31224bf1be525b84b9500304 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:49.859009 18985 tablet_bootstrap.cc:654] T b1792c160af346ffb4f047c5d4435c10 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:49.859149 18988 ts_tablet_manager.cc:1367] T 9f0b341a31224bf1be525b84b9500304 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.027s user 0.007s sys 0.001s
I20240430 20:11:49.859862 18988 raft_consensus.cc:348] T 9f0b341a31224bf1be525b84b9500304 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.860095 18988 raft_consensus.cc:374] T 9f0b341a31224bf1be525b84b9500304 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:49.860173 18988 raft_consensus.cc:729] T 9f0b341a31224bf1be525b84b9500304 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:49.860424 18988 consensus_queue.cc:260] T 9f0b341a31224bf1be525b84b9500304 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.861390 18988 ts_tablet_manager.cc:1398] T 9f0b341a31224bf1be525b84b9500304 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240430 20:11:49.861770 18988 tablet_bootstrap.cc:492] T d2381047f6dc4bdf962f9383e6c41548 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:49.864269 18988 tablet_bootstrap.cc:654] T d2381047f6dc4bdf962f9383e6c41548 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:49.866006 18983 tablet_bootstrap.cc:492] T 9c73d0a6bc414d3b9900570848d19a51 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:49.866178 18983 ts_tablet_manager.cc:1367] T 9c73d0a6bc414d3b9900570848d19a51 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.045s user 0.003s sys 0.000s
I20240430 20:11:49.867123 18983 raft_consensus.cc:348] T 9c73d0a6bc414d3b9900570848d19a51 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.867345 18983 raft_consensus.cc:374] T 9c73d0a6bc414d3b9900570848d19a51 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:49.867509 18983 raft_consensus.cc:729] T 9c73d0a6bc414d3b9900570848d19a51 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:49.867870 18983 consensus_queue.cc:260] T 9c73d0a6bc414d3b9900570848d19a51 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.868623 18983 ts_tablet_manager.cc:1398] T 9c73d0a6bc414d3b9900570848d19a51 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240430 20:11:49.870824 18985 tablet_bootstrap.cc:492] T b1792c160af346ffb4f047c5d4435c10 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:49.871013 18985 ts_tablet_manager.cc:1367] T b1792c160af346ffb4f047c5d4435c10 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.007s
I20240430 20:11:49.871752 18985 raft_consensus.cc:348] T b1792c160af346ffb4f047c5d4435c10 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.871977 18985 raft_consensus.cc:374] T b1792c160af346ffb4f047c5d4435c10 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:49.872058 18985 raft_consensus.cc:729] T b1792c160af346ffb4f047c5d4435c10 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:49.872344 18985 consensus_queue.cc:260] T b1792c160af346ffb4f047c5d4435c10 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.873884 20488 raft_consensus.cc:482] T e900f62a5aa143e791c7539660b60cd9 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:49.873962 18985 ts_tablet_manager.cc:1398] T b1792c160af346ffb4f047c5d4435c10 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240430 20:11:49.874047 20488 raft_consensus.cc:504] T e900f62a5aa143e791c7539660b60cd9 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.874573 18985 tablet_bootstrap.cc:492] T 0368d37dd9e2452982fb81f539c6805b P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:49.875886 18985 tablet_bootstrap.cc:654] T 0368d37dd9e2452982fb81f539c6805b P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:49.877570 18985 tablet_bootstrap.cc:492] T 0368d37dd9e2452982fb81f539c6805b P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:49.877595 18988 tablet_bootstrap.cc:492] T d2381047f6dc4bdf962f9383e6c41548 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:49.877725 18988 ts_tablet_manager.cc:1367] T d2381047f6dc4bdf962f9383e6c41548 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.000s
I20240430 20:11:49.877727 18985 ts_tablet_manager.cc:1367] T 0368d37dd9e2452982fb81f539c6805b P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240430 20:11:49.878510 18985 raft_consensus.cc:348] T 0368d37dd9e2452982fb81f539c6805b P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.878571 18988 raft_consensus.cc:348] T d2381047f6dc4bdf962f9383e6c41548 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.878753 18985 raft_consensus.cc:374] T 0368d37dd9e2452982fb81f539c6805b P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:49.878791 18988 raft_consensus.cc:374] T d2381047f6dc4bdf962f9383e6c41548 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:49.878834 18985 raft_consensus.cc:729] T 0368d37dd9e2452982fb81f539c6805b P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:49.878892 18988 raft_consensus.cc:729] T d2381047f6dc4bdf962f9383e6c41548 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:49.879084 18985 consensus_queue.cc:260] T 0368d37dd9e2452982fb81f539c6805b P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.879110 18988 consensus_queue.cc:260] T d2381047f6dc4bdf962f9383e6c41548 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.879792 18985 ts_tablet_manager.cc:1398] T 0368d37dd9e2452982fb81f539c6805b P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240430 20:11:49.880267 18985 tablet_bootstrap.cc:492] T 558e71ec370d46ca8fbdcec5b57eb2ec P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:49.880625 18988 ts_tablet_manager.cc:1398] T d2381047f6dc4bdf962f9383e6c41548 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240430 20:11:49.881196 18988 tablet_bootstrap.cc:492] T c1017a50d1d646c486c0b9a75cb88ce4 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:49.882189 18985 tablet_bootstrap.cc:654] T 558e71ec370d46ca8fbdcec5b57eb2ec P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:49.882926 18988 tablet_bootstrap.cc:654] T c1017a50d1d646c486c0b9a75cb88ce4 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:49.890154 19946 master_service.cc:946] ReplaceTablet: received request to replace tablet 2fc6908ecf954020a676b9d10a2b00e7 from {username='slave'} at 127.0.0.1:50746
I20240430 20:11:49.921798 20695 raft_consensus.cc:482] T 12795670f21447a8964fe1efc6e92398 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:49.921994 20695 raft_consensus.cc:504] T 12795670f21447a8964fe1efc6e92398 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.923082 20695 leader_election.cc:288] T 12795670f21447a8964fe1efc6e92398 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:49.923514 18475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12795670f21447a8964fe1efc6e92398" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:49.923744 18475 raft_consensus.cc:2454] T 12795670f21447a8964fe1efc6e92398 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:49.924127 18699 leader_election.cc:302] T 12795670f21447a8964fe1efc6e92398 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters:
I20240430 20:11:49.924466 20695 raft_consensus.cc:2781] T 12795670f21447a8964fe1efc6e92398 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:49.924580 20695 raft_consensus.cc:482] T 12795670f21447a8964fe1efc6e92398 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:49.924669 20695 raft_consensus.cc:3037] T 12795670f21447a8964fe1efc6e92398 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:49.927644 20695 raft_consensus.cc:504] T 12795670f21447a8964fe1efc6e92398 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.938361 18616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12795670f21447a8964fe1efc6e92398" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:49.938611 18616 raft_consensus.cc:2454] T 12795670f21447a8964fe1efc6e92398 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:49.939067 18616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12795670f21447a8964fe1efc6e92398" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:49.939218 18616 raft_consensus.cc:3037] T 12795670f21447a8964fe1efc6e92398 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:49.941834 18616 raft_consensus.cc:2454] T 12795670f21447a8964fe1efc6e92398 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:49.943198 20938 consensus_queue.cc:1035] T 90a694bd5db649afa50f3c4e752939c1 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:49.944304 18696 leader_election.cc:302] T 12795670f21447a8964fe1efc6e92398 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:49.944715 20198 raft_consensus.cc:2781] T 12795670f21447a8964fe1efc6e92398 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:49.944880 20198 raft_consensus.cc:686] T 12795670f21447a8964fe1efc6e92398 P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Becoming Leader. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Running, Role: LEADER
I20240430 20:11:49.945170 20198 consensus_queue.cc:237] T 12795670f21447a8964fe1efc6e92398 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:49.905073 19942 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50746:
name: "default.Table_cd541af5796d4b68abf97f9b05a1bb7b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:49.955478 19942 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cd541af5796d4b68abf97f9b05a1bb7b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:49.906013 19945 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50746:
name: "default.Table_a93c32ea6b7c49a1941650334c85b5e4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:49.958721 19945 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a93c32ea6b7c49a1941650334c85b5e4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:49.906813 19944 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50746:
name: "default.Table_93f5811e02a742e180954822432cacd1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:49.960594 19944 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_93f5811e02a742e180954822432cacd1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:49.961449 21012 consensus_queue.cc:1035] T 70247a4c910946c0afc24ce5ce0e85d6 P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:49.907631 19941 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50746:
table { table_name: "default.Table_94e5f2a77765434e8a76d9a558b23ae2" } new_table_name: "default.Table_144e5402c1864d818436015b26648024" modify_external_catalogs: true
I20240430 20:11:49.908233 19940 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50746:
table { table_name: "default.Table_f30868d759f64cceb682278f0dbb51ae" } modify_external_catalogs: true
I20240430 20:11:49.963778 19940 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50746:
table { table_name: "default.Table_f30868d759f64cceb682278f0dbb51ae" } modify_external_catalogs: true
I20240430 20:11:49.908862 19938 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50746:
table { table_name: "default.Table_b6ff64dbfc324fb199b8e6eebd51fd50" } modify_external_catalogs: true
I20240430 20:11:49.965679 19938 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50746:
table { table_name: "default.Table_b6ff64dbfc324fb199b8e6eebd51fd50" } modify_external_catalogs: true
I20240430 20:11:49.909564 19937 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50746:
table { table_name: "default.Table_ea4369e78de74045baa10ea2a61bfe38" } modify_external_catalogs: true
I20240430 20:11:49.967309 19937 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50746:
table { table_name: "default.Table_ea4369e78de74045baa10ea2a61bfe38" } modify_external_catalogs: true
I20240430 20:11:49.910292 19930 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50746:
table { table_name: "default.Table_7fb60e4b56f444009ab5da219238eec7" } new_table_name: "default.Table_9df4a0b4170f4f28a6c17b0cb052ead6" modify_external_catalogs: true
I20240430 20:11:49.910773 19929 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50746:
table { table_name: "default.Table_e860191f401c434e8cf0085ef41952b9" } new_table_name: "default.Table_c8b25b14d955429cbd79515081deceb1" modify_external_catalogs: true
W20240430 20:11:49.980073 18699 consensus_peers.cc:485] T 90a694bd5db649afa50f3c4e752939c1 P 3c51c33d5616482cafb68e26688820f4 -> Peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Couldn't send request to peer 2efb542aa55e439b99ccb98d1c5c884a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240430 20:11:49.985517 18475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12795670f21447a8964fe1efc6e92398" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:49.985939 18475 raft_consensus.cc:3037] T 12795670f21447a8964fe1efc6e92398 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:49.989720 18475 raft_consensus.cc:2454] T 12795670f21447a8964fe1efc6e92398 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:49.990530 20695 leader_election.cc:288] T 12795670f21447a8964fe1efc6e92398 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Requested vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:50.001909 18983 tablet_bootstrap.cc:492] T 47e98935eb21449abd3d4d24327778d5 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:50.002214 20489 consensus_queue.cc:1035] T 70247a4c910946c0afc24ce5ce0e85d6 P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:50.003510 18983 tablet_bootstrap.cc:654] T 47e98935eb21449abd3d4d24327778d5 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:50.021965 18988 tablet_bootstrap.cc:492] T c1017a50d1d646c486c0b9a75cb88ce4 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:50.022230 18988 ts_tablet_manager.cc:1367] T c1017a50d1d646c486c0b9a75cb88ce4 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.141s user 0.038s sys 0.011s
I20240430 20:11:50.022382 21013 raft_consensus.cc:482] T ed0c37a71ede4bfc9eb5ad1a4365f25e P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:50.022656 21013 raft_consensus.cc:504] T ed0c37a71ede4bfc9eb5ad1a4365f25e P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:50.023241 18988 raft_consensus.cc:348] T c1017a50d1d646c486c0b9a75cb88ce4 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:50.023561 18988 raft_consensus.cc:374] T c1017a50d1d646c486c0b9a75cb88ce4 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:50.023650 18988 raft_consensus.cc:729] T c1017a50d1d646c486c0b9a75cb88ce4 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:50.024129 18750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed0c37a71ede4bfc9eb5ad1a4365f25e" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:50.024662 18983 tablet_bootstrap.cc:492] T 47e98935eb21449abd3d4d24327778d5 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
W20240430 20:11:50.024801 18557 leader_election.cc:341] T ed0c37a71ede4bfc9eb5ad1a4365f25e P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:50.024866 18983 ts_tablet_manager.cc:1367] T 47e98935eb21449abd3d4d24327778d5 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.006s
I20240430 20:11:50.025247 21013 leader_election.cc:288] T ed0c37a71ede4bfc9eb5ad1a4365f25e P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:50.025866 18475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed0c37a71ede4bfc9eb5ad1a4365f25e" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:50.026180 18475 raft_consensus.cc:2454] T ed0c37a71ede4bfc9eb5ad1a4365f25e P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 0.
I20240430 20:11:50.026260 18983 raft_consensus.cc:348] T 47e98935eb21449abd3d4d24327778d5 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:50.026525 18983 raft_consensus.cc:374] T 47e98935eb21449abd3d4d24327778d5 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:50.026609 18983 raft_consensus.cc:729] T 47e98935eb21449abd3d4d24327778d5 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:50.026809 18558 leader_election.cc:302] T ed0c37a71ede4bfc9eb5ad1a4365f25e P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters: 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:50.027208 21013 raft_consensus.cc:2781] T ed0c37a71ede4bfc9eb5ad1a4365f25e P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:50.026887 18983 consensus_queue.cc:260] T 47e98935eb21449abd3d4d24327778d5 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:50.029275 21013 raft_consensus.cc:482] T ed0c37a71ede4bfc9eb5ad1a4365f25e P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:50.030273 18988 consensus_queue.cc:260] T c1017a50d1d646c486c0b9a75cb88ce4 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:50.030828 21013 raft_consensus.cc:3037] T ed0c37a71ede4bfc9eb5ad1a4365f25e P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:50.029786 18983 ts_tablet_manager.cc:1398] T 47e98935eb21449abd3d4d24327778d5 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240430 20:11:50.032447 18983 tablet_bootstrap.cc:492] T 2fc6908ecf954020a676b9d10a2b00e7 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:50.034655 18983 tablet_bootstrap.cc:654] T 2fc6908ecf954020a676b9d10a2b00e7 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:50.036049 18988 ts_tablet_manager.cc:1398] T c1017a50d1d646c486c0b9a75cb88ce4 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.014s user 0.003s sys 0.000s
I20240430 20:11:50.037451 18988 tablet_bootstrap.cc:492] T f7d3dea34cf54fe2b949d1b7ca68577a P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:50.038616 20384 raft_consensus.cc:482] T dad16ae02b8b4e29a724d8eec67bb4b8 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:50.038893 20384 raft_consensus.cc:504] T dad16ae02b8b4e29a724d8eec67bb4b8 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:50.040336 20384 leader_election.cc:288] T dad16ae02b8b4e29a724d8eec67bb4b8 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:50.040704 18474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dad16ae02b8b4e29a724d8eec67bb4b8" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:50.043109 20488 leader_election.cc:288] T e900f62a5aa143e791c7539660b60cd9 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:50.043924 18475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e900f62a5aa143e791c7539660b60cd9" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:50.044203 18475 raft_consensus.cc:2454] T e900f62a5aa143e791c7539660b60cd9 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 0.
I20240430 20:11:50.044596 18558 leader_election.cc:302] T e900f62a5aa143e791c7539660b60cd9 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:50.047499 21013 raft_consensus.cc:504] T ed0c37a71ede4bfc9eb5ad1a4365f25e P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:50.047832 18750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e900f62a5aa143e791c7539660b60cd9" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:50.040330 18756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dad16ae02b8b4e29a724d8eec67bb4b8" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:50.048707 18756 raft_consensus.cc:2379] T dad16ae02b8b4e29a724d8eec67bb4b8 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e1c6397f09747f091534faf93542710 in current term 1: Already voted for candidate 2efb542aa55e439b99ccb98d1c5c884a in this term.
W20240430 20:11:50.048563 18557 leader_election.cc:341] T e900f62a5aa143e791c7539660b60cd9 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:50.054109 18557 leader_election.cc:302] T dad16ae02b8b4e29a724d8eec67bb4b8 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:50.054783 18474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed0c37a71ede4bfc9eb5ad1a4365f25e" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:50.055081 18474 raft_consensus.cc:3037] T ed0c37a71ede4bfc9eb5ad1a4365f25e P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:50.055048 18756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed0c37a71ede4bfc9eb5ad1a4365f25e" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
W20240430 20:11:50.055703 18557 leader_election.cc:341] T ed0c37a71ede4bfc9eb5ad1a4365f25e P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:50.055959 20384 raft_consensus.cc:3037] T dad16ae02b8b4e29a724d8eec67bb4b8 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:50.057832 18474 raft_consensus.cc:2454] T ed0c37a71ede4bfc9eb5ad1a4365f25e P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
I20240430 20:11:50.058259 18558 leader_election.cc:302] T ed0c37a71ede4bfc9eb5ad1a4365f25e P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters: 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:50.058524 20489 raft_consensus.cc:2781] T ed0c37a71ede4bfc9eb5ad1a4365f25e P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:50.058969 20489 raft_consensus.cc:686] T ed0c37a71ede4bfc9eb5ad1a4365f25e P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Becoming Leader. State: Replica: 5e1c6397f09747f091534faf93542710, State: Running, Role: LEADER
I20240430 20:11:50.059212 20489 consensus_queue.cc:237] T ed0c37a71ede4bfc9eb5ad1a4365f25e P 5e1c6397f09747f091534faf93542710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:50.059631 21013 leader_election.cc:288] T ed0c37a71ede4bfc9eb5ad1a4365f25e P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:50.068241 21012 consensus_queue.cc:1035] T b1792c160af346ffb4f047c5d4435c10 P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:50.058455 20384 raft_consensus.cc:2726] T dad16ae02b8b4e29a724d8eec67bb4b8 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:50.076573 18988 tablet_bootstrap.cc:654] T f7d3dea34cf54fe2b949d1b7ca68577a P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:50.050305 20488 raft_consensus.cc:2781] T e900f62a5aa143e791c7539660b60cd9 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:50.089475 20488 raft_consensus.cc:482] T e900f62a5aa143e791c7539660b60cd9 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:50.089679 20488 raft_consensus.cc:3037] T e900f62a5aa143e791c7539660b60cd9 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
W20240430 20:11:50.100872 19941 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7a0176f8a74c47efbbe3d3343b3dc896 has no consensus state
I20240430 20:11:50.101614 19941 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50746:
table { table_name: "default.Table_5d3cb1a25c884804b05b07531e1b6965" } modify_external_catalogs: true
I20240430 20:11:50.101852 19941 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50746:
table { table_name: "default.Table_5d3cb1a25c884804b05b07531e1b6965" } modify_external_catalogs: true
I20240430 20:11:50.102807 19945 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50746:
name: "default.Table_11ef4a2c1255424bac9dd34fc0313926"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240430 20:11:50.103139 19944 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50746:
table { table_name: "default.Table_6c7df3aa7d8741ae9ac67cf4af868788" } modify_external_catalogs: true
I20240430 20:11:50.103320 19944 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50746:
table { table_name: "default.Table_6c7df3aa7d8741ae9ac67cf4af868788" } modify_external_catalogs: true
W20240430 20:11:50.103394 19945 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_11ef4a2c1255424bac9dd34fc0313926 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:50.104833 18528 heartbeater.cc:503] Master 127.17.177.62:36441 was elected leader, sending a full tablet report...
I20240430 20:11:50.105516 18983 tablet_bootstrap.cc:492] T 2fc6908ecf954020a676b9d10a2b00e7 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:50.105706 18983 ts_tablet_manager.cc:1367] T 2fc6908ecf954020a676b9d10a2b00e7 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.073s user 0.003s sys 0.000s
I20240430 20:11:50.106567 18983 raft_consensus.cc:348] T 2fc6908ecf954020a676b9d10a2b00e7 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:50.106832 18983 raft_consensus.cc:374] T 2fc6908ecf954020a676b9d10a2b00e7 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:50.106904 18983 raft_consensus.cc:729] T 2fc6908ecf954020a676b9d10a2b00e7 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:50.107172 18983 consensus_queue.cc:260] T 2fc6908ecf954020a676b9d10a2b00e7 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:50.107839 18983 ts_tablet_manager.cc:1398] T 2fc6908ecf954020a676b9d10a2b00e7 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240430 20:11:50.108700 18668 heartbeater.cc:503] Master 127.17.177.62:36441 was elected leader, sending a full tablet report...
I20240430 20:11:50.109935 18983 tablet_bootstrap.cc:492] T c1017a50d1d646c486c0b9a75cb88ce4 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:50.113550 18988 tablet_bootstrap.cc:492] T f7d3dea34cf54fe2b949d1b7ca68577a P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:50.113742 18988 ts_tablet_manager.cc:1367] T f7d3dea34cf54fe2b949d1b7ca68577a P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.077s user 0.003s sys 0.000s
I20240430 20:11:50.114707 18988 raft_consensus.cc:348] T f7d3dea34cf54fe2b949d1b7ca68577a P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:50.114980 18988 raft_consensus.cc:374] T f7d3dea34cf54fe2b949d1b7ca68577a P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:50.115088 18988 raft_consensus.cc:729] T f7d3dea34cf54fe2b949d1b7ca68577a P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:50.115391 18988 consensus_queue.cc:260] T f7d3dea34cf54fe2b949d1b7ca68577a P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:50.118675 18983 tablet_bootstrap.cc:654] T c1017a50d1d646c486c0b9a75cb88ce4 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:50.119997 19940 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: Sending DeleteTablet for 3 replicas of tablet d34a450a91d542c98261456d1f635e9c
I20240430 20:11:50.121496 19940 master_service.cc:946] ReplaceTablet: received request to replace tablet fd1b8407071e4e4287fc172009777414 from {username='slave'} at 127.0.0.1:50746
I20240430 20:11:50.121953 19938 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: Sending DeleteTablet for 3 replicas of tablet 0eb4f3e7ece6496293e01b63620d644b
I20240430 20:11:50.122820 19937 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet 9ea773945f294938a0352d28f8d9f93e
I20240430 20:11:50.131858 20545 consensus_queue.cc:1035] T dad16ae02b8b4e29a724d8eec67bb4b8 P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:50.154872 20695 raft_consensus.cc:482] T c1017a50d1d646c486c0b9a75cb88ce4 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240430 20:11:50.155081 19900 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 7a0176f8a74c47efbbe3d3343b3dc896 (table default.Table_144e5402c1864d818436015b26648024 [id=73e5c9f0470d47849109819c063e1ce4], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 7a0176f8a74c47efbbe3d3343b3dc896 has no consensus state
I20240430 20:11:50.155102 20695 raft_consensus.cc:504] T c1017a50d1d646c486c0b9a75cb88ce4 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:50.156646 18616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1017a50d1d646c486c0b9a75cb88ce4" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:50.159140 20695 leader_election.cc:288] T c1017a50d1d646c486c0b9a75cb88ce4 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
W20240430 20:11:50.159220 19929 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8e91920e7e36418d8c27d1f0bfd9b981 consensus state has no leader
I20240430 20:11:50.161811 18988 ts_tablet_manager.cc:1398] T f7d3dea34cf54fe2b949d1b7ca68577a P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.040s user 0.003s sys 0.000s
I20240430 20:11:50.161806 20198 consensus_queue.cc:1035] T 2a08e36c39824abf98b9b6214c8b84cb P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.044s
W20240430 20:11:50.162483 18696 leader_election.cc:341] T c1017a50d1d646c486c0b9a75cb88ce4 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Illegal state: must be running to vote when last-logged opid is not known
W20240430 20:11:50.162971 19930 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b285e3eb16d148779c74e057458d8461 consensus state has no leader
I20240430 20:11:50.168597 20198 consensus_queue.cc:1035] T b285e3eb16d148779c74e057458d8461 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:50.171483 18988 tablet_bootstrap.cc:492] T 558e71ec370d46ca8fbdcec5b57eb2ec P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:50.173439 18988 tablet_bootstrap.cc:654] T 558e71ec370d46ca8fbdcec5b57eb2ec P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:50.185097 18983 tablet_bootstrap.cc:492] T c1017a50d1d646c486c0b9a75cb88ce4 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:50.185514 18983 ts_tablet_manager.cc:1367] T c1017a50d1d646c486c0b9a75cb88ce4 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.076s user 0.002s sys 0.003s
I20240430 20:11:50.186429 18983 raft_consensus.cc:348] T c1017a50d1d646c486c0b9a75cb88ce4 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:50.191599 19937 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50746:
name: "default.Table_e158c1ec239f48a9be498828ad9a5a72"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:50.192443 19937 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e158c1ec239f48a9be498828ad9a5a72 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:50.194382 19944 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet 35875133118541ea909b9e644251a01e
I20240430 20:11:50.194680 19944 master_service.cc:946] ReplaceTablet: received request to replace tablet 1462ecf1176c4e66be42abddbe36b00c from {username='slave'} at 127.0.0.1:50746
I20240430 20:11:50.194677 19941 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet 7fceb89e8ee148999f747b08a15fd274
I20240430 20:11:50.195329 19945 master_service.cc:946] ReplaceTablet: received request to replace tablet 0eb4f3e7ece6496293e01b63620d644b from {username='slave'} at 127.0.0.1:50746
I20240430 20:11:50.195473 19930 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50746:
name: "default.Table_c1538175cbb94c18b8fd9e5de5436047"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240430 20:11:50.195963 19945 master_service.cc:946] ReplaceTablet: received request to replace tablet 84568f90b9c54cf292099808d4cbf5dc from {username='slave'} at 127.0.0.1:50746
W20240430 20:11:50.196138 19930 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c1538175cbb94c18b8fd9e5de5436047 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:50.196641 19945 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50746:
name: "default.Table_45b0755ed3e748f78fe856a4ab1831cc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:50.197247 19945 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_45b0755ed3e748f78fe856a4ab1831cc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:50.197779 19938 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50746:
name: "default.Table_31760cb86ebf4666abcffc4217aa2708"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:50.198379 19938 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_31760cb86ebf4666abcffc4217aa2708 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:50.201084 19941 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50746:
name: "default.Table_49d181ca513240d88696579ad5b4d3d8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:50.201682 19941 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_49d181ca513240d88696579ad5b4d3d8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:50.203011 19946 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: Sending DeleteTablet for 3 replicas of tablet 2fc6908ecf954020a676b9d10a2b00e7
I20240430 20:11:50.204000 19946 catalog_manager.cc:6554] ReplaceTablet: tablet 2fc6908ecf954020a676b9d10a2b00e7 deleted and replaced by tablet 9a5654fd913f4fbeb9d126f035ac32bf
I20240430 20:11:50.204491 19946 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50746:
table { table_name: "default.Table_c1669746f9114bf7b37a6f6a68fed20e" } new_table_name: "default.Table_0b4ca4cd0af148399dfa21f2d32114e3" modify_external_catalogs: true
I20240430 20:11:50.205241 19946 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50746:
name: "default.Table_e7097af3f530403a854d53e9d8e87108"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:50.206701 19946 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e7097af3f530403a854d53e9d8e87108 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:50.191679 18983 raft_consensus.cc:374] T c1017a50d1d646c486c0b9a75cb88ce4 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:50.211603 18983 raft_consensus.cc:729] T c1017a50d1d646c486c0b9a75cb88ce4 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:50.214401 21013 raft_consensus.cc:482] T 9c73d0a6bc414d3b9900570848d19a51 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:50.214579 21013 raft_consensus.cc:504] T 9c73d0a6bc414d3b9900570848d19a51 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:50.215310 21013 leader_election.cc:288] T 9c73d0a6bc414d3b9900570848d19a51 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:50.215670 18988 tablet_bootstrap.cc:492] T 558e71ec370d46ca8fbdcec5b57eb2ec P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:50.215912 18988 ts_tablet_manager.cc:1367] T 558e71ec370d46ca8fbdcec5b57eb2ec P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.045s user 0.003s sys 0.000s
I20240430 20:11:50.216849 18988 raft_consensus.cc:348] T 558e71ec370d46ca8fbdcec5b57eb2ec P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:50.217176 18988 raft_consensus.cc:374] T 558e71ec370d46ca8fbdcec5b57eb2ec P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:50.217271 18988 raft_consensus.cc:729] T 558e71ec370d46ca8fbdcec5b57eb2ec P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:50.217581 18988 consensus_queue.cc:260] T 558e71ec370d46ca8fbdcec5b57eb2ec P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:50.214455 18983 consensus_queue.cc:260] T c1017a50d1d646c486c0b9a75cb88ce4 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:50.228495 18983 ts_tablet_manager.cc:1398] T c1017a50d1d646c486c0b9a75cb88ce4 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.043s user 0.003s sys 0.000s
I20240430 20:11:50.228916 18983 tablet_bootstrap.cc:492] T d2381047f6dc4bdf962f9383e6c41548 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:50.233383 18983 tablet_bootstrap.cc:654] T d2381047f6dc4bdf962f9383e6c41548 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:50.239192 18985 tablet_bootstrap.cc:492] T 558e71ec370d46ca8fbdcec5b57eb2ec P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:50.239353 18985 ts_tablet_manager.cc:1367] T 558e71ec370d46ca8fbdcec5b57eb2ec P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.359s user 0.085s sys 0.135s
I20240430 20:11:50.239882 18988 ts_tablet_manager.cc:1398] T 558e71ec370d46ca8fbdcec5b57eb2ec P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.024s user 0.003s sys 0.001s
I20240430 20:11:50.240193 18985 raft_consensus.cc:348] T 558e71ec370d46ca8fbdcec5b57eb2ec P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:50.241958 18985 raft_consensus.cc:374] T 558e71ec370d46ca8fbdcec5b57eb2ec P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:50.242022 18985 raft_consensus.cc:729] T 558e71ec370d46ca8fbdcec5b57eb2ec P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:50.242472 18988 tablet_bootstrap.cc:492] T e900f62a5aa143e791c7539660b60cd9 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:50.242322 18985 consensus_queue.cc:260] T 558e71ec370d46ca8fbdcec5b57eb2ec P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:50.249305 18750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c73d0a6bc414d3b9900570848d19a51" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:50.249616 18750 raft_consensus.cc:2454] T 9c73d0a6bc414d3b9900570848d19a51 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 0.
I20240430 20:11:50.261858 18475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c73d0a6bc414d3b9900570848d19a51" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:50.262099 18475 raft_consensus.cc:2454] T 9c73d0a6bc414d3b9900570848d19a51 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 0.
I20240430 20:11:50.263552 21050 raft_consensus.cc:482] T 2873af6da928463f9809864b8eaf0c4a P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:50.263721 21050 raft_consensus.cc:504] T 2873af6da928463f9809864b8eaf0c4a P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:50.264509 21050 leader_election.cc:288] T 2873af6da928463f9809864b8eaf0c4a P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:50.266916 20488 raft_consensus.cc:504] T e900f62a5aa143e791c7539660b60cd9 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:50.267498 18988 tablet_bootstrap.cc:654] T e900f62a5aa143e791c7539660b60cd9 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:50.272202 18557 leader_election.cc:302] T 9c73d0a6bc414d3b9900570848d19a51 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:50.272575 21011 raft_consensus.cc:2781] T 9c73d0a6bc414d3b9900570848d19a51 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:50.272748 21011 raft_consensus.cc:482] T 9c73d0a6bc414d3b9900570848d19a51 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:50.272891 21011 raft_consensus.cc:3037] T 9c73d0a6bc414d3b9900570848d19a51 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:50.275766 21011 raft_consensus.cc:504] T 9c73d0a6bc414d3b9900570848d19a51 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:50.276808 21011 leader_election.cc:288] T 9c73d0a6bc414d3b9900570848d19a51 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:50.277128 20489 raft_consensus.cc:482] T a445fdfde01c4371a5e6b3daea5b00ab P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:50.277303 20489 raft_consensus.cc:504] T a445fdfde01c4371a5e6b3daea5b00ab P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:50.277900 20489 leader_election.cc:288] T a445fdfde01c4371a5e6b3daea5b00ab P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:50.279304 18616 raft_consensus.cc:3037] T 47e98935eb21449abd3d4d24327778d5 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:50.281749 18475 raft_consensus.cc:3037] T fda942d50b014f3d9589e1fb909fc2fc P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:50.282539 18473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1017a50d1d646c486c0b9a75cb88ce4" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:50.283958 20587 consensus_queue.cc:1035] T 47e98935eb21449abd3d4d24327778d5 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:50.285137 18988 tablet_bootstrap.cc:492] T e900f62a5aa143e791c7539660b60cd9 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:50.285316 18988 ts_tablet_manager.cc:1367] T e900f62a5aa143e791c7539660b60cd9 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.043s user 0.004s sys 0.000s
I20240430 20:11:50.286208 18988 raft_consensus.cc:348] T e900f62a5aa143e791c7539660b60cd9 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:50.286458 18988 raft_consensus.cc:374] T e900f62a5aa143e791c7539660b60cd9 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:50.286567 18988 raft_consensus.cc:729] T e900f62a5aa143e791c7539660b60cd9 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:50.286863 18988 consensus_queue.cc:260] T e900f62a5aa143e791c7539660b60cd9 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:50.295217 18608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2873af6da928463f9809864b8eaf0c4a" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:50.295470 18988 ts_tablet_manager.cc:1398] T e900f62a5aa143e791c7539660b60cd9 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240430 20:11:50.295521 18608 raft_consensus.cc:2396] T 2873af6da928463f9809864b8eaf0c4a P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2efb542aa55e439b99ccb98d1c5c884a for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20240430 20:11:50.295795 18988 tablet_bootstrap.cc:492] T 3e5225f1f5f544ecbf8dfa470af3daca P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:50.296867 20488 leader_election.cc:288] T e900f62a5aa143e791c7539660b60cd9 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:50.297453 21049 raft_consensus.cc:482] T 95383d9d5a234a4d9d841a8b114428b0 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:50.297608 21049 raft_consensus.cc:504] T 95383d9d5a234a4d9d841a8b114428b0 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:50.297780 18988 tablet_bootstrap.cc:654] T 3e5225f1f5f544ecbf8dfa470af3daca P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:50.299270 20587 consensus_queue.cc:1035] T fda942d50b014f3d9589e1fb909fc2fc P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:50.301800 18756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e900f62a5aa143e791c7539660b60cd9" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:50.303242 18756 raft_consensus.cc:3037] T e900f62a5aa143e791c7539660b60cd9 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
W20240430 20:11:50.306233 18415 leader_election.cc:341] T 95383d9d5a234a4d9d841a8b114428b0 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Not found: Tablet not found: 95383d9d5a234a4d9d841a8b114428b0
I20240430 20:11:50.307133 18756 raft_consensus.cc:2454] T e900f62a5aa143e791c7539660b60cd9 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
I20240430 20:11:50.307402 20629 raft_consensus.cc:482] T b285e3eb16d148779c74e057458d8461 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:50.307583 20629 raft_consensus.cc:504] T b285e3eb16d148779c74e057458d8461 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:50.314853 18478 raft_consensus.cc:3037] T b285e3eb16d148779c74e057458d8461 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
W20240430 20:11:50.315886 18417 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.177.3:44867, user_credentials={real_user=slave}} blocked reactor thread for 85961.3us
I20240430 20:11:50.319494 18478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c73d0a6bc414d3b9900570848d19a51" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:50.319691 18478 raft_consensus.cc:3037] T 9c73d0a6bc414d3b9900570848d19a51 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:50.319635 18475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a445fdfde01c4371a5e6b3daea5b00ab" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:50.320266 18482 raft_consensus.cc:3037] T b1792c160af346ffb4f047c5d4435c10 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:50.322276 18478 raft_consensus.cc:2454] T 9c73d0a6bc414d3b9900570848d19a51 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
I20240430 20:11:50.322902 20629 leader_election.cc:288] T b285e3eb16d148779c74e057458d8461 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:50.324595 18558 leader_election.cc:302] T 9c73d0a6bc414d3b9900570848d19a51 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:50.324959 21011 raft_consensus.cc:2781] T 9c73d0a6bc414d3b9900570848d19a51 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:50.325100 21011 raft_consensus.cc:686] T 9c73d0a6bc414d3b9900570848d19a51 P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Becoming Leader. State: Replica: 5e1c6397f09747f091534faf93542710, State: Running, Role: LEADER
I20240430 20:11:50.325650 21011 consensus_queue.cc:237] T 9c73d0a6bc414d3b9900570848d19a51 P 5e1c6397f09747f091534faf93542710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:50.326880 18474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e900f62a5aa143e791c7539660b60cd9" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:50.327149 18474 raft_consensus.cc:3037] T e900f62a5aa143e791c7539660b60cd9 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:50.329421 18985 ts_tablet_manager.cc:1398] T 558e71ec370d46ca8fbdcec5b57eb2ec P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.090s user 0.003s sys 0.002s
I20240430 20:11:50.329730 18985 tablet_bootstrap.cc:492] T 1701a15b115741aea13f8327e3d26b34 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:50.330688 20488 consensus_queue.cc:1035] T b1792c160af346ffb4f047c5d4435c10 P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:50.331213 18985 tablet_bootstrap.cc:654] T 1701a15b115741aea13f8327e3d26b34 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:50.332607 20384 raft_consensus.cc:482] T 2873af6da928463f9809864b8eaf0c4a P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3c51c33d5616482cafb68e26688820f4)
I20240430 20:11:50.332728 20384 raft_consensus.cc:504] T 2873af6da928463f9809864b8eaf0c4a P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:50.298439 21049 leader_election.cc:288] T 95383d9d5a234a4d9d841a8b114428b0 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:50.333321 20384 leader_election.cc:288] T 2873af6da928463f9809864b8eaf0c4a P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:50.336577 18608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b285e3eb16d148779c74e057458d8461" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:50.340613 18983 tablet_bootstrap.cc:492] T d2381047f6dc4bdf962f9383e6c41548 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:50.340750 18983 ts_tablet_manager.cc:1367] T d2381047f6dc4bdf962f9383e6c41548 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.112s user 0.010s sys 0.005s
I20240430 20:11:50.310937 18750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c73d0a6bc414d3b9900570848d19a51" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:50.341156 18750 raft_consensus.cc:3037] T 9c73d0a6bc414d3b9900570848d19a51 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:50.341091 18478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2873af6da928463f9809864b8eaf0c4a" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:50.341285 18478 raft_consensus.cc:2454] T 2873af6da928463f9809864b8eaf0c4a P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
I20240430 20:11:50.342803 18983 raft_consensus.cc:348] T d2381047f6dc4bdf962f9383e6c41548 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:50.343001 18983 raft_consensus.cc:374] T d2381047f6dc4bdf962f9383e6c41548 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:50.343053 18983 raft_consensus.cc:729] T d2381047f6dc4bdf962f9383e6c41548 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:50.344328 18750 raft_consensus.cc:2454] T 9c73d0a6bc414d3b9900570848d19a51 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
W20240430 20:11:50.314288 18699 leader_election.cc:341] T c1017a50d1d646c486c0b9a75cb88ce4 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:50.338615 18988 tablet_bootstrap.cc:492] T 3e5225f1f5f544ecbf8dfa470af3daca P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:50.344928 18699 leader_election.cc:302] T c1017a50d1d646c486c0b9a75cb88ce4 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710
I20240430 20:11:50.345049 18988 ts_tablet_manager.cc:1367] T 3e5225f1f5f544ecbf8dfa470af3daca P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.049s user 0.006s sys 0.000s
I20240430 20:11:50.345708 20695 raft_consensus.cc:2726] T c1017a50d1d646c486c0b9a75cb88ce4 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:50.345752 18988 raft_consensus.cc:348] T 3e5225f1f5f544ecbf8dfa470af3daca P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:50.345961 18988 raft_consensus.cc:374] T 3e5225f1f5f544ecbf8dfa470af3daca P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:50.346043 18988 raft_consensus.cc:729] T 3e5225f1f5f544ecbf8dfa470af3daca P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:50.346269 18988 consensus_queue.cc:260] T 3e5225f1f5f544ecbf8dfa470af3daca P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:50.346717 18988 ts_tablet_manager.cc:1398] T 3e5225f1f5f544ecbf8dfa470af3daca P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240430 20:11:50.347028 18988 tablet_bootstrap.cc:492] T 1701a15b115741aea13f8327e3d26b34 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:50.348587 18988 tablet_bootstrap.cc:654] T 1701a15b115741aea13f8327e3d26b34 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:50.348711 18750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a445fdfde01c4371a5e6b3daea5b00ab" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:50.348989 18750 raft_consensus.cc:2379] T a445fdfde01c4371a5e6b3daea5b00ab P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e1c6397f09747f091534faf93542710 in current term 1: Already voted for candidate 2efb542aa55e439b99ccb98d1c5c884a in this term.
I20240430 20:11:50.343236 18983 consensus_queue.cc:260] T d2381047f6dc4bdf962f9383e6c41548 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:50.349957 18983 ts_tablet_manager.cc:1398] T d2381047f6dc4bdf962f9383e6c41548 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240430 20:11:50.351280 18754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b285e3eb16d148779c74e057458d8461" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:50.350240 18983 tablet_bootstrap.cc:492] T f7d3dea34cf54fe2b949d1b7ca68577a P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:50.353377 20940 consensus_queue.cc:1035] T fda942d50b014f3d9589e1fb909fc2fc P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:50.353488 18983 tablet_bootstrap.cc:654] T f7d3dea34cf54fe2b949d1b7ca68577a P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:50.355182 20940 consensus_queue.cc:1035] T b285e3eb16d148779c74e057458d8461 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:50.358974 20587 raft_consensus.cc:482] T 2fc6908ecf954020a676b9d10a2b00e7 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:50.359167 20587 raft_consensus.cc:504] T 2fc6908ecf954020a676b9d10a2b00e7 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:50.359871 20587 leader_election.cc:288] T 2fc6908ecf954020a676b9d10a2b00e7 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:50.360677 18608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fc6908ecf954020a676b9d10a2b00e7" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:50.360916 18608 raft_consensus.cc:2454] T 2fc6908ecf954020a676b9d10a2b00e7 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:50.361470 18696 leader_election.cc:302] T 2fc6908ecf954020a676b9d10a2b00e7 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:50.361871 20198 raft_consensus.cc:2781] T 2fc6908ecf954020a676b9d10a2b00e7 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:50.361989 20198 raft_consensus.cc:482] T 2fc6908ecf954020a676b9d10a2b00e7 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:50.362109 20198 raft_consensus.cc:3037] T 2fc6908ecf954020a676b9d10a2b00e7 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:50.362990 18608 raft_consensus.cc:3037] T 2a08e36c39824abf98b9b6214c8b84cb P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:50.367311 20587 consensus_queue.cc:1035] T 2a08e36c39824abf98b9b6214c8b84cb P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240430 20:11:50.367625 18557 leader_election.cc:341] T 2873af6da928463f9809864b8eaf0c4a P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Not found: Tablet not found: 2873af6da928463f9809864b8eaf0c4a
I20240430 20:11:50.368119 18988 tablet_bootstrap.cc:492] T 1701a15b115741aea13f8327e3d26b34 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:50.368286 18988 ts_tablet_manager.cc:1367] T 1701a15b115741aea13f8327e3d26b34 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.000s
I20240430 20:11:50.373371 20198 raft_consensus.cc:504] T 2fc6908ecf954020a676b9d10a2b00e7 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:50.367841 18557 leader_election.cc:302] T a445fdfde01c4371a5e6b3daea5b00ab P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:50.379137 21011 raft_consensus.cc:3037] T a445fdfde01c4371a5e6b3daea5b00ab P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:50.382308 18616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fc6908ecf954020a676b9d10a2b00e7" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:50.382706 18616 raft_consensus.cc:3037] T 2fc6908ecf954020a676b9d10a2b00e7 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:50.383965 20587 raft_consensus.cc:482] T d2381047f6dc4bdf962f9383e6c41548 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:50.384171 20587 raft_consensus.cc:504] T d2381047f6dc4bdf962f9383e6c41548 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:50.384968 20587 leader_election.cc:288] T d2381047f6dc4bdf962f9383e6c41548 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
W20240430 20:11:50.385506 19899 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.177.61:34777, user_credentials={real_user=slave}} blocked reactor thread for 209294us
I20240430 20:11:50.385588 18620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2381047f6dc4bdf962f9383e6c41548" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:50.385984 18620 raft_consensus.cc:2454] T d2381047f6dc4bdf962f9383e6c41548 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:50.386139 18558 leader_election.cc:302] T 2873af6da928463f9809864b8eaf0c4a P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters: 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:50.386777 18696 leader_election.cc:302] T d2381047f6dc4bdf962f9383e6c41548 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:50.387331 20587 raft_consensus.cc:2781] T d2381047f6dc4bdf962f9383e6c41548 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:50.387444 20587 raft_consensus.cc:482] T d2381047f6dc4bdf962f9383e6c41548 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:50.387549 20587 raft_consensus.cc:3037] T d2381047f6dc4bdf962f9383e6c41548 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:50.387591 20488 raft_consensus.cc:2781] T 2873af6da928463f9809864b8eaf0c4a P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240430 20:11:50.387665 20488 raft_consensus.cc:482] T 2873af6da928463f9809864b8eaf0c4a P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3c51c33d5616482cafb68e26688820f4)
I20240430 20:11:50.387725 20488 raft_consensus.cc:3037] T 2873af6da928463f9809864b8eaf0c4a P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Advancing to term 2
I20240430 20:11:50.388132 18557 leader_election.cc:302] T e900f62a5aa143e791c7539660b60cd9 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:50.388403 21013 raft_consensus.cc:2781] T e900f62a5aa143e791c7539660b60cd9 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:50.388504 21013 raft_consensus.cc:686] T e900f62a5aa143e791c7539660b60cd9 P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Becoming Leader. State: Replica: 5e1c6397f09747f091534faf93542710, State: Running, Role: LEADER
I20240430 20:11:50.388721 21013 consensus_queue.cc:237] T e900f62a5aa143e791c7539660b60cd9 P 5e1c6397f09747f091534faf93542710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:50.390180 20587 raft_consensus.cc:504] T d2381047f6dc4bdf962f9383e6c41548 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:50.391225 20587 leader_election.cc:288] T d2381047f6dc4bdf962f9383e6c41548 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Requested vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:50.391532 18619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2381047f6dc4bdf962f9383e6c41548" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:50.391727 18619 raft_consensus.cc:3037] T d2381047f6dc4bdf962f9383e6c41548 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:50.392343 20198 leader_election.cc:288] T 2fc6908ecf954020a676b9d10a2b00e7 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Requested vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:50.397595 20198 raft_consensus.cc:482] T 9f0b341a31224bf1be525b84b9500304 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:50.397832 20198 raft_consensus.cc:504] T 9f0b341a31224bf1be525b84b9500304 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:50.398916 20198 leader_election.cc:288] T 9f0b341a31224bf1be525b84b9500304 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:50.401018 18740 tablet_service.cc:1514] Processing DeleteTablet for tablet d34a450a91d542c98261456d1f635e9c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:49 UTC) from {username='slave'} at 127.0.0.1:56540
W20240430 20:11:50.401125 18696 leader_election.cc:341] T 9f0b341a31224bf1be525b84b9500304 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Not found: Tablet not found: 9f0b341a31224bf1be525b84b9500304
I20240430 20:11:50.401399 18739 tablet_service.cc:1514] Processing DeleteTablet for tablet 0eb4f3e7ece6496293e01b63620d644b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:49 UTC) from {username='slave'} at 127.0.0.1:56540
I20240430 20:11:50.401628 18732 tablet_service.cc:1514] Processing DeleteTablet for tablet 2fc6908ecf954020a676b9d10a2b00e7 with delete_type TABLET_DATA_DELETED (replaced by tablet 9a5654fd913f4fbeb9d126f035ac32bf) from {username='slave'} at 127.0.0.1:56540
I20240430 20:11:50.403625 18988 raft_consensus.cc:348] T 1701a15b115741aea13f8327e3d26b34 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:50.403939 18988 raft_consensus.cc:374] T 1701a15b115741aea13f8327e3d26b34 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:50.404050 18988 raft_consensus.cc:729] T 1701a15b115741aea13f8327e3d26b34 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:50.409826 20629 consensus_queue.cc:1035] T dad16ae02b8b4e29a724d8eec67bb4b8 P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240430 20:11:50.411185 18417 leader_election.cc:341] T 2873af6da928463f9809864b8eaf0c4a P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Not found: Tablet not found: 2873af6da928463f9809864b8eaf0c4a
I20240430 20:11:50.411392 18417 leader_election.cc:302] T 2873af6da928463f9809864b8eaf0c4a P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a; no voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710
I20240430 20:11:50.411724 20629 raft_consensus.cc:2726] T 2873af6da928463f9809864b8eaf0c4a P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240430 20:11:50.412979 18417 leader_election.cc:341] T 95383d9d5a234a4d9d841a8b114428b0 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Not found: Tablet not found: 95383d9d5a234a4d9d841a8b114428b0
I20240430 20:11:50.404346 18988 consensus_queue.cc:260] T 1701a15b115741aea13f8327e3d26b34 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:50.418057 18988 ts_tablet_manager.cc:1398] T 1701a15b115741aea13f8327e3d26b34 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.050s user 0.003s sys 0.000s
I20240430 20:11:50.418474 18988 tablet_bootstrap.cc:492] T ed0c37a71ede4bfc9eb5ad1a4365f25e P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:50.419505 18983 tablet_bootstrap.cc:492] T f7d3dea34cf54fe2b949d1b7ca68577a P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:50.419663 18983 ts_tablet_manager.cc:1367] T f7d3dea34cf54fe2b949d1b7ca68577a P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.069s user 0.004s sys 0.002s
I20240430 20:11:50.420406 18983 raft_consensus.cc:348] T f7d3dea34cf54fe2b949d1b7ca68577a P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:50.420612 18983 raft_consensus.cc:374] T f7d3dea34cf54fe2b949d1b7ca68577a P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:50.420661 18983 raft_consensus.cc:729] T f7d3dea34cf54fe2b949d1b7ca68577a P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:50.420857 18983 consensus_queue.cc:260] T f7d3dea34cf54fe2b949d1b7ca68577a P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:50.421343 18983 ts_tablet_manager.cc:1398] T f7d3dea34cf54fe2b949d1b7ca68577a P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240430 20:11:50.421602 18983 tablet_bootstrap.cc:492] T 8e9711d0a382424f9f758bfa883866a0 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:50.434238 18417 leader_election.cc:302] T 95383d9d5a234a4d9d841a8b114428b0 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a; no voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710
I20240430 20:11:50.434937 20629 raft_consensus.cc:2726] T 95383d9d5a234a4d9d841a8b114428b0 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:50.436741 18983 tablet_bootstrap.cc:654] T 8e9711d0a382424f9f758bfa883866a0 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:50.414535 18985 tablet_bootstrap.cc:492] T 1701a15b115741aea13f8327e3d26b34 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:50.439467 18985 ts_tablet_manager.cc:1367] T 1701a15b115741aea13f8327e3d26b34 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.110s user 0.000s sys 0.009s
I20240430 20:11:50.440703 18985 raft_consensus.cc:348] T 1701a15b115741aea13f8327e3d26b34 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:50.440958 18985 raft_consensus.cc:374] T 1701a15b115741aea13f8327e3d26b34 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:50.441011 18985 raft_consensus.cc:729] T 1701a15b115741aea13f8327e3d26b34 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:50.441216 18985 consensus_queue.cc:260] T 1701a15b115741aea13f8327e3d26b34 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:50.441740 18603 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c73d0a6bc414d3b9900570848d19a51 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:41 UTC) from {username='slave'} at 127.0.0.1:45934
I20240430 20:11:50.441991 18602 tablet_service.cc:1514] Processing DeleteTablet for tablet 0eb4f3e7ece6496293e01b63620d644b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:49 UTC) from {username='slave'} at 127.0.0.1:45934
I20240430 20:11:50.442093 18590 tablet_service.cc:1514] Processing DeleteTablet for tablet d34a450a91d542c98261456d1f635e9c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:49 UTC) from {username='slave'} at 127.0.0.1:45934
I20240430 20:11:50.442418 18590 tablet_service.cc:1514] Processing DeleteTablet for tablet 2fc6908ecf954020a676b9d10a2b00e7 with delete_type TABLET_DATA_DELETED (replaced by tablet 9a5654fd913f4fbeb9d126f035ac32bf) from {username='slave'} at 127.0.0.1:45934
I20240430 20:11:50.451805 18985 ts_tablet_manager.cc:1398] T 1701a15b115741aea13f8327e3d26b34 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.012s user 0.000s sys 0.002s
I20240430 20:11:50.453989 18985 tablet_bootstrap.cc:492] T 90a694bd5db649afa50f3c4e752939c1 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:50.459769 21011 raft_consensus.cc:2726] T a445fdfde01c4371a5e6b3daea5b00ab P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:50.460714 18482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fc6908ecf954020a676b9d10a2b00e7" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:50.461256 18478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fc6908ecf954020a676b9d10a2b00e7" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:50.463657 18417 leader_election.cc:302] T b285e3eb16d148779c74e057458d8461 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a; no voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710
I20240430 20:11:50.330567 18474 raft_consensus.cc:2454] T e900f62a5aa143e791c7539660b60cd9 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
I20240430 20:11:50.464848 18985 tablet_bootstrap.cc:654] T 90a694bd5db649afa50f3c4e752939c1 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:50.462761 18616 raft_consensus.cc:2454] T 2fc6908ecf954020a676b9d10a2b00e7 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:50.467612 18481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2381047f6dc4bdf962f9383e6c41548" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:50.468215 18481 raft_consensus.cc:3037] T 488d0e609b224674932f890939664df2 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:50.470202 18619 raft_consensus.cc:2454] T d2381047f6dc4bdf962f9383e6c41548 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:50.471706 18481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2381047f6dc4bdf962f9383e6c41548" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:50.472169 18481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f0b341a31224bf1be525b84b9500304" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:50.472373 18481 raft_consensus.cc:2379] T 9f0b341a31224bf1be525b84b9500304 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c51c33d5616482cafb68e26688820f4 in current term 1: Already voted for candidate 5e1c6397f09747f091534faf93542710 in this term.
I20240430 20:11:50.472445 18985 tablet_bootstrap.cc:492] T 90a694bd5db649afa50f3c4e752939c1 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:50.472568 18985 ts_tablet_manager.cc:1367] T 90a694bd5db649afa50f3c4e752939c1 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.019s user 0.001s sys 0.002s
I20240430 20:11:50.479223 21058 consensus_queue.cc:1035] T 50e1c55a1a0a484984ad5cbb78eec2ed P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:50.481106 18985 raft_consensus.cc:348] T 90a694bd5db649afa50f3c4e752939c1 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:50.487784 18985 raft_consensus.cc:374] T 90a694bd5db649afa50f3c4e752939c1 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:50.488085 18985 raft_consensus.cc:729] T 90a694bd5db649afa50f3c4e752939c1 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:50.488485 18985 consensus_queue.cc:260] T 90a694bd5db649afa50f3c4e752939c1 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:50.488863 20488 raft_consensus.cc:504] T 2873af6da928463f9809864b8eaf0c4a P 5e1c6397f09747f091534faf93542710 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:50.493140 18696 leader_election.cc:302] T 2fc6908ecf954020a676b9d10a2b00e7 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:50.494264 20488 leader_election.cc:288] T 2873af6da928463f9809864b8eaf0c4a P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 2 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:50.494643 20940 consensus_queue.cc:1035] T 47e98935eb21449abd3d4d24327778d5 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.013s
I20240430 20:11:50.494876 18985 ts_tablet_manager.cc:1398] T 90a694bd5db649afa50f3c4e752939c1 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.022s user 0.001s sys 0.001s
I20240430 20:11:50.495177 18985 tablet_bootstrap.cc:492] T f3711610610542128c4a84e17bce4a3e P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
W20240430 20:11:50.496258 18699 leader_election.cc:341] T 2fc6908ecf954020a676b9d10a2b00e7 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:50.496497 18983 tablet_bootstrap.cc:492] T 8e9711d0a382424f9f758bfa883866a0 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:50.496920 18696 leader_election.cc:302] T d2381047f6dc4bdf962f9383e6c41548 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
W20240430 20:11:50.497186 18699 leader_election.cc:341] T 2fc6908ecf954020a676b9d10a2b00e7 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:50.498092 20198 raft_consensus.cc:2781] T 2fc6908ecf954020a676b9d10a2b00e7 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:50.498250 20198 raft_consensus.cc:686] T 2fc6908ecf954020a676b9d10a2b00e7 P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Becoming Leader. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Running, Role: LEADER
I20240430 20:11:50.496641 18983 ts_tablet_manager.cc:1367] T 8e9711d0a382424f9f758bfa883866a0 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.075s user 0.019s sys 0.007s
W20240430 20:11:50.498488 19900 catalog_manager.cc:4810] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): delete failed for tablet d34a450a91d542c98261456d1f635e9c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d34a450a91d542c98261456d1f635e9c already in progress: creating tablet
W20240430 20:11:50.499457 18699 leader_election.cc:341] T d2381047f6dc4bdf962f9383e6c41548 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
W20240430 20:11:50.499955 18699 leader_election.cc:341] T d2381047f6dc4bdf962f9383e6c41548 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:50.500017 20938 consensus_queue.cc:1035] T 488d0e609b224674932f890939664df2 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:50.500348 18699 leader_election.cc:302] T 9f0b341a31224bf1be525b84b9500304 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710
I20240430 20:11:50.500671 18988 tablet_bootstrap.cc:654] T ed0c37a71ede4bfc9eb5ad1a4365f25e P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:50.500777 20938 raft_consensus.cc:3037] T 9f0b341a31224bf1be525b84b9500304 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:50.501055 21081 tablet_replica.cc:330] T 0eb4f3e7ece6496293e01b63620d644b P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:50.501273 21081 raft_consensus.cc:2229] T 0eb4f3e7ece6496293e01b63620d644b P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:50.501530 21081 raft_consensus.cc:2258] T 0eb4f3e7ece6496293e01b63620d644b P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:50.502192 21081 ts_tablet_manager.cc:1875] T 0eb4f3e7ece6496293e01b63620d644b P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:50.500643 20940 raft_consensus.cc:2781] T d2381047f6dc4bdf962f9383e6c41548 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:50.503238 20940 raft_consensus.cc:686] T d2381047f6dc4bdf962f9383e6c41548 P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Becoming Leader. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Running, Role: LEADER
I20240430 20:11:50.503659 20940 consensus_queue.cc:237] T d2381047f6dc4bdf962f9383e6c41548 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:50.506629 20198 consensus_queue.cc:237] T 2fc6908ecf954020a676b9d10a2b00e7 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:50.506750 20938 raft_consensus.cc:2726] T 9f0b341a31224bf1be525b84b9500304 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:50.511372 21049 consensus_queue.cc:1035] T 50e1c55a1a0a484984ad5cbb78eec2ed P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:50.513088 21081 ts_tablet_manager.cc:1888] T 0eb4f3e7ece6496293e01b63620d644b P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:50.513227 21081 log.cc:1196] T 0eb4f3e7ece6496293e01b63620d644b P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/0eb4f3e7ece6496293e01b63620d644b
I20240430 20:11:50.513679 21081 ts_tablet_manager.cc:1909] T 0eb4f3e7ece6496293e01b63620d644b P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:50.514709 19900 catalog_manager.cc:4830] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): tablet 0eb4f3e7ece6496293e01b63620d644b (table default.Table_b6ff64dbfc324fb199b8e6eebd51fd50 [id=4b13edffc2a4446dba14079d2b0f2ca2]) successfully deleted
W20240430 20:11:50.515024 18557 leader_election.cc:341] T 2873af6da928463f9809864b8eaf0c4a P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Not found: Tablet not found: 2873af6da928463f9809864b8eaf0c4a
I20240430 20:11:50.516589 18983 raft_consensus.cc:348] T 8e9711d0a382424f9f758bfa883866a0 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:50.516772 18983 raft_consensus.cc:374] T 8e9711d0a382424f9f758bfa883866a0 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:50.516817 18983 raft_consensus.cc:729] T 8e9711d0a382424f9f758bfa883866a0 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:50.517011 18983 consensus_queue.cc:260] T 8e9711d0a382424f9f758bfa883866a0 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:50.517501 18983 ts_tablet_manager.cc:1398] T 8e9711d0a382424f9f758bfa883866a0 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.019s user 0.000s sys 0.000s
I20240430 20:11:50.517871 21081 tablet_replica.cc:330] T 2fc6908ecf954020a676b9d10a2b00e7 P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:50.518110 21081 raft_consensus.cc:2229] T 2fc6908ecf954020a676b9d10a2b00e7 P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Raft consensus shutting down.
I20240430 20:11:50.518326 21081 pending_rounds.cc:62] T 2fc6908ecf954020a676b9d10a2b00e7 P 3c51c33d5616482cafb68e26688820f4: Trying to abort 1 pending ops.
I20240430 20:11:50.518404 21081 pending_rounds.cc:68] T 2fc6908ecf954020a676b9d10a2b00e7 P 3c51c33d5616482cafb68e26688820f4: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7022624401440350208 op_type: NO_OP noop_request { }
I20240430 20:11:50.518515 21081 raft_consensus.cc:2866] T 2fc6908ecf954020a676b9d10a2b00e7 P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240430 20:11:50.518646 21081 raft_consensus.cc:2258] T 2fc6908ecf954020a676b9d10a2b00e7 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:50.518864 21058 consensus_queue.cc:1035] T a445fdfde01c4371a5e6b3daea5b00ab P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:50.519157 18983 tablet_bootstrap.cc:492] T 898b2c91adf34545808df9c786e218c9 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:50.521029 18983 tablet_bootstrap.cc:654] T 898b2c91adf34545808df9c786e218c9 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:50.531929 18985 tablet_bootstrap.cc:654] T f3711610610542128c4a84e17bce4a3e P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:50.544585 21054 raft_consensus.cc:2726] T b285e3eb16d148779c74e057458d8461 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:50.545073 21054 raft_consensus.cc:482] T 488d0e609b224674932f890939664df2 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:50.545245 21054 raft_consensus.cc:504] T 488d0e609b224674932f890939664df2 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:50.546066 21054 leader_election.cc:288] T 488d0e609b224674932f890939664df2 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:50.546473 21054 raft_consensus.cc:482] T 0368d37dd9e2452982fb81f539c6805b P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:50.546564 18756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "488d0e609b224674932f890939664df2" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:50.546627 21054 raft_consensus.cc:504] T 0368d37dd9e2452982fb81f539c6805b P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:50.547159 21054 leader_election.cc:288] T 0368d37dd9e2452982fb81f539c6805b P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:50.547995 18756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0368d37dd9e2452982fb81f539c6805b" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:50.531522 21084 tablet_replica.cc:330] T 9c73d0a6bc414d3b9900570848d19a51 P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:50.534013 20488 raft_consensus.cc:482] T 558e71ec370d46ca8fbdcec5b57eb2ec P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:50.547065 18619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "488d0e609b224674932f890939664df2" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:50.548374 18619 raft_consensus.cc:2454] T 488d0e609b224674932f890939664df2 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 1.
I20240430 20:11:50.548296 20488 raft_consensus.cc:504] T 558e71ec370d46ca8fbdcec5b57eb2ec P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:50.548429 21084 raft_consensus.cc:2229] T 9c73d0a6bc414d3b9900570848d19a51 P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Raft consensus shutting down.
I20240430 20:11:50.547748 18616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0368d37dd9e2452982fb81f539c6805b" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:50.548662 18616 raft_consensus.cc:2379] T 0368d37dd9e2452982fb81f539c6805b P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2efb542aa55e439b99ccb98d1c5c884a in current term 1: Already voted for candidate 3c51c33d5616482cafb68e26688820f4 in this term.
I20240430 20:11:50.548704 21084 pending_rounds.cc:62] T 9c73d0a6bc414d3b9900570848d19a51 P 5e1c6397f09747f091534faf93542710: Trying to abort 1 pending ops.
I20240430 20:11:50.548749 21084 pending_rounds.cc:68] T 9c73d0a6bc414d3b9900570848d19a51 P 5e1c6397f09747f091534faf93542710: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7022624400706379776 op_type: NO_OP noop_request { }
I20240430 20:11:50.548790 18415 leader_election.cc:302] T 488d0e609b224674932f890939664df2 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:50.548871 21084 raft_consensus.cc:2866] T 9c73d0a6bc414d3b9900570848d19a51 P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240430 20:11:50.548902 20488 leader_election.cc:288] T 558e71ec370d46ca8fbdcec5b57eb2ec P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:50.549021 21054 raft_consensus.cc:2781] T 488d0e609b224674932f890939664df2 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240430 20:11:50.549093 21054 raft_consensus.cc:482] T 488d0e609b224674932f890939664df2 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:50.549156 21054 raft_consensus.cc:3037] T 488d0e609b224674932f890939664df2 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Advancing to term 2
I20240430 20:11:50.549557 18756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "558e71ec370d46ca8fbdcec5b57eb2ec" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:50.549773 18756 raft_consensus.cc:2454] T 558e71ec370d46ca8fbdcec5b57eb2ec P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 0.
I20240430 20:11:50.550081 18557 leader_election.cc:302] T 558e71ec370d46ca8fbdcec5b57eb2ec P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:50.550323 20488 raft_consensus.cc:2781] T 558e71ec370d46ca8fbdcec5b57eb2ec P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:50.550382 20488 raft_consensus.cc:482] T 558e71ec370d46ca8fbdcec5b57eb2ec P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:50.550436 20488 raft_consensus.cc:3037] T 558e71ec370d46ca8fbdcec5b57eb2ec P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:50.552179 21054 raft_consensus.cc:504] T 488d0e609b224674932f890939664df2 P 2efb542aa55e439b99ccb98d1c5c884a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:50.552767 21054 leader_election.cc:288] T 488d0e609b224674932f890939664df2 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:50.553210 18616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "488d0e609b224674932f890939664df2" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:50.553249 20488 raft_consensus.cc:504] T 558e71ec370d46ca8fbdcec5b57eb2ec P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:50.553400 18616 raft_consensus.cc:3037] T 488d0e609b224674932f890939664df2 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Advancing to term 2
I20240430 20:11:50.554311 18756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "558e71ec370d46ca8fbdcec5b57eb2ec" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:50.554574 18756 raft_consensus.cc:3037] T 558e71ec370d46ca8fbdcec5b57eb2ec P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:50.557106 18481 raft_consensus.cc:3037] T f90a5c5818c54390a2ae323fed7ebe57 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:50.557116 18756 raft_consensus.cc:2454] T 558e71ec370d46ca8fbdcec5b57eb2ec P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
I20240430 20:11:50.557809 18557 leader_election.cc:302] T 558e71ec370d46ca8fbdcec5b57eb2ec P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:50.558099 21011 raft_consensus.cc:2781] T 558e71ec370d46ca8fbdcec5b57eb2ec P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:50.558211 21011 raft_consensus.cc:686] T 558e71ec370d46ca8fbdcec5b57eb2ec P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Becoming Leader. State: Replica: 5e1c6397f09747f091534faf93542710, State: Running, Role: LEADER
I20240430 20:11:50.558455 21011 consensus_queue.cc:237] T 558e71ec370d46ca8fbdcec5b57eb2ec P 5e1c6397f09747f091534faf93542710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:50.563184 21081 ts_tablet_manager.cc:1875] T 2fc6908ecf954020a676b9d10a2b00e7 P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:50.563952 18481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2873af6da928463f9809864b8eaf0c4a" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:50.564163 18481 raft_consensus.cc:3037] T 2873af6da928463f9809864b8eaf0c4a P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Advancing to term 2
I20240430 20:11:50.564567 18983 tablet_bootstrap.cc:492] T 898b2c91adf34545808df9c786e218c9 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:50.564690 18983 ts_tablet_manager.cc:1367] T 898b2c91adf34545808df9c786e218c9 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.047s user 0.011s sys 0.005s
I20240430 20:11:50.565336 18616 raft_consensus.cc:2454] T 488d0e609b224674932f890939664df2 P 5e1c6397f09747f091534faf93542710 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 2.
I20240430 20:11:50.565817 18415 leader_election.cc:302] T 488d0e609b224674932f890939664df2 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:50.565953 18983 raft_consensus.cc:348] T 898b2c91adf34545808df9c786e218c9 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:50.566133 18983 raft_consensus.cc:374] T 898b2c91adf34545808df9c786e218c9 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:50.566185 18983 raft_consensus.cc:729] T 898b2c91adf34545808df9c786e218c9 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:50.566365 18983 consensus_queue.cc:260] T 898b2c91adf34545808df9c786e218c9 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:50.566913 18983 ts_tablet_manager.cc:1398] T 898b2c91adf34545808df9c786e218c9 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240430 20:11:50.566962 21054 raft_consensus.cc:2781] T 488d0e609b224674932f890939664df2 P 2efb542aa55e439b99ccb98d1c5c884a [term 2 FOLLOWER]: Leader election won for term 2
I20240430 20:11:50.567119 21058 consensus_queue.cc:1035] T a445fdfde01c4371a5e6b3daea5b00ab P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:50.567929 21054 raft_consensus.cc:686] T 488d0e609b224674932f890939664df2 P 2efb542aa55e439b99ccb98d1c5c884a [term 2 LEADER]: Becoming Leader. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Running, Role: LEADER
I20240430 20:11:50.568825 20695 raft_consensus.cc:482] T 1701a15b115741aea13f8327e3d26b34 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:50.569038 20695 raft_consensus.cc:504] T 1701a15b115741aea13f8327e3d26b34 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:50.569147 21054 consensus_queue.cc:237] T 488d0e609b224674932f890939664df2 P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:50.570268 18616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1701a15b115741aea13f8327e3d26b34" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:50.570448 18616 raft_consensus.cc:2454] T 1701a15b115741aea13f8327e3d26b34 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:50.571310 18481 raft_consensus.cc:2454] T 2873af6da928463f9809864b8eaf0c4a P 2efb542aa55e439b99ccb98d1c5c884a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 2.
I20240430 20:11:50.572911 18481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "558e71ec370d46ca8fbdcec5b57eb2ec" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:50.573096 18481 raft_consensus.cc:2454] T 558e71ec370d46ca8fbdcec5b57eb2ec P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 0.
I20240430 20:11:50.574481 18988 tablet_bootstrap.cc:492] T ed0c37a71ede4bfc9eb5ad1a4365f25e P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:50.574458 20489 consensus_queue.cc:1035] T f90a5c5818c54390a2ae323fed7ebe57 P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:50.574462 18696 leader_election.cc:302] T 1701a15b115741aea13f8327e3d26b34 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:50.574483 20695 leader_election.cc:288] T 1701a15b115741aea13f8327e3d26b34 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:50.574889 18988 ts_tablet_manager.cc:1367] T ed0c37a71ede4bfc9eb5ad1a4365f25e P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.157s user 0.007s sys 0.011s
I20240430 20:11:50.548942 21084 raft_consensus.cc:2258] T 9c73d0a6bc414d3b9900570848d19a51 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:50.575394 20198 raft_consensus.cc:2781] T 1701a15b115741aea13f8327e3d26b34 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:50.576906 20488 leader_election.cc:288] T 558e71ec370d46ca8fbdcec5b57eb2ec P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:50.575414 21081 ts_tablet_manager.cc:1888] T 2fc6908ecf954020a676b9d10a2b00e7 P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:50.577345 21081 log.cc:1196] T 2fc6908ecf954020a676b9d10a2b00e7 P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/2fc6908ecf954020a676b9d10a2b00e7
I20240430 20:11:50.577354 20198 raft_consensus.cc:482] T 1701a15b115741aea13f8327e3d26b34 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:50.577656 18558 leader_election.cc:302] T 2873af6da928463f9809864b8eaf0c4a P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters: 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:50.578174 20488 raft_consensus.cc:2781] T 2873af6da928463f9809864b8eaf0c4a P 5e1c6397f09747f091534faf93542710 [term 2 FOLLOWER]: Leader election won for term 2
I20240430 20:11:50.578193 20198 raft_consensus.cc:3037] T 1701a15b115741aea13f8327e3d26b34 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:50.578295 18988 raft_consensus.cc:348] T ed0c37a71ede4bfc9eb5ad1a4365f25e P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:50.578660 20488 raft_consensus.cc:686] T 2873af6da928463f9809864b8eaf0c4a P 5e1c6397f09747f091534faf93542710 [term 2 LEADER]: Becoming Leader. State: Replica: 5e1c6397f09747f091534faf93542710, State: Running, Role: LEADER
I20240430 20:11:50.579037 20488 consensus_queue.cc:237] T 2873af6da928463f9809864b8eaf0c4a P 5e1c6397f09747f091534faf93542710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:50.582684 21081 ts_tablet_manager.cc:1909] T 2fc6908ecf954020a676b9d10a2b00e7 P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:50.583492 19900 catalog_manager.cc:4830] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): tablet 2fc6908ecf954020a676b9d10a2b00e7 (table default.Table_766e3df58895427b9e5179eb555fe7e4 [id=5c11311047b34a3eb06cc06e8cc10223]) successfully deleted
I20240430 20:11:50.567206 18983 tablet_bootstrap.cc:492] T 828ca913e2aa4102a6277bdbc6686ef5 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:50.586676 18988 raft_consensus.cc:374] T ed0c37a71ede4bfc9eb5ad1a4365f25e P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:50.586843 18988 raft_consensus.cc:729] T ed0c37a71ede4bfc9eb5ad1a4365f25e P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:50.586709 20198 raft_consensus.cc:504] T 1701a15b115741aea13f8327e3d26b34 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:50.587941 20198 leader_election.cc:288] T 1701a15b115741aea13f8327e3d26b34 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Requested vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:50.587906 18616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1701a15b115741aea13f8327e3d26b34" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:50.588085 18616 raft_consensus.cc:3037] T 1701a15b115741aea13f8327e3d26b34 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:50.588457 18417 leader_election.cc:302] T 0368d37dd9e2452982fb81f539c6805b P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a; no voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710
I20240430 20:11:50.588922 21058 raft_consensus.cc:3037] T 0368d37dd9e2452982fb81f539c6805b P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:50.591077 18616 raft_consensus.cc:2454] T 1701a15b115741aea13f8327e3d26b34 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:50.591372 18988 consensus_queue.cc:260] T ed0c37a71ede4bfc9eb5ad1a4365f25e P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:50.591594 21058 raft_consensus.cc:2726] T 0368d37dd9e2452982fb81f539c6805b P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:50.592608 18696 leader_election.cc:302] T 1701a15b115741aea13f8327e3d26b34 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:50.594106 21063 raft_consensus.cc:482] T 1f4e860484784aa9a2d28af2e9fb611c P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:50.594234 21063 raft_consensus.cc:504] T 1f4e860484784aa9a2d28af2e9fb611c P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:50.594889 21063 leader_election.cc:288] T 1f4e860484784aa9a2d28af2e9fb611c P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:50.595551 18446 tablet_service.cc:1514] Processing DeleteTablet for tablet d34a450a91d542c98261456d1f635e9c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:49 UTC) from {username='slave'} at 127.0.0.1:41450
I20240430 20:11:50.595666 18457 tablet_service.cc:1514] Processing DeleteTablet for tablet 0eb4f3e7ece6496293e01b63620d644b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:49 UTC) from {username='slave'} at 127.0.0.1:41450
I20240430 20:11:50.595808 18619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f4e860484784aa9a2d28af2e9fb611c" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:50.596009 18619 raft_consensus.cc:2454] T 1f4e860484784aa9a2d28af2e9fb611c P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 0.
I20240430 20:11:50.596206 18457 tablet_service.cc:1514] Processing DeleteTablet for tablet 2fc6908ecf954020a676b9d10a2b00e7 with delete_type TABLET_DATA_DELETED (replaced by tablet 9a5654fd913f4fbeb9d126f035ac32bf) from {username='slave'} at 127.0.0.1:41450
I20240430 20:11:50.596259 18983 tablet_bootstrap.cc:654] T 828ca913e2aa4102a6277bdbc6686ef5 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:50.597028 18415 leader_election.cc:302] T 1f4e860484784aa9a2d28af2e9fb611c P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:50.597267 20630 raft_consensus.cc:2781] T 1f4e860484784aa9a2d28af2e9fb611c P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:50.597342 20630 raft_consensus.cc:482] T 1f4e860484784aa9a2d28af2e9fb611c P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:50.597417 20630 raft_consensus.cc:3037] T 1f4e860484784aa9a2d28af2e9fb611c P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:50.600191 20630 raft_consensus.cc:504] T 1f4e860484784aa9a2d28af2e9fb611c P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:50.601594 20587 raft_consensus.cc:2781] T 1701a15b115741aea13f8327e3d26b34 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:50.601910 20587 raft_consensus.cc:686] T 1701a15b115741aea13f8327e3d26b34 P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Becoming Leader. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Running, Role: LEADER
I20240430 20:11:50.602232 20587 consensus_queue.cc:237] T 1701a15b115741aea13f8327e3d26b34 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:50.603405 21084 ts_tablet_manager.cc:1875] T 9c73d0a6bc414d3b9900570848d19a51 P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:50.603562 18619 raft_consensus.cc:1226] T 488d0e609b224674932f890939664df2 P 5e1c6397f09747f091534faf93542710 [term 2 FOLLOWER]: Rejecting Update request from peer 3c51c33d5616482cafb68e26688820f4 for earlier term 1. Current term is 2. Ops: []
I20240430 20:11:50.604393 20198 consensus_queue.cc:1046] T 488d0e609b224674932f890939664df2 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:50.604658 20695 raft_consensus.cc:3032] T 488d0e609b224674932f890939664df2 P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Stepping down as leader of term 1
I20240430 20:11:50.604779 20695 raft_consensus.cc:729] T 488d0e609b224674932f890939664df2 P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Running, Role: LEADER
I20240430 20:11:50.605086 20695 consensus_queue.cc:260] T 488d0e609b224674932f890939664df2 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:50.605386 20695 raft_consensus.cc:3037] T 488d0e609b224674932f890939664df2 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Advancing to term 2
I20240430 20:11:50.607026 20630 leader_election.cc:288] T 1f4e860484784aa9a2d28af2e9fb611c P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:50.607666 18616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f4e860484784aa9a2d28af2e9fb611c" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:50.607844 18616 raft_consensus.cc:3037] T 1f4e860484784aa9a2d28af2e9fb611c P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:50.611603 18762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "488d0e609b224674932f890939664df2" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:50.611920 18762 raft_consensus.cc:2396] T 488d0e609b224674932f890939664df2 P 3c51c33d5616482cafb68e26688820f4 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 2efb542aa55e439b99ccb98d1c5c884a for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20240430 20:11:50.612645 18988 ts_tablet_manager.cc:1398] T ed0c37a71ede4bfc9eb5ad1a4365f25e P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.038s user 0.001s sys 0.001s
I20240430 20:11:50.613618 18988 tablet_bootstrap.cc:492] T df92eb85261d47d88cbf16d29d9ad30e P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:50.615487 18988 tablet_bootstrap.cc:654] T df92eb85261d47d88cbf16d29d9ad30e P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:50.615896 18756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f4e860484784aa9a2d28af2e9fb611c" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:50.616153 18756 raft_consensus.cc:2454] T 1f4e860484784aa9a2d28af2e9fb611c P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 0.
I20240430 20:11:50.616775 18616 raft_consensus.cc:2454] T 1f4e860484784aa9a2d28af2e9fb611c P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 1.
I20240430 20:11:50.617106 18756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f4e860484784aa9a2d28af2e9fb611c" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:50.617318 18756 raft_consensus.cc:3037] T 1f4e860484784aa9a2d28af2e9fb611c P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:50.617269 18415 leader_election.cc:302] T 1f4e860484784aa9a2d28af2e9fb611c P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:50.617611 21055 raft_consensus.cc:2781] T 1f4e860484784aa9a2d28af2e9fb611c P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:50.617748 21055 raft_consensus.cc:686] T 1f4e860484784aa9a2d28af2e9fb611c P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Becoming Leader. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Running, Role: LEADER
I20240430 20:11:50.618052 21055 consensus_queue.cc:237] T 1f4e860484784aa9a2d28af2e9fb611c P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:50.621114 18756 raft_consensus.cc:2454] T 1f4e860484784aa9a2d28af2e9fb611c P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 1.
I20240430 20:11:50.626042 18762 raft_consensus.cc:3037] T 3e5225f1f5f544ecbf8dfa470af3daca P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:50.626807 18475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1701a15b115741aea13f8327e3d26b34" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:50.626991 18475 raft_consensus.cc:3037] T 1701a15b115741aea13f8327e3d26b34 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:50.629033 18473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1701a15b115741aea13f8327e3d26b34" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:50.629318 20489 consensus_queue.cc:1035] T 3e5225f1f5f544ecbf8dfa470af3daca P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:50.629568 21013 raft_consensus.cc:482] T f7d3dea34cf54fe2b949d1b7ca68577a P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:50.629695 21013 raft_consensus.cc:504] T f7d3dea34cf54fe2b949d1b7ca68577a P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:50.630347 21013 leader_election.cc:288] T f7d3dea34cf54fe2b949d1b7ca68577a P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:50.631455 18756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7d3dea34cf54fe2b949d1b7ca68577a" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:50.631875 18756 raft_consensus.cc:2454] T f7d3dea34cf54fe2b949d1b7ca68577a P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 0.
I20240430 20:11:50.632212 18557 leader_election.cc:302] T f7d3dea34cf54fe2b949d1b7ca68577a P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:50.632444 21013 raft_consensus.cc:2781] T f7d3dea34cf54fe2b949d1b7ca68577a P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:50.632500 21013 raft_consensus.cc:482] T f7d3dea34cf54fe2b949d1b7ca68577a P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:50.632555 21013 raft_consensus.cc:3037] T f7d3dea34cf54fe2b949d1b7ca68577a P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:50.632500 18475 raft_consensus.cc:2454] T 1701a15b115741aea13f8327e3d26b34 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:50.635275 21013 raft_consensus.cc:504] T f7d3dea34cf54fe2b949d1b7ca68577a P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:50.636242 21013 leader_election.cc:288] T f7d3dea34cf54fe2b949d1b7ca68577a P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:50.636884 18758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7d3dea34cf54fe2b949d1b7ca68577a" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:50.637095 18758 raft_consensus.cc:3037] T f7d3dea34cf54fe2b949d1b7ca68577a P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:50.638103 18482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7d3dea34cf54fe2b949d1b7ca68577a" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:50.638648 18473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "558e71ec370d46ca8fbdcec5b57eb2ec" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:50.638828 18473 raft_consensus.cc:3037] T 558e71ec370d46ca8fbdcec5b57eb2ec P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:50.640115 18758 raft_consensus.cc:2454] T f7d3dea34cf54fe2b949d1b7ca68577a P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
I20240430 20:11:50.640715 18557 leader_election.cc:302] T f7d3dea34cf54fe2b949d1b7ca68577a P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:50.641126 21013 raft_consensus.cc:2781] T f7d3dea34cf54fe2b949d1b7ca68577a P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:50.641311 21013 raft_consensus.cc:686] T f7d3dea34cf54fe2b949d1b7ca68577a P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Becoming Leader. State: Replica: 5e1c6397f09747f091534faf93542710, State: Running, Role: LEADER
I20240430 20:11:50.641672 21013 consensus_queue.cc:237] T f7d3dea34cf54fe2b949d1b7ca68577a P 5e1c6397f09747f091534faf93542710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:50.642653 18473 raft_consensus.cc:2454] T 558e71ec370d46ca8fbdcec5b57eb2ec P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
I20240430 20:11:50.644117 18985 tablet_bootstrap.cc:492] T f3711610610542128c4a84e17bce4a3e P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:50.644338 18985 ts_tablet_manager.cc:1367] T f3711610610542128c4a84e17bce4a3e P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.149s user 0.003s sys 0.007s
I20240430 20:11:50.645291 18985 raft_consensus.cc:348] T f3711610610542128c4a84e17bce4a3e P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:50.645749 18985 raft_consensus.cc:374] T f3711610610542128c4a84e17bce4a3e P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:50.645864 18985 raft_consensus.cc:729] T f3711610610542128c4a84e17bce4a3e P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:50.646364 18985 consensus_queue.cc:260] T f3711610610542128c4a84e17bce4a3e P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:50.648087 18985 ts_tablet_manager.cc:1398] T f3711610610542128c4a84e17bce4a3e P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240430 20:11:50.651549 18481 raft_consensus.cc:1226] T 488d0e609b224674932f890939664df2 P 2efb542aa55e439b99ccb98d1c5c884a [term 2 LEADER]: Rejecting Update request from peer 3c51c33d5616482cafb68e26688820f4 for earlier term 1. Current term is 2. Ops: [1.1-1.1]
W20240430 20:11:50.652812 18558 leader_election.cc:341] T f7d3dea34cf54fe2b949d1b7ca68577a P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:50.653757 20384 consensus_queue.cc:1035] T 3e5225f1f5f544ecbf8dfa470af3daca P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:50.656288 18985 tablet_bootstrap.cc:492] T d2381047f6dc4bdf962f9383e6c41548 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:50.663372 18985 tablet_bootstrap.cc:654] T d2381047f6dc4bdf962f9383e6c41548 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:50.664503 18988 tablet_bootstrap.cc:492] T df92eb85261d47d88cbf16d29d9ad30e P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:50.664731 18988 ts_tablet_manager.cc:1367] T df92eb85261d47d88cbf16d29d9ad30e P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.051s user 0.000s sys 0.004s
I20240430 20:11:50.665652 18988 raft_consensus.cc:348] T df92eb85261d47d88cbf16d29d9ad30e P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:50.665931 18988 raft_consensus.cc:374] T df92eb85261d47d88cbf16d29d9ad30e P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:50.666054 18988 raft_consensus.cc:729] T df92eb85261d47d88cbf16d29d9ad30e P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:50.666332 18988 consensus_queue.cc:260] T df92eb85261d47d88cbf16d29d9ad30e P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:50.667016 18988 ts_tablet_manager.cc:1398] T df92eb85261d47d88cbf16d29d9ad30e P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240430 20:11:50.667330 18988 tablet_bootstrap.cc:492] T 898b2c91adf34545808df9c786e218c9 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:50.670931 18985 tablet_bootstrap.cc:492] T d2381047f6dc4bdf962f9383e6c41548 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:50.671092 18985 ts_tablet_manager.cc:1367] T d2381047f6dc4bdf962f9383e6c41548 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.001s
I20240430 20:11:50.676102 18985 raft_consensus.cc:348] T d2381047f6dc4bdf962f9383e6c41548 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:50.676345 18985 raft_consensus.cc:374] T d2381047f6dc4bdf962f9383e6c41548 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:50.676417 18985 raft_consensus.cc:729] T d2381047f6dc4bdf962f9383e6c41548 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:50.676805 21084 ts_tablet_manager.cc:1888] T 9c73d0a6bc414d3b9900570848d19a51 P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:50.677644 18985 consensus_queue.cc:260] T d2381047f6dc4bdf962f9383e6c41548 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:50.681329 18988 tablet_bootstrap.cc:654] T 898b2c91adf34545808df9c786e218c9 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:50.682296 21084 log.cc:1196] T 9c73d0a6bc414d3b9900570848d19a51 P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/9c73d0a6bc414d3b9900570848d19a51
I20240430 20:11:50.682614 20940 consensus_queue.cc:1035] T 0368d37dd9e2452982fb81f539c6805b P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:50.696611 21054 raft_consensus.cc:482] T f90a5c5818c54390a2ae323fed7ebe57 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:50.697047 21054 raft_consensus.cc:504] T f90a5c5818c54390a2ae323fed7ebe57 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:50.698055 21054 leader_election.cc:288] T f90a5c5818c54390a2ae323fed7ebe57 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:50.698683 18619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f90a5c5818c54390a2ae323fed7ebe57" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:50.698701 21084 ts_tablet_manager.cc:1909] T 9c73d0a6bc414d3b9900570848d19a51 P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:50.699714 21084 tablet_replica.cc:330] T 0eb4f3e7ece6496293e01b63620d644b P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:50.700050 21084 raft_consensus.cc:2229] T 0eb4f3e7ece6496293e01b63620d644b P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Raft consensus shutting down.
I20240430 20:11:50.700487 21084 raft_consensus.cc:2258] T 0eb4f3e7ece6496293e01b63620d644b P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:50.701308 21084 ts_tablet_manager.cc:1875] T 0eb4f3e7ece6496293e01b63620d644b P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:50.717216 18482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7d3dea34cf54fe2b949d1b7ca68577a" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
W20240430 20:11:50.719300 18558 leader_election.cc:341] T f7d3dea34cf54fe2b949d1b7ca68577a P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:50.721143 18762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f90a5c5818c54390a2ae323fed7ebe57" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:50.721827 18762 raft_consensus.cc:2454] T f90a5c5818c54390a2ae323fed7ebe57 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 1.
I20240430 20:11:50.723814 18988 tablet_bootstrap.cc:492] T 898b2c91adf34545808df9c786e218c9 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:50.727087 18417 leader_election.cc:302] T f90a5c5818c54390a2ae323fed7ebe57 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters: 5e1c6397f09747f091534faf93542710
I20240430 20:11:50.727380 21055 raft_consensus.cc:2781] T f90a5c5818c54390a2ae323fed7ebe57 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240430 20:11:50.727504 21055 raft_consensus.cc:482] T f90a5c5818c54390a2ae323fed7ebe57 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 5e1c6397f09747f091534faf93542710)
I20240430 20:11:50.727620 21055 raft_consensus.cc:3037] T f90a5c5818c54390a2ae323fed7ebe57 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Advancing to term 2
I20240430 20:11:50.728109 18988 ts_tablet_manager.cc:1367] T 898b2c91adf34545808df9c786e218c9 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.061s user 0.003s sys 0.003s
I20240430 20:11:50.728775 18988 raft_consensus.cc:348] T 898b2c91adf34545808df9c786e218c9 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:50.729880 18988 raft_consensus.cc:374] T 898b2c91adf34545808df9c786e218c9 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:50.730326 18988 raft_consensus.cc:729] T 898b2c91adf34545808df9c786e218c9 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:50.730726 18988 consensus_queue.cc:260] T 898b2c91adf34545808df9c786e218c9 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:50.731225 21055 raft_consensus.cc:504] T f90a5c5818c54390a2ae323fed7ebe57 P 2efb542aa55e439b99ccb98d1c5c884a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:50.732175 21055 leader_election.cc:288] T f90a5c5818c54390a2ae323fed7ebe57 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:50.732451 18619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f90a5c5818c54390a2ae323fed7ebe57" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:50.733620 21054 raft_consensus.cc:482] T c6378f1495144c458af49b3c19a2b67c P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:50.733789 21054 raft_consensus.cc:504] T c6378f1495144c458af49b3c19a2b67c P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:50.735383 21054 leader_election.cc:288] T c6378f1495144c458af49b3c19a2b67c P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:50.736152 18988 ts_tablet_manager.cc:1398] T 898b2c91adf34545808df9c786e218c9 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
W20240430 20:11:50.736321 18415 leader_election.cc:341] T c6378f1495144c458af49b3c19a2b67c P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Not found: Tablet not found: c6378f1495144c458af49b3c19a2b67c
I20240430 20:11:50.740079 21108 tablet_replica.cc:330] T 0eb4f3e7ece6496293e01b63620d644b P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
W20240430 20:11:50.740679 19902 catalog_manager.cc:4810] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): delete failed for tablet d34a450a91d542c98261456d1f635e9c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d34a450a91d542c98261456d1f635e9c already in progress: creating tablet
W20240430 20:11:50.742803 18557 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.177.3:44867, user_credentials={real_user=slave}} blocked reactor thread for 51012.3us
I20240430 20:11:50.743301 19895 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet 9c73d0a6bc414d3b9900570848d19a51 (table default.Table_49a012a7c4c8457fadb0a8f0fc9654c6 [id=eecb2f8eeef547c5ab1abe6a2430e592]) successfully deleted
W20240430 20:11:50.741258 21105 log.cc:924] Time spent T 488d0e609b224674932f890939664df2 P 2efb542aa55e439b99ccb98d1c5c884a: Append to log took a long time: real 0.091s user 0.000s sys 0.000s
I20240430 20:11:50.751137 21084 ts_tablet_manager.cc:1888] T 0eb4f3e7ece6496293e01b63620d644b P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:50.751832 21084 log.cc:1196] T 0eb4f3e7ece6496293e01b63620d644b P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/0eb4f3e7ece6496293e01b63620d644b
I20240430 20:11:50.753640 21084 ts_tablet_manager.cc:1909] T 0eb4f3e7ece6496293e01b63620d644b P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:50.755070 21084 tablet_replica.cc:330] T 2fc6908ecf954020a676b9d10a2b00e7 P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:50.755261 18988 tablet_bootstrap.cc:492] T 417351cbc9d54f9d84bb7155f39da119 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:50.756525 21084 raft_consensus.cc:2229] T 2fc6908ecf954020a676b9d10a2b00e7 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:50.756757 18988 tablet_bootstrap.cc:654] T 417351cbc9d54f9d84bb7155f39da119 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:50.757323 21084 raft_consensus.cc:2258] T 2fc6908ecf954020a676b9d10a2b00e7 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:50.757531 18985 ts_tablet_manager.cc:1398] T d2381047f6dc4bdf962f9383e6c41548 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.086s user 0.002s sys 0.000s
I20240430 20:11:50.763264 21122 raft_consensus.cc:482] T 00000000000000000000000000000000 P 89df84ccf5394c9bb60d4e4ed872fc2e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 88b3f98793354c7aaa5c106e8ec31ed8)
I20240430 20:11:50.763587 21122 raft_consensus.cc:504] T 00000000000000000000000000000000 P 89df84ccf5394c9bb60d4e4ed872fc2e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b3f98793354c7aaa5c106e8ec31ed8" member_type: VOTER last_known_addr { host: "127.17.177.62" port: 36441 } } peers { permanent_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e" member_type: VOTER last_known_addr { host: "127.17.177.61" port: 34777 } } peers { permanent_uuid: "b632d6ef4c104a058a6db208024cab33" member_type: VOTER last_known_addr { host: "127.17.177.60" port: 37225 } }
I20240430 20:11:50.764264 19940 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: Sending DeleteTablet for 3 replicas of tablet fd1b8407071e4e4287fc172009777414
I20240430 20:11:50.764987 21122 leader_election.cc:288] T 00000000000000000000000000000000 P 89df84ccf5394c9bb60d4e4ed872fc2e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 88b3f98793354c7aaa5c106e8ec31ed8 (127.17.177.62:36441), b632d6ef4c104a058a6db208024cab33 (127.17.177.60:37225)
I20240430 20:11:50.766993 18740 tablet_service.cc:1514] Processing DeleteTablet for tablet fd1b8407071e4e4287fc172009777414 with delete_type TABLET_DATA_DELETED (replaced by tablet 8abf0b43f8a447dc902135ce94e4fc78) from {username='slave'} at 127.0.0.1:56540
W20240430 20:11:50.769508 19900 catalog_manager.cc:4810] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): delete failed for tablet fd1b8407071e4e4287fc172009777414 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fd1b8407071e4e4287fc172009777414 already in progress: creating tablet
I20240430 20:11:50.769937 19895 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet 0eb4f3e7ece6496293e01b63620d644b (table default.Table_b6ff64dbfc324fb199b8e6eebd51fd50 [id=4b13edffc2a4446dba14079d2b0f2ca2]) successfully deleted
I20240430 20:11:50.771225 18446 tablet_service.cc:1514] Processing DeleteTablet for tablet fd1b8407071e4e4287fc172009777414 with delete_type TABLET_DATA_DELETED (replaced by tablet 8abf0b43f8a447dc902135ce94e4fc78) from {username='slave'} at 127.0.0.1:41450
W20240430 20:11:50.771881 19895 catalog_manager.cc:4810] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): delete failed for tablet d34a450a91d542c98261456d1f635e9c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d34a450a91d542c98261456d1f635e9c already in progress: creating tablet
I20240430 20:11:50.772711 19980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "89df84ccf5394c9bb60d4e4ed872fc2e" candidate_term: 7 candidate_status { last_received { term: 6 index: 771 } } ignore_live_leader: false dest_uuid: "88b3f98793354c7aaa5c106e8ec31ed8" is_pre_election: true
I20240430 20:11:50.773463 19940 catalog_manager.cc:6554] ReplaceTablet: tablet fd1b8407071e4e4287fc172009777414 deleted and replaced by tablet 8abf0b43f8a447dc902135ce94e4fc78
I20240430 20:11:50.783425 19938 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50746:
name: "default.Table_6ff76cc99d73469da3e9b8522abec263"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:50.784119 19938 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6ff76cc99d73469da3e9b8522abec263 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:50.784797 21049 raft_consensus.cc:482] T 3ec3993042f5498b818bdf32d5087928 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:50.784966 21049 raft_consensus.cc:504] T 3ec3993042f5498b818bdf32d5087928 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:50.785738 21049 leader_election.cc:288] T 3ec3993042f5498b818bdf32d5087928 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:50.785949 19937 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50746:
name: "default.Table_8bcd1481133a4f45aac58ee0460b7ef8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:50.786693 19937 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8bcd1481133a4f45aac58ee0460b7ef8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:50.788249 19945 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50746:
table { table_name: "default.Table_0c5d4686f2ed40fb8d62c31b0166dc05" } new_table_name: "default.Table_19df33614a6c4f94b6419a5b441e3ef8" modify_external_catalogs: true
I20240430 20:11:50.788857 19945 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50746:
table { table_name: "default.Table_2eed103638364cde94cd8af68d3cdf9b" } new_table_name: "default.Table_0dd55b391be94213b4af60ea6e5a979f" modify_external_catalogs: true
I20240430 20:11:50.791913 19940 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50746:
name: "default.Table_066abbbc2a9a4c8f8025b62ceb68ca8a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240430 20:11:50.770637 18603 tablet_service.cc:1514] Processing DeleteTablet for tablet fd1b8407071e4e4287fc172009777414 with delete_type TABLET_DATA_DELETED (replaced by tablet 8abf0b43f8a447dc902135ce94e4fc78) from {username='slave'} at 127.0.0.1:45934
W20240430 20:11:50.792471 19940 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_066abbbc2a9a4c8f8025b62ceb68ca8a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:50.793771 21084 ts_tablet_manager.cc:1875] T 2fc6908ecf954020a676b9d10a2b00e7 P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240430 20:11:50.793840 18415 leader_election.cc:341] T 3ec3993042f5498b818bdf32d5087928 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Not found: Tablet not found: 3ec3993042f5498b818bdf32d5087928
I20240430 20:11:50.794549 19941 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50746:
table { table_name: "default.Table_6d9a8c8bd13b42eca6834e707653a66b" } modify_external_catalogs: true
I20240430 20:11:50.794770 19941 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50746:
table { table_name: "default.Table_6d9a8c8bd13b42eca6834e707653a66b" } modify_external_catalogs: true
I20240430 20:11:50.795225 19946 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50746:
table { table_name: "default.Table_7199d7c36d3f45f79cf9e4e2b4145592" } modify_external_catalogs: true
I20240430 20:11:50.795415 19946 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50746:
table { table_name: "default.Table_7199d7c36d3f45f79cf9e4e2b4145592" } modify_external_catalogs: true
I20240430 20:11:50.803177 18983 tablet_bootstrap.cc:492] T 828ca913e2aa4102a6277bdbc6686ef5 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:50.803408 18983 ts_tablet_manager.cc:1367] T 828ca913e2aa4102a6277bdbc6686ef5 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.236s user 0.028s sys 0.051s
I20240430 20:11:50.804366 18983 raft_consensus.cc:348] T 828ca913e2aa4102a6277bdbc6686ef5 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:50.804607 18983 raft_consensus.cc:374] T 828ca913e2aa4102a6277bdbc6686ef5 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:50.804698 18983 raft_consensus.cc:729] T 828ca913e2aa4102a6277bdbc6686ef5 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:50.805003 18983 consensus_queue.cc:260] T 828ca913e2aa4102a6277bdbc6686ef5 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:50.805588 18983 ts_tablet_manager.cc:1398] T 828ca913e2aa4102a6277bdbc6686ef5 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240430 20:11:50.805836 18983 tablet_bootstrap.cc:492] T 417351cbc9d54f9d84bb7155f39da119 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:50.807624 18983 tablet_bootstrap.cc:654] T 417351cbc9d54f9d84bb7155f39da119 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:50.812690 21084 ts_tablet_manager.cc:1888] T 2fc6908ecf954020a676b9d10a2b00e7 P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:50.813441 19944 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: Sending DeleteTablet for 3 replicas of tablet 1462ecf1176c4e66be42abddbe36b00c
I20240430 20:11:50.814018 19944 catalog_manager.cc:6554] ReplaceTablet: tablet 1462ecf1176c4e66be42abddbe36b00c deleted and replaced by tablet dd1de890e4394db287cf38bb411f5586
I20240430 20:11:50.814416 19944 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50746:
table { table_name: "default.Table_3776a4b9d4f24aa6b7f882bb4d0116ef" } modify_external_catalogs: true
I20240430 20:11:50.814652 19944 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50746:
table { table_name: "default.Table_3776a4b9d4f24aa6b7f882bb4d0116ef" } modify_external_catalogs: true
I20240430 20:11:50.815950 18740 tablet_service.cc:1514] Processing DeleteTablet for tablet 1462ecf1176c4e66be42abddbe36b00c with delete_type TABLET_DATA_DELETED (replaced by tablet dd1de890e4394db287cf38bb411f5586) from {username='slave'} at 127.0.0.1:56540
I20240430 20:11:50.816082 18446 tablet_service.cc:1514] Processing DeleteTablet for tablet 1462ecf1176c4e66be42abddbe36b00c with delete_type TABLET_DATA_DELETED (replaced by tablet dd1de890e4394db287cf38bb411f5586) from {username='slave'} at 127.0.0.1:41450
I20240430 20:11:50.816264 21081 tablet_replica.cc:330] T 1462ecf1176c4e66be42abddbe36b00c P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:50.816483 21081 raft_consensus.cc:2229] T 1462ecf1176c4e66be42abddbe36b00c P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:50.816671 21081 raft_consensus.cc:2258] T 1462ecf1176c4e66be42abddbe36b00c P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:50.817250 21081 ts_tablet_manager.cc:1875] T 1462ecf1176c4e66be42abddbe36b00c P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:50.820201 21084 log.cc:1196] T 2fc6908ecf954020a676b9d10a2b00e7 P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/2fc6908ecf954020a676b9d10a2b00e7
I20240430 20:11:50.820716 21084 ts_tablet_manager.cc:1909] T 2fc6908ecf954020a676b9d10a2b00e7 P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
W20240430 20:11:50.821219 18211 leader_election.cc:334] T 00000000000000000000000000000000 P 89df84ccf5394c9bb60d4e4ed872fc2e [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer b632d6ef4c104a058a6db208024cab33 (127.17.177.60:37225): Network error: Client connection negotiation failed: client connection to 127.17.177.60:37225: connect: Connection refused (error 111)
I20240430 20:11:50.815255 18983 tablet_bootstrap.cc:492] T 417351cbc9d54f9d84bb7155f39da119 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:50.821611 18983 ts_tablet_manager.cc:1367] T 417351cbc9d54f9d84bb7155f39da119 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.000s
I20240430 20:11:50.821434 18211 leader_election.cc:302] T 00000000000000000000000000000000 P 89df84ccf5394c9bb60d4e4ed872fc2e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89df84ccf5394c9bb60d4e4ed872fc2e; no voters: 88b3f98793354c7aaa5c106e8ec31ed8, b632d6ef4c104a058a6db208024cab33
I20240430 20:11:50.822005 21122 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 89df84ccf5394c9bb60d4e4ed872fc2e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240430 20:11:50.822227 20489 raft_consensus.cc:482] T 828ca913e2aa4102a6277bdbc6686ef5 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:50.822353 20489 raft_consensus.cc:504] T 828ca913e2aa4102a6277bdbc6686ef5 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:50.822386 18983 raft_consensus.cc:348] T 417351cbc9d54f9d84bb7155f39da119 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:50.822568 18983 raft_consensus.cc:374] T 417351cbc9d54f9d84bb7155f39da119 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:50.822643 18983 raft_consensus.cc:729] T 417351cbc9d54f9d84bb7155f39da119 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:50.822839 18983 consensus_queue.cc:260] T 417351cbc9d54f9d84bb7155f39da119 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:50.823047 20489 leader_election.cc:288] T 828ca913e2aa4102a6277bdbc6686ef5 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:50.823339 18983 ts_tablet_manager.cc:1398] T 417351cbc9d54f9d84bb7155f39da119 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240430 20:11:50.823601 18983 tablet_bootstrap.cc:492] T 62732af95c6a4c749c5b726f0460dc72 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:50.823803 18474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "828ca913e2aa4102a6277bdbc6686ef5" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
W20240430 20:11:50.824410 18558 leader_election.cc:341] T 828ca913e2aa4102a6277bdbc6686ef5 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:50.825165 21081 ts_tablet_manager.cc:1888] T 1462ecf1176c4e66be42abddbe36b00c P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240430 20:11:50.825337 21081 log.cc:1196] T 1462ecf1176c4e66be42abddbe36b00c P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/1462ecf1176c4e66be42abddbe36b00c
I20240430 20:11:50.825824 21081 ts_tablet_manager.cc:1909] T 1462ecf1176c4e66be42abddbe36b00c P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:50.827371 19900 catalog_manager.cc:4830] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): tablet 1462ecf1176c4e66be42abddbe36b00c (table default.Table_b49a7942aa8c4f58acb94c35d1cdbb70 [id=6257559b0e9545c58848a76af0c5a5f7]) successfully deleted
I20240430 20:11:50.830222 19895 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet 2fc6908ecf954020a676b9d10a2b00e7 (table default.Table_766e3df58895427b9e5179eb555fe7e4 [id=5c11311047b34a3eb06cc06e8cc10223]) successfully deleted
W20240430 20:11:50.830847 19895 catalog_manager.cc:4810] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): delete failed for tablet fd1b8407071e4e4287fc172009777414 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fd1b8407071e4e4287fc172009777414 already in progress: creating tablet
I20240430 20:11:50.831456 18603 tablet_service.cc:1514] Processing DeleteTablet for tablet 1462ecf1176c4e66be42abddbe36b00c with delete_type TABLET_DATA_DELETED (replaced by tablet dd1de890e4394db287cf38bb411f5586) from {username='slave'} at 127.0.0.1:45934
I20240430 20:11:50.831513 18985 tablet_bootstrap.cc:492] T f7d3dea34cf54fe2b949d1b7ca68577a P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:50.831797 21084 tablet_replica.cc:330] T 1462ecf1176c4e66be42abddbe36b00c P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:50.832039 21084 raft_consensus.cc:2229] T 1462ecf1176c4e66be42abddbe36b00c P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:50.832265 21084 raft_consensus.cc:2258] T 1462ecf1176c4e66be42abddbe36b00c P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:50.832484 18474 raft_consensus.cc:3037] T f3711610610542128c4a84e17bce4a3e P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:50.832895 21084 ts_tablet_manager.cc:1875] T 1462ecf1176c4e66be42abddbe36b00c P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:50.833138 18985 tablet_bootstrap.cc:654] T f7d3dea34cf54fe2b949d1b7ca68577a P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:50.835007 18983 tablet_bootstrap.cc:654] T 62732af95c6a4c749c5b726f0460dc72 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:50.841876 19937 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50746:
table { table_name: "default.Table_b7c7025fbf2e4039aadd143d021bfef5" } new_table_name: "default.Table_f1dc481c6d8c43f589fe2b1892a4cb10" modify_external_catalogs: true
I20240430 20:11:50.841961 21084 ts_tablet_manager.cc:1888] T 1462ecf1176c4e66be42abddbe36b00c P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240430 20:11:50.842074 21084 log.cc:1196] T 1462ecf1176c4e66be42abddbe36b00c P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/1462ecf1176c4e66be42abddbe36b00c
I20240430 20:11:50.842449 21084 ts_tablet_manager.cc:1909] T 1462ecf1176c4e66be42abddbe36b00c P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:50.844066 19940 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50746:
table { table_name: "default.Table_57208e9f76f14782a806f0a2971a3a58" } new_table_name: "default.Table_f2f38f5d69e34eaa9e3dbf24cfd1cdd6" modify_external_catalogs: true
I20240430 20:11:50.844882 19938 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50746:
table { table_name: "default.Table_b49a7942aa8c4f58acb94c35d1cdbb70" } new_table_name: "default.Table_84084a27f7504dd992fce07448c10df3" modify_external_catalogs: true
I20240430 20:11:50.846796 21108 raft_consensus.cc:2229] T 0eb4f3e7ece6496293e01b63620d644b P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:50.849267 21108 raft_consensus.cc:2258] T 0eb4f3e7ece6496293e01b63620d644b P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:50.850056 21108 ts_tablet_manager.cc:1875] T 0eb4f3e7ece6496293e01b63620d644b P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:50.851384 21011 consensus_queue.cc:1035] T f3711610610542128c4a84e17bce4a3e P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240430 20:11:50.854615 18415 consensus_peers.cc:485] T 1462ecf1176c4e66be42abddbe36b00c P 2efb542aa55e439b99ccb98d1c5c884a -> Peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Couldn't send request to peer 5e1c6397f09747f091534faf93542710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1462ecf1176c4e66be42abddbe36b00c. This is attempt 1: this message will repeat every 5th retry.
I20240430 20:11:50.862831 19895 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet 1462ecf1176c4e66be42abddbe36b00c (table default.Table_b49a7942aa8c4f58acb94c35d1cdbb70 [id=6257559b0e9545c58848a76af0c5a5f7]) successfully deleted
I20240430 20:11:50.865494 20198 consensus_queue.cc:1035] T 12795670f21447a8964fe1efc6e92398 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:50.868417 21055 raft_consensus.cc:482] T 90a694bd5db649afa50f3c4e752939c1 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:50.868680 21055 raft_consensus.cc:504] T 90a694bd5db649afa50f3c4e752939c1 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:50.869846 21055 leader_election.cc:288] T 90a694bd5db649afa50f3c4e752939c1 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:50.869976 18619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90a694bd5db649afa50f3c4e752939c1" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:50.874815 18475 raft_consensus.cc:3037] T 90a694bd5db649afa50f3c4e752939c1 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:50.879199 18988 tablet_bootstrap.cc:492] T 417351cbc9d54f9d84bb7155f39da119 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:50.879352 18988 ts_tablet_manager.cc:1367] T 417351cbc9d54f9d84bb7155f39da119 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.131s user 0.006s sys 0.018s
I20240430 20:11:50.880123 18988 raft_consensus.cc:348] T 417351cbc9d54f9d84bb7155f39da119 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:50.880318 18988 raft_consensus.cc:374] T 417351cbc9d54f9d84bb7155f39da119 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:50.880375 18988 raft_consensus.cc:729] T 417351cbc9d54f9d84bb7155f39da119 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:50.880560 18988 consensus_queue.cc:260] T 417351cbc9d54f9d84bb7155f39da119 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:50.881942 18988 ts_tablet_manager.cc:1398] T 417351cbc9d54f9d84bb7155f39da119 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240430 20:11:50.883284 18988 tablet_bootstrap.cc:492] T 828ca913e2aa4102a6277bdbc6686ef5 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:50.884668 18988 tablet_bootstrap.cc:654] T 828ca913e2aa4102a6277bdbc6686ef5 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:50.885737 18985 tablet_bootstrap.cc:492] T f7d3dea34cf54fe2b949d1b7ca68577a P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:50.887989 18985 ts_tablet_manager.cc:1367] T f7d3dea34cf54fe2b949d1b7ca68577a P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.130s user 0.003s sys 0.002s
I20240430 20:11:50.888886 18985 raft_consensus.cc:348] T f7d3dea34cf54fe2b949d1b7ca68577a P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:50.889107 18985 raft_consensus.cc:374] T f7d3dea34cf54fe2b949d1b7ca68577a P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:50.889194 18985 raft_consensus.cc:729] T f7d3dea34cf54fe2b949d1b7ca68577a P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:50.889430 18985 consensus_queue.cc:260] T f7d3dea34cf54fe2b949d1b7ca68577a P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:50.889948 18985 ts_tablet_manager.cc:1398] T f7d3dea34cf54fe2b949d1b7ca68577a P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240430 20:11:50.890234 18985 tablet_bootstrap.cc:492] T c1017a50d1d646c486c0b9a75cb88ce4 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:50.900573 18988 tablet_bootstrap.cc:492] T 828ca913e2aa4102a6277bdbc6686ef5 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:50.900839 18988 ts_tablet_manager.cc:1367] T 828ca913e2aa4102a6277bdbc6686ef5 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.018s user 0.001s sys 0.001s
I20240430 20:11:50.902734 18988 raft_consensus.cc:348] T 828ca913e2aa4102a6277bdbc6686ef5 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:50.903016 18988 raft_consensus.cc:374] T 828ca913e2aa4102a6277bdbc6686ef5 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:50.903077 18988 raft_consensus.cc:729] T 828ca913e2aa4102a6277bdbc6686ef5 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:50.903316 18988 consensus_queue.cc:260] T 828ca913e2aa4102a6277bdbc6686ef5 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:50.903919 18988 ts_tablet_manager.cc:1398] T 828ca913e2aa4102a6277bdbc6686ef5 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20240430 20:11:50.904208 18988 tablet_bootstrap.cc:492] T 8e9711d0a382424f9f758bfa883866a0 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:50.906335 18988 tablet_bootstrap.cc:654] T 8e9711d0a382424f9f758bfa883866a0 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:50.924430 18988 tablet_bootstrap.cc:492] T 8e9711d0a382424f9f758bfa883866a0 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:50.924634 18988 ts_tablet_manager.cc:1367] T 8e9711d0a382424f9f758bfa883866a0 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.001s
I20240430 20:11:50.925523 18988 raft_consensus.cc:348] T 8e9711d0a382424f9f758bfa883866a0 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:50.925791 18988 raft_consensus.cc:374] T 8e9711d0a382424f9f758bfa883866a0 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:50.925856 18988 raft_consensus.cc:729] T 8e9711d0a382424f9f758bfa883866a0 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:50.926119 18988 consensus_queue.cc:260] T 8e9711d0a382424f9f758bfa883866a0 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:50.927927 18988 ts_tablet_manager.cc:1398] T 8e9711d0a382424f9f758bfa883866a0 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240430 20:11:50.929128 18988 tablet_bootstrap.cc:492] T 6b51cbc731794083a88fe4bed1ddeebf P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:50.930842 18988 tablet_bootstrap.cc:654] T 6b51cbc731794083a88fe4bed1ddeebf P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:50.936978 20938 raft_consensus.cc:482] T 898b2c91adf34545808df9c786e218c9 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:50.937131 20938 raft_consensus.cc:504] T 898b2c91adf34545808df9c786e218c9 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:50.937701 20938 leader_election.cc:288] T 898b2c91adf34545808df9c786e218c9 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:50.938174 18616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "898b2c91adf34545808df9c786e218c9" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:50.938366 18616 raft_consensus.cc:2454] T 898b2c91adf34545808df9c786e218c9 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:50.938827 18696 leader_election.cc:302] T 898b2c91adf34545808df9c786e218c9 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:50.939100 20587 raft_consensus.cc:2781] T 898b2c91adf34545808df9c786e218c9 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:50.939180 20587 raft_consensus.cc:482] T 898b2c91adf34545808df9c786e218c9 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:50.939237 20587 raft_consensus.cc:3037] T 898b2c91adf34545808df9c786e218c9 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:50.940621 21108 ts_tablet_manager.cc:1888] T 0eb4f3e7ece6496293e01b63620d644b P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:50.940821 21108 log.cc:1196] T 0eb4f3e7ece6496293e01b63620d644b P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/0eb4f3e7ece6496293e01b63620d644b
I20240430 20:11:50.941298 21108 ts_tablet_manager.cc:1909] T 0eb4f3e7ece6496293e01b63620d644b P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:50.941831 20587 raft_consensus.cc:504] T 898b2c91adf34545808df9c786e218c9 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:50.944948 18603 tablet_service.cc:1467] Processing CreateTablet for tablet a9f74fd7ef87468d83cf6cf7fa9106e8 (DEFAULT_TABLE table=default.Table_cd541af5796d4b68abf97f9b05a1bb7b [id=0553d780657b416ebc60685be30fe8a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:50.945200 18756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "828ca913e2aa4102a6277bdbc6686ef5" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:50.945367 18603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9f74fd7ef87468d83cf6cf7fa9106e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:50.945376 18756 raft_consensus.cc:2454] T 828ca913e2aa4102a6277bdbc6686ef5 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 0.
I20240430 20:11:50.946244 20587 leader_election.cc:288] T 898b2c91adf34545808df9c786e218c9 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Requested vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:50.946882 18619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "898b2c91adf34545808df9c786e218c9" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:50.947067 18619 raft_consensus.cc:3037] T 898b2c91adf34545808df9c786e218c9 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:50.950009 18619 raft_consensus.cc:2454] T 898b2c91adf34545808df9c786e218c9 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:50.947304 18740 tablet_service.cc:1467] Processing CreateTablet for tablet a9f74fd7ef87468d83cf6cf7fa9106e8 (DEFAULT_TABLE table=default.Table_cd541af5796d4b68abf97f9b05a1bb7b [id=0553d780657b416ebc60685be30fe8a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:50.951189 18740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9f74fd7ef87468d83cf6cf7fa9106e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:50.953114 19902 catalog_manager.cc:4830] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): tablet 0eb4f3e7ece6496293e01b63620d644b (table default.Table_b6ff64dbfc324fb199b8e6eebd51fd50 [id=4b13edffc2a4446dba14079d2b0f2ca2]) successfully deleted
W20240430 20:11:50.953805 19902 catalog_manager.cc:4810] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): delete failed for tablet 2fc6908ecf954020a676b9d10a2b00e7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2fc6908ecf954020a676b9d10a2b00e7 already in progress: creating tablet
W20240430 20:11:50.954344 19902 catalog_manager.cc:4810] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): delete failed for tablet fd1b8407071e4e4287fc172009777414 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fd1b8407071e4e4287fc172009777414 already in progress: creating tablet
I20240430 20:11:50.942996 18985 tablet_bootstrap.cc:654] T c1017a50d1d646c486c0b9a75cb88ce4 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:50.950546 18446 tablet_service.cc:1467] Processing CreateTablet for tablet a9f74fd7ef87468d83cf6cf7fa9106e8 (DEFAULT_TABLE table=default.Table_cd541af5796d4b68abf97f9b05a1bb7b [id=0553d780657b416ebc60685be30fe8a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:50.954871 18446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9f74fd7ef87468d83cf6cf7fa9106e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:50.958832 21055 raft_consensus.cc:482] T d2381047f6dc4bdf962f9383e6c41548 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:50.950453 18696 leader_election.cc:302] T 898b2c91adf34545808df9c786e218c9 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:50.959009 21055 raft_consensus.cc:504] T d2381047f6dc4bdf962f9383e6c41548 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:50.959205 20938 raft_consensus.cc:2781] T 898b2c91adf34545808df9c786e218c9 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:50.959321 20938 raft_consensus.cc:686] T 898b2c91adf34545808df9c786e218c9 P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Becoming Leader. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Running, Role: LEADER
I20240430 20:11:50.959555 20938 consensus_queue.cc:237] T 898b2c91adf34545808df9c786e218c9 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:50.959708 21055 leader_election.cc:288] T d2381047f6dc4bdf962f9383e6c41548 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:50.960875 18619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2381047f6dc4bdf962f9383e6c41548" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:50.961208 18619 raft_consensus.cc:2379] T d2381047f6dc4bdf962f9383e6c41548 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2efb542aa55e439b99ccb98d1c5c884a in current term 1: Already voted for candidate 3c51c33d5616482cafb68e26688820f4 in this term.
I20240430 20:11:50.961469 21108 tablet_replica.cc:330] T 1462ecf1176c4e66be42abddbe36b00c P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:50.962934 18756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f90a5c5818c54390a2ae323fed7ebe57" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:50.963119 18756 raft_consensus.cc:3037] T f90a5c5818c54390a2ae323fed7ebe57 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Advancing to term 2
I20240430 20:11:50.966130 18756 raft_consensus.cc:2454] T f90a5c5818c54390a2ae323fed7ebe57 P 3c51c33d5616482cafb68e26688820f4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 2.
I20240430 20:11:50.966724 21108 raft_consensus.cc:2229] T 1462ecf1176c4e66be42abddbe36b00c P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Raft consensus shutting down.
I20240430 20:11:50.967854 18590 tablet_service.cc:1467] Processing CreateTablet for tablet d78dd72c3a254fa5a98382a2d24b41f8 (DEFAULT_TABLE table=default.Table_df40d2a666d94cada72ebdb96b7d7f9e [id=f33b27d82b54476eb1eab57b39b91108]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:50.968322 18590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d78dd72c3a254fa5a98382a2d24b41f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:50.966223 18985 tablet_bootstrap.cc:492] T c1017a50d1d646c486c0b9a75cb88ce4 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:50.968864 18985 ts_tablet_manager.cc:1367] T c1017a50d1d646c486c0b9a75cb88ce4 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.079s user 0.004s sys 0.000s
I20240430 20:11:50.970185 18457 tablet_service.cc:1467] Processing CreateTablet for tablet d78dd72c3a254fa5a98382a2d24b41f8 (DEFAULT_TABLE table=default.Table_df40d2a666d94cada72ebdb96b7d7f9e [id=f33b27d82b54476eb1eab57b39b91108]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:50.971496 18985 raft_consensus.cc:348] T c1017a50d1d646c486c0b9a75cb88ce4 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:50.976074 18740 tablet_service.cc:1467] Processing CreateTablet for tablet d78dd72c3a254fa5a98382a2d24b41f8 (DEFAULT_TABLE table=default.Table_df40d2a666d94cada72ebdb96b7d7f9e [id=f33b27d82b54476eb1eab57b39b91108]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:50.977135 18740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d78dd72c3a254fa5a98382a2d24b41f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:50.977823 18985 raft_consensus.cc:374] T c1017a50d1d646c486c0b9a75cb88ce4 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:50.977933 18985 raft_consensus.cc:729] T c1017a50d1d646c486c0b9a75cb88ce4 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:50.978199 18985 consensus_queue.cc:260] T c1017a50d1d646c486c0b9a75cb88ce4 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:50.978873 18985 ts_tablet_manager.cc:1398] T c1017a50d1d646c486c0b9a75cb88ce4 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240430 20:11:50.978857 18756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90a694bd5db649afa50f3c4e752939c1" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:50.978896 18590 tablet_service.cc:1467] Processing CreateTablet for tablet ddae8bc160b34f5590e5b0f09d36749d (DEFAULT_TABLE table=default.Table_8236b26459a348f99005b9ef3033a30f [id=ec659350534140e89c40849e2a7dac27]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:50.972106 21108 raft_consensus.cc:2258] T 1462ecf1176c4e66be42abddbe36b00c P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:50.979332 18590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddae8bc160b34f5590e5b0f09d36749d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:50.976887 18457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d78dd72c3a254fa5a98382a2d24b41f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:50.980413 18732 tablet_service.cc:1467] Processing CreateTablet for tablet ddae8bc160b34f5590e5b0f09d36749d (DEFAULT_TABLE table=default.Table_8236b26459a348f99005b9ef3033a30f [id=ec659350534140e89c40849e2a7dac27]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:50.980820 18732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddae8bc160b34f5590e5b0f09d36749d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:50.986351 21108 ts_tablet_manager.cc:1875] T 1462ecf1176c4e66be42abddbe36b00c P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:50.994443 18983 tablet_bootstrap.cc:492] T 62732af95c6a4c749c5b726f0460dc72 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:50.994642 18983 ts_tablet_manager.cc:1367] T 62732af95c6a4c749c5b726f0460dc72 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.171s user 0.020s sys 0.017s
I20240430 20:11:50.995625 18756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2381047f6dc4bdf962f9383e6c41548" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:50.995652 18446 tablet_service.cc:1467] Processing CreateTablet for tablet ddae8bc160b34f5590e5b0f09d36749d (DEFAULT_TABLE table=default.Table_8236b26459a348f99005b9ef3033a30f [id=ec659350534140e89c40849e2a7dac27]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:50.995733 18603 tablet_service.cc:1467] Processing CreateTablet for tablet 43ec03fd93da43569bb9d3be01de43bc (DEFAULT_TABLE table=default.Table_cfb6af12a3114f96837868eaa33be09b [id=cdf3c4662ece4b0ba1c88057affd26e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:50.996129 18603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43ec03fd93da43569bb9d3be01de43bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:50.996820 18988 tablet_bootstrap.cc:492] T 6b51cbc731794083a88fe4bed1ddeebf P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:50.996955 18988 ts_tablet_manager.cc:1367] T 6b51cbc731794083a88fe4bed1ddeebf P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.069s user 0.003s sys 0.000s
I20240430 20:11:50.997665 18988 raft_consensus.cc:348] T 6b51cbc731794083a88fe4bed1ddeebf P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:50.997841 18985 tablet_bootstrap.cc:492] T 2fc6908ecf954020a676b9d10a2b00e7 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:50.997841 18988 raft_consensus.cc:374] T 6b51cbc731794083a88fe4bed1ddeebf P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:50.997892 18988 raft_consensus.cc:729] T 6b51cbc731794083a88fe4bed1ddeebf P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:50.998072 18988 consensus_queue.cc:260] T 6b51cbc731794083a88fe4bed1ddeebf P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:50.999500 18985 tablet_bootstrap.cc:654] T 2fc6908ecf954020a676b9d10a2b00e7 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:51.002657 18988 ts_tablet_manager.cc:1398] T 6b51cbc731794083a88fe4bed1ddeebf P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240430 20:11:51.002831 20489 consensus_queue.cc:1035] T f3711610610542128c4a84e17bce4a3e P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:51.002997 18988 tablet_bootstrap.cc:492] T 62732af95c6a4c749c5b726f0460dc72 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:51.003432 18557 leader_election.cc:302] T 828ca913e2aa4102a6277bdbc6686ef5 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a
I20240430 20:11:51.003672 20489 raft_consensus.cc:2781] T 828ca913e2aa4102a6277bdbc6686ef5 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:51.003741 20489 raft_consensus.cc:482] T 828ca913e2aa4102a6277bdbc6686ef5 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:51.003798 20489 raft_consensus.cc:3037] T 828ca913e2aa4102a6277bdbc6686ef5 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:51.006637 20489 raft_consensus.cc:504] T 828ca913e2aa4102a6277bdbc6686ef5 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:51.007234 20489 leader_election.cc:288] T 828ca913e2aa4102a6277bdbc6686ef5 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:51.007737 20489 consensus_queue.cc:1035] T f90a5c5818c54390a2ae323fed7ebe57 P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:51.009344 18475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "898b2c91adf34545808df9c786e218c9" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:51.009800 18481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "898b2c91adf34545808df9c786e218c9" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:51.010741 18475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "828ca913e2aa4102a6277bdbc6686ef5" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
W20240430 20:11:51.011260 18558 leader_election.cc:341] T 828ca913e2aa4102a6277bdbc6686ef5 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:51.013358 18446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddae8bc160b34f5590e5b0f09d36749d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:51.013751 18732 tablet_service.cc:1467] Processing CreateTablet for tablet 43ec03fd93da43569bb9d3be01de43bc (DEFAULT_TABLE table=default.Table_cfb6af12a3114f96837868eaa33be09b [id=cdf3c4662ece4b0ba1c88057affd26e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:51.014165 18732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43ec03fd93da43569bb9d3be01de43bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:51.018002 18988 tablet_bootstrap.cc:654] T 62732af95c6a4c749c5b726f0460dc72 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:51.023293 18983 raft_consensus.cc:348] T 62732af95c6a4c749c5b726f0460dc72 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:51.023550 18983 raft_consensus.cc:374] T 62732af95c6a4c749c5b726f0460dc72 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:51.023610 18983 raft_consensus.cc:729] T 62732af95c6a4c749c5b726f0460dc72 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:51.023883 18983 consensus_queue.cc:260] T 62732af95c6a4c749c5b726f0460dc72 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:51.028605 18983 ts_tablet_manager.cc:1398] T 62732af95c6a4c749c5b726f0460dc72 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.034s user 0.002s sys 0.000s
I20240430 20:11:51.032445 18740 tablet_service.cc:1467] Processing CreateTablet for tablet dac141d1b70c48d785caddd7182811ee (DEFAULT_TABLE table=default.Table_fc0a4540f98b4eea95099b8ab46ea235 [id=f3c53ccffe64410495cc8c379a05cfae]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:51.032995 18740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dac141d1b70c48d785caddd7182811ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:51.033432 18739 tablet_service.cc:1467] Processing CreateTablet for tablet 13f0a0a87da242c8acb9892bc2897616 (DEFAULT_TABLE table=default.Table_2e4e483bcd6c4dd6bdfcc938cb0e2da6 [id=afdab68d2df0433f9102de19b55c60b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:51.033778 18739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13f0a0a87da242c8acb9892bc2897616. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:51.039404 18461 tablet_service.cc:1467] Processing CreateTablet for tablet dac141d1b70c48d785caddd7182811ee (DEFAULT_TABLE table=default.Table_fc0a4540f98b4eea95099b8ab46ea235 [id=f3c53ccffe64410495cc8c379a05cfae]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:51.039922 20795 rpc_server.cc:307] RPC server started. Bound to: 127.17.177.60:37225
I20240430 20:11:51.040033 18457 tablet_service.cc:1467] Processing CreateTablet for tablet 43ec03fd93da43569bb9d3be01de43bc (DEFAULT_TABLE table=default.Table_cfb6af12a3114f96837868eaa33be09b [id=cdf3c4662ece4b0ba1c88057affd26e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:51.040494 18985 tablet_bootstrap.cc:492] T 2fc6908ecf954020a676b9d10a2b00e7 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:51.040661 18985 ts_tablet_manager.cc:1367] T 2fc6908ecf954020a676b9d10a2b00e7 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.043s user 0.003s sys 0.000s
I20240430 20:11:51.041188 18417 leader_election.cc:302] T f90a5c5818c54390a2ae323fed7ebe57 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters: 5e1c6397f09747f091534faf93542710
I20240430 20:11:51.041611 21054 raft_consensus.cc:2781] T f90a5c5818c54390a2ae323fed7ebe57 P 2efb542aa55e439b99ccb98d1c5c884a [term 2 FOLLOWER]: Leader election won for term 2
I20240430 20:11:51.041755 21054 raft_consensus.cc:686] T f90a5c5818c54390a2ae323fed7ebe57 P 2efb542aa55e439b99ccb98d1c5c884a [term 2 LEADER]: Becoming Leader. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Running, Role: LEADER
I20240430 20:11:51.042012 21054 consensus_queue.cc:237] T f90a5c5818c54390a2ae323fed7ebe57 P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:51.042197 18983 tablet_bootstrap.cc:492] T 8901327b3efb4201adce61a9266142ea P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:51.042609 18985 raft_consensus.cc:348] T 2fc6908ecf954020a676b9d10a2b00e7 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:51.042853 18985 raft_consensus.cc:374] T 2fc6908ecf954020a676b9d10a2b00e7 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:51.042954 18985 raft_consensus.cc:729] T 2fc6908ecf954020a676b9d10a2b00e7 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:51.043154 18985 consensus_queue.cc:260] T 2fc6908ecf954020a676b9d10a2b00e7 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:51.043468 20795 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/master-2/data/info.pb
I20240430 20:11:51.043702 18985 ts_tablet_manager.cc:1398] T 2fc6908ecf954020a676b9d10a2b00e7 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240430 20:11:51.043938 18983 tablet_bootstrap.cc:654] T 8901327b3efb4201adce61a9266142ea P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
W20240430 20:11:51.044106 18417 leader_election.cc:341] T c6378f1495144c458af49b3c19a2b67c P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Not found: Tablet not found: c6378f1495144c458af49b3c19a2b67c
I20240430 20:11:51.044246 18417 leader_election.cc:302] T c6378f1495144c458af49b3c19a2b67c P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a; no voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710
I20240430 20:11:51.044624 21055 raft_consensus.cc:2726] T c6378f1495144c458af49b3c19a2b67c P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240430 20:11:51.045576 18417 leader_election.cc:341] T 3ec3993042f5498b818bdf32d5087928 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Not found: Tablet not found: 3ec3993042f5498b818bdf32d5087928
I20240430 20:11:51.045703 18417 leader_election.cc:302] T 3ec3993042f5498b818bdf32d5087928 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a; no voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710
I20240430 20:11:51.046111 21055 raft_consensus.cc:2726] T 3ec3993042f5498b818bdf32d5087928 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240430 20:11:51.046826 18461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dac141d1b70c48d785caddd7182811ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:51.048355 18417 leader_election.cc:302] T 90a694bd5db649afa50f3c4e752939c1 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a; no voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710
I20240430 20:11:51.048853 21055 raft_consensus.cc:2726] T 90a694bd5db649afa50f3c4e752939c1 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:51.050273 18116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNG3HqZ/build/asan/bin/kudu as pid 20795
I20240430 20:11:51.050526 18417 leader_election.cc:302] T d2381047f6dc4bdf962f9383e6c41548 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a; no voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710
I20240430 20:11:51.051030 21055 raft_consensus.cc:3037] T d2381047f6dc4bdf962f9383e6c41548 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:51.051239 18988 tablet_bootstrap.cc:492] T 62732af95c6a4c749c5b726f0460dc72 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:51.051385 18988 ts_tablet_manager.cc:1367] T 62732af95c6a4c749c5b726f0460dc72 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.048s user 0.009s sys 0.000s
I20240430 20:11:51.052073 18988 raft_consensus.cc:348] T 62732af95c6a4c749c5b726f0460dc72 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:51.052281 18988 raft_consensus.cc:374] T 62732af95c6a4c749c5b726f0460dc72 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:51.052340 18988 raft_consensus.cc:729] T 62732af95c6a4c749c5b726f0460dc72 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:51.052515 18988 consensus_queue.cc:260] T 62732af95c6a4c749c5b726f0460dc72 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:51.053547 21055 raft_consensus.cc:2726] T d2381047f6dc4bdf962f9383e6c41548 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:51.053583 18988 ts_tablet_manager.cc:1398] T 62732af95c6a4c749c5b726f0460dc72 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240430 20:11:51.054463 18457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43ec03fd93da43569bb9d3be01de43bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:51.055047 18988 tablet_bootstrap.cc:492] T 8901327b3efb4201adce61a9266142ea P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:51.056638 18988 tablet_bootstrap.cc:654] T 8901327b3efb4201adce61a9266142ea P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:51.067188 18983 tablet_bootstrap.cc:492] T 8901327b3efb4201adce61a9266142ea P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:51.067363 18983 ts_tablet_manager.cc:1367] T 8901327b3efb4201adce61a9266142ea P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.025s user 0.000s sys 0.009s
I20240430 20:11:51.068149 18983 raft_consensus.cc:348] T 8901327b3efb4201adce61a9266142ea P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:51.068378 18983 raft_consensus.cc:374] T 8901327b3efb4201adce61a9266142ea P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:51.068454 18983 raft_consensus.cc:729] T 8901327b3efb4201adce61a9266142ea P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:51.068651 21175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.177.60:37225 every 8 connection(s)
I20240430 20:11:51.068761 18983 consensus_queue.cc:260] T 8901327b3efb4201adce61a9266142ea P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:51.069195 18985 tablet_bootstrap.cc:492] T 8e9711d0a382424f9f758bfa883866a0 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:51.071252 18985 tablet_bootstrap.cc:654] T 8e9711d0a382424f9f758bfa883866a0 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:51.076018 20489 raft_consensus.cc:482] T 8901327b3efb4201adce61a9266142ea P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:51.076205 20489 raft_consensus.cc:504] T 8901327b3efb4201adce61a9266142ea P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:51.076933 20489 leader_election.cc:288] T 8901327b3efb4201adce61a9266142ea P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:51.077061 18983 ts_tablet_manager.cc:1398] T 8901327b3efb4201adce61a9266142ea P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.010s user 0.000s sys 0.002s
I20240430 20:11:51.077777 18474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8901327b3efb4201adce61a9266142ea" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
W20240430 20:11:51.078512 18558 leader_election.cc:341] T 8901327b3efb4201adce61a9266142ea P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:51.082123 18983 tablet_bootstrap.cc:492] T 6b51cbc731794083a88fe4bed1ddeebf P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:51.084204 21108 ts_tablet_manager.cc:1888] T 1462ecf1176c4e66be42abddbe36b00c P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240430 20:11:51.084467 21108 log.cc:1196] T 1462ecf1176c4e66be42abddbe36b00c P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/1462ecf1176c4e66be42abddbe36b00c
I20240430 20:11:51.084877 18603 tablet_service.cc:1467] Processing CreateTablet for tablet dac141d1b70c48d785caddd7182811ee (DEFAULT_TABLE table=default.Table_fc0a4540f98b4eea95099b8ab46ea235 [id=f3c53ccffe64410495cc8c379a05cfae]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:51.084954 18590 tablet_service.cc:1467] Processing CreateTablet for tablet 13f0a0a87da242c8acb9892bc2897616 (DEFAULT_TABLE table=default.Table_2e4e483bcd6c4dd6bdfcc938cb0e2da6 [id=afdab68d2df0433f9102de19b55c60b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:51.085295 18603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dac141d1b70c48d785caddd7182811ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:51.088626 20587 raft_consensus.cc:482] T 115c4a12440647a4bb46eb6824f7ea98 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5e1c6397f09747f091534faf93542710)
I20240430 20:11:51.088784 20587 raft_consensus.cc:504] T 115c4a12440647a4bb46eb6824f7ea98 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:51.089445 20587 leader_election.cc:288] T 115c4a12440647a4bb46eb6824f7ea98 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:51.089666 20489 raft_consensus.cc:482] T c1017a50d1d646c486c0b9a75cb88ce4 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:51.089792 20489 raft_consensus.cc:504] T c1017a50d1d646c486c0b9a75cb88ce4 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
W20240430 20:11:51.090364 18696 leader_election.cc:341] T 115c4a12440647a4bb46eb6824f7ea98 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Not found: Tablet not found: 115c4a12440647a4bb46eb6824f7ea98
I20240430 20:11:51.090528 20489 leader_election.cc:288] T c1017a50d1d646c486c0b9a75cb88ce4 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:51.090981 18590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13f0a0a87da242c8acb9892bc2897616. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:51.091909 20587 consensus_queue.cc:1035] T 0368d37dd9e2452982fb81f539c6805b P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:51.093732 20587 consensus_queue.cc:1035] T 12795670f21447a8964fe1efc6e92398 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240430 20:11:51.094342 18699 leader_election.cc:341] T 898b2c91adf34545808df9c786e218c9 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:51.096904 19929 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 6b51cbc731794083a88fe4bed1ddeebf (table default.Table_c6c962da746e4fd8a4d7200d09f08e9e [id=8e0f5d8c941d423ea7b30a89513d5170]). Expected version 1 got 0
I20240430 20:11:51.097286 19929 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 62732af95c6a4c749c5b726f0460dc72 (table default.Table_e3160f5ced1b4718b479401ca4df675c [id=2d44ec94131148048021b2ee6366ba30]). Expected version 3 got 0
I20240430 20:11:51.099100 18983 tablet_bootstrap.cc:654] T 6b51cbc731794083a88fe4bed1ddeebf P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:51.101152 18740 tablet_service.cc:1467] Processing CreateTablet for tablet 58c5ac58847e4e769e8e88e7e27494ae (DEFAULT_TABLE table=default.Table_71775acb989f4c2f87d670dff67a93b6 [id=a6c9697e1e9949b1a14d62c9f9ba4fed]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:51.101675 18740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58c5ac58847e4e769e8e88e7e27494ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:51.103119 20489 consensus_queue.cc:1035] T ed0c37a71ede4bfc9eb5ad1a4365f25e P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:51.091037 18474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1017a50d1d646c486c0b9a75cb88ce4" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:51.092628 18985 tablet_bootstrap.cc:492] T 8e9711d0a382424f9f758bfa883866a0 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:51.115413 18985 ts_tablet_manager.cc:1367] T 8e9711d0a382424f9f758bfa883866a0 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.046s user 0.002s sys 0.008s
I20240430 20:11:51.115391 18474 raft_consensus.cc:2454] T c1017a50d1d646c486c0b9a75cb88ce4 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 0.
I20240430 20:11:51.116178 18558 leader_election.cc:302] T c1017a50d1d646c486c0b9a75cb88ce4 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:51.116376 21108 ts_tablet_manager.cc:1909] T 1462ecf1176c4e66be42abddbe36b00c P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:51.116497 21013 raft_consensus.cc:2781] T c1017a50d1d646c486c0b9a75cb88ce4 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:51.116567 21013 raft_consensus.cc:482] T c1017a50d1d646c486c0b9a75cb88ce4 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:51.116622 21013 raft_consensus.cc:3037] T c1017a50d1d646c486c0b9a75cb88ce4 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:51.116479 18985 raft_consensus.cc:348] T 8e9711d0a382424f9f758bfa883866a0 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:51.116680 18985 raft_consensus.cc:374] T 8e9711d0a382424f9f758bfa883866a0 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:51.116782 18985 raft_consensus.cc:729] T 8e9711d0a382424f9f758bfa883866a0 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:51.116979 18985 consensus_queue.cc:260] T 8e9711d0a382424f9f758bfa883866a0 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:51.117532 19902 catalog_manager.cc:4830] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): tablet 1462ecf1176c4e66be42abddbe36b00c (table default.Table_b49a7942aa8c4f58acb94c35d1cdbb70 [id=6257559b0e9545c58848a76af0c5a5f7]) successfully deleted
I20240430 20:11:51.117635 18985 ts_tablet_manager.cc:1398] T 8e9711d0a382424f9f758bfa883866a0 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240430 20:11:51.106964 18446 tablet_service.cc:1467] Processing CreateTablet for tablet 13f0a0a87da242c8acb9892bc2897616 (DEFAULT_TABLE table=default.Table_2e4e483bcd6c4dd6bdfcc938cb0e2da6 [id=afdab68d2df0433f9102de19b55c60b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:51.118050 18985 tablet_bootstrap.cc:492] T 898b2c91adf34545808df9c786e218c9 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:51.118140 18446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13f0a0a87da242c8acb9892bc2897616. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:51.119442 21013 raft_consensus.cc:504] T c1017a50d1d646c486c0b9a75cb88ce4 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:51.120112 18985 tablet_bootstrap.cc:654] T 898b2c91adf34545808df9c786e218c9 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:51.120129 21013 leader_election.cc:288] T c1017a50d1d646c486c0b9a75cb88ce4 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:51.126964 18474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1017a50d1d646c486c0b9a75cb88ce4" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:51.127207 18474 raft_consensus.cc:3037] T c1017a50d1d646c486c0b9a75cb88ce4 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:51.130264 18474 raft_consensus.cc:2454] T c1017a50d1d646c486c0b9a75cb88ce4 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
I20240430 20:11:51.131307 18558 leader_election.cc:302] T c1017a50d1d646c486c0b9a75cb88ce4 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:51.132377 21013 raft_consensus.cc:2781] T c1017a50d1d646c486c0b9a75cb88ce4 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:51.132817 18528 heartbeater.cc:503] Master 127.17.177.62:36441 was elected leader, sending a full tablet report...
I20240430 20:11:51.132849 21013 raft_consensus.cc:686] T c1017a50d1d646c486c0b9a75cb88ce4 P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Becoming Leader. State: Replica: 5e1c6397f09747f091534faf93542710, State: Running, Role: LEADER
I20240430 20:11:51.133186 21013 consensus_queue.cc:237] T c1017a50d1d646c486c0b9a75cb88ce4 P 5e1c6397f09747f091534faf93542710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
W20240430 20:11:51.138355 18699 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.177.1:41217, user_credentials={real_user=slave}} blocked reactor thread for 43997.1us
I20240430 20:11:51.139986 18603 tablet_service.cc:1467] Processing CreateTablet for tablet 58c5ac58847e4e769e8e88e7e27494ae (DEFAULT_TABLE table=default.Table_71775acb989f4c2f87d670dff67a93b6 [id=a6c9697e1e9949b1a14d62c9f9ba4fed]), partition=RANGE (key) PARTITION UNBOUNDED
W20240430 20:11:51.140205 18699 leader_election.cc:341] T 898b2c91adf34545808df9c786e218c9 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:51.140555 18603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58c5ac58847e4e769e8e88e7e27494ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:51.143651 18457 tablet_service.cc:1467] Processing CreateTablet for tablet 58c5ac58847e4e769e8e88e7e27494ae (DEFAULT_TABLE table=default.Table_71775acb989f4c2f87d670dff67a93b6 [id=a6c9697e1e9949b1a14d62c9f9ba4fed]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:51.143903 18590 tablet_service.cc:1467] Processing CreateTablet for tablet 2cd6744ae3354f31ae8fa68f751c5483 (DEFAULT_TABLE table=default.Table_f86df7f6c95e4da48b80ae4733dc6994 [id=a29411bd07ba400c942bedc9e8e19884]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:51.144294 18590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cd6744ae3354f31ae8fa68f751c5483. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:51.144928 18457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58c5ac58847e4e769e8e88e7e27494ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:51.144874 18740 tablet_service.cc:1467] Processing CreateTablet for tablet 2cd6744ae3354f31ae8fa68f751c5483 (DEFAULT_TABLE table=default.Table_f86df7f6c95e4da48b80ae4733dc6994 [id=a29411bd07ba400c942bedc9e8e19884]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:51.145259 18740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cd6744ae3354f31ae8fa68f751c5483. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:51.149922 18461 tablet_service.cc:1467] Processing CreateTablet for tablet 2cd6744ae3354f31ae8fa68f751c5483 (DEFAULT_TABLE table=default.Table_f86df7f6c95e4da48b80ae4733dc6994 [id=a29411bd07ba400c942bedc9e8e19884]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:51.156703 18985 tablet_bootstrap.cc:492] T 898b2c91adf34545808df9c786e218c9 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:51.156900 18985 ts_tablet_manager.cc:1367] T 898b2c91adf34545808df9c786e218c9 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.039s user 0.005s sys 0.000s
I20240430 20:11:51.157624 18985 raft_consensus.cc:348] T 898b2c91adf34545808df9c786e218c9 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:51.157832 18985 raft_consensus.cc:374] T 898b2c91adf34545808df9c786e218c9 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:51.157914 18985 raft_consensus.cc:729] T 898b2c91adf34545808df9c786e218c9 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:51.158155 18985 consensus_queue.cc:260] T 898b2c91adf34545808df9c786e218c9 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:51.158744 18985 ts_tablet_manager.cc:1398] T 898b2c91adf34545808df9c786e218c9 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240430 20:11:51.165645 21055 raft_consensus.cc:482] T 8e9711d0a382424f9f758bfa883866a0 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:51.165820 21055 raft_consensus.cc:504] T 8e9711d0a382424f9f758bfa883866a0 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:51.165863 18602 tablet_service.cc:1467] Processing CreateTablet for tablet f133adb974b94afd8ab4626f9b26949e (DEFAULT_TABLE table=default.Table_dfbb6c73df814ce49886f7b8f0e0968d [id=9b01fa1f94b24da799aed1617be3c892]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:51.166386 18602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f133adb974b94afd8ab4626f9b26949e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:51.166529 21055 leader_election.cc:288] T 8e9711d0a382424f9f758bfa883866a0 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:51.167026 18616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e9711d0a382424f9f758bfa883866a0" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:51.098340 19929 catalog_manager.cc:5484] T a0baee91f5b44945b595d0e234af018e P 2efb542aa55e439b99ccb98d1c5c884a reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e1c6397f09747f091534faf93542710 (127.17.177.2). New cstate: current_term: 1 leader_uuid: "5e1c6397f09747f091534faf93542710" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } }
I20240430 20:11:51.174819 19929 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet a0baee91f5b44945b595d0e234af018e (table default.Table_5ed3ea587cf843db9a1ff1ffab40b678 [id=b3d1c2f97c4648518dbea83a46cadd85]). Expected version 2 got 0
I20240430 20:11:51.177203 18445 tablet_service.cc:1467] Processing CreateTablet for tablet f133adb974b94afd8ab4626f9b26949e (DEFAULT_TABLE table=default.Table_dfbb6c73df814ce49886f7b8f0e0968d [id=9b01fa1f94b24da799aed1617be3c892]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:51.181034 18461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cd6744ae3354f31ae8fa68f751c5483. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:51.181160 18983 tablet_bootstrap.cc:492] T 6b51cbc731794083a88fe4bed1ddeebf P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:51.181295 18983 ts_tablet_manager.cc:1367] T 6b51cbc731794083a88fe4bed1ddeebf P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.099s user 0.016s sys 0.016s
I20240430 20:11:51.182034 18983 raft_consensus.cc:348] T 6b51cbc731794083a88fe4bed1ddeebf P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:51.182219 18983 raft_consensus.cc:374] T 6b51cbc731794083a88fe4bed1ddeebf P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:51.182261 18983 raft_consensus.cc:729] T 6b51cbc731794083a88fe4bed1ddeebf P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:51.182478 18983 consensus_queue.cc:260] T 6b51cbc731794083a88fe4bed1ddeebf P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:51.183009 18983 ts_tablet_manager.cc:1398] T 6b51cbc731794083a88fe4bed1ddeebf P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240430 20:11:51.183290 18983 tablet_bootstrap.cc:492] T 489dc838744a49eba542009c86688aa5 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:51.184641 18983 tablet_bootstrap.cc:654] T 489dc838744a49eba542009c86688aa5 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:51.185585 18602 tablet_service.cc:1467] Processing CreateTablet for tablet 36a07d27145a46e48c3f10e8211a389e (DEFAULT_TABLE table=default.Table_326878a52ab041609f3c339cd65b7013 [id=d4763eaeacb546ae94b77b5a78fef986]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:51.185945 18602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36a07d27145a46e48c3f10e8211a389e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:51.188763 19929 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 266fd23786bc435db5083370f3330ed0 (table default.Table_cf60b652e9dc49e7a948ce66fd1cbffe [id=9734bb8dd05a400ca4244b3829f01dee]). Expected version 1 got 0
I20240430 20:11:51.190291 18603 tablet_service.cc:1467] Processing CreateTablet for tablet da3e038a135846f38e98de970ece273d (DEFAULT_TABLE table=default.Table_b869ed2a85d64fcf83a0eb41c1a86446 [id=9d43a387b6884f6dbb32a65cc1f0bb97]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:51.190701 18603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da3e038a135846f38e98de970ece273d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:51.191891 18669 heartbeater.cc:347] Connected to a master server at 127.17.177.60:37225
I20240430 20:11:51.194976 18445 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f133adb974b94afd8ab4626f9b26949e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:51.196815 19929 catalog_manager.cc:5484] T ae24e75a0d42441ebc97400ffa94145e P 2efb542aa55e439b99ccb98d1c5c884a reported cstate change: term changed from 0 to 1, leader changed from <none> to 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1). New cstate: current_term: 1 leader_uuid: "2efb542aa55e439b99ccb98d1c5c884a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } health_report { overall_health: HEALTHY } } }
I20240430 20:11:51.199775 19929 catalog_manager.cc:5484] T 50e1c55a1a0a484984ad5cbb78eec2ed P 2efb542aa55e439b99ccb98d1c5c884a reported cstate change: term changed from 0 to 1, leader changed from <none> to 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1). New cstate: current_term: 1 leader_uuid: "2efb542aa55e439b99ccb98d1c5c884a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } health_report { overall_health: HEALTHY } } }
I20240430 20:11:51.200145 18529 heartbeater.cc:347] Connected to a master server at 127.17.177.60:37225
I20240430 20:11:51.202761 21055 raft_consensus.cc:482] T f7d3dea34cf54fe2b949d1b7ca68577a P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:51.202948 21055 raft_consensus.cc:504] T f7d3dea34cf54fe2b949d1b7ca68577a P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:51.203470 19929 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 1701a15b115741aea13f8327e3d26b34 (table default.Table_3776a4b9d4f24aa6b7f882bb4d0116ef [id=7127dbd368864f7ba80a755549fe7ede]). Expected version 1 got 0
I20240430 20:11:51.203684 21055 leader_election.cc:288] T f7d3dea34cf54fe2b949d1b7ca68577a P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:51.204579 19929 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 89b774967624498fa2be499878b71124 (table default.Table_2ec9e316bc3347b189a13b7a8bbcf829 [id=9fa22833979f4c2f978de651145acef9]). Expected version 1 got 0
I20240430 20:11:51.204533 18619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7d3dea34cf54fe2b949d1b7ca68577a" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:51.204912 19929 catalog_manager.cc:5484] T 130007acc8ba4e709c3eb85fabe75903 P 2efb542aa55e439b99ccb98d1c5c884a reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c51c33d5616482cafb68e26688820f4 (127.17.177.3). New cstate: current_term: 1 leader_uuid: "3c51c33d5616482cafb68e26688820f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } }
I20240430 20:11:51.205732 18983 tablet_bootstrap.cc:492] T 489dc838744a49eba542009c86688aa5 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:51.205888 18983 ts_tablet_manager.cc:1367] T 489dc838744a49eba542009c86688aa5 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.001s
I20240430 20:11:51.206017 19929 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 130007acc8ba4e709c3eb85fabe75903 (table default.Table_78987875650940c293247bc43db345ab [id=36f478a20a1f4a53a52fc29ade62b7f6]). Expected version 1 got 0
I20240430 20:11:51.206434 19929 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet ed0c37a71ede4bfc9eb5ad1a4365f25e (table default.Table_a2e85efc6b454524b0b122102ec07002 [id=86d795a82e674d859104741ebd04f4e8]). Expected version 1 got 0
I20240430 20:11:51.207652 18590 tablet_service.cc:1467] Processing CreateTablet for tablet 7c61dc93df2541f09e9e525181a02b22 (DEFAULT_TABLE table=default.Table_9c955414a012419eb8f904a0878a6336 [id=0e5a8ce8c9a440caa87263100895a12e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:51.208076 18590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c61dc93df2541f09e9e525181a02b22. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:51.208014 18586 tablet_service.cc:1467] Processing CreateTablet for tablet 8db2038dbc5845489f414441898288f2 (DEFAULT_TABLE table=default.Table_b99234d57c5c42d58e4a6516fc1482d4 [id=5b3ce682abaf4acc8b62f9d34ff63c64]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:51.208308 18586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8db2038dbc5845489f414441898288f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:51.208770 21177 sys_catalog.cc:263] Verifying existing consensus state
I20240430 20:11:51.209249 18983 raft_consensus.cc:348] T 489dc838744a49eba542009c86688aa5 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:51.209437 18983 raft_consensus.cc:374] T 489dc838744a49eba542009c86688aa5 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:51.209488 18983 raft_consensus.cc:729] T 489dc838744a49eba542009c86688aa5 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:51.209671 18983 consensus_queue.cc:260] T 489dc838744a49eba542009c86688aa5 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:51.207712 18601 tablet_service.cc:1467] Processing CreateTablet for tablet 40b4a8641a9b4d5195c9f9857019acd7 (DEFAULT_TABLE table=default.Table_8db024fb9ef241ed86cb98d1c3939607 [id=1ad13ddf3cfd402a84d2d4a5a55030fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:51.210916 18601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40b4a8641a9b4d5195c9f9857019acd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:51.214042 19929 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 12795670f21447a8964fe1efc6e92398 (table default.Table_896bbae1f6944005b0272ca8ebc97218 [id=d5cea90cbc2c48df9d8b80da1eccc949]). Expected version 3 got 0
I20240430 20:11:51.214383 18983 ts_tablet_manager.cc:1398] T 489dc838744a49eba542009c86688aa5 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20240430 20:11:51.214679 19929 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 3e5225f1f5f544ecbf8dfa470af3daca (table default.Table_c7c87a8e019944cab3ec99e9563ac01a [id=abf3765074eb4d7d9d2bcc4ab721e6bd]). Expected version 1 got 0
I20240430 20:11:51.215761 18590 tablet_service.cc:1467] Processing CreateTablet for tablet 2cd41b407d6b444d844c4c7c0ebf82e5 (DEFAULT_TABLE table=default.Table_673d1448679f4b15a5ee2a1369c8f8ea [id=3813b5e9778c4bd683a06598cef0d86f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:51.215946 18983 tablet_bootstrap.cc:492] T 35fdb052135541cfb42a8aeb03b64dc8 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:51.216230 18590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cd41b407d6b444d844c4c7c0ebf82e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:51.219044 21177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: Bootstrap starting.
I20240430 20:11:51.219480 19929 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 724e51ff05ec4baebc2bb55e05c8eddc (table default.Table_19df33614a6c4f94b6419a5b441e3ef8 [id=4dce8a767f0541e1beb5464b2ed95d06]). Expected version 2 got 1
I20240430 20:11:51.221505 18616 raft_consensus.cc:2454] T 8e9711d0a382424f9f758bfa883866a0 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 0.
I20240430 20:11:51.222046 18415 leader_election.cc:302] T 8e9711d0a382424f9f758bfa883866a0 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:51.222465 21055 raft_consensus.cc:2781] T 8e9711d0a382424f9f758bfa883866a0 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:51.222564 21055 raft_consensus.cc:482] T 8e9711d0a382424f9f758bfa883866a0 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:51.222733 21055 raft_consensus.cc:3037] T 8e9711d0a382424f9f758bfa883866a0 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:51.226342 19929 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet f1073fc792954c5795a05aa7d0ec24df (table default.Table_ba8b218e936b4bfcb073f8182f474cca [id=aedf11fda8914a3b843a57f8a7b69a53]). Expected version 1 got 0
I20240430 20:11:51.226346 18459 tablet_service.cc:1467] Processing CreateTablet for tablet da3e038a135846f38e98de970ece273d (DEFAULT_TABLE table=default.Table_b869ed2a85d64fcf83a0eb41c1a86446 [id=9d43a387b6884f6dbb32a65cc1f0bb97]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:51.226505 18454 tablet_service.cc:1467] Processing CreateTablet for tablet 108e88d8caf6449092b3df372cdb4f56 (DEFAULT_TABLE table=default.Table_70ba80736dca466e9ca9c231a40e7bd4 [id=1399e5a2de60445ab758e25dbc54ba34]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:51.226923 18459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da3e038a135846f38e98de970ece273d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:51.226797 19929 catalog_manager.cc:5484] T 66dd767bd3f8494f94061fb89e5e5680 P 2efb542aa55e439b99ccb98d1c5c884a reported cstate change: term changed from 0 to 1, leader changed from <none> to 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1). New cstate: current_term: 1 leader_uuid: "2efb542aa55e439b99ccb98d1c5c884a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } health_report { overall_health: HEALTHY } } }
I20240430 20:11:51.227262 18454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 108e88d8caf6449092b3df372cdb4f56. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:51.227453 19929 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 66dd767bd3f8494f94061fb89e5e5680 (table default.Table_b064d5659f7243f5b0ec167fb6c3f404 [id=5227206f44d04fa89b4842b899f33afb]). Expected version 2 got 0
I20240430 20:11:51.226378 18446 tablet_service.cc:1467] Processing CreateTablet for tablet 36a07d27145a46e48c3f10e8211a389e (DEFAULT_TABLE table=default.Table_326878a52ab041609f3c339cd65b7013 [id=d4763eaeacb546ae94b77b5a78fef986]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:51.227838 18446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36a07d27145a46e48c3f10e8211a389e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:51.228367 19929 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 37b2ea4be7494f0985808ade788dbe2b (table default.Table_5f6e0fce999b46de927f9271e5de5071 [id=0baad83a8c0f4aeebc7ddad068625a69]). Expected version 1 got 0
I20240430 20:11:51.228655 19929 catalog_manager.cc:5484] T 22fef8bb6b034db098c3372ae992f7a7 P 2efb542aa55e439b99ccb98d1c5c884a reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c51c33d5616482cafb68e26688820f4 (127.17.177.3). New cstate: current_term: 1 leader_uuid: "3c51c33d5616482cafb68e26688820f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } }
I20240430 20:11:51.228997 19929 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 22fef8bb6b034db098c3372ae992f7a7 (table default.Table_4440df38610340758047115845b42b95 [id=7d186ca20c094cc19aef792fe1c094e9]). Expected version 1 got 0
I20240430 20:11:51.229156 19929 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 330453ddeea6456e8ef93fa92608b0d3 (table default.Table_c1f36f7df5ae4e7c8eb8b4e99b89748b [id=a50abf28790d4284a76ac6ee22c02606]). Expected version 1 got 0
I20240430 20:11:51.229312 19929 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet acd23f0a63d5448aaa7ed78422e67bad (table default.Table_972f13be4dca4cd19b9a04534dc59f51 [id=69f37db4de634a3185613e40013e0ed7]). Expected version 1 got 0
I20240430 20:11:51.229496 19929 catalog_manager.cc:5484] T 2e79df8744e74b5991a79ce978ff0202 P 2efb542aa55e439b99ccb98d1c5c884a reported cstate change: term changed from 0 to 1, leader changed from <none> to 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1). New cstate: current_term: 1 leader_uuid: "2efb542aa55e439b99ccb98d1c5c884a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } health_report { overall_health: HEALTHY } } }
I20240430 20:11:51.229887 19929 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 2e79df8744e74b5991a79ce978ff0202 (table default.Table_276bbdc7cfe542d1a2936485a2aad1ec [id=f146a15e6e8a4acbb4c734c1c863da57]). Expected version 1 got 0
I20240430 20:11:51.230082 19929 catalog_manager.cc:5484] T ca64de16d40f4a44b8b411f5526ba11a P 2efb542aa55e439b99ccb98d1c5c884a reported cstate change: term changed from 1 to 2, leader changed from 5e1c6397f09747f091534faf93542710 (127.17.177.2) to 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1). New cstate: current_term: 2 leader_uuid: "2efb542aa55e439b99ccb98d1c5c884a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } health_report { overall_health: HEALTHY } } }
I20240430 20:11:51.230489 19929 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet ca64de16d40f4a44b8b411f5526ba11a (table default.Table_1cf914acf7fe4e98b8dbc297b6372f21 [id=f4ba9536e9c141deb68f4f62b7e8fa70]). Expected version 1 got 0
I20240430 20:11:51.230736 19929 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet eb6c96047b114ebba73db7a371f2da8e (table default.Table_7d5f6f18cbd64c869739872a7dff9be2 [id=e67d619e36c0485f8f22b4a81cb4cb6c]). Expected version 1 got 0
I20240430 20:11:51.231226 18449 tablet_service.cc:1467] Processing CreateTablet for tablet 7c61dc93df2541f09e9e525181a02b22 (DEFAULT_TABLE table=default.Table_9c955414a012419eb8f904a0878a6336 [id=0e5a8ce8c9a440caa87263100895a12e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:51.231308 18450 tablet_service.cc:1467] Processing CreateTablet for tablet 40b4a8641a9b4d5195c9f9857019acd7 (DEFAULT_TABLE table=default.Table_8db024fb9ef241ed86cb98d1c3939607 [id=1ad13ddf3cfd402a84d2d4a5a55030fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:51.231634 18449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c61dc93df2541f09e9e525181a02b22. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:51.231879 18450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40b4a8641a9b4d5195c9f9857019acd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:51.231853 18601 tablet_service.cc:1467] Processing CreateTablet for tablet 7a0176f8a74c47efbbe3d3343b3dc896 (DEFAULT_TABLE table=default.Table_144e5402c1864d818436015b26648024 [id=73e5c9f0470d47849109819c063e1ce4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:51.232237 18601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a0176f8a74c47efbbe3d3343b3dc896. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:51.236404 18458 tablet_service.cc:1467] Processing CreateTablet for tablet 8db2038dbc5845489f414441898288f2 (DEFAULT_TABLE table=default.Table_b99234d57c5c42d58e4a6516fc1482d4 [id=5b3ce682abaf4acc8b62f9d34ff63c64]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:51.236711 18983 tablet_bootstrap.cc:654] T 35fdb052135541cfb42a8aeb03b64dc8 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:51.236797 18458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8db2038dbc5845489f414441898288f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:51.241526 18447 tablet_service.cc:1467] Processing CreateTablet for tablet 2cd41b407d6b444d844c4c7c0ebf82e5 (DEFAULT_TABLE table=default.Table_673d1448679f4b15a5ee2a1369c8f8ea [id=3813b5e9778c4bd683a06598cef0d86f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:51.241904 18447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cd41b407d6b444d844c4c7c0ebf82e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:51.245851 18448 tablet_service.cc:1467] Processing CreateTablet for tablet 7a0176f8a74c47efbbe3d3343b3dc896 (DEFAULT_TABLE table=default.Table_144e5402c1864d818436015b26648024 [id=73e5c9f0470d47849109819c063e1ce4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:51.246274 18448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a0176f8a74c47efbbe3d3343b3dc896. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:51.248430 20198 raft_consensus.cc:482] T ed0c37a71ede4bfc9eb5ad1a4365f25e P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:51.248576 20198 raft_consensus.cc:504] T ed0c37a71ede4bfc9eb5ad1a4365f25e P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:51.248662 18603 tablet_service.cc:1467] Processing CreateTablet for tablet 108e88d8caf6449092b3df372cdb4f56 (DEFAULT_TABLE table=default.Table_70ba80736dca466e9ca9c231a40e7bd4 [id=1399e5a2de60445ab758e25dbc54ba34]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:51.249034 18603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 108e88d8caf6449092b3df372cdb4f56. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:51.249213 20198 leader_election.cc:288] T ed0c37a71ede4bfc9eb5ad1a4365f25e P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:51.249874 18616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed0c37a71ede4bfc9eb5ad1a4365f25e" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:51.258841 18983 tablet_bootstrap.cc:492] T 35fdb052135541cfb42a8aeb03b64dc8 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:51.259101 18983 ts_tablet_manager.cc:1367] T 35fdb052135541cfb42a8aeb03b64dc8 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.043s user 0.001s sys 0.004s
I20240430 20:11:51.259871 18590 tablet_service.cc:1467] Processing CreateTablet for tablet e2f65b284e794ac28a18670f152a82b2 (DEFAULT_TABLE table=default.Table_4adefb6ee63f4301a479e1d042ff1c8b [id=16b11b30a9ab46b5a2c5da5c6a494ac8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:51.194945 18985 tablet_bootstrap.cc:492] T 417351cbc9d54f9d84bb7155f39da119 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:51.260552 18453 tablet_service.cc:1467] Processing CreateTablet for tablet e2f65b284e794ac28a18670f152a82b2 (DEFAULT_TABLE table=default.Table_4adefb6ee63f4301a479e1d042ff1c8b [id=16b11b30a9ab46b5a2c5da5c6a494ac8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:51.260010 18983 raft_consensus.cc:348] T 35fdb052135541cfb42a8aeb03b64dc8 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:51.260957 18983 raft_consensus.cc:374] T 35fdb052135541cfb42a8aeb03b64dc8 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:51.261055 18983 raft_consensus.cc:729] T 35fdb052135541cfb42a8aeb03b64dc8 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:51.261257 18590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2f65b284e794ac28a18670f152a82b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:51.265527 18601 tablet_service.cc:1467] Processing CreateTablet for tablet bdc9e34851a9418ba7576163a266d74c (DEFAULT_TABLE table=default.Table_ae525cb900474e458017ab35d825fc1c [id=c8d3ef652a764c3a9d658fbfc9f41ede]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:51.266002 18601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdc9e34851a9418ba7576163a266d74c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:51.261315 18983 consensus_queue.cc:260] T 35fdb052135541cfb42a8aeb03b64dc8 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:51.269207 18983 ts_tablet_manager.cc:1398] T 35fdb052135541cfb42a8aeb03b64dc8 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.010s user 0.001s sys 0.000s
I20240430 20:11:51.269616 18983 tablet_bootstrap.cc:492] T eb6c96047b114ebba73db7a371f2da8e P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:51.269730 18586 tablet_service.cc:1467] Processing CreateTablet for tablet 2736c1920d264d7b9d7fd5da930543a5 (DEFAULT_TABLE table=default.Table_123c6be593f44fc2ac4115d192c43e68 [id=a7adb9bbadef40609bee9bd9e86ad818]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:51.270087 18586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2736c1920d264d7b9d7fd5da930543a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:51.271440 18983 tablet_bootstrap.cc:654] T eb6c96047b114ebba73db7a371f2da8e P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:51.275501 18739 tablet_service.cc:1467] Processing CreateTablet for tablet da3e038a135846f38e98de970ece273d (DEFAULT_TABLE table=default.Table_b869ed2a85d64fcf83a0eb41c1a86446 [id=9d43a387b6884f6dbb32a65cc1f0bb97]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:51.275830 18732 tablet_service.cc:1467] Processing CreateTablet for tablet f133adb974b94afd8ab4626f9b26949e (DEFAULT_TABLE table=default.Table_dfbb6c73df814ce49886f7b8f0e0968d [id=9b01fa1f94b24da799aed1617be3c892]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:51.275929 18739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da3e038a135846f38e98de970ece273d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:51.278842 18733 tablet_service.cc:1467] Processing CreateTablet for tablet 2cd41b407d6b444d844c4c7c0ebf82e5 (DEFAULT_TABLE table=default.Table_673d1448679f4b15a5ee2a1369c8f8ea [id=3813b5e9778c4bd683a06598cef0d86f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:51.279057 19929 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 69f88532952a4591bf1577541c3430e6 (table default.Table_1177828cad1f4a24bd397619dad96170 [id=a95fed1bc3fa4865bf9a35dfc51b659c]). Expected version 2 got 0
I20240430 20:11:51.279217 18733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cd41b407d6b444d844c4c7c0ebf82e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:51.279531 19929 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 086b3dfa84c045a18c8c4f177dce606a (table default.Table_df19cb8a6f4d46699350ec5b69c14e5a [id=197923633fc54a4582207942b51530e2]). Expected version 2 got 0
I20240430 20:11:51.279696 19929 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 1050ab528c7d41559831652a1437d358 (table default.Table_acddc4cbf6334546b5fb3e4752fef2a5 [id=274cb15fd751410e83327ac4e8dc4629]). Expected version 2 got 0
I20240430 20:11:51.280157 18729 tablet_service.cc:1467] Processing CreateTablet for tablet 0c42e66f0e4f4291ad906b66f212165d (DEFAULT_TABLE table=default.Table_43043172400c49598042d752412c3a28 [id=c4cfc76234794e428b432eb2f5c74bb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:51.280481 18729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c42e66f0e4f4291ad906b66f212165d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:51.280575 18737 tablet_service.cc:1467] Processing CreateTablet for tablet 7c61dc93df2541f09e9e525181a02b22 (DEFAULT_TABLE table=default.Table_9c955414a012419eb8f904a0878a6336 [id=0e5a8ce8c9a440caa87263100895a12e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:51.280910 18737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c61dc93df2541f09e9e525181a02b22. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:51.282817 18744 tablet_service.cc:1467] Processing CreateTablet for tablet 7a0176f8a74c47efbbe3d3343b3dc896 (DEFAULT_TABLE table=default.Table_144e5402c1864d818436015b26648024 [id=73e5c9f0470d47849109819c063e1ce4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:51.283131 18744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a0176f8a74c47efbbe3d3343b3dc896. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:51.284032 18726 tablet_service.cc:1467] Processing CreateTablet for tablet 40b4a8641a9b4d5195c9f9857019acd7 (DEFAULT_TABLE table=default.Table_8db024fb9ef241ed86cb98d1c3939607 [id=1ad13ddf3cfd402a84d2d4a5a55030fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:51.284301 18726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40b4a8641a9b4d5195c9f9857019acd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:51.283944 19929 catalog_manager.cc:5484] T 8e33df70a1504dbd852fe0b3c3f61622 P 2efb542aa55e439b99ccb98d1c5c884a reported cstate change: term changed from 0 to 1, leader changed from <none> to 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1). New cstate: current_term: 1 leader_uuid: "2efb542aa55e439b99ccb98d1c5c884a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } health_report { overall_health: HEALTHY } } }
I20240430 20:11:51.286036 19929 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet aaef85eea87f4443943fd3692bb2a38f (table default.Table_3da5b4c8668c470483ae2a0c7773eaaf [id=e292d818ee3c45df90522f08eb2d6227]). Expected version 1 got 0
I20240430 20:11:51.286115 18732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f133adb974b94afd8ab4626f9b26949e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:51.286240 19929 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet a59ef6c545dd41a28265a4c1b8ce10e0 (table default.Table_07e40af8f1c546048975c8edf88170d7 [id=86df5d161d8641c89070e7eed9a39360]). Expected version 2 got 0
I20240430 20:11:51.286242 18736 tablet_service.cc:1467] Processing CreateTablet for tablet e2f65b284e794ac28a18670f152a82b2 (DEFAULT_TABLE table=default.Table_4adefb6ee63f4301a479e1d042ff1c8b [id=16b11b30a9ab46b5a2c5da5c6a494ac8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:51.286540 18736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2f65b284e794ac28a18670f152a82b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:51.286711 19929 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 81787eccca6c42c1b421377a3bf9ebe6 (table default.Table_0b4ca4cd0af148399dfa21f2d32114e3 [id=7f602584cef9486eb7d9b4389be3f43e]). Expected version 3 got 2
I20240430 20:11:51.286972 19929 catalog_manager.cc:5484] T 47bf64ecc41b445da4ff697e141243dd P 2efb542aa55e439b99ccb98d1c5c884a reported cstate change: term changed from 0 to 1, leader changed from <none> to 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1). New cstate: current_term: 1 leader_uuid: "2efb542aa55e439b99ccb98d1c5c884a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } health_report { overall_health: HEALTHY } } }
I20240430 20:11:51.287147 18734 tablet_service.cc:1467] Processing CreateTablet for tablet bdc9e34851a9418ba7576163a266d74c (DEFAULT_TABLE table=default.Table_ae525cb900474e458017ab35d825fc1c [id=c8d3ef652a764c3a9d658fbfc9f41ede]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:51.287390 19929 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 47bf64ecc41b445da4ff697e141243dd (table default.Table_f1ad29d5986d4b799cfc67afb2b0dfdf [id=838fc8b61a864dea8877a6f144e4cb4f]). Expected version 2 got 0
I20240430 20:11:51.287456 18734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdc9e34851a9418ba7576163a266d74c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:51.287612 19929 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 5df364628561475391c9a5e10ff5a05f (table default.Table_3c8f9f7b22f64fb1bdc3f1550987233a [id=7fd6f8576f4c4cbc8ffcdb4ab95e15dc]). Expected version 1 got 0
I20240430 20:11:51.287750 19929 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 821c55edb3414d72b1489eea97d3f73a (table default.Table_4d68c38d9ba7452a829f8318b2b267b8 [id=d06e3e9045b1429aaeebd1fd67d841de]). Expected version 1 got 0
I20240430 20:11:51.287917 19929 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 94fbd4a57a1e438891fcb456f6b40be3 (table default.Table_25ddc3adef6b4ccf8b9c00cc6772031c [id=4fa623ffdcf347639150a355806038af]). Expected version 1 got 0
I20240430 20:11:51.288218 19929 catalog_manager.cc:5484] T 70247a4c910946c0afc24ce5ce0e85d6 P 2efb542aa55e439b99ccb98d1c5c884a reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e1c6397f09747f091534faf93542710 (127.17.177.2). New cstate: current_term: 1 leader_uuid: "5e1c6397f09747f091534faf93542710" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } }
I20240430 20:11:51.288549 18758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e9711d0a382424f9f758bfa883866a0" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:51.288723 18758 raft_consensus.cc:2454] T 8e9711d0a382424f9f758bfa883866a0 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 0.
I20240430 20:11:51.288856 19929 catalog_manager.cc:5484] T 99203cc522de424184c6f97453f166fd P 2efb542aa55e439b99ccb98d1c5c884a reported cstate change: term changed from 0 to 1, leader changed from <none> to 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1). New cstate: current_term: 1 leader_uuid: "2efb542aa55e439b99ccb98d1c5c884a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } health_report { overall_health: HEALTHY } } }
I20240430 20:11:51.289232 19929 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 99203cc522de424184c6f97453f166fd (table default.Table_cb2a4b9ceaf4431facfff2c9bcf94c5d [id=3f8690a437ae49a3beb009fa7899d5d0]). Expected version 1 got 0
I20240430 20:11:51.289198 18758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7d3dea34cf54fe2b949d1b7ca68577a" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:51.289387 18758 raft_consensus.cc:2379] T f7d3dea34cf54fe2b949d1b7ca68577a P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2efb542aa55e439b99ccb98d1c5c884a in current term 1: Already voted for candidate 5e1c6397f09747f091534faf93542710 in this term.
I20240430 20:11:51.289618 18730 tablet_service.cc:1467] Processing CreateTablet for tablet 2736c1920d264d7b9d7fd5da930543a5 (DEFAULT_TABLE table=default.Table_123c6be593f44fc2ac4115d192c43e68 [id=a7adb9bbadef40609bee9bd9e86ad818]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:51.289526 19929 catalog_manager.cc:5484] T a482f4622f774d60b4efd6827b8be7d7 P 2efb542aa55e439b99ccb98d1c5c884a reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c51c33d5616482cafb68e26688820f4 (127.17.177.3). New cstate: current_term: 1 leader_uuid: "3c51c33d5616482cafb68e26688820f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } }
I20240430 20:11:51.289865 19929 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet a482f4622f774d60b4efd6827b8be7d7 (table default.Table_e686cdfcbcc046749fa5b51578a64794 [id=c3dc9f05316d46d6a3ef4d03f40877ec]). Expected version 1 got 0
I20240430 20:11:51.289932 18730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2736c1920d264d7b9d7fd5da930543a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:51.290061 19929 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet e041756c962b4a9f97c2cb314e7d8c05 (table default.Table_de88cafd92b149f9b545369bff71033c [id=c4bf4806f3d84579b16a3038d8d982c3]). Expected version 1 got 0
I20240430 20:11:51.291270 18602 tablet_service.cc:1467] Processing CreateTablet for tablet 1ba8b96864214bb3bc71ab99026632f3 (DEFAULT_TABLE table=default.Table_fac4c5c0ce954200838069be87be2ad9 [id=873474f62b8243e4b9068504b9f189a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:51.291630 18602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ba8b96864214bb3bc71ab99026632f3. 1 dirs total, 0 dirs full, 0 dirs failed
W20240430 20:11:51.296173 19900 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 7a0176f8a74c47efbbe3d3343b3dc896 (table default.Table_144e5402c1864d818436015b26648024 [id=73e5c9f0470d47849109819c063e1ce4], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 7a0176f8a74c47efbbe3d3343b3dc896 consensus state has no leader
I20240430 20:11:51.278539 18725 tablet_service.cc:1467] Processing CreateTablet for tablet 8db2038dbc5845489f414441898288f2 (DEFAULT_TABLE table=default.Table_b99234d57c5c42d58e4a6516fc1482d4 [id=5b3ce682abaf4acc8b62f9d34ff63c64]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:51.298678 18725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8db2038dbc5845489f414441898288f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:51.275893 18740 tablet_service.cc:1467] Processing CreateTablet for tablet 36a07d27145a46e48c3f10e8211a389e (DEFAULT_TABLE table=default.Table_326878a52ab041609f3c339cd65b7013 [id=d4763eaeacb546ae94b77b5a78fef986]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:51.302992 18740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36a07d27145a46e48c3f10e8211a389e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:51.276274 18742 tablet_service.cc:1467] Processing CreateTablet for tablet 108e88d8caf6449092b3df372cdb4f56 (DEFAULT_TABLE table=default.Table_70ba80736dca466e9ca9c231a40e7bd4 [id=1399e5a2de60445ab758e25dbc54ba34]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:51.304658 18742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 108e88d8caf6449092b3df372cdb4f56. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:51.306347 18762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "828ca913e2aa4102a6277bdbc6686ef5" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:51.306537 18762 raft_consensus.cc:3037] T 828ca913e2aa4102a6277bdbc6686ef5 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:51.308459 18756 raft_consensus.cc:1226] T f90a5c5818c54390a2ae323fed7ebe57 P 3c51c33d5616482cafb68e26688820f4 [term 2 FOLLOWER]: Rejecting Update request from peer 5e1c6397f09747f091534faf93542710 for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20240430 20:11:51.308749 18763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8901327b3efb4201adce61a9266142ea" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:51.309340 18763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1017a50d1d646c486c0b9a75cb88ce4" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:51.309511 18763 raft_consensus.cc:2454] T c1017a50d1d646c486c0b9a75cb88ce4 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 0.
I20240430 20:11:51.311630 18763 raft_consensus.cc:3037] T ed0c37a71ede4bfc9eb5ad1a4365f25e P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:51.312193 18756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1017a50d1d646c486c0b9a75cb88ce4" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:51.312366 18756 raft_consensus.cc:3037] T c1017a50d1d646c486c0b9a75cb88ce4 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:51.290459 19929 catalog_manager.cc:5484] T c091701480c4412faabad3d071d85d03 P 2efb542aa55e439b99ccb98d1c5c884a reported cstate change: term changed from 0 to 1, leader changed from <none> to 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1). New cstate: current_term: 1 leader_uuid: "2efb542aa55e439b99ccb98d1c5c884a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } health_report { overall_health: HEALTHY } } }
I20240430 20:11:51.313277 18601 tablet_service.cc:1467] Processing CreateTablet for tablet 0c42e66f0e4f4291ad906b66f212165d (DEFAULT_TABLE table=default.Table_43043172400c49598042d752412c3a28 [id=c4cfc76234794e428b432eb2f5c74bb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:51.313338 19929 catalog_manager.cc:5484] T 0a16d98f700c46e29f6dd8568752ef29 P 2efb542aa55e439b99ccb98d1c5c884a reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e1c6397f09747f091534faf93542710 (127.17.177.2). New cstate: current_term: 1 leader_uuid: "5e1c6397f09747f091534faf93542710" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } }
I20240430 20:11:51.314034 19929 catalog_manager.cc:5484] T f5eaf2413e1d4e69b56fae61f768f0ce P 2efb542aa55e439b99ccb98d1c5c884a reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e1c6397f09747f091534faf93542710 (127.17.177.2). New cstate: current_term: 1 leader_uuid: "5e1c6397f09747f091534faf93542710" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } }
I20240430 20:11:51.314415 19929 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet f5eaf2413e1d4e69b56fae61f768f0ce (table default.Table_1280cfe32d5b41abb4d3eb1bd70641bf [id=8489b872074a42b6abb713e420968141]). Expected version 1 got 0
I20240430 20:11:51.315017 19929 catalog_manager.cc:5484] T fbdeb4714b8c4b93ab72822c43329aae P 2efb542aa55e439b99ccb98d1c5c884a reported cstate change: term changed from 0 to 1, leader changed from <none> to 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1). New cstate: current_term: 1 leader_uuid: "2efb542aa55e439b99ccb98d1c5c884a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } health_report { overall_health: HEALTHY } } }
I20240430 20:11:51.315439 19929 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet fbdeb4714b8c4b93ab72822c43329aae (table default.Table_c12ca6938d424fbfa2e1df5cdb00fdbc [id=0fb93f36bc17408ea56bc4cafcfd8580]). Expected version 1 got 0
I20240430 20:11:51.315846 19929 catalog_manager.cc:5484] T 06e82c706b9c4f35bf53faddf57b3fd7 P 2efb542aa55e439b99ccb98d1c5c884a reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c51c33d5616482cafb68e26688820f4 (127.17.177.3). New cstate: current_term: 1 leader_uuid: "3c51c33d5616482cafb68e26688820f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } }
I20240430 20:11:51.316404 19929 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 488d0e609b224674932f890939664df2 (table default.Table_58f35a2120874472978ca0e5f816e86c [id=ff18001c6d814afc8c1768d0c84a6bac]). Expected version 1 got 0
I20240430 20:11:51.316952 19929 catalog_manager.cc:5484] T 78ba48dadef548708153878810cc46f8 P 2efb542aa55e439b99ccb98d1c5c884a reported cstate change: term changed from 0 to 1, leader changed from <none> to 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1). New cstate: current_term: 1 leader_uuid: "2efb542aa55e439b99ccb98d1c5c884a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } health_report { overall_health: HEALTHY } } }
I20240430 20:11:51.316058 18601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c42e66f0e4f4291ad906b66f212165d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:51.317325 19929 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 78ba48dadef548708153878810cc46f8 (table default.Table_b9983e101a6547649eb44ee04b5e1147 [id=29769be2b9ec4ecba9de5c0a65692f85]). Expected version 1 got 0
I20240430 20:11:51.319895 18735 tablet_service.cc:1467] Processing CreateTablet for tablet 1ba8b96864214bb3bc71ab99026632f3 (DEFAULT_TABLE table=default.Table_fac4c5c0ce954200838069be87be2ad9 [id=873474f62b8243e4b9068504b9f189a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:51.320288 18735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ba8b96864214bb3bc71ab99026632f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:51.319062 18983 tablet_bootstrap.cc:492] T eb6c96047b114ebba73db7a371f2da8e P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:51.327731 18983 ts_tablet_manager.cc:1367] T eb6c96047b114ebba73db7a371f2da8e P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.058s user 0.000s sys 0.004s
I20240430 20:11:51.329165 18810 heartbeater.cc:347] Connected to a master server at 127.17.177.60:37225
I20240430 20:11:51.329367 18810 heartbeater.cc:503] Master 127.17.177.60:37225 was elected leader, sending a full tablet report...
I20240430 20:11:51.329319 18983 raft_consensus.cc:348] T eb6c96047b114ebba73db7a371f2da8e P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:51.329511 18983 raft_consensus.cc:374] T eb6c96047b114ebba73db7a371f2da8e P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:51.329557 18983 raft_consensus.cc:729] T eb6c96047b114ebba73db7a371f2da8e P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:51.329756 18983 consensus_queue.cc:260] T eb6c96047b114ebba73db7a371f2da8e P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:51.330240 18983 ts_tablet_manager.cc:1398] T eb6c96047b114ebba73db7a371f2da8e P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240430 20:11:51.330613 18983 tablet_bootstrap.cc:492] T 2aa2a489116940dea3173cad16d94fa5 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:51.332158 18983 tablet_bootstrap.cc:654] T 2aa2a489116940dea3173cad16d94fa5 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:51.335889 21055 raft_consensus.cc:504] T 8e9711d0a382424f9f758bfa883866a0 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:51.336697 21055 leader_election.cc:288] T 8e9711d0a382424f9f758bfa883866a0 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:51.340025 18616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e9711d0a382424f9f758bfa883866a0" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:51.340236 18616 raft_consensus.cc:3037] T 8e9711d0a382424f9f758bfa883866a0 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:51.343703 18452 tablet_service.cc:1467] Processing CreateTablet for tablet bdc9e34851a9418ba7576163a266d74c (DEFAULT_TABLE table=default.Table_ae525cb900474e458017ab35d825fc1c [id=c8d3ef652a764c3a9d658fbfc9f41ede]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:51.343858 18453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2f65b284e794ac28a18670f152a82b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:51.344178 18452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdc9e34851a9418ba7576163a266d74c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:51.346797 18616 raft_consensus.cc:2454] T 8e9711d0a382424f9f758bfa883866a0 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 1.
I20240430 20:11:51.317613 19929 catalog_manager.cc:5484] T 53da4b3855344b6d92211af767195dc8 P 2efb542aa55e439b99ccb98d1c5c884a reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c51c33d5616482cafb68e26688820f4 (127.17.177.3). New cstate: current_term: 1 leader_uuid: "3c51c33d5616482cafb68e26688820f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } }
I20240430 20:11:51.351794 19929 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 772eba5f24c549c791cad75565858285 (table default.Table_5fc77d5f60ae44f4bb4ca9f7f183672c [id=6ca223ca6354449c959d85ed2ac6a9c8]). Expected version 1 got 0
I20240430 20:11:51.351985 18762 raft_consensus.cc:2454] T 828ca913e2aa4102a6277bdbc6686ef5 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
I20240430 20:11:51.352986 19929 catalog_manager.cc:5484] T 885d27d4f8df48dd996d5b309b783f3b P 2efb542aa55e439b99ccb98d1c5c884a reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c51c33d5616482cafb68e26688820f4 (127.17.177.3). New cstate: current_term: 1 leader_uuid: "3c51c33d5616482cafb68e26688820f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } }
I20240430 20:11:51.354807 18756 raft_consensus.cc:2454] T c1017a50d1d646c486c0b9a75cb88ce4 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
I20240430 20:11:51.355211 19929 catalog_manager.cc:5484] T c6e3d09402574f1e8fe9e4160c70a3ee P 2efb542aa55e439b99ccb98d1c5c884a reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c51c33d5616482cafb68e26688820f4 (127.17.177.3). New cstate: current_term: 1 leader_uuid: "3c51c33d5616482cafb68e26688820f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } }
I20240430 20:11:51.359480 18983 tablet_bootstrap.cc:492] T 2aa2a489116940dea3173cad16d94fa5 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:51.359622 18983 ts_tablet_manager.cc:1367] T 2aa2a489116940dea3173cad16d94fa5 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.029s user 0.004s sys 0.000s
I20240430 20:11:51.360625 18983 raft_consensus.cc:348] T 2aa2a489116940dea3173cad16d94fa5 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:51.360836 18983 raft_consensus.cc:374] T 2aa2a489116940dea3173cad16d94fa5 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:51.360920 18983 raft_consensus.cc:729] T 2aa2a489116940dea3173cad16d94fa5 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:51.361141 18983 consensus_queue.cc:260] T 2aa2a489116940dea3173cad16d94fa5 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:51.361224 19929 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet c6e3d09402574f1e8fe9e4160c70a3ee (table default.Table_656db97355f64d7c8aa29b1352dfe9c8 [id=450f5d1c7e084a1bb88d035c0ccfcf82]). Expected version 1 got 0
I20240430 20:11:51.361699 19929 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 3a03b6997921408cb5a65f4cae223e9c (table default.Table_744b5c96ea014bdb8feae740356a790e [id=83b1ac2e9c3d4a10b02f96ebb17faf44]). Expected version 1 got 0
I20240430 20:11:51.361775 18983 ts_tablet_manager.cc:1398] T 2aa2a489116940dea3173cad16d94fa5 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240430 20:11:51.362043 18983 tablet_bootstrap.cc:492] T 89b774967624498fa2be499878b71124 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:51.362253 19929 catalog_manager.cc:5484] T ae20f44df50343289b0a512a4ea53692 P 2efb542aa55e439b99ccb98d1c5c884a reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e1c6397f09747f091534faf93542710 (127.17.177.2). New cstate: current_term: 1 leader_uuid: "5e1c6397f09747f091534faf93542710" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } }
I20240430 20:11:51.362753 19929 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet ae20f44df50343289b0a512a4ea53692 (table default.Table_1844f6b1bda84812825384e14a2fd94e [id=71da9acc3cff4d95bbac4a686b708fd5]). Expected version 1 got 0
I20240430 20:11:51.363145 19929 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 0368d37dd9e2452982fb81f539c6805b (table default.Table_404b186c544e4b30bb36cb77d1a14120 [id=4330626e342a48699ea939add5c91b11]). Expected version 1 got 0
I20240430 20:11:51.363531 18983 tablet_bootstrap.cc:654] T 89b774967624498fa2be499878b71124 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:51.363349 19929 catalog_manager.cc:5484] T 7ce09fcbe1ac457a9c05efaf000c3263 P 2efb542aa55e439b99ccb98d1c5c884a reported cstate change: term changed from 1 to 2, leader changed from 5e1c6397f09747f091534faf93542710 (127.17.177.2) to 3c51c33d5616482cafb68e26688820f4 (127.17.177.3). New cstate: current_term: 2 leader_uuid: "3c51c33d5616482cafb68e26688820f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } }
I20240430 20:11:51.371105 19929 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 7ce09fcbe1ac457a9c05efaf000c3263 (table default.Table_edc4c27ed846454288f73f6a0d88cb69 [id=78fd3f516083489ab3b42d90dcef7e4d]). Expected version 1 got 0
I20240430 20:11:51.371551 19929 catalog_manager.cc:5484] T 331c031d44854d679a27d10d9151d0ea P 2efb542aa55e439b99ccb98d1c5c884a reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e1c6397f09747f091534faf93542710 (127.17.177.2). New cstate: current_term: 1 leader_uuid: "5e1c6397f09747f091534faf93542710" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } }
I20240430 20:11:51.372007 19929 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 331c031d44854d679a27d10d9151d0ea (table default.Table_763f0800e8bb4fa69976886416250f42 [id=61c8a5b99ef34a0baeeb343fef7cfd05]). Expected version 1 got 0
I20240430 20:11:51.372529 19929 catalog_manager.cc:5484] T 30a521916bda46678be20a066bd1d3f3 P 2efb542aa55e439b99ccb98d1c5c884a reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c51c33d5616482cafb68e26688820f4 (127.17.177.3). New cstate: current_term: 1 leader_uuid: "3c51c33d5616482cafb68e26688820f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } }
I20240430 20:11:51.373126 19929 catalog_manager.cc:5484] T 2da900b6bbfa4fa99fe8494fd2e68da8 P 2efb542aa55e439b99ccb98d1c5c884a reported cstate change: term changed from 0 to 1, leader changed from <none> to 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1). New cstate: current_term: 1 leader_uuid: "2efb542aa55e439b99ccb98d1c5c884a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } health_report { overall_health: HEALTHY } } }
I20240430 20:11:51.373582 19929 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 2da900b6bbfa4fa99fe8494fd2e68da8 (table default.Table_19a91a7f17ff456ebb145e8dd89c0940 [id=67283dd3995a44c6a7a7bc8722e4f955]). Expected version 1 got 0
I20240430 20:11:51.373837 19929 catalog_manager.cc:5484] T 8e91920e7e36418d8c27d1f0bfd9b981 P 2efb542aa55e439b99ccb98d1c5c884a reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e1c6397f09747f091534faf93542710 (127.17.177.2). New cstate: current_term: 1 leader_uuid: "5e1c6397f09747f091534faf93542710" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } }
I20240430 20:11:51.374195 19929 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 8e91920e7e36418d8c27d1f0bfd9b981 (table default.Table_c8b25b14d955429cbd79515081deceb1 [id=f393bcc9669c4ea985a2d40deb161f0c]). Expected version 2 got 0
I20240430 20:11:51.374461 19929 catalog_manager.cc:5484] T 677481aee5fa4eb4b189322d7a3f8f2e P 2efb542aa55e439b99ccb98d1c5c884a reported cstate change: term changed from 0 to 1, leader changed from <none> to 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1). New cstate: current_term: 1 leader_uuid: "2efb542aa55e439b99ccb98d1c5c884a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } health_report { overall_health: HEALTHY } } }
I20240430 20:11:51.374902 19929 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 677481aee5fa4eb4b189322d7a3f8f2e (table default.Table_1ce204e15fce4c469816cf4e7f80cac7 [id=d268c31d82df4871bc1a78aea5871b33]). Expected version 1 got 0
I20240430 20:11:51.375144 19929 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 5b26319488644d6a8b6be60460a5e7af (table default.Table_69892a8264a4477db48ee6248758e046 [id=5a1c4a7be2fc465fac89a50854eb3322]). Expected version 3 got 0
I20240430 20:11:51.375836 19929 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet b285e3eb16d148779c74e057458d8461 (table default.Table_9df4a0b4170f4f28a6c17b0cb052ead6 [id=d1655c507b434f129ad7ae0b04365e51]). Expected version 2 got 0
I20240430 20:11:51.375823 21013 raft_consensus.cc:482] T 417351cbc9d54f9d84bb7155f39da119 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:51.375973 21013 raft_consensus.cc:504] T 417351cbc9d54f9d84bb7155f39da119 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:51.376619 21013 leader_election.cc:288] T 417351cbc9d54f9d84bb7155f39da119 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:51.375365 18481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed0c37a71ede4bfc9eb5ad1a4365f25e" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:51.376984 18415 leader_election.cc:302] T 8e9711d0a382424f9f758bfa883866a0 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:51.378621 21049 raft_consensus.cc:2781] T 8e9711d0a382424f9f758bfa883866a0 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:51.380856 21049 raft_consensus.cc:686] T 8e9711d0a382424f9f758bfa883866a0 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Becoming Leader. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Running, Role: LEADER
I20240430 20:11:51.397492 18459 tablet_service.cc:1467] Processing CreateTablet for tablet 1ba8b96864214bb3bc71ab99026632f3 (DEFAULT_TABLE table=default.Table_fac4c5c0ce954200838069be87be2ad9 [id=873474f62b8243e4b9068504b9f189a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:51.397959 18459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ba8b96864214bb3bc71ab99026632f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:51.402577 18476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "417351cbc9d54f9d84bb7155f39da119" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:51.406325 21049 consensus_queue.cc:237] T 8e9711d0a382424f9f758bfa883866a0 P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:51.406961 18985 tablet_bootstrap.cc:654] T 417351cbc9d54f9d84bb7155f39da119 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:51.408270 18447 tablet_service.cc:1467] Processing CreateTablet for tablet 2736c1920d264d7b9d7fd5da930543a5 (DEFAULT_TABLE table=default.Table_123c6be593f44fc2ac4115d192c43e68 [id=a7adb9bbadef40609bee9bd9e86ad818]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:51.408711 18447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2736c1920d264d7b9d7fd5da930543a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:51.391140 18417 leader_election.cc:302] T f7d3dea34cf54fe2b949d1b7ca68577a P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a; no voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710
I20240430 20:11:51.411094 20630 raft_consensus.cc:3037] T f7d3dea34cf54fe2b949d1b7ca68577a P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:51.413939 20630 raft_consensus.cc:2726] T f7d3dea34cf54fe2b949d1b7ca68577a P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240430 20:11:51.415082 18558 leader_election.cc:341] T 417351cbc9d54f9d84bb7155f39da119 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:51.415750 21011 consensus_queue.cc:1035] T e900f62a5aa143e791c7539660b60cd9 P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:51.415814 18445 tablet_service.cc:1467] Processing CreateTablet for tablet 0c42e66f0e4f4291ad906b66f212165d (DEFAULT_TABLE table=default.Table_43043172400c49598042d752412c3a28 [id=c4cfc76234794e428b432eb2f5c74bb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:51.425210 18474 raft_consensus.cc:1261] T 2873af6da928463f9809864b8eaf0c4a P 2efb542aa55e439b99ccb98d1c5c884a [term 2 FOLLOWER]: Refusing update from remote peer 5e1c6397f09747f091534faf93542710: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240430 20:11:51.425941 20489 consensus_queue.cc:1035] T 2873af6da928463f9809864b8eaf0c4a P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:51.429136 20938 raft_consensus.cc:482] T 417351cbc9d54f9d84bb7155f39da119 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:51.429343 20938 raft_consensus.cc:504] T 417351cbc9d54f9d84bb7155f39da119 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:51.430269 20938 leader_election.cc:288] T 417351cbc9d54f9d84bb7155f39da119 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:51.430752 18616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "417351cbc9d54f9d84bb7155f39da119" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:51.430970 18616 raft_consensus.cc:2454] T 417351cbc9d54f9d84bb7155f39da119 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:51.431557 18696 leader_election.cc:302] T 417351cbc9d54f9d84bb7155f39da119 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:51.431942 20938 raft_consensus.cc:2781] T 417351cbc9d54f9d84bb7155f39da119 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:51.432073 20938 raft_consensus.cc:482] T 417351cbc9d54f9d84bb7155f39da119 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:51.432169 20938 raft_consensus.cc:3037] T 417351cbc9d54f9d84bb7155f39da119 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:51.432082 18988 tablet_bootstrap.cc:492] T 8901327b3efb4201adce61a9266142ea P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:51.433791 18988 ts_tablet_manager.cc:1367] T 8901327b3efb4201adce61a9266142ea P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.379s user 0.007s sys 0.033s
I20240430 20:11:51.436453 18445 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c42e66f0e4f4291ad906b66f212165d. 1 dirs total, 0 dirs full, 0 dirs failed
W20240430 20:11:51.439354 19942 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.17.177.3:39881 (request call id 135) took 1527 ms (client timeout 1000 ms). Trace:
W20240430 20:11:51.439486 19942 rpcz_store.cc:269] 0430 20:11:49.911858 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:50.064611 (+152753us) service_pool.cc:225] Handling call
0430 20:11:51.439285 (+1374674us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":1}
I20240430 20:11:51.440382 18988 raft_consensus.cc:348] T 8901327b3efb4201adce61a9266142ea P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:51.443670 18988 raft_consensus.cc:374] T 8901327b3efb4201adce61a9266142ea P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:51.444108 18730 tablet_service.cc:1514] Processing DeleteTablet for tablet 2fc6908ecf954020a676b9d10a2b00e7 with delete_type TABLET_DATA_DELETED (replaced by tablet 9a5654fd913f4fbeb9d126f035ac32bf) from {username='slave'} at 127.0.0.1:56540
I20240430 20:11:51.444254 18988 raft_consensus.cc:729] T 8901327b3efb4201adce61a9266142ea P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
W20240430 20:11:51.445319 19900 catalog_manager.cc:4797] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): delete failed for tablet 2fc6908ecf954020a676b9d10a2b00e7 because the tablet was not found. No further retry: Not found: Tablet not found: 2fc6908ecf954020a676b9d10a2b00e7
I20240430 20:11:51.445936 18985 tablet_bootstrap.cc:492] T 417351cbc9d54f9d84bb7155f39da119 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:51.446108 18985 ts_tablet_manager.cc:1367] T 417351cbc9d54f9d84bb7155f39da119 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.251s user 0.007s sys 0.003s
I20240430 20:11:51.446208 20938 raft_consensus.cc:504] T 417351cbc9d54f9d84bb7155f39da119 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:51.446991 20938 leader_election.cc:288] T 417351cbc9d54f9d84bb7155f39da119 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Requested vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:51.447420 18616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "417351cbc9d54f9d84bb7155f39da119" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:51.447618 18616 raft_consensus.cc:3037] T 417351cbc9d54f9d84bb7155f39da119 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:51.450698 18616 raft_consensus.cc:2454] T 417351cbc9d54f9d84bb7155f39da119 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:51.451246 18696 leader_election.cc:302] T 417351cbc9d54f9d84bb7155f39da119 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:51.453017 20938 raft_consensus.cc:2781] T 417351cbc9d54f9d84bb7155f39da119 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:51.452988 18985 raft_consensus.cc:348] T 417351cbc9d54f9d84bb7155f39da119 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:51.453322 18985 raft_consensus.cc:374] T 417351cbc9d54f9d84bb7155f39da119 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:51.453406 20938 raft_consensus.cc:686] T 417351cbc9d54f9d84bb7155f39da119 P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Becoming Leader. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Running, Role: LEADER
I20240430 20:11:51.453461 18985 raft_consensus.cc:729] T 417351cbc9d54f9d84bb7155f39da119 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:51.453858 20938 consensus_queue.cc:237] T 417351cbc9d54f9d84bb7155f39da119 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:51.453861 18985 consensus_queue.cc:260] T 417351cbc9d54f9d84bb7155f39da119 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:51.459245 18985 ts_tablet_manager.cc:1398] T 417351cbc9d54f9d84bb7155f39da119 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.013s user 0.004s sys 0.000s
I20240430 20:11:51.458778 18988 consensus_queue.cc:260] T 8901327b3efb4201adce61a9266142ea P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:51.466259 18985 tablet_bootstrap.cc:492] T 828ca913e2aa4102a6277bdbc6686ef5 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:51.472170 21054 consensus_queue.cc:1035] T 1f4e860484784aa9a2d28af2e9fb611c P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:51.480055 20489 consensus_queue.cc:1046] T f90a5c5818c54390a2ae323fed7ebe57 P 5e1c6397f09747f091534faf93542710 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240430 20:11:51.480226 18557 leader_election.cc:341] T 8901327b3efb4201adce61a9266142ea P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:51.480327 18557 leader_election.cc:302] T 8901327b3efb4201adce61a9266142ea P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:51.480326 20489 raft_consensus.cc:3032] T f90a5c5818c54390a2ae323fed7ebe57 P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Stepping down as leader of term 1
I20240430 20:11:51.480518 20489 raft_consensus.cc:729] T f90a5c5818c54390a2ae323fed7ebe57 P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Running, Role: LEADER
I20240430 20:11:51.480546 18668 heartbeater.cc:503] Master 127.17.177.62:36441 was elected leader, sending a full tablet report...
I20240430 20:11:51.480736 20489 consensus_queue.cc:260] T f90a5c5818c54390a2ae323fed7ebe57 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:51.480930 21011 raft_consensus.cc:2726] T 8901327b3efb4201adce61a9266142ea P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:51.481029 20489 raft_consensus.cc:3037] T f90a5c5818c54390a2ae323fed7ebe57 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Advancing to term 2
W20240430 20:11:51.482939 18699 leader_election.cc:341] T 115c4a12440647a4bb46eb6824f7ea98 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Not found: Tablet not found: 115c4a12440647a4bb46eb6824f7ea98
I20240430 20:11:51.483556 18699 leader_election.cc:302] T 115c4a12440647a4bb46eb6824f7ea98 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710
I20240430 20:11:51.484135 20198 raft_consensus.cc:2726] T 115c4a12440647a4bb46eb6824f7ea98 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240430 20:11:51.486181 18699 leader_election.cc:302] T ed0c37a71ede4bfc9eb5ad1a4365f25e P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710
I20240430 20:11:51.486426 20198 raft_consensus.cc:2726] T ed0c37a71ede4bfc9eb5ad1a4365f25e P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:51.490258 19930 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 3a03b6997921408cb5a65f4cae223e9c (table default.Table_744b5c96ea014bdb8feae740356a790e [id=83b1ac2e9c3d4a10b02f96ebb17faf44]). Expected version 1 got 0
I20240430 20:11:51.490670 19930 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet ae20f44df50343289b0a512a4ea53692 (table default.Table_1844f6b1bda84812825384e14a2fd94e [id=71da9acc3cff4d95bbac4a686b708fd5]). Expected version 1 got 0
I20240430 20:11:51.491098 19930 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet eb6c96047b114ebba73db7a371f2da8e (table default.Table_7d5f6f18cbd64c869739872a7dff9be2 [id=e67d619e36c0485f8f22b4a81cb4cb6c]). Expected version 1 got 0
I20240430 20:11:51.491348 19930 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 6b51cbc731794083a88fe4bed1ddeebf (table default.Table_c6c962da746e4fd8a4d7200d09f08e9e [id=8e0f5d8c941d423ea7b30a89513d5170]). Expected version 1 got 0
I20240430 20:11:51.491593 19930 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 62732af95c6a4c749c5b726f0460dc72 (table default.Table_e3160f5ced1b4718b479401ca4df675c [id=2d44ec94131148048021b2ee6366ba30]). Expected version 3 got 0
I20240430 20:11:51.491923 19930 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet a0baee91f5b44945b595d0e234af018e (table default.Table_5ed3ea587cf843db9a1ff1ffab40b678 [id=b3d1c2f97c4648518dbea83a46cadd85]). Expected version 2 got 0
I20240430 20:11:51.492179 19930 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 266fd23786bc435db5083370f3330ed0 (table default.Table_cf60b652e9dc49e7a948ce66fd1cbffe [id=9734bb8dd05a400ca4244b3829f01dee]). Expected version 1 got 0
I20240430 20:11:51.492564 19930 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet ca64de16d40f4a44b8b411f5526ba11a (table default.Table_1cf914acf7fe4e98b8dbc297b6372f21 [id=f4ba9536e9c141deb68f4f62b7e8fa70]). Expected version 1 got 0
I20240430 20:11:51.492844 19930 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 724e51ff05ec4baebc2bb55e05c8eddc (table default.Table_19df33614a6c4f94b6419a5b441e3ef8 [id=4dce8a767f0541e1beb5464b2ed95d06]). Expected version 2 got 1
I20240430 20:11:51.492921 18810 heartbeater.cc:503] Master 127.17.177.60:37225 was elected leader, sending a full tablet report...
I20240430 20:11:51.493199 19930 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 37b2ea4be7494f0985808ade788dbe2b (table default.Table_5f6e0fce999b46de927f9271e5de5071 [id=0baad83a8c0f4aeebc7ddad068625a69]). Expected version 1 got 0
I20240430 20:11:51.493623 19930 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 2e79df8744e74b5991a79ce978ff0202 (table default.Table_276bbdc7cfe542d1a2936485a2aad1ec [id=f146a15e6e8a4acbb4c734c1c863da57]). Expected version 1 got 0
I20240430 20:11:51.494194 19930 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 22fef8bb6b034db098c3372ae992f7a7 (table default.Table_4440df38610340758047115845b42b95 [id=7d186ca20c094cc19aef792fe1c094e9]). Expected version 1 got 0
I20240430 20:11:51.494477 19930 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 330453ddeea6456e8ef93fa92608b0d3 (table default.Table_c1f36f7df5ae4e7c8eb8b4e99b89748b [id=a50abf28790d4284a76ac6ee22c02606]). Expected version 1 got 0
I20240430 20:11:51.494767 19930 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet acd23f0a63d5448aaa7ed78422e67bad (table default.Table_972f13be4dca4cd19b9a04534dc59f51 [id=69f37db4de634a3185613e40013e0ed7]). Expected version 1 got 0
I20240430 20:11:51.495126 19930 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 69f88532952a4591bf1577541c3430e6 (table default.Table_1177828cad1f4a24bd397619dad96170 [id=a95fed1bc3fa4865bf9a35dfc51b659c]). Expected version 2 got 0
I20240430 20:11:51.497090 18988 ts_tablet_manager.cc:1398] T 8901327b3efb4201adce61a9266142ea P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.062s user 0.001s sys 0.002s
I20240430 20:11:51.498044 19930 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 1050ab528c7d41559831652a1437d358 (table default.Table_acddc4cbf6334546b5fb3e4752fef2a5 [id=274cb15fd751410e83327ac4e8dc4629]). Expected version 2 got 0
I20240430 20:11:51.498337 19930 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 086b3dfa84c045a18c8c4f177dce606a (table default.Table_df19cb8a6f4d46699350ec5b69c14e5a [id=197923633fc54a4582207942b51530e2]). Expected version 2 got 0
I20240430 20:11:51.498778 18669 heartbeater.cc:347] Connected to a master server at 127.17.177.60:37225
I20240430 20:11:51.500977 18988 tablet_bootstrap.cc:492] T eb6c96047b114ebba73db7a371f2da8e P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:51.509964 19930 catalog_manager.cc:5484] T ed0c37a71ede4bfc9eb5ad1a4365f25e P 5e1c6397f09747f091534faf93542710 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e1c6397f09747f091534faf93542710 (127.17.177.2). New cstate: current_term: 1 leader_uuid: "5e1c6397f09747f091534faf93542710" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } health_report { overall_health: UNKNOWN } } }
I20240430 20:11:51.511888 18988 tablet_bootstrap.cc:654] T eb6c96047b114ebba73db7a371f2da8e P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:51.511878 19930 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet ed0c37a71ede4bfc9eb5ad1a4365f25e (table default.Table_a2e85efc6b454524b0b122102ec07002 [id=86d795a82e674d859104741ebd04f4e8]). Expected version 1 got 0
I20240430 20:11:51.512646 18985 tablet_bootstrap.cc:654] T 828ca913e2aa4102a6277bdbc6686ef5 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:51.514513 20489 consensus_queue.cc:1035] T f7d3dea34cf54fe2b949d1b7ca68577a P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:51.516496 20587 consensus_queue.cc:1035] T d2381047f6dc4bdf962f9383e6c41548 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:51.512542 19930 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 821c55edb3414d72b1489eea97d3f73a (table default.Table_4d68c38d9ba7452a829f8318b2b267b8 [id=d06e3e9045b1429aaeebd1fd67d841de]). Expected version 1 got 0
I20240430 20:11:51.518720 19930 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 47bf64ecc41b445da4ff697e141243dd (table default.Table_f1ad29d5986d4b799cfc67afb2b0dfdf [id=838fc8b61a864dea8877a6f144e4cb4f]). Expected version 2 got 0
I20240430 20:11:51.519479 19930 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 94fbd4a57a1e438891fcb456f6b40be3 (table default.Table_25ddc3adef6b4ccf8b9c00cc6772031c [id=4fa623ffdcf347639150a355806038af]). Expected version 1 got 0
I20240430 20:11:51.524338 21013 consensus_queue.cc:1035] T 558e71ec370d46ca8fbdcec5b57eb2ec P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:51.529299 20489 raft_consensus.cc:482] T eb6c96047b114ebba73db7a371f2da8e P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:51.529455 20489 raft_consensus.cc:504] T eb6c96047b114ebba73db7a371f2da8e P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:51.520773 19930 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 99203cc522de424184c6f97453f166fd (table default.Table_cb2a4b9ceaf4431facfff2c9bcf94c5d [id=3f8690a437ae49a3beb009fa7899d5d0]). Expected version 1 got 0
I20240430 20:11:51.530050 20489 leader_election.cc:288] T eb6c96047b114ebba73db7a371f2da8e P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:51.530498 19930 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet aaef85eea87f4443943fd3692bb2a38f (table default.Table_3da5b4c8668c470483ae2a0c7773eaaf [id=e292d818ee3c45df90522f08eb2d6227]). Expected version 1 got 0
I20240430 20:11:51.530823 19930 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet a59ef6c545dd41a28265a4c1b8ce10e0 (table default.Table_07e40af8f1c546048975c8edf88170d7 [id=86df5d161d8641c89070e7eed9a39360]). Expected version 2 got 0
I20240430 20:11:51.531143 19930 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet e041756c962b4a9f97c2cb314e7d8c05 (table default.Table_de88cafd92b149f9b545369bff71033c [id=c4bf4806f3d84579b16a3038d8d982c3]). Expected version 1 got 0
I20240430 20:11:51.532022 18475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb6c96047b114ebba73db7a371f2da8e" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
W20240430 20:11:51.532709 18558 leader_election.cc:341] T eb6c96047b114ebba73db7a371f2da8e P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:51.536269 18985 tablet_bootstrap.cc:492] T 828ca913e2aa4102a6277bdbc6686ef5 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:51.536474 18985 ts_tablet_manager.cc:1367] T 828ca913e2aa4102a6277bdbc6686ef5 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.075s user 0.003s sys 0.003s
I20240430 20:11:51.537290 19930 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet a482f4622f774d60b4efd6827b8be7d7 (table default.Table_e686cdfcbcc046749fa5b51578a64794 [id=c3dc9f05316d46d6a3ef4d03f40877ec]). Expected version 1 got 0
I20240430 20:11:51.537757 19930 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 5df364628561475391c9a5e10ff5a05f (table default.Table_3c8f9f7b22f64fb1bdc3f1550987233a [id=7fd6f8576f4c4cbc8ffcdb4ab95e15dc]). Expected version 1 got 0
I20240430 20:11:51.538103 19930 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet b285e3eb16d148779c74e057458d8461 (table default.Table_9df4a0b4170f4f28a6c17b0cb052ead6 [id=d1655c507b434f129ad7ae0b04365e51]). Expected version 2 got 0
I20240430 20:11:51.538718 19930 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet f5eaf2413e1d4e69b56fae61f768f0ce (table default.Table_1280cfe32d5b41abb4d3eb1bd70641bf [id=8489b872074a42b6abb713e420968141]). Expected version 1 got 0
I20240430 20:11:51.539361 19930 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet fbdeb4714b8c4b93ab72822c43329aae (table default.Table_c12ca6938d424fbfa2e1df5cdb00fdbc [id=0fb93f36bc17408ea56bc4cafcfd8580]). Expected version 1 got 0
I20240430 20:11:51.539762 19930 catalog_manager.cc:5484] T b1792c160af346ffb4f047c5d4435c10 P 5e1c6397f09747f091534faf93542710 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e1c6397f09747f091534faf93542710 (127.17.177.2). New cstate: current_term: 1 leader_uuid: "5e1c6397f09747f091534faf93542710" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } health_report { overall_health: UNKNOWN } } }
I20240430 20:11:51.541298 19930 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 772eba5f24c549c791cad75565858285 (table default.Table_5fc77d5f60ae44f4bb4ca9f7f183672c [id=6ca223ca6354449c959d85ed2ac6a9c8]). Expected version 1 got 0
I20240430 20:11:51.541648 19930 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet c6e3d09402574f1e8fe9e4160c70a3ee (table default.Table_656db97355f64d7c8aa29b1352dfe9c8 [id=450f5d1c7e084a1bb88d035c0ccfcf82]). Expected version 1 got 0
I20240430 20:11:51.541935 19930 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 78ba48dadef548708153878810cc46f8 (table default.Table_b9983e101a6547649eb44ee04b5e1147 [id=29769be2b9ec4ecba9de5c0a65692f85]). Expected version 1 got 0
I20240430 20:11:51.543292 18985 raft_consensus.cc:348] T 828ca913e2aa4102a6277bdbc6686ef5 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:51.543728 18985 raft_consensus.cc:374] T 828ca913e2aa4102a6277bdbc6686ef5 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:51.543877 18985 raft_consensus.cc:729] T 828ca913e2aa4102a6277bdbc6686ef5 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:51.544505 18985 consensus_queue.cc:260] T 828ca913e2aa4102a6277bdbc6686ef5 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:51.552776 18985 ts_tablet_manager.cc:1398] T 828ca913e2aa4102a6277bdbc6686ef5 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.016s user 0.004s sys 0.000s
I20240430 20:11:51.554177 19930 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 488d0e609b224674932f890939664df2 (table default.Table_58f35a2120874472978ca0e5f816e86c [id=ff18001c6d814afc8c1768d0c84a6bac]). Expected version 1 got 0
I20240430 20:11:51.554567 19930 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 2da900b6bbfa4fa99fe8494fd2e68da8 (table default.Table_19a91a7f17ff456ebb145e8dd89c0940 [id=67283dd3995a44c6a7a7bc8722e4f955]). Expected version 1 got 0
I20240430 20:11:51.554790 19930 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 8e91920e7e36418d8c27d1f0bfd9b981 (table default.Table_c8b25b14d955429cbd79515081deceb1 [id=f393bcc9669c4ea985a2d40deb161f0c]). Expected version 2 got 0
I20240430 20:11:51.554981 19930 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 0368d37dd9e2452982fb81f539c6805b (table default.Table_404b186c544e4b30bb36cb77d1a14120 [id=4330626e342a48699ea939add5c91b11]). Expected version 1 got 0
I20240430 20:11:51.555218 19930 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 7ce09fcbe1ac457a9c05efaf000c3263 (table default.Table_edc4c27ed846454288f73f6a0d88cb69 [id=78fd3f516083489ab3b42d90dcef7e4d]). Expected version 1 got 0
I20240430 20:11:51.555472 19930 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 331c031d44854d679a27d10d9151d0ea (table default.Table_763f0800e8bb4fa69976886416250f42 [id=61c8a5b99ef34a0baeeb343fef7cfd05]). Expected version 1 got 0
I20240430 20:11:51.555641 19930 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 677481aee5fa4eb4b189322d7a3f8f2e (table default.Table_1ce204e15fce4c469816cf4e7f80cac7 [id=d268c31d82df4871bc1a78aea5871b33]). Expected version 1 got 0
I20240430 20:11:51.555804 19930 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 5b26319488644d6a8b6be60460a5e7af (table default.Table_69892a8264a4477db48ee6248758e046 [id=5a1c4a7be2fc465fac89a50854eb3322]). Expected version 3 got 0
I20240430 20:11:51.555974 19930 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet f1073fc792954c5795a05aa7d0ec24df (table default.Table_ba8b218e936b4bfcb073f8182f474cca [id=aedf11fda8914a3b843a57f8a7b69a53]). Expected version 1 got 0
I20240430 20:11:51.556180 19930 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 66dd767bd3f8494f94061fb89e5e5680 (table default.Table_b064d5659f7243f5b0ec167fb6c3f404 [id=5227206f44d04fa89b4842b899f33afb]). Expected version 2 got 0
I20240430 20:11:51.556339 19930 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 81787eccca6c42c1b421377a3bf9ebe6 (table default.Table_0b4ca4cd0af148399dfa21f2d32114e3 [id=7f602584cef9486eb7d9b4389be3f43e]). Expected version 3 got 2
I20240430 20:11:51.556651 19930 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 130007acc8ba4e709c3eb85fabe75903 (table default.Table_78987875650940c293247bc43db345ab [id=36f478a20a1f4a53a52fc29ade62b7f6]). Expected version 1 got 0
I20240430 20:11:51.556996 19930 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 12795670f21447a8964fe1efc6e92398 (table default.Table_896bbae1f6944005b0272ca8ebc97218 [id=d5cea90cbc2c48df9d8b80da1eccc949]). Expected version 3 got 0
I20240430 20:11:51.557199 19930 catalog_manager.cc:5484] T 3e5225f1f5f544ecbf8dfa470af3daca P 5e1c6397f09747f091534faf93542710 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e1c6397f09747f091534faf93542710 (127.17.177.2). New cstate: current_term: 1 leader_uuid: "5e1c6397f09747f091534faf93542710" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } health_report { overall_health: UNKNOWN } } }
I20240430 20:11:51.557657 19930 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 3e5225f1f5f544ecbf8dfa470af3daca (table default.Table_c7c87a8e019944cab3ec99e9563ac01a [id=abf3765074eb4d7d9d2bcc4ab721e6bd]). Expected version 1 got 0
I20240430 20:11:51.557956 19930 catalog_manager.cc:5484] T f90a5c5818c54390a2ae323fed7ebe57 P 5e1c6397f09747f091534faf93542710 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e1c6397f09747f091534faf93542710 (127.17.177.2). New cstate: current_term: 1 leader_uuid: "5e1c6397f09747f091534faf93542710" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } health_report { overall_health: UNKNOWN } } }
I20240430 20:11:51.559325 19930 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 89b774967624498fa2be499878b71124 (table default.Table_2ec9e316bc3347b189a13b7a8bbcf829 [id=9fa22833979f4c2f978de651145acef9]). Expected version 1 got 0
W20240430 20:11:51.560346 19945 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1faa2f2c84184b129ba6e4d2857822f0 consensus state has no leader
I20240430 20:11:51.560556 19945 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:50746 (request call id 51) took 1171 ms. Trace:
I20240430 20:11:51.560696 19945 rpcz_store.cc:276] 0430 20:11:50.388662 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:50.788692 (+400030us) service_pool.cc:225] Handling call
0430 20:11:50.789028 (+ 336us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0430 20:11:50.789332 (+ 304us) catalog_manager.cc:3613] Apply alter schema
0430 20:11:50.789564 (+ 232us) catalog_manager.cc:3636] Acquired catalog manager lock
0430 20:11:50.789704 (+ 140us) catalog_manager.cc:3783] Updating metadata on disk
0430 20:11:50.843248 (+ 53544us) catalog_manager.cc:3814] Committing alterations to in-memory state
0430 20:11:51.560520 (+717272us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:50.797279 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:50.797387 (+ 108us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:50.797395 (+ 8us) write_op.cc:427] Acquired schema lock
0430 20:11:50.797399 (+ 4us) tablet.cc:660] Decoding operations
0430 20:11:50.797508 (+ 109us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:50.797518 (+ 10us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:50.797523 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0430 20:11:50.797588 (+ 65us) tablet.cc:699] Row locks acquired
0430 20:11:50.797590 (+ 2us) write_op.cc:260] PREPARE: finished
0430 20:11:50.797622 (+ 32us) write_op.cc:270] Start()
0430 20:11:50.797651 (+ 29us) write_op.cc:276] Timestamp: P: 1714507910797590 usec, L: 0
0430 20:11:50.797654 (+ 3us) op_driver.cc:331] REPLICATION: starting
0430 20:11:50.797794 (+ 140us) log.cc:842] Serialized 405 byte log entry
0430 20:11:50.839792 (+ 41998us) op_driver.cc:444] REPLICATION: finished
0430 20:11:50.840054 (+ 262us) write_op.cc:301] APPLY: starting
0430 20:11:50.840063 (+ 9us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:50.840082 (+ 19us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:50.840110 (+ 28us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:50.840144 (+ 34us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:50.840190 (+ 46us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0430 20:11:50.840199 (+ 9us) write_op.cc:312] APPLY: finished
0430 20:11:50.840283 (+ 84us) log.cc:842] Serialized 23 byte log entry
0430 20:11:50.840367 (+ 84us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:50.840388 (+ 21us) write_op.cc:446] Released schema lock
0430 20:11:50.840395 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:50.840461 (+ 66us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:50.840464 (+ 3us) write_op.cc:446] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":228,"apply.run_cpu_time_us":464,"apply.run_wall_time_us":464,"num_ops":1,"prepare.queue_time_us":7034,"prepare.run_cpu_time_us":590,"prepare.run_wall_time_us":590,"raft.queue_time_us":724,"raft.run_cpu_time_us":3,"raft.run_wall_time_us":3,"replication_time_us":42118}]]}
I20240430 20:11:51.567719 21011 raft_consensus.cc:482] T 6b51cbc731794083a88fe4bed1ddeebf P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:51.567925 21011 raft_consensus.cc:504] T 6b51cbc731794083a88fe4bed1ddeebf P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:51.568746 21011 leader_election.cc:288] T 6b51cbc731794083a88fe4bed1ddeebf P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:51.569617 18475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b51cbc731794083a88fe4bed1ddeebf" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
W20240430 20:11:51.570305 18558 leader_election.cc:341] T 6b51cbc731794083a88fe4bed1ddeebf P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
W20240430 20:11:51.570964 19938 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet dd1de890e4394db287cf38bb411f5586 has no consensus state
I20240430 20:11:51.571228 19941 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: Sending DeleteTablet for 3 replicas of tablet 8668f7a285de43f48d83abecb4b2e96c
I20240430 20:11:51.571847 19941 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:50746 (request call id 52) took 1183 ms. Trace:
I20240430 20:11:51.571962 19941 rpcz_store.cc:276] 0430 20:11:50.388737 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:50.794376 (+405639us) service_pool.cc:225] Handling call
0430 20:11:50.794917 (+ 541us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0430 20:11:50.795420 (+ 503us) catalog_manager.cc:2749] Modifying in-memory table state
0430 20:11:50.795687 (+ 267us) catalog_manager.cc:2758] Locking tablets
0430 20:11:50.922784 (+127097us) catalog_manager.cc:2772] Removing table and tablets from system table
0430 20:11:50.948756 (+ 25972us) catalog_manager.cc:2790] Removing table from by-name map
0430 20:11:50.948847 (+ 91us) catalog_manager.cc:2806] Committing in-memory state
0430 20:11:51.571185 (+622338us) catalog_manager.cc:2810] Aborting table tasks
0430 20:11:51.571779 (+ 594us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:50.927716 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:50.927852 (+ 136us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:50.927858 (+ 6us) write_op.cc:427] Acquired schema lock
0430 20:11:50.927861 (+ 3us) tablet.cc:660] Decoding operations
0430 20:11:50.927999 (+ 138us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:50.928015 (+ 16us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:50.928021 (+ 6us) tablet.cc:683] Acquiring locks for 2 operations
0430 20:11:50.928200 (+ 179us) tablet.cc:699] Row locks acquired
0430 20:11:50.928203 (+ 3us) write_op.cc:260] PREPARE: finished
0430 20:11:50.928250 (+ 47us) write_op.cc:270] Start()
0430 20:11:50.928290 (+ 40us) write_op.cc:276] Timestamp: P: 1714507910928201 usec, L: 0
0430 20:11:50.928296 (+ 6us) op_driver.cc:331] REPLICATION: starting
0430 20:11:50.929006 (+ 710us) log.cc:842] Serialized 728 byte log entry
0430 20:11:50.947921 (+ 18915us) op_driver.cc:444] REPLICATION: finished
0430 20:11:50.948107 (+ 186us) write_op.cc:301] APPLY: starting
0430 20:11:50.948125 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:50.948173 (+ 48us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:50.948200 (+ 27us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:50.948270 (+ 70us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:50.948304 (+ 34us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0430 20:11:50.948312 (+ 8us) write_op.cc:312] APPLY: finished
0430 20:11:50.948392 (+ 80us) log.cc:842] Serialized 29 byte log entry
0430 20:11:50.948642 (+ 250us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:50.948668 (+ 26us) write_op.cc:446] Released schema lock
0430 20:11:50.948676 (+ 8us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:50.948939 (+ 263us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:50.948944 (+ 5us) write_op.cc:446] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":137,"apply.run_cpu_time_us":536,"apply.run_wall_time_us":907,"num_ops":2,"prepare.queue_time_us":3972,"prepare.run_cpu_time_us":989,"prepare.run_wall_time_us":2406,"raft.queue_time_us":1211,"raft.run_cpu_time_us":298,"raft.run_wall_time_us":299,"replication_time_us":19594}]]}
I20240430 20:11:51.572750 21055 raft_consensus.cc:482] T 2fc6908ecf954020a676b9d10a2b00e7 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:51.572821 19941 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50746:
table { table_name: "default.Table_96aea8c4ff80428da8a0394f1bf7b493" } modify_external_catalogs: true
I20240430 20:11:51.572938 21055 raft_consensus.cc:504] T 2fc6908ecf954020a676b9d10a2b00e7 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:51.573626 18730 tablet_service.cc:1514] Processing DeleteTablet for tablet 8668f7a285de43f48d83abecb4b2e96c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:50 UTC) from {username='slave'} at 127.0.0.1:56540
I20240430 20:11:51.573885 18602 tablet_service.cc:1514] Processing DeleteTablet for tablet 8668f7a285de43f48d83abecb4b2e96c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:50 UTC) from {username='slave'} at 127.0.0.1:45934
W20240430 20:11:51.574708 19900 catalog_manager.cc:4810] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): delete failed for tablet 8668f7a285de43f48d83abecb4b2e96c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8668f7a285de43f48d83abecb4b2e96c already in progress: creating tablet
I20240430 20:11:51.574901 21055 leader_election.cc:288] T 2fc6908ecf954020a676b9d10a2b00e7 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
W20240430 20:11:51.575842 18415 leader_election.cc:341] T 2fc6908ecf954020a676b9d10a2b00e7 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Not found: Tablet not found: 2fc6908ecf954020a676b9d10a2b00e7
I20240430 20:11:51.576721 18445 tablet_service.cc:1514] Processing DeleteTablet for tablet 8668f7a285de43f48d83abecb4b2e96c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:50 UTC) from {username='slave'} at 127.0.0.1:41450
W20240430 20:11:51.577471 19902 catalog_manager.cc:4810] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): delete failed for tablet 8668f7a285de43f48d83abecb4b2e96c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8668f7a285de43f48d83abecb4b2e96c already in progress: creating tablet
I20240430 20:11:51.579831 21049 raft_consensus.cc:482] T 417351cbc9d54f9d84bb7155f39da119 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:51.580086 21049 raft_consensus.cc:504] T 417351cbc9d54f9d84bb7155f39da119 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:51.581710 18616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "417351cbc9d54f9d84bb7155f39da119" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:51.582018 18616 raft_consensus.cc:2379] T 417351cbc9d54f9d84bb7155f39da119 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2efb542aa55e439b99ccb98d1c5c884a in current term 1: Already voted for candidate 3c51c33d5616482cafb68e26688820f4 in this term.
I20240430 20:11:51.583861 21049 leader_election.cc:288] T 417351cbc9d54f9d84bb7155f39da119 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
W20240430 20:11:51.584022 19940 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 17ddcaf971154a52b606bbf29a539bf0 consensus state has no leader
W20240430 20:11:51.585099 19937 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8e9711d0a382424f9f758bfa883866a0 consensus state has no leader
I20240430 20:11:51.585685 19946 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: Sending DeleteTablet for 3 replicas of tablet 7bde1d6934784c9aa5911181e8df9eeb
I20240430 20:11:51.586273 19946 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:50746 (request call id 53) took 1197 ms. Trace:
W20240430 20:11:51.586352 18536 tablet.cc:2369] T 417351cbc9d54f9d84bb7155f39da119 P 2efb542aa55e439b99ccb98d1c5c884a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240430 20:11:51.586367 19946 rpcz_store.cc:276] 0430 20:11:50.388823 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:50.795097 (+406274us) service_pool.cc:225] Handling call
0430 20:11:50.795520 (+ 423us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0430 20:11:50.795666 (+ 146us) catalog_manager.cc:2749] Modifying in-memory table state
0430 20:11:50.795717 (+ 51us) catalog_manager.cc:2758] Locking tablets
0430 20:11:50.795868 (+ 151us) catalog_manager.cc:2772] Removing table and tablets from system table
0430 20:11:50.842427 (+ 46559us) catalog_manager.cc:2790] Removing table from by-name map
0430 20:11:50.842517 (+ 90us) catalog_manager.cc:2806] Committing in-memory state
0430 20:11:51.585644 (+743127us) catalog_manager.cc:2810] Aborting table tasks
0430 20:11:51.586211 (+ 567us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:50.817558 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:50.817665 (+ 107us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:50.817673 (+ 8us) write_op.cc:427] Acquired schema lock
0430 20:11:50.817676 (+ 3us) tablet.cc:660] Decoding operations
0430 20:11:50.817939 (+ 263us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:50.817950 (+ 11us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:50.817956 (+ 6us) tablet.cc:683] Acquiring locks for 2 operations
0430 20:11:50.818087 (+ 131us) tablet.cc:699] Row locks acquired
0430 20:11:50.818090 (+ 3us) write_op.cc:260] PREPARE: finished
0430 20:11:50.818124 (+ 34us) write_op.cc:270] Start()
0430 20:11:50.818157 (+ 33us) write_op.cc:276] Timestamp: P: 1714507910818090 usec, L: 0
0430 20:11:50.818160 (+ 3us) op_driver.cc:331] REPLICATION: starting
0430 20:11:50.818338 (+ 178us) log.cc:842] Serialized 762 byte log entry
0430 20:11:50.839940 (+ 21602us) op_driver.cc:444] REPLICATION: finished
0430 20:11:50.841406 (+ 1466us) write_op.cc:301] APPLY: starting
0430 20:11:50.841415 (+ 9us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:50.841441 (+ 26us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:50.841462 (+ 21us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:50.841509 (+ 47us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:50.841529 (+ 20us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0430 20:11:50.841534 (+ 5us) write_op.cc:312] APPLY: finished
0430 20:11:50.841575 (+ 41us) log.cc:842] Serialized 29 byte log entry
0430 20:11:50.841609 (+ 34us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:50.841622 (+ 13us) write_op.cc:446] Released schema lock
0430 20:11:50.841628 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:50.841679 (+ 51us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:50.841686 (+ 7us) write_op.cc:446] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1442,"apply.run_cpu_time_us":320,"apply.run_wall_time_us":319,"num_ops":2,"prepare.queue_time_us":21127,"prepare.run_cpu_time_us":787,"prepare.run_wall_time_us":1226,"raft.queue_time_us":139,"raft.run_cpu_time_us":2,"raft.run_wall_time_us":3,"replication_time_us":21765}]]}
I20240430 20:11:51.587260 19946 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50746:
table { table_name: "default.Table_c7c87a8e019944cab3ec99e9563ac01a" } modify_external_catalogs: true
I20240430 20:11:51.588048 18590 tablet_service.cc:1514] Processing DeleteTablet for tablet 7bde1d6934784c9aa5911181e8df9eeb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:50 UTC) from {username='slave'} at 127.0.0.1:45934
I20240430 20:11:51.589004 18730 tablet_service.cc:1514] Processing DeleteTablet for tablet 7bde1d6934784c9aa5911181e8df9eeb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:50 UTC) from {username='slave'} at 127.0.0.1:56540
I20240430 20:11:51.589246 21203 tablet_replica.cc:330] T 7bde1d6934784c9aa5911181e8df9eeb P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:51.589473 21203 raft_consensus.cc:2229] T 7bde1d6934784c9aa5911181e8df9eeb P 3c51c33d5616482cafb68e26688820f4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:51.589676 21203 raft_consensus.cc:2258] T 7bde1d6934784c9aa5911181e8df9eeb P 3c51c33d5616482cafb68e26688820f4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:51.590276 21203 ts_tablet_manager.cc:1875] T 7bde1d6934784c9aa5911181e8df9eeb P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:51.594832 20198 raft_consensus.cc:482] T df92eb85261d47d88cbf16d29d9ad30e P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:51.594985 20198 raft_consensus.cc:504] T df92eb85261d47d88cbf16d29d9ad30e P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
W20240430 20:11:51.600872 18696 leader_election.cc:341] T df92eb85261d47d88cbf16d29d9ad30e P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Not found: Tablet not found: df92eb85261d47d88cbf16d29d9ad30e
I20240430 20:11:51.604569 20198 leader_election.cc:288] T df92eb85261d47d88cbf16d29d9ad30e P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:51.611256 18758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e9711d0a382424f9f758bfa883866a0" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:51.611467 18758 raft_consensus.cc:3037] T 8e9711d0a382424f9f758bfa883866a0 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:51.611696 18983 tablet_bootstrap.cc:492] T 89b774967624498fa2be499878b71124 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:51.611840 18983 ts_tablet_manager.cc:1367] T 89b774967624498fa2be499878b71124 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.250s user 0.021s sys 0.023s
I20240430 20:11:51.612588 18983 raft_consensus.cc:348] T 89b774967624498fa2be499878b71124 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:51.612779 18983 raft_consensus.cc:374] T 89b774967624498fa2be499878b71124 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:51.612834 18983 raft_consensus.cc:729] T 89b774967624498fa2be499878b71124 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:51.614154 18758 raft_consensus.cc:2454] T 8e9711d0a382424f9f758bfa883866a0 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 1.
I20240430 20:11:51.615577 18988 tablet_bootstrap.cc:492] T eb6c96047b114ebba73db7a371f2da8e P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:51.615727 18988 ts_tablet_manager.cc:1367] T eb6c96047b114ebba73db7a371f2da8e P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.116s user 0.009s sys 0.009s
I20240430 20:11:51.616580 18988 raft_consensus.cc:348] T eb6c96047b114ebba73db7a371f2da8e P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:51.616802 18988 raft_consensus.cc:374] T eb6c96047b114ebba73db7a371f2da8e P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:51.616869 18988 raft_consensus.cc:729] T eb6c96047b114ebba73db7a371f2da8e P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:51.617142 18756 raft_consensus.cc:1261] T 488d0e609b224674932f890939664df2 P 3c51c33d5616482cafb68e26688820f4 [term 2 FOLLOWER]: Refusing update from remote peer 2efb542aa55e439b99ccb98d1c5c884a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 1. (term mismatch)
I20240430 20:11:51.617231 18756 pending_rounds.cc:76] T 488d0e609b224674932f890939664df2 P 3c51c33d5616482cafb68e26688820f4: Aborting all ops after (but not including) 0
I20240430 20:11:51.617097 18988 consensus_queue.cc:260] T eb6c96047b114ebba73db7a371f2da8e P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:51.617271 18756 pending_rounds.cc:97] T 488d0e609b224674932f890939664df2 P 3c51c33d5616482cafb68e26688820f4: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20240430 20:11:51.617322 18756 raft_consensus.cc:2866] T 488d0e609b224674932f890939664df2 P 3c51c33d5616482cafb68e26688820f4 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240430 20:11:51.617761 18988 ts_tablet_manager.cc:1398] T eb6c96047b114ebba73db7a371f2da8e P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240430 20:11:51.624434 18557 leader_election.cc:302] T 828ca913e2aa4102a6277bdbc6686ef5 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a
I20240430 20:11:51.624961 20488 raft_consensus.cc:2781] T 828ca913e2aa4102a6277bdbc6686ef5 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:51.625108 20488 raft_consensus.cc:686] T 828ca913e2aa4102a6277bdbc6686ef5 P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Becoming Leader. State: Replica: 5e1c6397f09747f091534faf93542710, State: Running, Role: LEADER
I20240430 20:11:51.625384 20488 consensus_queue.cc:237] T 828ca913e2aa4102a6277bdbc6686ef5 P 5e1c6397f09747f091534faf93542710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:51.626827 18983 consensus_queue.cc:260] T 89b774967624498fa2be499878b71124 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:51.627612 18983 ts_tablet_manager.cc:1398] T 89b774967624498fa2be499878b71124 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.016s user 0.001s sys 0.001s
I20240430 20:11:51.631390 18983 tablet_bootstrap.cc:492] T aa9998f4e17147aa8f2b35c01980463a P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:51.631573 18445 tablet_service.cc:1514] Processing DeleteTablet for tablet 7bde1d6934784c9aa5911181e8df9eeb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:50 UTC) from {username='slave'} at 127.0.0.1:41450
I20240430 20:11:51.631881 21108 tablet_replica.cc:330] T 7bde1d6934784c9aa5911181e8df9eeb P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:51.632151 21108 raft_consensus.cc:2229] T 7bde1d6934784c9aa5911181e8df9eeb P 2efb542aa55e439b99ccb98d1c5c884a [term 2 LEADER]: Raft consensus shutting down.
I20240430 20:11:51.618094 18988 tablet_bootstrap.cc:492] T 35fdb052135541cfb42a8aeb03b64dc8 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
W20240430 20:11:51.628880 18815 tablet.cc:2369] T 8901327b3efb4201adce61a9266142ea P 3c51c33d5616482cafb68e26688820f4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240430 20:11:51.632627 21108 raft_consensus.cc:2258] T 7bde1d6934784c9aa5911181e8df9eeb P 2efb542aa55e439b99ccb98d1c5c884a [term 2 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:51.633513 21108 ts_tablet_manager.cc:1875] T 7bde1d6934784c9aa5911181e8df9eeb P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:51.622087 18756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "417351cbc9d54f9d84bb7155f39da119" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:51.635587 18983 tablet_bootstrap.cc:654] T aa9998f4e17147aa8f2b35c01980463a P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:51.644474 18988 tablet_bootstrap.cc:654] T 35fdb052135541cfb42a8aeb03b64dc8 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:51.646925 21108 ts_tablet_manager.cc:1888] T 7bde1d6934784c9aa5911181e8df9eeb P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.2
I20240430 20:11:51.647218 21108 log.cc:1196] T 7bde1d6934784c9aa5911181e8df9eeb P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/7bde1d6934784c9aa5911181e8df9eeb
I20240430 20:11:51.647814 21108 ts_tablet_manager.cc:1909] T 7bde1d6934784c9aa5911181e8df9eeb P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:51.648643 19902 catalog_manager.cc:4830] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): tablet 7bde1d6934784c9aa5911181e8df9eeb (table default.Table_7199d7c36d3f45f79cf9e4e2b4145592 [id=f6aef8f6a9164dd78f49e03f723b45a9]) successfully deleted
I20240430 20:11:51.657366 18983 tablet_bootstrap.cc:492] T aa9998f4e17147aa8f2b35c01980463a P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:51.657547 18983 ts_tablet_manager.cc:1367] T aa9998f4e17147aa8f2b35c01980463a P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.026s user 0.000s sys 0.003s
W20240430 20:11:51.666508 19895 catalog_manager.cc:4810] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): delete failed for tablet 8668f7a285de43f48d83abecb4b2e96c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8668f7a285de43f48d83abecb4b2e96c already in progress: creating tablet
I20240430 20:11:51.668794 21213 tablet_replica.cc:330] T 7bde1d6934784c9aa5911181e8df9eeb P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:51.675915 20938 consensus_queue.cc:1035] T 1701a15b115741aea13f8327e3d26b34 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:51.680171 18472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "417351cbc9d54f9d84bb7155f39da119" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:51.680608 18472 raft_consensus.cc:2454] T 417351cbc9d54f9d84bb7155f39da119 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:51.658447 18983 raft_consensus.cc:348] T aa9998f4e17147aa8f2b35c01980463a P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:51.691107 18983 raft_consensus.cc:374] T aa9998f4e17147aa8f2b35c01980463a P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:51.691231 18983 raft_consensus.cc:729] T aa9998f4e17147aa8f2b35c01980463a P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:51.691519 18983 consensus_queue.cc:260] T aa9998f4e17147aa8f2b35c01980463a P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:51.689882 21213 raft_consensus.cc:2229] T 7bde1d6934784c9aa5911181e8df9eeb P 5e1c6397f09747f091534faf93542710 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:51.693450 18472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "417351cbc9d54f9d84bb7155f39da119" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:51.693765 18472 raft_consensus.cc:3037] T 417351cbc9d54f9d84bb7155f39da119 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:51.695729 21213 raft_consensus.cc:2258] T 7bde1d6934784c9aa5911181e8df9eeb P 5e1c6397f09747f091534faf93542710 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:51.701038 18472 raft_consensus.cc:2454] T 417351cbc9d54f9d84bb7155f39da119 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
W20240430 20:11:51.704751 18671 tablet.cc:2369] T 89b774967624498fa2be499878b71124 P 5e1c6397f09747f091534faf93542710: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240430 20:11:51.711210 18988 tablet_bootstrap.cc:492] T 35fdb052135541cfb42a8aeb03b64dc8 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:51.711421 18988 ts_tablet_manager.cc:1367] T 35fdb052135541cfb42a8aeb03b64dc8 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.093s user 0.003s sys 0.003s
I20240430 20:11:51.722424 21198 raft_consensus.cc:482] T 6b51cbc731794083a88fe4bed1ddeebf P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:51.723224 21198 raft_consensus.cc:504] T 6b51cbc731794083a88fe4bed1ddeebf P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:51.723878 21198 leader_election.cc:288] T 6b51cbc731794083a88fe4bed1ddeebf P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:51.725016 18619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b51cbc731794083a88fe4bed1ddeebf" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:51.725224 18619 raft_consensus.cc:2454] T 6b51cbc731794083a88fe4bed1ddeebf P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:51.725522 18696 leader_election.cc:302] T 6b51cbc731794083a88fe4bed1ddeebf P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:51.725823 21198 raft_consensus.cc:2781] T 6b51cbc731794083a88fe4bed1ddeebf P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:51.725893 21198 raft_consensus.cc:482] T 6b51cbc731794083a88fe4bed1ddeebf P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:51.725953 21198 raft_consensus.cc:3037] T 6b51cbc731794083a88fe4bed1ddeebf P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:51.726084 18988 raft_consensus.cc:348] T 35fdb052135541cfb42a8aeb03b64dc8 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:51.726354 18988 raft_consensus.cc:374] T 35fdb052135541cfb42a8aeb03b64dc8 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:51.726457 18988 raft_consensus.cc:729] T 35fdb052135541cfb42a8aeb03b64dc8 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:51.726812 18988 consensus_queue.cc:260] T 35fdb052135541cfb42a8aeb03b64dc8 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
W20240430 20:11:51.727043 18557 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.177.3:44867, user_credentials={real_user=slave}} blocked reactor thread for 49040.6us
I20240430 20:11:51.727572 18988 ts_tablet_manager.cc:1398] T 35fdb052135541cfb42a8aeb03b64dc8 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.016s user 0.003s sys 0.000s
I20240430 20:11:51.711135 18477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df92eb85261d47d88cbf16d29d9ad30e" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:51.713225 21054 consensus_queue.cc:1035] T 488d0e609b224674932f890939664df2 P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:51.730571 18477 raft_consensus.cc:2379] T df92eb85261d47d88cbf16d29d9ad30e P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c51c33d5616482cafb68e26688820f4 in current term 1: Already voted for candidate 5e1c6397f09747f091534faf93542710 in this term.
I20240430 20:11:51.733520 21228 tablet.cc:1722] T 8e91920e7e36418d8c27d1f0bfd9b981 P 2efb542aa55e439b99ccb98d1c5c884a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240430 20:11:51.734145 21213 ts_tablet_manager.cc:1875] T 7bde1d6934784c9aa5911181e8df9eeb P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:51.739845 19942 catalog_manager.cc:5484] T d2381047f6dc4bdf962f9383e6c41548 P 3c51c33d5616482cafb68e26688820f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c51c33d5616482cafb68e26688820f4 (127.17.177.3). New cstate: current_term: 1 leader_uuid: "3c51c33d5616482cafb68e26688820f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } health_report { overall_health: UNKNOWN } } }
I20240430 20:11:51.740689 21198 raft_consensus.cc:504] T 6b51cbc731794083a88fe4bed1ddeebf P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:51.740964 19942 catalog_manager.cc:5484] T 0368d37dd9e2452982fb81f539c6805b P 3c51c33d5616482cafb68e26688820f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c51c33d5616482cafb68e26688820f4 (127.17.177.3). New cstate: current_term: 1 leader_uuid: "3c51c33d5616482cafb68e26688820f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } health_report { overall_health: UNKNOWN } } }
I20240430 20:11:51.741505 19942 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 0368d37dd9e2452982fb81f539c6805b (table default.Table_404b186c544e4b30bb36cb77d1a14120 [id=4330626e342a48699ea939add5c91b11]). Expected version 1 got 0
I20240430 20:11:51.741753 19942 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet ed0c37a71ede4bfc9eb5ad1a4365f25e (table default.Table_a2e85efc6b454524b0b122102ec07002 [id=86d795a82e674d859104741ebd04f4e8]). Expected version 1 got 0
I20240430 20:11:51.741847 21198 leader_election.cc:288] T 6b51cbc731794083a88fe4bed1ddeebf P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Requested vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:51.742169 18988 tablet_bootstrap.cc:492] T 489dc838744a49eba542009c86688aa5 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:51.742146 19942 catalog_manager.cc:5484] T b285e3eb16d148779c74e057458d8461 P 3c51c33d5616482cafb68e26688820f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c51c33d5616482cafb68e26688820f4 (127.17.177.3). New cstate: current_term: 1 leader_uuid: "3c51c33d5616482cafb68e26688820f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } health_report { overall_health: UNKNOWN } } }
I20240430 20:11:51.742524 19942 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet b285e3eb16d148779c74e057458d8461 (table default.Table_9df4a0b4170f4f28a6c17b0cb052ead6 [id=d1655c507b434f129ad7ae0b04365e51]). Expected version 2 got 0
I20240430 20:11:51.743196 19942 catalog_manager.cc:5484] T 2a08e36c39824abf98b9b6214c8b84cb P 3c51c33d5616482cafb68e26688820f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c51c33d5616482cafb68e26688820f4 (127.17.177.3). New cstate: current_term: 1 leader_uuid: "3c51c33d5616482cafb68e26688820f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } health_report { overall_health: HEALTHY } } }
I20240430 20:11:51.743947 19942 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 130007acc8ba4e709c3eb85fabe75903 (table default.Table_78987875650940c293247bc43db345ab [id=36f478a20a1f4a53a52fc29ade62b7f6]). Expected version 1 got 0
I20240430 20:11:51.744442 19942 catalog_manager.cc:5484] T fda942d50b014f3d9589e1fb909fc2fc P 3c51c33d5616482cafb68e26688820f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c51c33d5616482cafb68e26688820f4 (127.17.177.3). New cstate: current_term: 1 leader_uuid: "3c51c33d5616482cafb68e26688820f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } health_report { overall_health: UNKNOWN } } }
I20240430 20:11:51.745849 19942 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 3e5225f1f5f544ecbf8dfa470af3daca (table default.Table_c7c87a8e019944cab3ec99e9563ac01a [id=abf3765074eb4d7d9d2bcc4ab721e6bd]). Expected version 1 got 0
I20240430 20:11:51.747305 19942 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 8e91920e7e36418d8c27d1f0bfd9b981 (table default.Table_c8b25b14d955429cbd79515081deceb1 [id=f393bcc9669c4ea985a2d40deb161f0c]). Expected version 2 got 0
I20240430 20:11:51.747987 19942 catalog_manager.cc:5484] T 12795670f21447a8964fe1efc6e92398 P 3c51c33d5616482cafb68e26688820f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c51c33d5616482cafb68e26688820f4 (127.17.177.3). New cstate: current_term: 1 leader_uuid: "3c51c33d5616482cafb68e26688820f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } health_report { overall_health: UNKNOWN } } }
I20240430 20:11:51.749079 21220 consensus_queue.cc:1035] T ed0c37a71ede4bfc9eb5ad1a4365f25e P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:51.749209 19942 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 12795670f21447a8964fe1efc6e92398 (table default.Table_896bbae1f6944005b0272ca8ebc97218 [id=d5cea90cbc2c48df9d8b80da1eccc949]). Expected version 3 got 0
I20240430 20:11:51.749898 21220 raft_consensus.cc:482] T 62732af95c6a4c749c5b726f0460dc72 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:51.749925 19944 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: Sending DeleteTablet for 3 replicas of tablet 1701a15b115741aea13f8327e3d26b34
I20240430 20:11:51.750048 21220 raft_consensus.cc:504] T 62732af95c6a4c749c5b726f0460dc72 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:51.750478 19944 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:50746 (request call id 54) took 1361 ms. Trace:
I20240430 20:11:51.750839 21220 leader_election.cc:288] T 62732af95c6a4c749c5b726f0460dc72 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:51.750620 19944 rpcz_store.cc:276] 0430 20:11:50.388905 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:50.814216 (+425311us) service_pool.cc:225] Handling call
0430 20:11:50.814780 (+ 564us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0430 20:11:50.814945 (+ 165us) catalog_manager.cc:2749] Modifying in-memory table state
0430 20:11:50.815040 (+ 95us) catalog_manager.cc:2758] Locking tablets
0430 20:11:51.419074 (+604034us) catalog_manager.cc:2772] Removing table and tablets from system table
0430 20:11:51.426851 (+ 7777us) catalog_manager.cc:2790] Removing table from by-name map
0430 20:11:51.426953 (+ 102us) catalog_manager.cc:2806] Committing in-memory state
0430 20:11:51.749862 (+322909us) catalog_manager.cc:2810] Aborting table tasks
0430 20:11:51.750440 (+ 578us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:51.419933 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:51.420071 (+ 138us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:51.420078 (+ 7us) write_op.cc:427] Acquired schema lock
0430 20:11:51.420082 (+ 4us) tablet.cc:660] Decoding operations
0430 20:11:51.420219 (+ 137us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:51.420233 (+ 14us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:51.420238 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0430 20:11:51.420345 (+ 107us) tablet.cc:699] Row locks acquired
0430 20:11:51.420347 (+ 2us) write_op.cc:260] PREPARE: finished
0430 20:11:51.420393 (+ 46us) write_op.cc:270] Start()
0430 20:11:51.420430 (+ 37us) write_op.cc:276] Timestamp: P: 1714507911420345 usec, L: 0
0430 20:11:51.420436 (+ 6us) op_driver.cc:331] REPLICATION: starting
0430 20:11:51.420619 (+ 183us) log.cc:842] Serialized 759 byte log entry
0430 20:11:51.425686 (+ 5067us) op_driver.cc:444] REPLICATION: finished
0430 20:11:51.426147 (+ 461us) write_op.cc:301] APPLY: starting
0430 20:11:51.426164 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:51.426211 (+ 47us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:51.426250 (+ 39us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:51.426330 (+ 80us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:51.426369 (+ 39us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0430 20:11:51.426390 (+ 21us) write_op.cc:312] APPLY: finished
0430 20:11:51.426460 (+ 70us) log.cc:842] Serialized 29 byte log entry
0430 20:11:51.426518 (+ 58us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:51.426539 (+ 21us) write_op.cc:446] Released schema lock
0430 20:11:51.426548 (+ 9us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:51.426997 (+ 449us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:51.427002 (+ 5us) write_op.cc:446] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":397,"apply.run_cpu_time_us":556,"apply.run_wall_time_us":915,"num_ops":2,"prepare.queue_time_us":63,"prepare.run_cpu_time_us":808,"prepare.run_wall_time_us":808,"raft.queue_time_us":223,"raft.run_cpu_time_us":290,"raft.run_wall_time_us":290,"replication_time_us":5222}]]}
I20240430 20:11:51.751531 19944 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50746:
name: "default.Table_22d93bfc850f42bab0f67f0e0545601f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:51.752148 19944 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_22d93bfc850f42bab0f67f0e0545601f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:51.752240 21218 tablet.cc:1722] T 8e91920e7e36418d8c27d1f0bfd9b981 P 5e1c6397f09747f091534faf93542710: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240430 20:11:51.756719 18983 ts_tablet_manager.cc:1398] T aa9998f4e17147aa8f2b35c01980463a P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.099s user 0.006s sys 0.003s
I20240430 20:11:51.756747 18445 tablet_service.cc:1514] Processing DeleteTablet for tablet 1701a15b115741aea13f8327e3d26b34 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:50 UTC) from {username='slave'} at 127.0.0.1:41450
I20240430 20:11:51.757022 18983 tablet_bootstrap.cc:492] T d4d6c37b578c4fe1aa666b0a6b0bc7c9 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:51.758870 18983 tablet_bootstrap.cc:654] T d4d6c37b578c4fe1aa666b0a6b0bc7c9 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:51.759428 21108 tablet_replica.cc:330] T 1701a15b115741aea13f8327e3d26b34 P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:51.759732 21108 raft_consensus.cc:2229] T 1701a15b115741aea13f8327e3d26b34 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:51.759959 21108 raft_consensus.cc:2258] T 1701a15b115741aea13f8327e3d26b34 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:51.760619 21108 ts_tablet_manager.cc:1875] T 1701a15b115741aea13f8327e3d26b34 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240430 20:11:51.768738 18417 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.177.3:44867, user_credentials={real_user=slave}} blocked reactor thread for 56124.9us
I20240430 20:11:51.769572 20630 consensus_queue.cc:1035] T 1f4e860484784aa9a2d28af2e9fb611c P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240430 20:11:51.771349 18417 leader_election.cc:341] T 2fc6908ecf954020a676b9d10a2b00e7 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Not found: Tablet not found: 2fc6908ecf954020a676b9d10a2b00e7
I20240430 20:11:51.771628 18619 raft_consensus.cc:1261] T 488d0e609b224674932f890939664df2 P 5e1c6397f09747f091534faf93542710 [term 2 FOLLOWER]: Refusing update from remote peer 2efb542aa55e439b99ccb98d1c5c884a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20240430 20:11:51.771632 18417 leader_election.cc:302] T 2fc6908ecf954020a676b9d10a2b00e7 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a; no voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710
I20240430 20:11:51.772341 20630 raft_consensus.cc:2726] T 2fc6908ecf954020a676b9d10a2b00e7 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:51.772816 20630 consensus_queue.cc:1035] T 488d0e609b224674932f890939664df2 P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:51.780918 21108 ts_tablet_manager.cc:1888] T 1701a15b115741aea13f8327e3d26b34 P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:51.781188 18983 tablet_bootstrap.cc:492] T d4d6c37b578c4fe1aa666b0a6b0bc7c9 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:51.781343 18983 ts_tablet_manager.cc:1367] T d4d6c37b578c4fe1aa666b0a6b0bc7c9 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.002s
I20240430 20:11:51.781389 21108 log.cc:1196] T 1701a15b115741aea13f8327e3d26b34 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/1701a15b115741aea13f8327e3d26b34
I20240430 20:11:51.781534 21055 raft_consensus.cc:482] T 898b2c91adf34545808df9c786e218c9 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:51.781733 21055 raft_consensus.cc:504] T 898b2c91adf34545808df9c786e218c9 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:51.782280 21108 ts_tablet_manager.cc:1909] T 1701a15b115741aea13f8327e3d26b34 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:51.782524 21055 leader_election.cc:288] T 898b2c91adf34545808df9c786e218c9 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:51.786075 19902 catalog_manager.cc:4830] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): tablet 1701a15b115741aea13f8327e3d26b34 (table default.Table_3776a4b9d4f24aa6b7f882bb4d0116ef [id=7127dbd368864f7ba80a755549fe7ede]) successfully deleted
I20240430 20:11:51.789031 18985 tablet_bootstrap.cc:492] T 8901327b3efb4201adce61a9266142ea P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:51.790498 18985 tablet_bootstrap.cc:654] T 8901327b3efb4201adce61a9266142ea P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:51.799353 18616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "898b2c91adf34545808df9c786e218c9" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:51.799651 18616 raft_consensus.cc:2379] T 898b2c91adf34545808df9c786e218c9 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2efb542aa55e439b99ccb98d1c5c884a in current term 1: Already voted for candidate 3c51c33d5616482cafb68e26688820f4 in this term.
I20240430 20:11:51.800912 18985 tablet_bootstrap.cc:492] T 8901327b3efb4201adce61a9266142ea P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:51.801074 18985 ts_tablet_manager.cc:1367] T 8901327b3efb4201adce61a9266142ea P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.002s
I20240430 20:11:51.801756 18985 raft_consensus.cc:348] T 8901327b3efb4201adce61a9266142ea P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:51.801973 18985 raft_consensus.cc:374] T 8901327b3efb4201adce61a9266142ea P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:51.802042 18985 raft_consensus.cc:729] T 8901327b3efb4201adce61a9266142ea P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:51.805290 18985 consensus_queue.cc:260] T 8901327b3efb4201adce61a9266142ea P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:51.805932 18985 ts_tablet_manager.cc:1398] T 8901327b3efb4201adce61a9266142ea P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240430 20:11:51.806496 18985 tablet_bootstrap.cc:492] T 62732af95c6a4c749c5b726f0460dc72 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:51.808128 18985 tablet_bootstrap.cc:654] T 62732af95c6a4c749c5b726f0460dc72 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:51.809269 21203 ts_tablet_manager.cc:1888] T 7bde1d6934784c9aa5911181e8df9eeb P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.2
I20240430 20:11:51.809768 21203 log.cc:1196] T 7bde1d6934784c9aa5911181e8df9eeb P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/7bde1d6934784c9aa5911181e8df9eeb
I20240430 20:11:51.810468 20489 raft_consensus.cc:482] T 35fdb052135541cfb42a8aeb03b64dc8 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:51.810658 20489 raft_consensus.cc:504] T 35fdb052135541cfb42a8aeb03b64dc8 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:51.811247 20489 leader_election.cc:288] T 35fdb052135541cfb42a8aeb03b64dc8 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:51.816372 18730 tablet_service.cc:1514] Processing DeleteTablet for tablet 1701a15b115741aea13f8327e3d26b34 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:50 UTC) from {username='slave'} at 127.0.0.1:56540
I20240430 20:11:51.818037 18985 tablet_bootstrap.cc:492] T 62732af95c6a4c749c5b726f0460dc72 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:51.818203 18985 ts_tablet_manager.cc:1367] T 62732af95c6a4c749c5b726f0460dc72 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.000s
I20240430 20:11:51.820437 21055 raft_consensus.cc:482] T 8901327b3efb4201adce61a9266142ea P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:51.820565 21055 raft_consensus.cc:504] T 8901327b3efb4201adce61a9266142ea P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:51.821242 21055 leader_election.cc:288] T 8901327b3efb4201adce61a9266142ea P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:51.822057 18616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8901327b3efb4201adce61a9266142ea" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:51.822274 18616 raft_consensus.cc:2454] T 8901327b3efb4201adce61a9266142ea P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 0.
I20240430 20:11:51.820122 18985 raft_consensus.cc:348] T 62732af95c6a4c749c5b726f0460dc72 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:51.822619 18985 raft_consensus.cc:374] T 62732af95c6a4c749c5b726f0460dc72 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:51.822700 18985 raft_consensus.cc:729] T 62732af95c6a4c749c5b726f0460dc72 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:51.822907 18415 leader_election.cc:302] T 8901327b3efb4201adce61a9266142ea P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:51.822935 18985 consensus_queue.cc:260] T 62732af95c6a4c749c5b726f0460dc72 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:51.823334 21049 raft_consensus.cc:2781] T 8901327b3efb4201adce61a9266142ea P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:51.823415 21049 raft_consensus.cc:482] T 8901327b3efb4201adce61a9266142ea P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:51.823478 21049 raft_consensus.cc:3037] T 8901327b3efb4201adce61a9266142ea P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:51.823514 18985 ts_tablet_manager.cc:1398] T 62732af95c6a4c749c5b726f0460dc72 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240430 20:11:51.823813 18985 tablet_bootstrap.cc:492] T 6b51cbc731794083a88fe4bed1ddeebf P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:51.825577 18985 tablet_bootstrap.cc:654] T 6b51cbc731794083a88fe4bed1ddeebf P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:51.827098 21049 raft_consensus.cc:504] T 8901327b3efb4201adce61a9266142ea P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:51.828145 21049 leader_election.cc:288] T 8901327b3efb4201adce61a9266142ea P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:51.829046 18616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8901327b3efb4201adce61a9266142ea" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:51.829267 18616 raft_consensus.cc:3037] T 8901327b3efb4201adce61a9266142ea P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:51.832612 18616 raft_consensus.cc:2454] T 8901327b3efb4201adce61a9266142ea P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 1.
I20240430 20:11:51.833421 18415 leader_election.cc:302] T 8901327b3efb4201adce61a9266142ea P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:51.833911 21049 raft_consensus.cc:2781] T 8901327b3efb4201adce61a9266142ea P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:51.834100 21049 raft_consensus.cc:686] T 8901327b3efb4201adce61a9266142ea P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Becoming Leader. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Running, Role: LEADER
I20240430 20:11:51.834448 21049 consensus_queue.cc:237] T 8901327b3efb4201adce61a9266142ea P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:51.843967 18758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "417351cbc9d54f9d84bb7155f39da119" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:51.844995 21055 raft_consensus.cc:482] T 9f0b341a31224bf1be525b84b9500304 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:51.845155 21055 raft_consensus.cc:504] T 9f0b341a31224bf1be525b84b9500304 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:51.845875 21055 leader_election.cc:288] T 9f0b341a31224bf1be525b84b9500304 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
W20240430 20:11:51.847018 18415 leader_election.cc:341] T 9f0b341a31224bf1be525b84b9500304 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Not found: Tablet not found: 9f0b341a31224bf1be525b84b9500304
I20240430 20:11:51.848304 18985 tablet_bootstrap.cc:492] T 6b51cbc731794083a88fe4bed1ddeebf P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:51.848453 18985 ts_tablet_manager.cc:1367] T 6b51cbc731794083a88fe4bed1ddeebf P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.025s user 0.006s sys 0.003s
I20240430 20:11:51.849179 18985 raft_consensus.cc:348] T 6b51cbc731794083a88fe4bed1ddeebf P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:51.849371 18985 raft_consensus.cc:374] T 6b51cbc731794083a88fe4bed1ddeebf P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:51.849428 18985 raft_consensus.cc:729] T 6b51cbc731794083a88fe4bed1ddeebf P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:51.849622 18985 consensus_queue.cc:260] T 6b51cbc731794083a88fe4bed1ddeebf P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:51.850749 18985 ts_tablet_manager.cc:1398] T 6b51cbc731794083a88fe4bed1ddeebf P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240430 20:11:51.851014 18985 tablet_bootstrap.cc:492] T 35fdb052135541cfb42a8aeb03b64dc8 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:51.852607 18985 tablet_bootstrap.cc:654] T 35fdb052135541cfb42a8aeb03b64dc8 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:51.854632 18985 tablet_bootstrap.cc:492] T 35fdb052135541cfb42a8aeb03b64dc8 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:51.854763 18985 ts_tablet_manager.cc:1367] T 35fdb052135541cfb42a8aeb03b64dc8 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240430 20:11:51.856236 18983 raft_consensus.cc:348] T d4d6c37b578c4fe1aa666b0a6b0bc7c9 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:51.856441 18983 raft_consensus.cc:374] T d4d6c37b578c4fe1aa666b0a6b0bc7c9 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:51.856510 18983 raft_consensus.cc:729] T d4d6c37b578c4fe1aa666b0a6b0bc7c9 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:51.856710 18983 consensus_queue.cc:260] T d4d6c37b578c4fe1aa666b0a6b0bc7c9 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:51.857235 18983 ts_tablet_manager.cc:1398] T d4d6c37b578c4fe1aa666b0a6b0bc7c9 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.076s user 0.001s sys 0.001s
I20240430 20:11:51.857501 18983 tablet_bootstrap.cc:492] T 3a03b6997921408cb5a65f4cae223e9c P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:51.860033 18983 tablet_bootstrap.cc:654] T 3a03b6997921408cb5a65f4cae223e9c P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:51.862363 18985 raft_consensus.cc:348] T 35fdb052135541cfb42a8aeb03b64dc8 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:51.862895 18988 tablet_bootstrap.cc:654] T 489dc838744a49eba542009c86688aa5 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:51.862720 18985 raft_consensus.cc:374] T 35fdb052135541cfb42a8aeb03b64dc8 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:51.872727 18985 raft_consensus.cc:729] T 35fdb052135541cfb42a8aeb03b64dc8 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:51.873304 18985 consensus_queue.cc:260] T 35fdb052135541cfb42a8aeb03b64dc8 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:51.874231 18985 ts_tablet_manager.cc:1398] T 35fdb052135541cfb42a8aeb03b64dc8 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.019s user 0.002s sys 0.000s
I20240430 20:11:51.875032 18985 tablet_bootstrap.cc:492] T 489dc838744a49eba542009c86688aa5 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:51.877022 18985 tablet_bootstrap.cc:654] T 489dc838744a49eba542009c86688aa5 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:51.885874 21203 ts_tablet_manager.cc:1909] T 7bde1d6934784c9aa5911181e8df9eeb P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:51.886679 19900 catalog_manager.cc:4830] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): tablet 7bde1d6934784c9aa5911181e8df9eeb (table default.Table_7199d7c36d3f45f79cf9e4e2b4145592 [id=f6aef8f6a9164dd78f49e03f723b45a9]) successfully deleted
I20240430 20:11:51.889859 18985 tablet_bootstrap.cc:492] T 489dc838744a49eba542009c86688aa5 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:51.890815 18983 tablet_bootstrap.cc:492] T 3a03b6997921408cb5a65f4cae223e9c P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:51.891039 18983 ts_tablet_manager.cc:1367] T 3a03b6997921408cb5a65f4cae223e9c P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.034s user 0.004s sys 0.004s
I20240430 20:11:51.891855 18983 raft_consensus.cc:348] T 3a03b6997921408cb5a65f4cae223e9c P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:51.892143 18983 raft_consensus.cc:374] T 3a03b6997921408cb5a65f4cae223e9c P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:51.892249 18983 raft_consensus.cc:729] T 3a03b6997921408cb5a65f4cae223e9c P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:51.892514 18983 consensus_queue.cc:260] T 3a03b6997921408cb5a65f4cae223e9c P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
W20240430 20:11:51.893369 18558 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.177.1:41217, user_credentials={real_user=slave}} blocked reactor thread for 154489us
I20240430 20:11:51.893648 18985 ts_tablet_manager.cc:1367] T 489dc838744a49eba542009c86688aa5 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.003s
I20240430 20:11:51.893441 18983 ts_tablet_manager.cc:1398] T 3a03b6997921408cb5a65f4cae223e9c P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240430 20:11:51.894433 18985 raft_consensus.cc:348] T 489dc838744a49eba542009c86688aa5 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:51.894794 18985 raft_consensus.cc:374] T 489dc838744a49eba542009c86688aa5 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:51.894914 18985 raft_consensus.cc:729] T 489dc838744a49eba542009c86688aa5 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:51.894945 18472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62732af95c6a4c749c5b726f0460dc72" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:51.895208 18472 raft_consensus.cc:2454] T 62732af95c6a4c749c5b726f0460dc72 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 0.
I20240430 20:11:51.895207 18985 consensus_queue.cc:260] T 489dc838744a49eba542009c86688aa5 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:51.896891 18985 ts_tablet_manager.cc:1398] T 489dc838744a49eba542009c86688aa5 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240430 20:11:51.898362 18985 tablet_bootstrap.cc:492] T eb6c96047b114ebba73db7a371f2da8e P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:51.900031 18985 tablet_bootstrap.cc:654] T eb6c96047b114ebba73db7a371f2da8e P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:51.906375 18763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb6c96047b114ebba73db7a371f2da8e" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:51.906654 18763 raft_consensus.cc:2454] T eb6c96047b114ebba73db7a371f2da8e P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 0.
I20240430 20:11:51.906980 18417 leader_election.cc:302] T 417351cbc9d54f9d84bb7155f39da119 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a; no voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710
I20240430 20:11:51.907522 20630 raft_consensus.cc:2726] T 417351cbc9d54f9d84bb7155f39da119 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:51.907773 18764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b51cbc731794083a88fe4bed1ddeebf" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:51.908017 18764 raft_consensus.cc:2379] T 6b51cbc731794083a88fe4bed1ddeebf P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e1c6397f09747f091534faf93542710 in current term 1: Already voted for candidate 3c51c33d5616482cafb68e26688820f4 in this term.
I20240430 20:11:51.908103 21203 tablet_replica.cc:330] T 1701a15b115741aea13f8327e3d26b34 P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:51.915741 21229 raft_consensus.cc:482] T aa9998f4e17147aa8f2b35c01980463a P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:51.917673 19938 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:50746 (request call id 57) took 1528 ms. Trace:
I20240430 20:11:51.918128 18472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35fdb052135541cfb42a8aeb03b64dc8" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:51.916849 21229 raft_consensus.cc:504] T aa9998f4e17147aa8f2b35c01980463a P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:51.918340 18472 raft_consensus.cc:2454] T 35fdb052135541cfb42a8aeb03b64dc8 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 0.
I20240430 20:11:51.918042 19938 rpcz_store.cc:276] 0430 20:11:50.389136 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:50.844695 (+455559us) service_pool.cc:225] Handling call
0430 20:11:50.845096 (+ 401us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0430 20:11:50.845361 (+ 265us) catalog_manager.cc:3613] Apply alter schema
0430 20:11:50.845533 (+ 172us) catalog_manager.cc:3636] Acquired catalog manager lock
0430 20:11:50.845587 (+ 54us) catalog_manager.cc:3783] Updating metadata on disk
0430 20:11:50.900580 (+ 54993us) catalog_manager.cc:3814] Committing alterations to in-memory state
0430 20:11:51.917627 (+1017047us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:50.846028 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:50.847213 (+ 1185us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:50.847221 (+ 8us) write_op.cc:427] Acquired schema lock
0430 20:11:50.847226 (+ 5us) tablet.cc:660] Decoding operations
0430 20:11:50.847355 (+ 129us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:50.847368 (+ 13us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:50.847376 (+ 8us) tablet.cc:683] Acquiring locks for 1 operations
0430 20:11:50.847451 (+ 75us) tablet.cc:699] Row locks acquired
0430 20:11:50.847455 (+ 4us) write_op.cc:260] PREPARE: finished
0430 20:11:50.847494 (+ 39us) write_op.cc:270] Start()
0430 20:11:50.847531 (+ 37us) write_op.cc:276] Timestamp: P: 1714507910847455 usec, L: 0
0430 20:11:50.847537 (+ 6us) op_driver.cc:331] REPLICATION: starting
0430 20:11:50.847759 (+ 222us) log.cc:842] Serialized 405 byte log entry
0430 20:11:50.899084 (+ 51325us) op_driver.cc:444] REPLICATION: finished
0430 20:11:50.899784 (+ 700us) write_op.cc:301] APPLY: starting
0430 20:11:50.899794 (+ 10us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:50.899821 (+ 27us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:50.899857 (+ 36us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:50.899896 (+ 39us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:50.899921 (+ 25us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0430 20:11:50.899928 (+ 7us) write_op.cc:312] APPLY: finished
0430 20:11:50.899985 (+ 57us) log.cc:842] Serialized 23 byte log entry
0430 20:11:50.900021 (+ 36us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:50.900034 (+ 13us) write_op.cc:446] Released schema lock
0430 20:11:50.900038 (+ 4us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:50.900081 (+ 43us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:50.900085 (+ 4us) write_op.cc:446] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":671,"apply.run_cpu_time_us":337,"apply.run_wall_time_us":338,"num_ops":1,"prepare.queue_time_us":42,"prepare.run_cpu_time_us":789,"prepare.run_wall_time_us":2205,"replication_time_us":51531}]]}
I20240430 20:11:51.921516 19946 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50746:
table { table_name: "default.Table_c7c87a8e019944cab3ec99e9563ac01a" } modify_external_catalogs: true
I20240430 20:11:51.922124 19937 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:50746 (request call id 55) took 1533 ms. Trace:
I20240430 20:11:51.922307 21229 leader_election.cc:288] T aa9998f4e17147aa8f2b35c01980463a P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:51.922319 19937 rpcz_store.cc:276] 0430 20:11:50.388995 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:50.841702 (+452707us) service_pool.cc:225] Handling call
0430 20:11:50.842106 (+ 404us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0430 20:11:50.842529 (+ 423us) catalog_manager.cc:3613] Apply alter schema
0430 20:11:50.842786 (+ 257us) catalog_manager.cc:3636] Acquired catalog manager lock
0430 20:11:50.842909 (+ 123us) catalog_manager.cc:3783] Updating metadata on disk
0430 20:11:50.899731 (+ 56822us) catalog_manager.cc:3814] Committing alterations to in-memory state
0430 20:11:51.922092 (+1022361us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:50.844598 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:50.844748 (+ 150us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:50.844756 (+ 8us) write_op.cc:427] Acquired schema lock
0430 20:11:50.844760 (+ 4us) tablet.cc:660] Decoding operations
0430 20:11:50.844923 (+ 163us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:50.844942 (+ 19us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:50.844949 (+ 7us) tablet.cc:683] Acquiring locks for 1 operations
0430 20:11:50.845043 (+ 94us) tablet.cc:699] Row locks acquired
0430 20:11:50.845047 (+ 4us) write_op.cc:260] PREPARE: finished
0430 20:11:50.845094 (+ 47us) write_op.cc:270] Start()
0430 20:11:50.845134 (+ 40us) write_op.cc:276] Timestamp: P: 1714507910845055 usec, L: 0
0430 20:11:50.845140 (+ 6us) op_driver.cc:331] REPLICATION: starting
0430 20:11:50.845377 (+ 237us) log.cc:842] Serialized 405 byte log entry
0430 20:11:50.899016 (+ 53639us) op_driver.cc:444] REPLICATION: finished
0430 20:11:50.899241 (+ 225us) write_op.cc:301] APPLY: starting
0430 20:11:50.899259 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:50.899304 (+ 45us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:50.899334 (+ 30us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:50.899389 (+ 55us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:50.899426 (+ 37us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0430 20:11:50.899434 (+ 8us) write_op.cc:312] APPLY: finished
0430 20:11:50.899526 (+ 92us) log.cc:842] Serialized 23 byte log entry
0430 20:11:50.899597 (+ 71us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:50.899618 (+ 21us) write_op.cc:446] Released schema lock
0430 20:11:50.899626 (+ 8us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:50.899695 (+ 69us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:50.899699 (+ 4us) write_op.cc:446] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":175,"apply.run_cpu_time_us":509,"apply.run_wall_time_us":508,"num_ops":1,"prepare.queue_time_us":999,"prepare.run_cpu_time_us":906,"prepare.run_wall_time_us":1386,"raft.queue_time_us":46,"raft.run_cpu_time_us":5,"raft.run_wall_time_us":6,"replication_time_us":53844}]]}
I20240430 20:11:51.923776 19937 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50746:
table { table_name: "default.Table_673d1448679f4b15a5ee2a1369c8f8ea" } new_table_name: "default.Table_6f8e0eff8f5a484dbd6a9c3fdb5d03f6" modify_external_catalogs: true
I20240430 20:11:51.926136 19940 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:50746 (request call id 56) took 1537 ms. Trace:
I20240430 20:11:51.926442 18985 tablet_bootstrap.cc:492] T eb6c96047b114ebba73db7a371f2da8e P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:51.926682 18985 ts_tablet_manager.cc:1367] T eb6c96047b114ebba73db7a371f2da8e P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.028s user 0.006s sys 0.004s
I20240430 20:11:51.926303 19940 rpcz_store.cc:276] 0430 20:11:50.389074 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:50.843807 (+454733us) service_pool.cc:225] Handling call
0430 20:11:50.844494 (+ 687us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0430 20:11:50.848541 (+ 4047us) catalog_manager.cc:3613] Apply alter schema
0430 20:11:50.848754 (+ 213us) catalog_manager.cc:3636] Acquired catalog manager lock
0430 20:11:50.848856 (+ 102us) catalog_manager.cc:3783] Updating metadata on disk
0430 20:11:50.901493 (+ 52637us) catalog_manager.cc:3814] Committing alterations to in-memory state
0430 20:11:51.926082 (+1024589us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:50.849580 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:50.849702 (+ 122us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:50.849711 (+ 9us) write_op.cc:427] Acquired schema lock
0430 20:11:50.849715 (+ 4us) tablet.cc:660] Decoding operations
0430 20:11:50.849833 (+ 118us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:50.849847 (+ 14us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:50.849856 (+ 9us) tablet.cc:683] Acquiring locks for 1 operations
0430 20:11:50.849931 (+ 75us) tablet.cc:699] Row locks acquired
0430 20:11:50.849935 (+ 4us) write_op.cc:260] PREPARE: finished
0430 20:11:50.849988 (+ 53us) write_op.cc:270] Start()
0430 20:11:50.850027 (+ 39us) write_op.cc:276] Timestamp: P: 1714507910849949 usec, L: 0
0430 20:11:50.850032 (+ 5us) op_driver.cc:331] REPLICATION: starting
0430 20:11:50.850226 (+ 194us) log.cc:842] Serialized 405 byte log entry
0430 20:11:50.899124 (+ 48898us) op_driver.cc:444] REPLICATION: finished
0430 20:11:50.900160 (+ 1036us) write_op.cc:301] APPLY: starting
0430 20:11:50.900169 (+ 9us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:50.900193 (+ 24us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:50.900220 (+ 27us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:50.900259 (+ 39us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:50.900279 (+ 20us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0430 20:11:50.900286 (+ 7us) write_op.cc:312] APPLY: finished
0430 20:11:50.900336 (+ 50us) log.cc:842] Serialized 23 byte log entry
0430 20:11:50.900376 (+ 40us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:50.900391 (+ 15us) write_op.cc:446] Released schema lock
0430 20:11:50.900398 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:50.900445 (+ 47us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:50.900451 (+ 6us) write_op.cc:446] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":997,"apply.run_cpu_time_us":342,"apply.run_wall_time_us":342,"num_ops":1,"prepare.queue_time_us":99,"prepare.run_cpu_time_us":768,"prepare.run_wall_time_us":1247,"raft.queue_time_us":45,"raft.run_cpu_time_us":5,"raft.run_wall_time_us":6,"replication_time_us":49079}]]}
I20240430 20:11:51.927474 18985 raft_consensus.cc:348] T eb6c96047b114ebba73db7a371f2da8e P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:51.927765 19940 master_service.cc:946] ReplaceTablet: received request to replace tablet ebf74809b52f47f3b0ed21b6ac0f541a from {username='slave'} at 127.0.0.1:50746
I20240430 20:11:51.927767 21225 raft_consensus.cc:482] T 89b774967624498fa2be499878b71124 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:51.927942 21225 raft_consensus.cc:504] T 89b774967624498fa2be499878b71124 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:51.928531 21225 leader_election.cc:288] T 89b774967624498fa2be499878b71124 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:51.928704 18985 raft_consensus.cc:374] T eb6c96047b114ebba73db7a371f2da8e P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:51.928817 18985 raft_consensus.cc:729] T eb6c96047b114ebba73db7a371f2da8e P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:51.929025 19941 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50746:
table { table_name: "default.Table_96aea8c4ff80428da8a0394f1bf7b493" } modify_external_catalogs: true
I20240430 20:11:51.929121 18985 consensus_queue.cc:260] T eb6c96047b114ebba73db7a371f2da8e P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:51.929725 18985 ts_tablet_manager.cc:1398] T eb6c96047b114ebba73db7a371f2da8e P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240430 20:11:51.930060 18985 tablet_bootstrap.cc:492] T 2aa2a489116940dea3173cad16d94fa5 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:51.931674 18985 tablet_bootstrap.cc:654] T 2aa2a489116940dea3173cad16d94fa5 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:51.937016 18616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b51cbc731794083a88fe4bed1ddeebf" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:51.937230 18616 raft_consensus.cc:3037] T 6b51cbc731794083a88fe4bed1ddeebf P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:51.938156 21225 raft_consensus.cc:482] T 489dc838744a49eba542009c86688aa5 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:51.938294 21225 raft_consensus.cc:504] T 489dc838744a49eba542009c86688aa5 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:51.939296 21203 raft_consensus.cc:2229] T 1701a15b115741aea13f8327e3d26b34 P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Raft consensus shutting down.
I20240430 20:11:51.939342 21225 leader_election.cc:288] T 489dc838744a49eba542009c86688aa5 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:51.939750 21203 raft_consensus.cc:2258] T 1701a15b115741aea13f8327e3d26b34 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:51.942850 18616 raft_consensus.cc:2454] T 6b51cbc731794083a88fe4bed1ddeebf P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:51.946846 18985 tablet_bootstrap.cc:492] T 2aa2a489116940dea3173cad16d94fa5 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:51.947008 18985 ts_tablet_manager.cc:1367] T 2aa2a489116940dea3173cad16d94fa5 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.000s
I20240430 20:11:51.947736 18985 raft_consensus.cc:348] T 2aa2a489116940dea3173cad16d94fa5 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:51.947952 18985 raft_consensus.cc:374] T 2aa2a489116940dea3173cad16d94fa5 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:51.947736 18472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa9998f4e17147aa8f2b35c01980463a" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:51.948057 18985 raft_consensus.cc:729] T 2aa2a489116940dea3173cad16d94fa5 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:51.948408 18985 consensus_queue.cc:260] T 2aa2a489116940dea3173cad16d94fa5 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:51.949074 18985 ts_tablet_manager.cc:1398] T 2aa2a489116940dea3173cad16d94fa5 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240430 20:11:51.949211 18472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89b774967624498fa2be499878b71124" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:51.949398 18985 tablet_bootstrap.cc:492] T 3a03b6997921408cb5a65f4cae223e9c P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:51.951037 18985 tablet_bootstrap.cc:654] T 3a03b6997921408cb5a65f4cae223e9c P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:51.954835 18472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "489dc838744a49eba542009c86688aa5" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:51.955096 18472 raft_consensus.cc:2454] T 489dc838744a49eba542009c86688aa5 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 0.
I20240430 20:11:51.956480 18696 leader_election.cc:302] T 6b51cbc731794083a88fe4bed1ddeebf P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:51.956849 20198 raft_consensus.cc:2781] T 6b51cbc731794083a88fe4bed1ddeebf P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:51.956986 20198 raft_consensus.cc:686] T 6b51cbc731794083a88fe4bed1ddeebf P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Becoming Leader. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Running, Role: LEADER
I20240430 20:11:51.957237 20198 consensus_queue.cc:237] T 6b51cbc731794083a88fe4bed1ddeebf P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:51.964645 19944 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:50746 (request call id 60) took 1371 ms. Trace:
I20240430 20:11:51.964866 19944 rpcz_store.cc:276] 0430 20:11:50.593411 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:51.751236 (+1157825us) service_pool.cc:225] Handling call
0430 20:11:51.925100 (+173864us) catalog_manager.cc:2072] Acquired catalog manager lock
0430 20:11:51.925602 (+ 502us) catalog_manager.cc:2118] Created new table and tablet info
0430 20:11:51.964303 (+ 38701us) catalog_manager.cc:2178] Wrote table and tablets to system table
0430 20:11:51.964454 (+ 151us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0430 20:11:51.964588 (+ 134us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:51.937437 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:51.937553 (+ 116us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:51.937558 (+ 5us) write_op.cc:427] Acquired schema lock
0430 20:11:51.937560 (+ 2us) tablet.cc:660] Decoding operations
0430 20:11:51.937665 (+ 105us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:51.937675 (+ 10us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:51.937680 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0430 20:11:51.937755 (+ 75us) tablet.cc:699] Row locks acquired
0430 20:11:51.937758 (+ 3us) write_op.cc:260] PREPARE: finished
0430 20:11:51.937790 (+ 32us) write_op.cc:270] Start()
0430 20:11:51.937816 (+ 26us) write_op.cc:276] Timestamp: P: 1714507911937753 usec, L: 0
0430 20:11:51.937820 (+ 4us) op_driver.cc:331] REPLICATION: starting
0430 20:11:51.937982 (+ 162us) log.cc:842] Serialized 433 byte log entry
0430 20:11:51.945585 (+ 7603us) op_driver.cc:444] REPLICATION: finished
0430 20:11:51.945886 (+ 301us) write_op.cc:301] APPLY: starting
0430 20:11:51.945902 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:51.945944 (+ 42us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:51.945976 (+ 32us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:51.946092 (+ 116us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:51.946125 (+ 33us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0430 20:11:51.946132 (+ 7us) write_op.cc:312] APPLY: finished
0430 20:11:51.946192 (+ 60us) log.cc:842] Serialized 29 byte log entry
0430 20:11:51.946245 (+ 53us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:51.946262 (+ 17us) write_op.cc:446] Released schema lock
0430 20:11:51.946268 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:51.946313 (+ 45us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:51.946317 (+ 4us) write_op.cc:446] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":243,"apply.run_cpu_time_us":487,"apply.run_wall_time_us":487,"mutex_wait_us":11267,"num_ops":2,"prepare.queue_time_us":11380,"prepare.run_cpu_time_us":611,"prepare.run_wall_time_us":612,"replication_time_us":7739}]]}
I20240430 20:11:51.966357 19944 master_service.cc:946] ReplaceTablet: received request to replace tablet 13c2e69ae91f4957887fd64b90f6e435 from {username='slave'} at 127.0.0.1:50746
I20240430 20:11:51.967733 18985 tablet_bootstrap.cc:492] T 3a03b6997921408cb5a65f4cae223e9c P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:51.967897 18985 ts_tablet_manager.cc:1367] T 3a03b6997921408cb5a65f4cae223e9c P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.000s
I20240430 20:11:51.968649 18985 raft_consensus.cc:348] T 3a03b6997921408cb5a65f4cae223e9c P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:51.968852 18985 raft_consensus.cc:374] T 3a03b6997921408cb5a65f4cae223e9c P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:51.968919 18985 raft_consensus.cc:729] T 3a03b6997921408cb5a65f4cae223e9c P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:51.969144 18985 consensus_queue.cc:260] T 3a03b6997921408cb5a65f4cae223e9c P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:51.969805 18985 ts_tablet_manager.cc:1398] T 3a03b6997921408cb5a65f4cae223e9c P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240430 20:11:51.970115 18985 tablet_bootstrap.cc:492] T d4d6c37b578c4fe1aa666b0a6b0bc7c9 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:51.971732 18985 tablet_bootstrap.cc:654] T d4d6c37b578c4fe1aa666b0a6b0bc7c9 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:51.971971 19940 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: Sending DeleteTablet for 3 replicas of tablet ebf74809b52f47f3b0ed21b6ac0f541a
I20240430 20:11:51.972725 18730 tablet_service.cc:1514] Processing DeleteTablet for tablet ebf74809b52f47f3b0ed21b6ac0f541a with delete_type TABLET_DATA_DELETED (replaced by tablet 9e8ab75e7d9c4b64a7b7dd2678569490) from {username='slave'} at 127.0.0.1:56540
I20240430 20:11:51.973187 18558 leader_election.cc:302] T 62732af95c6a4c749c5b726f0460dc72 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:51.973645 21229 raft_consensus.cc:2781] T 62732af95c6a4c749c5b726f0460dc72 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:51.973757 19940 catalog_manager.cc:6554] ReplaceTablet: tablet ebf74809b52f47f3b0ed21b6ac0f541a deleted and replaced by tablet 9e8ab75e7d9c4b64a7b7dd2678569490
I20240430 20:11:51.973877 19940 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:50746 (request call id 62) took 1099 ms. Trace:
I20240430 20:11:51.973924 18445 tablet_service.cc:1514] Processing DeleteTablet for tablet ebf74809b52f47f3b0ed21b6ac0f541a with delete_type TABLET_DATA_DELETED (replaced by tablet 9e8ab75e7d9c4b64a7b7dd2678569490) from {username='slave'} at 127.0.0.1:41450
I20240430 20:11:51.973798 21229 raft_consensus.cc:482] T 62732af95c6a4c749c5b726f0460dc72 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:51.974171 21108 tablet_replica.cc:330] T ebf74809b52f47f3b0ed21b6ac0f541a P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:51.974417 21108 raft_consensus.cc:2229] T ebf74809b52f47f3b0ed21b6ac0f541a P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:51.974638 21108 raft_consensus.cc:2258] T ebf74809b52f47f3b0ed21b6ac0f541a P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:51.974056 19940 rpcz_store.cc:276] 0430 20:11:50.874178 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:51.927677 (+1053499us) service_pool.cc:225] Handling call
0430 20:11:51.973845 (+ 46168us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:51.938089 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:51.938172 (+ 83us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:51.938178 (+ 6us) write_op.cc:427] Acquired schema lock
0430 20:11:51.938180 (+ 2us) tablet.cc:660] Decoding operations
0430 20:11:51.938288 (+ 108us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:51.938298 (+ 10us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:51.938304 (+ 6us) tablet.cc:683] Acquiring locks for 2 operations
0430 20:11:51.938376 (+ 72us) tablet.cc:699] Row locks acquired
0430 20:11:51.938378 (+ 2us) write_op.cc:260] PREPARE: finished
0430 20:11:51.938425 (+ 47us) write_op.cc:270] Start()
0430 20:11:51.938455 (+ 30us) write_op.cc:276] Timestamp: P: 1714507911938385 usec, L: 0
0430 20:11:51.938459 (+ 4us) op_driver.cc:331] REPLICATION: starting
0430 20:11:51.938647 (+ 188us) log.cc:842] Serialized 659 byte log entry
0430 20:11:51.945644 (+ 6997us) op_driver.cc:444] REPLICATION: finished
0430 20:11:51.946404 (+ 760us) write_op.cc:301] APPLY: starting
0430 20:11:51.946413 (+ 9us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:51.946444 (+ 31us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:51.946466 (+ 22us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:51.946546 (+ 80us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:51.946626 (+ 80us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0430 20:11:51.946634 (+ 8us) write_op.cc:312] APPLY: finished
0430 20:11:51.946692 (+ 58us) log.cc:842] Serialized 29 byte log entry
0430 20:11:51.946732 (+ 40us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:51.946748 (+ 16us) write_op.cc:446] Released schema lock
0430 20:11:51.946755 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:51.946803 (+ 48us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:51.946806 (+ 3us) write_op.cc:446] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":723,"apply.run_cpu_time_us":441,"apply.run_wall_time_us":440,"mutex_wait_us":8407,"num_ops":2,"prepare.queue_time_us":9107,"prepare.run_cpu_time_us":607,"prepare.run_wall_time_us":606,"replication_time_us":7172,"threads_started":1}]]}
I20240430 20:11:51.975241 19940 master_service.cc:946] ReplaceTablet: received request to replace tablet f9bf604e6c234349a9f72f45a4c4aef6 from {username='slave'} at 127.0.0.1:50746
I20240430 20:11:51.975277 21108 ts_tablet_manager.cc:1875] T ebf74809b52f47f3b0ed21b6ac0f541a P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240430 20:11:51.976907 19937 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2cd41b407d6b444d844c4c7c0ebf82e5 consensus state has no leader
I20240430 20:11:51.977080 19937 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:50746 (request call id 61) took 1185 ms. Trace:
I20240430 20:11:51.977177 19937 rpcz_store.cc:276] 0430 20:11:50.791673 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:51.923549 (+1131876us) service_pool.cc:225] Handling call
0430 20:11:51.924056 (+ 507us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0430 20:11:51.924271 (+ 215us) catalog_manager.cc:3613] Apply alter schema
0430 20:11:51.924505 (+ 234us) catalog_manager.cc:3636] Acquired catalog manager lock
0430 20:11:51.924586 (+ 81us) catalog_manager.cc:3783] Updating metadata on disk
0430 20:11:51.976632 (+ 52046us) catalog_manager.cc:3814] Committing alterations to in-memory state
0430 20:11:51.977048 (+ 416us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:51.934600 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:51.934723 (+ 123us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:51.934731 (+ 8us) write_op.cc:427] Acquired schema lock
0430 20:11:51.934734 (+ 3us) tablet.cc:660] Decoding operations
0430 20:11:51.934850 (+ 116us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:51.934864 (+ 14us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:51.934870 (+ 6us) tablet.cc:683] Acquiring locks for 1 operations
0430 20:11:51.934939 (+ 69us) tablet.cc:699] Row locks acquired
0430 20:11:51.934942 (+ 3us) write_op.cc:260] PREPARE: finished
0430 20:11:51.934980 (+ 38us) write_op.cc:270] Start()
0430 20:11:51.935013 (+ 33us) write_op.cc:276] Timestamp: P: 1714507911934941 usec, L: 0
0430 20:11:51.935017 (+ 4us) op_driver.cc:331] REPLICATION: starting
0430 20:11:51.935208 (+ 191us) log.cc:842] Serialized 405 byte log entry
0430 20:11:51.941426 (+ 6218us) op_driver.cc:444] REPLICATION: finished
0430 20:11:51.941537 (+ 111us) write_op.cc:301] APPLY: starting
0430 20:11:51.941553 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:51.941594 (+ 41us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:51.941630 (+ 36us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:51.941678 (+ 48us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:51.941710 (+ 32us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0430 20:11:51.941717 (+ 7us) write_op.cc:312] APPLY: finished
0430 20:11:51.941791 (+ 74us) log.cc:842] Serialized 23 byte log entry
0430 20:11:51.941854 (+ 63us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:51.941871 (+ 17us) write_op.cc:446] Released schema lock
0430 20:11:51.941877 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:51.941931 (+ 54us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:51.941935 (+ 4us) write_op.cc:446] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":67,"apply.run_cpu_time_us":447,"apply.run_wall_time_us":446,"num_ops":1,"prepare.queue_time_us":9472,"prepare.run_cpu_time_us":730,"prepare.run_wall_time_us":729,"raft.queue_time_us":480,"raft.run_cpu_time_us":266,"raft.run_wall_time_us":267,"replication_time_us":6386}]]}
I20240430 20:11:51.979339 21229 raft_consensus.cc:3037] T 62732af95c6a4c749c5b726f0460dc72 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:51.979646 18985 tablet_bootstrap.cc:492] T d4d6c37b578c4fe1aa666b0a6b0bc7c9 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:51.979775 18985 ts_tablet_manager.cc:1367] T d4d6c37b578c4fe1aa666b0a6b0bc7c9 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.000s
I20240430 20:11:51.973569 18558 leader_election.cc:302] T 35fdb052135541cfb42a8aeb03b64dc8 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:51.980521 18985 raft_consensus.cc:348] T d4d6c37b578c4fe1aa666b0a6b0bc7c9 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:51.980701 18985 raft_consensus.cc:374] T d4d6c37b578c4fe1aa666b0a6b0bc7c9 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:51.980806 18985 raft_consensus.cc:729] T d4d6c37b578c4fe1aa666b0a6b0bc7c9 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:51.981052 21225 raft_consensus.cc:2781] T 35fdb052135541cfb42a8aeb03b64dc8 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:51.981051 18985 consensus_queue.cc:260] T d4d6c37b578c4fe1aa666b0a6b0bc7c9 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:51.981150 21225 raft_consensus.cc:482] T 35fdb052135541cfb42a8aeb03b64dc8 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:51.981227 21225 raft_consensus.cc:3037] T 35fdb052135541cfb42a8aeb03b64dc8 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:51.981647 18985 ts_tablet_manager.cc:1398] T d4d6c37b578c4fe1aa666b0a6b0bc7c9 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240430 20:11:51.981982 18985 tablet_bootstrap.cc:492] T aa9998f4e17147aa8f2b35c01980463a P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:51.984097 18985 tablet_bootstrap.cc:654] T aa9998f4e17147aa8f2b35c01980463a P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:51.984102 21225 raft_consensus.cc:504] T 35fdb052135541cfb42a8aeb03b64dc8 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:51.985101 21225 leader_election.cc:288] T 35fdb052135541cfb42a8aeb03b64dc8 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
W20240430 20:11:51.986562 18558 leader_election.cc:341] T aa9998f4e17147aa8f2b35c01980463a P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
W20240430 20:11:51.987030 18558 leader_election.cc:341] T 89b774967624498fa2be499878b71124 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:51.988435 18558 leader_election.cc:302] T 489dc838744a49eba542009c86688aa5 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:51.989581 18472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b51cbc731794083a88fe4bed1ddeebf" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:51.988021 21229 raft_consensus.cc:504] T 62732af95c6a4c749c5b726f0460dc72 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:51.989778 21225 raft_consensus.cc:2781] T 489dc838744a49eba542009c86688aa5 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:51.989769 18472 raft_consensus.cc:2454] T 6b51cbc731794083a88fe4bed1ddeebf P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:51.989843 21225 raft_consensus.cc:482] T 489dc838744a49eba542009c86688aa5 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:51.989912 21225 raft_consensus.cc:3037] T 489dc838744a49eba542009c86688aa5 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:51.990468 18472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b51cbc731794083a88fe4bed1ddeebf" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:51.990661 18472 raft_consensus.cc:3037] T 6b51cbc731794083a88fe4bed1ddeebf P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:51.992250 21229 leader_election.cc:288] T 62732af95c6a4c749c5b726f0460dc72 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:51.992594 21225 raft_consensus.cc:504] T 489dc838744a49eba542009c86688aa5 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:51.993095 18472 raft_consensus.cc:2454] T 6b51cbc731794083a88fe4bed1ddeebf P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:51.993770 21225 leader_election.cc:288] T 489dc838744a49eba542009c86688aa5 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:51.995000 18472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35fdb052135541cfb42a8aeb03b64dc8" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:51.995208 18472 raft_consensus.cc:3037] T 35fdb052135541cfb42a8aeb03b64dc8 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:51.997992 18472 raft_consensus.cc:2454] T 35fdb052135541cfb42a8aeb03b64dc8 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
I20240430 20:11:51.998961 21203 ts_tablet_manager.cc:1875] T 1701a15b115741aea13f8327e3d26b34 P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:51.999425 21108 ts_tablet_manager.cc:1888] T ebf74809b52f47f3b0ed21b6ac0f541a P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240430 20:11:51.999914 18472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62732af95c6a4c749c5b726f0460dc72" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:52.000182 18472 raft_consensus.cc:3037] T 62732af95c6a4c749c5b726f0460dc72 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:52.000247 21108 log.cc:1196] T ebf74809b52f47f3b0ed21b6ac0f541a P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/ebf74809b52f47f3b0ed21b6ac0f541a
W20240430 20:11:52.001739 18544 kernel_stack_watchdog.cc:198] Thread 18557 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:308 for 168ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7ffa8ed33980 (unknown) at ??:0
@ 0x90ef77 __sanitizer::internal_sched_yield() at ??:?
@ 0x87462d __interceptor_pthread_create at ??:?
@ 0x7ffa8b9a1d11 kudu::Thread::StartThread() at ??:0
@ 0xa4d187 kudu::Thread::Create() at ??:?
@ 0x7ffa8b9c27ab kudu::ThreadPool::CreateThread() at ??:0
@ 0x7ffa8b9be3ad kudu::ThreadPool::DoSubmit() at ??:0
@ 0x7ffa8b9bc84f kudu::ThreadPoolToken::Submit() at ??:0
@ 0x7ffa970ef9e8 kudu::consensus::Peer::ProcessResponse() at ??:0
@ 0x7ffa970f6061 kudu::consensus::Peer::SendNextRequest()::$_3::operator()() at ??:0
@ 0x7ffa970f5e6e std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7ffa925f38cf kudu::rpc::Proxy::AsyncRequest()::$_1::operator()() at ??:0
@ 0x7ffa925f34ae std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7ffa925df4e0 kudu::rpc::OutboundCall::CallCallback() at ??:0
I20240430 20:11:52.002844 18472 raft_consensus.cc:2454] T 62732af95c6a4c749c5b726f0460dc72 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
I20240430 20:11:52.003576 18476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "489dc838744a49eba542009c86688aa5" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:52.003742 18476 raft_consensus.cc:3037] T 489dc838744a49eba542009c86688aa5 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:52.003898 21108 ts_tablet_manager.cc:1909] T ebf74809b52f47f3b0ed21b6ac0f541a P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:52.005335 21203 ts_tablet_manager.cc:1888] T 1701a15b115741aea13f8327e3d26b34 P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:52.005450 21203 log.cc:1196] T 1701a15b115741aea13f8327e3d26b34 P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/1701a15b115741aea13f8327e3d26b34
I20240430 20:11:52.006285 18476 raft_consensus.cc:2454] T 489dc838744a49eba542009c86688aa5 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
I20240430 20:11:52.010221 19902 catalog_manager.cc:4830] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): tablet ebf74809b52f47f3b0ed21b6ac0f541a (table default.Table_b2d07520b9944a89ad62fca70396174e [id=1c8e0a3a44ec4a798770c0165ed3dffa]) successfully deleted
I20240430 20:11:52.015486 20938 raft_consensus.cc:482] T 62732af95c6a4c749c5b726f0460dc72 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:52.015630 20938 raft_consensus.cc:504] T 62732af95c6a4c749c5b726f0460dc72 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:52.016263 20938 leader_election.cc:288] T 62732af95c6a4c749c5b726f0460dc72 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:52.018637 18763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8901327b3efb4201adce61a9266142ea" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:52.018707 18752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f0b341a31224bf1be525b84b9500304" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:52.018800 18763 raft_consensus.cc:3037] T 8901327b3efb4201adce61a9266142ea P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:52.018905 18752 raft_consensus.cc:2454] T 9f0b341a31224bf1be525b84b9500304 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 1.
I20240430 20:11:52.018436 18756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "898b2c91adf34545808df9c786e218c9" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:52.019400 18762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8901327b3efb4201adce61a9266142ea" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:52.021874 18763 raft_consensus.cc:2454] T 8901327b3efb4201adce61a9266142ea P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 1.
I20240430 20:11:52.026371 20630 raft_consensus.cc:482] T d4d6c37b578c4fe1aa666b0a6b0bc7c9 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:52.026528 20630 raft_consensus.cc:504] T d4d6c37b578c4fe1aa666b0a6b0bc7c9 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:52.027168 20630 leader_election.cc:288] T d4d6c37b578c4fe1aa666b0a6b0bc7c9 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:52.028221 18619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4d6c37b578c4fe1aa666b0a6b0bc7c9" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:52.028427 18619 raft_consensus.cc:2454] T d4d6c37b578c4fe1aa666b0a6b0bc7c9 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 0.
I20240430 20:11:52.028792 18415 leader_election.cc:302] T d4d6c37b578c4fe1aa666b0a6b0bc7c9 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:52.029052 20630 raft_consensus.cc:2781] T d4d6c37b578c4fe1aa666b0a6b0bc7c9 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:52.029114 20630 raft_consensus.cc:482] T d4d6c37b578c4fe1aa666b0a6b0bc7c9 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:52.029163 20630 raft_consensus.cc:3037] T d4d6c37b578c4fe1aa666b0a6b0bc7c9 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:52.029645 21213 ts_tablet_manager.cc:1888] T 7bde1d6934784c9aa5911181e8df9eeb P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.2
I20240430 20:11:52.029744 21213 log.cc:1196] T 7bde1d6934784c9aa5911181e8df9eeb P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/7bde1d6934784c9aa5911181e8df9eeb
I20240430 20:11:52.030059 21213 ts_tablet_manager.cc:1909] T 7bde1d6934784c9aa5911181e8df9eeb P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
W20240430 20:11:52.031248 18557 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.177.3:44867, user_credentials={real_user=slave}} blocked reactor thread for 303187us
I20240430 20:11:52.031919 20630 raft_consensus.cc:504] T d4d6c37b578c4fe1aa666b0a6b0bc7c9 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:52.033198 18601 tablet_service.cc:1514] Processing DeleteTablet for tablet 1701a15b115741aea13f8327e3d26b34 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:50 UTC) from {username='slave'} at 127.0.0.1:45934
I20240430 20:11:52.033249 18602 tablet_service.cc:1514] Processing DeleteTablet for tablet ebf74809b52f47f3b0ed21b6ac0f541a with delete_type TABLET_DATA_DELETED (replaced by tablet 9e8ab75e7d9c4b64a7b7dd2678569490) from {username='slave'} at 127.0.0.1:45934
I20240430 20:11:52.033686 20630 leader_election.cc:288] T d4d6c37b578c4fe1aa666b0a6b0bc7c9 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:52.033881 21220 consensus_queue.cc:1035] T e900f62a5aa143e791c7539660b60cd9 P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:52.034341 18619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4d6c37b578c4fe1aa666b0a6b0bc7c9" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:52.034519 18619 raft_consensus.cc:3037] T d4d6c37b578c4fe1aa666b0a6b0bc7c9 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:52.037290 18619 raft_consensus.cc:2454] T d4d6c37b578c4fe1aa666b0a6b0bc7c9 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 1.
I20240430 20:11:52.038022 18985 tablet_bootstrap.cc:492] T aa9998f4e17147aa8f2b35c01980463a P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:52.038154 18985 ts_tablet_manager.cc:1367] T aa9998f4e17147aa8f2b35c01980463a P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.056s user 0.005s sys 0.009s
I20240430 20:11:52.038343 18415 leader_election.cc:302] T d4d6c37b578c4fe1aa666b0a6b0bc7c9 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:52.038853 20630 raft_consensus.cc:2781] T d4d6c37b578c4fe1aa666b0a6b0bc7c9 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:52.039067 20630 raft_consensus.cc:686] T d4d6c37b578c4fe1aa666b0a6b0bc7c9 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Becoming Leader. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Running, Role: LEADER
I20240430 20:11:52.038995 18985 raft_consensus.cc:348] T aa9998f4e17147aa8f2b35c01980463a P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:52.039175 18985 raft_consensus.cc:374] T aa9998f4e17147aa8f2b35c01980463a P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:52.039220 18985 raft_consensus.cc:729] T aa9998f4e17147aa8f2b35c01980463a P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:52.039309 20630 consensus_queue.cc:237] T d4d6c37b578c4fe1aa666b0a6b0bc7c9 P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:52.039404 18985 consensus_queue.cc:260] T aa9998f4e17147aa8f2b35c01980463a P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:52.039940 18985 ts_tablet_manager.cc:1398] T aa9998f4e17147aa8f2b35c01980463a P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240430 20:11:52.040207 18985 tablet_bootstrap.cc:492] T 89b774967624498fa2be499878b71124 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:52.041791 18985 tablet_bootstrap.cc:654] T 89b774967624498fa2be499878b71124 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:52.044708 18983 tablet_bootstrap.cc:492] T 1fe7c6f1ab54499facd4d01573d3e7bc P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:52.048310 19937 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50746:
name: "default.Table_5ab8136f886143f2a4a7afb5d64a3690"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:52.048955 19937 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5ab8136f886143f2a4a7afb5d64a3690 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:52.057507 18985 tablet_bootstrap.cc:492] T 89b774967624498fa2be499878b71124 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:52.057667 18985 ts_tablet_manager.cc:1367] T 89b774967624498fa2be499878b71124 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.004s
I20240430 20:11:52.058506 18985 raft_consensus.cc:348] T 89b774967624498fa2be499878b71124 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:52.058729 18985 raft_consensus.cc:374] T 89b774967624498fa2be499878b71124 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:52.058785 18985 raft_consensus.cc:729] T 89b774967624498fa2be499878b71124 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:52.058995 18985 consensus_queue.cc:260] T 89b774967624498fa2be499878b71124 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:52.059592 18985 ts_tablet_manager.cc:1398] T 89b774967624498fa2be499878b71124 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240430 20:11:52.060174 18985 tablet_bootstrap.cc:492] T 2e1cac64cefe4336bd3207b5ca58b631 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:52.061822 18985 tablet_bootstrap.cc:654] T 2e1cac64cefe4336bd3207b5ca58b631 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:52.065423 21213 tablet_replica.cc:330] T 1701a15b115741aea13f8327e3d26b34 P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:52.065616 21213 raft_consensus.cc:2229] T 1701a15b115741aea13f8327e3d26b34 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:52.065753 21213 raft_consensus.cc:2258] T 1701a15b115741aea13f8327e3d26b34 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:52.066622 21213 ts_tablet_manager.cc:1875] T 1701a15b115741aea13f8327e3d26b34 P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:52.073918 21213 ts_tablet_manager.cc:1888] T 1701a15b115741aea13f8327e3d26b34 P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:52.074004 18557 leader_election.cc:302] T 417351cbc9d54f9d84bb7155f39da119 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:52.074126 21213 log.cc:1196] T 1701a15b115741aea13f8327e3d26b34 P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/1701a15b115741aea13f8327e3d26b34
I20240430 20:11:52.074244 21221 raft_consensus.cc:2726] T 417351cbc9d54f9d84bb7155f39da119 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:52.074849 21213 ts_tablet_manager.cc:1909] T 1701a15b115741aea13f8327e3d26b34 P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:52.076603 18619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62732af95c6a4c749c5b726f0460dc72" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:52.076879 18619 raft_consensus.cc:2379] T 62732af95c6a4c749c5b726f0460dc72 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c51c33d5616482cafb68e26688820f4 in current term 1: Already voted for candidate 5e1c6397f09747f091534faf93542710 in this term.
I20240430 20:11:52.083053 20630 raft_consensus.cc:482] T 828ca913e2aa4102a6277bdbc6686ef5 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240430 20:11:52.083164 18557 consensus_peers.cc:485] T 2873af6da928463f9809864b8eaf0c4a P 5e1c6397f09747f091534faf93542710 -> Peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Couldn't send request to peer 3c51c33d5616482cafb68e26688820f4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2873af6da928463f9809864b8eaf0c4a. This is attempt 1: this message will repeat every 5th retry.
I20240430 20:11:52.083199 20630 raft_consensus.cc:504] T 828ca913e2aa4102a6277bdbc6686ef5 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:52.083825 20630 leader_election.cc:288] T 828ca913e2aa4102a6277bdbc6686ef5 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:52.084301 19895 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet 7bde1d6934784c9aa5911181e8df9eeb (table default.Table_7199d7c36d3f45f79cf9e4e2b4145592 [id=f6aef8f6a9164dd78f49e03f723b45a9]) successfully deleted
I20240430 20:11:52.084508 18616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "828ca913e2aa4102a6277bdbc6686ef5" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:52.084758 19895 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet 1701a15b115741aea13f8327e3d26b34 (table default.Table_3776a4b9d4f24aa6b7f882bb4d0116ef [id=7127dbd368864f7ba80a755549fe7ede]) successfully deleted
I20240430 20:11:52.085115 21238 raft_consensus.cc:482] T 2aa2a489116940dea3173cad16d94fa5 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:52.085283 21238 raft_consensus.cc:504] T 2aa2a489116940dea3173cad16d94fa5 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:52.086064 21238 leader_election.cc:288] T 2aa2a489116940dea3173cad16d94fa5 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:52.086783 18760 raft_consensus.cc:1261] T f90a5c5818c54390a2ae323fed7ebe57 P 3c51c33d5616482cafb68e26688820f4 [term 2 FOLLOWER]: Refusing update from remote peer 2efb542aa55e439b99ccb98d1c5c884a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240430 20:11:52.095721 18616 raft_consensus.cc:1261] T f90a5c5818c54390a2ae323fed7ebe57 P 5e1c6397f09747f091534faf93542710 [term 2 FOLLOWER]: Refusing update from remote peer 2efb542aa55e439b99ccb98d1c5c884a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240430 20:11:52.096177 20630 consensus_queue.cc:1035] T f90a5c5818c54390a2ae323fed7ebe57 P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240430 20:11:52.096719 21238 raft_consensus.cc:482] T 3a03b6997921408cb5a65f4cae223e9c P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:52.096725 18472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aa2a489116940dea3173cad16d94fa5" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:52.096895 18472 raft_consensus.cc:2454] T 2aa2a489116940dea3173cad16d94fa5 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 0.
I20240430 20:11:52.096890 21238 raft_consensus.cc:504] T 3a03b6997921408cb5a65f4cae223e9c P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:52.097484 21238 leader_election.cc:288] T 3a03b6997921408cb5a65f4cae223e9c P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:52.098479 18558 leader_election.cc:302] T 35fdb052135541cfb42a8aeb03b64dc8 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:52.099529 18983 tablet_bootstrap.cc:654] T 1fe7c6f1ab54499facd4d01573d3e7bc P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:52.099826 21238 raft_consensus.cc:2781] T 35fdb052135541cfb42a8aeb03b64dc8 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:52.100032 21238 raft_consensus.cc:686] T 35fdb052135541cfb42a8aeb03b64dc8 P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Becoming Leader. State: Replica: 5e1c6397f09747f091534faf93542710, State: Running, Role: LEADER
I20240430 20:11:52.100273 21238 consensus_queue.cc:237] T 35fdb052135541cfb42a8aeb03b64dc8 P 5e1c6397f09747f091534faf93542710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:52.103549 18558 leader_election.cc:302] T 62732af95c6a4c749c5b726f0460dc72 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:52.105021 18558 leader_election.cc:302] T 489dc838744a49eba542009c86688aa5 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:52.105333 21221 raft_consensus.cc:2781] T 62732af95c6a4c749c5b726f0460dc72 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:52.105463 21225 raft_consensus.cc:2781] T 489dc838744a49eba542009c86688aa5 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:52.105551 21225 raft_consensus.cc:686] T 489dc838744a49eba542009c86688aa5 P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Becoming Leader. State: Replica: 5e1c6397f09747f091534faf93542710, State: Running, Role: LEADER
I20240430 20:11:52.105885 21225 consensus_queue.cc:237] T 489dc838744a49eba542009c86688aa5 P 5e1c6397f09747f091534faf93542710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:52.107049 21221 raft_consensus.cc:686] T 62732af95c6a4c749c5b726f0460dc72 P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Becoming Leader. State: Replica: 5e1c6397f09747f091534faf93542710, State: Running, Role: LEADER
I20240430 20:11:52.107303 21221 consensus_queue.cc:237] T 62732af95c6a4c749c5b726f0460dc72 P 5e1c6397f09747f091534faf93542710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:52.094147 20198 consensus_queue.cc:1035] T 898b2c91adf34545808df9c786e218c9 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.015s
I20240430 20:11:52.077153 21213 tablet_replica.cc:330] T ebf74809b52f47f3b0ed21b6ac0f541a P 5e1c6397f09747f091534faf93542710: stopping tablet replica
W20240430 20:11:52.112565 18558 consensus_peers.cc:485] T ebf74809b52f47f3b0ed21b6ac0f541a P 5e1c6397f09747f091534faf93542710 -> Peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Couldn't send request to peer 2efb542aa55e439b99ccb98d1c5c884a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ebf74809b52f47f3b0ed21b6ac0f541a. This is attempt 1: this message will repeat every 5th retry.
I20240430 20:11:52.112429 21213 raft_consensus.cc:2229] T ebf74809b52f47f3b0ed21b6ac0f541a P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Raft consensus shutting down.
I20240430 20:11:52.113211 18760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4d6c37b578c4fe1aa666b0a6b0bc7c9" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:52.114219 18760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4d6c37b578c4fe1aa666b0a6b0bc7c9" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:52.116005 21213 raft_consensus.cc:2258] T ebf74809b52f47f3b0ed21b6ac0f541a P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:52.117091 18558 leader_election.cc:302] T 2aa2a489116940dea3173cad16d94fa5 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:52.117663 21221 raft_consensus.cc:2781] T 2aa2a489116940dea3173cad16d94fa5 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:52.117725 21221 raft_consensus.cc:482] T 2aa2a489116940dea3173cad16d94fa5 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:52.117779 21221 raft_consensus.cc:3037] T 2aa2a489116940dea3173cad16d94fa5 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:52.120510 21221 raft_consensus.cc:504] T 2aa2a489116940dea3173cad16d94fa5 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:52.121155 21221 leader_election.cc:288] T 2aa2a489116940dea3173cad16d94fa5 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:52.122376 18476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a03b6997921408cb5a65f4cae223e9c" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:52.122352 21213 ts_tablet_manager.cc:1875] T ebf74809b52f47f3b0ed21b6ac0f541a P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:52.122576 18476 raft_consensus.cc:2454] T 3a03b6997921408cb5a65f4cae223e9c P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 0.
I20240430 20:11:52.123788 18472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aa2a489116940dea3173cad16d94fa5" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:52.123961 18472 raft_consensus.cc:3037] T 2aa2a489116940dea3173cad16d94fa5 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:52.127072 18472 raft_consensus.cc:2454] T 2aa2a489116940dea3173cad16d94fa5 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
I20240430 20:11:52.128554 18988 tablet_bootstrap.cc:492] T 489dc838744a49eba542009c86688aa5 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:52.128702 18988 ts_tablet_manager.cc:1367] T 489dc838744a49eba542009c86688aa5 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.387s user 0.006s sys 0.010s
I20240430 20:11:52.131711 18558 leader_election.cc:302] T 3a03b6997921408cb5a65f4cae223e9c P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:52.133253 21221 raft_consensus.cc:2781] T 3a03b6997921408cb5a65f4cae223e9c P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:52.133334 21221 raft_consensus.cc:482] T 3a03b6997921408cb5a65f4cae223e9c P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:52.133400 21221 raft_consensus.cc:3037] T 3a03b6997921408cb5a65f4cae223e9c P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:52.136041 21221 raft_consensus.cc:504] T 3a03b6997921408cb5a65f4cae223e9c P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:52.136631 21221 leader_election.cc:288] T 3a03b6997921408cb5a65f4cae223e9c P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:52.138307 18558 leader_election.cc:302] T 2aa2a489116940dea3173cad16d94fa5 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:52.141359 18472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a03b6997921408cb5a65f4cae223e9c" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:52.141556 18472 raft_consensus.cc:3037] T 3a03b6997921408cb5a65f4cae223e9c P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:52.144259 18472 raft_consensus.cc:2454] T 3a03b6997921408cb5a65f4cae223e9c P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
I20240430 20:11:52.147994 18558 leader_election.cc:302] T 3a03b6997921408cb5a65f4cae223e9c P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:52.148941 21238 raft_consensus.cc:2781] T 3a03b6997921408cb5a65f4cae223e9c P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:52.149178 21238 raft_consensus.cc:686] T 3a03b6997921408cb5a65f4cae223e9c P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Becoming Leader. State: Replica: 5e1c6397f09747f091534faf93542710, State: Running, Role: LEADER
I20240430 20:11:52.149426 21238 consensus_queue.cc:237] T 3a03b6997921408cb5a65f4cae223e9c P 5e1c6397f09747f091534faf93542710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:52.150226 21054 raft_consensus.cc:482] T 89b774967624498fa2be499878b71124 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:52.150358 21054 raft_consensus.cc:504] T 89b774967624498fa2be499878b71124 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:52.150971 21054 leader_election.cc:288] T 89b774967624498fa2be499878b71124 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:52.152897 20198 consensus_queue.cc:1035] T d2381047f6dc4bdf962f9383e6c41548 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:52.154220 18699 leader_election.cc:302] T df92eb85261d47d88cbf16d29d9ad30e P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710
I20240430 20:11:52.154444 20198 raft_consensus.cc:3037] T df92eb85261d47d88cbf16d29d9ad30e P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:52.156289 18760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "828ca913e2aa4102a6277bdbc6686ef5" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:52.156530 18760 raft_consensus.cc:2379] T 828ca913e2aa4102a6277bdbc6686ef5 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2efb542aa55e439b99ccb98d1c5c884a in current term 1: Already voted for candidate 5e1c6397f09747f091534faf93542710 in this term.
I20240430 20:11:52.159359 20198 raft_consensus.cc:2726] T df92eb85261d47d88cbf16d29d9ad30e P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:52.160324 18619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89b774967624498fa2be499878b71124" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:52.160607 18619 raft_consensus.cc:2454] T 89b774967624498fa2be499878b71124 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 0.
I20240430 20:11:52.161026 18415 leader_election.cc:302] T 89b774967624498fa2be499878b71124 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:52.161396 21049 raft_consensus.cc:2781] T 89b774967624498fa2be499878b71124 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:52.161480 21049 raft_consensus.cc:482] T 89b774967624498fa2be499878b71124 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:52.161556 21049 raft_consensus.cc:3037] T 89b774967624498fa2be499878b71124 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:52.162631 18983 tablet_bootstrap.cc:492] T 1fe7c6f1ab54499facd4d01573d3e7bc P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:52.162761 18983 ts_tablet_manager.cc:1367] T 1fe7c6f1ab54499facd4d01573d3e7bc P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.118s user 0.006s sys 0.004s
I20240430 20:11:52.163491 18983 raft_consensus.cc:348] T 1fe7c6f1ab54499facd4d01573d3e7bc P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:52.163736 18983 raft_consensus.cc:374] T 1fe7c6f1ab54499facd4d01573d3e7bc P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:52.163784 18983 raft_consensus.cc:729] T 1fe7c6f1ab54499facd4d01573d3e7bc P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:52.164044 18983 consensus_queue.cc:260] T 1fe7c6f1ab54499facd4d01573d3e7bc P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:52.164184 21049 raft_consensus.cc:504] T 89b774967624498fa2be499878b71124 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:52.164592 18983 ts_tablet_manager.cc:1398] T 1fe7c6f1ab54499facd4d01573d3e7bc P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240430 20:11:52.164896 18983 tablet_bootstrap.cc:492] T 13c2e69ae91f4957887fd64b90f6e435 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:52.165647 21049 leader_election.cc:288] T 89b774967624498fa2be499878b71124 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Requested vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:52.166224 18620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89b774967624498fa2be499878b71124" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:52.166389 18620 raft_consensus.cc:3037] T 89b774967624498fa2be499878b71124 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:52.167654 18983 tablet_bootstrap.cc:654] T 13c2e69ae91f4957887fd64b90f6e435 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:52.169104 18620 raft_consensus.cc:2454] T 89b774967624498fa2be499878b71124 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 1.
I20240430 20:11:52.170454 18415 leader_election.cc:302] T 89b774967624498fa2be499878b71124 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:52.170776 21049 raft_consensus.cc:2781] T 89b774967624498fa2be499878b71124 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:52.170984 21049 raft_consensus.cc:686] T 89b774967624498fa2be499878b71124 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Becoming Leader. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Running, Role: LEADER
I20240430 20:11:52.171226 21049 consensus_queue.cc:237] T 89b774967624498fa2be499878b71124 P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:52.175297 21177 log.cc:824] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: Log is configured to *not* fsync() on all Append() calls
I20240430 20:11:52.199146 21213 ts_tablet_manager.cc:1888] T ebf74809b52f47f3b0ed21b6ac0f541a P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240430 20:11:52.200222 21213 log.cc:1196] T ebf74809b52f47f3b0ed21b6ac0f541a P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/ebf74809b52f47f3b0ed21b6ac0f541a
I20240430 20:11:52.202402 21213 ts_tablet_manager.cc:1909] T ebf74809b52f47f3b0ed21b6ac0f541a P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:52.207159 19895 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet ebf74809b52f47f3b0ed21b6ac0f541a (table default.Table_b2d07520b9944a89ad62fca70396174e [id=1c8e0a3a44ec4a798770c0165ed3dffa]) successfully deleted
I20240430 20:11:52.211172 18528 heartbeater.cc:503] Master 127.17.177.62:36441 was elected leader, sending a full tablet report...
I20240430 20:11:52.213897 21049 raft_consensus.cc:482] T df92eb85261d47d88cbf16d29d9ad30e P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:52.214103 21049 raft_consensus.cc:504] T df92eb85261d47d88cbf16d29d9ad30e P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
W20240430 20:11:52.216786 18415 leader_election.cc:341] T df92eb85261d47d88cbf16d29d9ad30e P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Not found: Tablet not found: df92eb85261d47d88cbf16d29d9ad30e
I20240430 20:11:52.218312 21049 leader_election.cc:288] T df92eb85261d47d88cbf16d29d9ad30e P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:52.224246 21221 raft_consensus.cc:2781] T 2aa2a489116940dea3173cad16d94fa5 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:52.224447 21221 raft_consensus.cc:686] T 2aa2a489116940dea3173cad16d94fa5 P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Becoming Leader. State: Replica: 5e1c6397f09747f091534faf93542710, State: Running, Role: LEADER
I20240430 20:11:52.224762 21221 consensus_queue.cc:237] T 2aa2a489116940dea3173cad16d94fa5 P 5e1c6397f09747f091534faf93542710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:52.236295 18477 raft_consensus.cc:3037] T 898b2c91adf34545808df9c786e218c9 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:52.236507 18474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62732af95c6a4c749c5b726f0460dc72" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:52.236819 18474 raft_consensus.cc:2379] T 62732af95c6a4c749c5b726f0460dc72 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c51c33d5616482cafb68e26688820f4 in current term 1: Already voted for candidate 5e1c6397f09747f091534faf93542710 in this term.
I20240430 20:11:52.237735 18557 leader_election.cc:302] T eb6c96047b114ebba73db7a371f2da8e P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a
I20240430 20:11:52.238206 21221 raft_consensus.cc:2781] T eb6c96047b114ebba73db7a371f2da8e P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:52.238283 21221 raft_consensus.cc:482] T eb6c96047b114ebba73db7a371f2da8e P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:52.238349 21221 raft_consensus.cc:3037] T eb6c96047b114ebba73db7a371f2da8e P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:52.241037 21221 raft_consensus.cc:504] T eb6c96047b114ebba73db7a371f2da8e P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:52.242003 21221 leader_election.cc:288] T eb6c96047b114ebba73db7a371f2da8e P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:52.242493 18477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb6c96047b114ebba73db7a371f2da8e" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:52.242695 18477 raft_consensus.cc:3037] T eb6c96047b114ebba73db7a371f2da8e P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:52.243791 21238 consensus_queue.cc:1035] T f7d3dea34cf54fe2b949d1b7ca68577a P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:52.244782 18557 leader_election.cc:302] T 6b51cbc731794083a88fe4bed1ddeebf P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:52.255575 21203 ts_tablet_manager.cc:1909] T 1701a15b115741aea13f8327e3d26b34 P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:52.259086 18988 raft_consensus.cc:348] T 489dc838744a49eba542009c86688aa5 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:52.272855 18810 heartbeater.cc:347] Connected to a master server at 127.17.177.60:37225
I20240430 20:11:52.477111 18988 raft_consensus.cc:374] T 489dc838744a49eba542009c86688aa5 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:52.477279 18988 raft_consensus.cc:729] T 489dc838744a49eba542009c86688aa5 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:52.477406 18810 heartbeater.cc:503] Master 127.17.177.60:37225 was elected leader, sending a full tablet report...
I20240430 20:11:52.477576 18988 consensus_queue.cc:260] T 489dc838744a49eba542009c86688aa5 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:52.245226 21238 raft_consensus.cc:2726] T 6b51cbc731794083a88fe4bed1ddeebf P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:52.301484 18983 tablet_bootstrap.cc:492] T 13c2e69ae91f4957887fd64b90f6e435 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:52.478405 18983 ts_tablet_manager.cc:1367] T 13c2e69ae91f4957887fd64b90f6e435 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.314s user 0.003s sys 0.001s
W20240430 20:11:52.301724 19895 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 2cd41b407d6b444d844c4c7c0ebf82e5 (table default.Table_6f8e0eff8f5a484dbd6a9c3fdb5d03f6 [id=3813b5e9778c4bd683a06598cef0d86f], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 2cd41b407d6b444d844c4c7c0ebf82e5 consensus state has no leader
I20240430 20:11:52.479218 18983 raft_consensus.cc:348] T 13c2e69ae91f4957887fd64b90f6e435 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:52.479415 18983 raft_consensus.cc:374] T 13c2e69ae91f4957887fd64b90f6e435 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:52.479467 18983 raft_consensus.cc:729] T 13c2e69ae91f4957887fd64b90f6e435 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:52.245322 18477 raft_consensus.cc:2454] T eb6c96047b114ebba73db7a371f2da8e P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
I20240430 20:11:52.274995 18985 tablet_bootstrap.cc:492] T 2e1cac64cefe4336bd3207b5ca58b631 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:52.479692 18985 ts_tablet_manager.cc:1367] T 2e1cac64cefe4336bd3207b5ca58b631 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.420s user 0.006s sys 0.039s
I20240430 20:11:52.310202 18750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35fdb052135541cfb42a8aeb03b64dc8" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:52.479631 18983 consensus_queue.cc:260] T 13c2e69ae91f4957887fd64b90f6e435 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:52.355814 20630 consensus_queue.cc:1035] T 8e9711d0a382424f9f758bfa883866a0 P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:52.334709 21221 consensus_queue.cc:1035] T c1017a50d1d646c486c0b9a75cb88ce4 P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:52.480031 18983 ts_tablet_manager.cc:1398] T 13c2e69ae91f4957887fd64b90f6e435 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240430 20:11:52.307235 19900 catalog_manager.cc:4830] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): tablet 1701a15b115741aea13f8327e3d26b34 (table default.Table_3776a4b9d4f24aa6b7f882bb4d0116ef [id=7127dbd368864f7ba80a755549fe7ede]) successfully deleted
I20240430 20:11:52.480311 18983 tablet_bootstrap.cc:492] T a692b253db914531b8fc6ffaf3e350b5 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
W20240430 20:11:52.480398 19900 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.17.177.3:44867, user_credentials={real_user=slave}} blocked reactor thread for 173189us
I20240430 20:11:52.480453 18558 leader_election.cc:302] T eb6c96047b114ebba73db7a371f2da8e P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:52.310206 18760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62732af95c6a4c749c5b726f0460dc72" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:52.310549 21203 tablet_replica.cc:330] T ebf74809b52f47f3b0ed21b6ac0f541a P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:52.413110 21229 consensus_queue.cc:1035] T 558e71ec370d46ca8fbdcec5b57eb2ec P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:52.355994 18459 tablet_service.cc:1467] Processing CreateTablet for tablet a1754ef24e86421f83ca3476dbc3a43a (DEFAULT_TABLE table=default.Table_e7097af3f530403a854d53e9d8e87108 [id=c352f291842e42f59a7204c765a63041]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:52.356125 18445 tablet_service.cc:1467] Processing CreateTablet for tablet 8991518bfe3a44c9bb85a52b4e06040b (DEFAULT_TABLE table=default.Table_8bcd1481133a4f45aac58ee0460b7ef8 [id=cb8d130400a1468fb08e0811d07dba95]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:52.481927 18459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1754ef24e86421f83ca3476dbc3a43a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:52.481906 21229 raft_consensus.cc:2781] T eb6c96047b114ebba73db7a371f2da8e P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:52.481957 18983 tablet_bootstrap.cc:654] T a692b253db914531b8fc6ffaf3e350b5 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:52.482020 21229 raft_consensus.cc:686] T eb6c96047b114ebba73db7a371f2da8e P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Becoming Leader. State: Replica: 5e1c6397f09747f091534faf93542710, State: Running, Role: LEADER
I20240430 20:11:52.482287 21229 consensus_queue.cc:237] T eb6c96047b114ebba73db7a371f2da8e P 5e1c6397f09747f091534faf93542710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:52.310885 18763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa9998f4e17147aa8f2b35c01980463a" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:52.363821 18758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89b774967624498fa2be499878b71124" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:52.483309 21227 consensus_queue.cc:1035] T c1017a50d1d646c486c0b9a75cb88ce4 P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:52.483390 18988 ts_tablet_manager.cc:1398] T 489dc838744a49eba542009c86688aa5 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.349s user 0.000s sys 0.002s
I20240430 20:11:52.483984 18601 tablet_service.cc:1467] Processing CreateTablet for tablet a1754ef24e86421f83ca3476dbc3a43a (DEFAULT_TABLE table=default.Table_e7097af3f530403a854d53e9d8e87108 [id=c352f291842e42f59a7204c765a63041]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:52.364342 18764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "489dc838744a49eba542009c86688aa5" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:52.356417 18447 tablet_service.cc:1467] Processing CreateTablet for tablet 06e2dd1f7e8d434d9094f1acdec95e09 (DEFAULT_TABLE table=default.Table_31760cb86ebf4666abcffc4217aa2708 [id=f235b4c2340d45a2b348f1488dc3a6c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:52.364837 18762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35fdb052135541cfb42a8aeb03b64dc8" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:52.484375 18601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1754ef24e86421f83ca3476dbc3a43a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:52.365077 18756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62732af95c6a4c749c5b726f0460dc72" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:52.365276 18752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "489dc838744a49eba542009c86688aa5" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:52.484898 18764 raft_consensus.cc:2454] T 489dc838744a49eba542009c86688aa5 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 0.
I20240430 20:11:52.351454 19937 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:50746 (request call id 65) took 1386 ms. Trace:
I20240430 20:11:52.485484 19937 rpcz_store.cc:276] 0430 20:11:50.964952 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:51.978188 (+1013236us) service_pool.cc:225] Handling call
0430 20:11:52.048197 (+ 70009us) spinlock_profiling.cc:243] Waited 69.7 ms on lock 0x616000070654. stack: 00007f6d23ac8756 00007f6d23ac8510 00007f6d22afb48d 0000000000955804 00000000009557cd 00007f6d3260715d 00007f6d32607253 00007f6d2f339d41 00007f6d36f2534d 00007f6d370b1cb3 00007f6d2de9a2a6 00007f6d2de99fd5 00007f6d2a7d004e 00007f6d2a7cef2e 00007f6d2a7d5405 00007f6d2a7d5d2f
0430 20:11:52.049054 (+ 857us) catalog_manager.cc:2072] Acquired catalog manager lock
0430 20:11:52.049649 (+ 595us) catalog_manager.cc:2118] Created new table and tablet info
0430 20:11:52.094280 (+ 44631us) catalog_manager.cc:2178] Wrote table and tablets to system table
0430 20:11:52.351237 (+256957us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0430 20:11:52.351398 (+ 161us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:52.050185 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:52.050279 (+ 94us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:52.050286 (+ 7us) write_op.cc:427] Acquired schema lock
0430 20:11:52.050289 (+ 3us) tablet.cc:660] Decoding operations
0430 20:11:52.050376 (+ 87us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:52.050399 (+ 23us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:52.050405 (+ 6us) tablet.cc:683] Acquiring locks for 2 operations
0430 20:11:52.050480 (+ 75us) tablet.cc:699] Row locks acquired
0430 20:11:52.050482 (+ 2us) write_op.cc:260] PREPARE: finished
0430 20:11:52.050521 (+ 39us) write_op.cc:270] Start()
0430 20:11:52.050555 (+ 34us) write_op.cc:276] Timestamp: P: 1714507912050469 usec, L: 0
0430 20:11:52.050558 (+ 3us) op_driver.cc:331] REPLICATION: starting
0430 20:11:52.050785 (+ 227us) log.cc:842] Serialized 433 byte log entry
0430 20:11:52.067694 (+ 16909us) op_driver.cc:444] REPLICATION: finished
0430 20:11:52.068652 (+ 958us) write_op.cc:301] APPLY: starting
0430 20:11:52.068677 (+ 25us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:52.068743 (+ 66us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:52.068806 (+ 63us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:52.068948 (+ 142us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:52.068997 (+ 49us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0430 20:11:52.069009 (+ 12us) write_op.cc:312] APPLY: finished
0430 20:11:52.069114 (+ 105us) log.cc:842] Serialized 29 byte log entry
0430 20:11:52.069478 (+ 364us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:52.069508 (+ 30us) write_op.cc:446] Released schema lock
0430 20:11:52.069518 (+ 10us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:52.069586 (+ 68us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:52.069592 (+ 6us) write_op.cc:446] Released schema lock
Metrics: {"spinlock_wait_cycles":160244480,"child_traces":[["op",{"apply.queue_time_us":823,"apply.run_cpu_time_us":738,"apply.run_wall_time_us":1010,"num_ops":2,"prepare.queue_time_us":56,"prepare.run_cpu_time_us":723,"prepare.run_wall_time_us":724,"raft.queue_time_us":477,"raft.run_cpu_time_us":263,"raft.run_wall_time_us":648,"replication_time_us":17099}]]}
I20240430 20:11:52.486884 18600 tablet_service.cc:1467] Processing CreateTablet for tablet f6ab6cf7858348f7a9c6aba1d61ade87 (DEFAULT_TABLE table=default.Table_a93c32ea6b7c49a1941650334c85b5e4 [id=52c66e0eee004fad9ad1a493645a99ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:52.365443 18757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a03b6997921408cb5a65f4cae223e9c" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:52.487227 18600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6ab6cf7858348f7a9c6aba1d61ade87. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:52.365517 18755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aa2a489116940dea3173cad16d94fa5" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:52.356741 18453 tablet_service.cc:1467] Processing CreateTablet for tablet a37ff93ed9bf4b70b57fea7c038d2195 (DEFAULT_TABLE table=default.Table_e158c1ec239f48a9be498828ad9a5a72 [id=eb0231e25f204632808706605865d342]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:52.490031 18595 tablet_service.cc:1467] Processing CreateTablet for tablet daeef5d233f24602ad5d64f7685a50eb (DEFAULT_TABLE table=default.Table_6ff76cc99d73469da3e9b8522abec263 [id=128df55b93534de6aaddabb25d21fcfe]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:52.490821 18730 tablet_service.cc:1467] Processing CreateTablet for tablet 8991518bfe3a44c9bb85a52b4e06040b (DEFAULT_TABLE table=default.Table_8bcd1481133a4f45aac58ee0460b7ef8 [id=cb8d130400a1468fb08e0811d07dba95]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:52.491255 18736 tablet_service.cc:1467] Processing CreateTablet for tablet a37ff93ed9bf4b70b57fea7c038d2195 (DEFAULT_TABLE table=default.Table_e158c1ec239f48a9be498828ad9a5a72 [id=eb0231e25f204632808706605865d342]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:52.366441 18751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aa2a489116940dea3173cad16d94fa5" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:52.366802 18761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a03b6997921408cb5a65f4cae223e9c" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:52.491590 18732 tablet_service.cc:1467] Processing CreateTablet for tablet ca952d9886304b1593160338d7d30458 (DEFAULT_TABLE table=default.Table_45b0755ed3e748f78fe856a4ab1831cc [id=71f2b0943f0f4b4e96617108202740fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:52.491698 18735 tablet_service.cc:1467] Processing CreateTablet for tablet a1754ef24e86421f83ca3476dbc3a43a (DEFAULT_TABLE table=default.Table_e7097af3f530403a854d53e9d8e87108 [id=c352f291842e42f59a7204c765a63041]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:52.492004 18725 tablet_service.cc:1467] Processing CreateTablet for tablet 06e2dd1f7e8d434d9094f1acdec95e09 (DEFAULT_TABLE table=default.Table_31760cb86ebf4666abcffc4217aa2708 [id=f235b4c2340d45a2b348f1488dc3a6c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:52.362053 18452 tablet_service.cc:1467] Processing CreateTablet for tablet ca952d9886304b1593160338d7d30458 (DEFAULT_TABLE table=default.Table_45b0755ed3e748f78fe856a4ab1831cc [id=71f2b0943f0f4b4e96617108202740fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:52.492326 18740 tablet_service.cc:1467] Processing CreateTablet for tablet a81c2d2790aa4e94809a2edd7cd0555e (DEFAULT_TABLE table=default.Table_93f5811e02a742e180954822432cacd1 [id=f3912511868f48fb8124ebe82b0c5fa0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:52.492434 18726 tablet_service.cc:1467] Processing CreateTablet for tablet daeef5d233f24602ad5d64f7685a50eb (DEFAULT_TABLE table=default.Table_6ff76cc99d73469da3e9b8522abec263 [id=128df55b93534de6aaddabb25d21fcfe]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:52.492700 18729 tablet_service.cc:1467] Processing CreateTablet for tablet 9743ebed69784fda9d26dc125d205fbf (DEFAULT_TABLE table=default.Table_49d181ca513240d88696579ad5b4d3d8 [id=4de11fd888f645519d9398eb7b556d85]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:52.492756 18734 tablet_service.cc:1467] Processing CreateTablet for tablet 9a5654fd913f4fbeb9d126f035ac32bf (DEFAULT_TABLE table=default.Table_766e3df58895427b9e5179eb555fe7e4 [id=5c11311047b34a3eb06cc06e8cc10223]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:52.493081 18744 tablet_service.cc:1467] Processing CreateTablet for tablet dd1de890e4394db287cf38bb411f5586 (DEFAULT_TABLE table=default.Table_84084a27f7504dd992fce07448c10df3 [id=6257559b0e9545c58848a76af0c5a5f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:52.493176 18737 tablet_service.cc:1467] Processing CreateTablet for tablet 85baf4cd31cb40d281bae432bafd2791 (DEFAULT_TABLE table=default.Table_11ef4a2c1255424bac9dd34fc0313926 [id=1906e2cda0f74f568d0e00f7b3badeba]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:52.389111 21281 tablet.cc:1722] T 8e91920e7e36418d8c27d1f0bfd9b981 P 3c51c33d5616482cafb68e26688820f4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240430 20:11:52.493423 18724 tablet_service.cc:1467] Processing CreateTablet for tablet 8abf0b43f8a447dc902135ce94e4fc78 (DEFAULT_TABLE table=default.Table_5572f0acadaa432e8570b159fd9b76bc [id=d0f3b1709ac94aed8ec568c0327eab34]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:52.363436 18454 tablet_service.cc:1467] Processing CreateTablet for tablet c556f1aa75ac4c908e654e01d72d45aa (DEFAULT_TABLE table=default.Table_c1538175cbb94c18b8fd9e5de5436047 [id=1a4baeb013fb478c96f8802a6f0b419f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:52.367280 18759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb6c96047b114ebba73db7a371f2da8e" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:52.363752 18450 tablet_service.cc:1467] Processing CreateTablet for tablet a81c2d2790aa4e94809a2edd7cd0555e (DEFAULT_TABLE table=default.Table_93f5811e02a742e180954822432cacd1 [id=f3912511868f48fb8124ebe82b0c5fa0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:52.404316 18746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89b774967624498fa2be499878b71124" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:52.493795 18759 raft_consensus.cc:3037] T eb6c96047b114ebba73db7a371f2da8e P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:52.364948 18449 tablet_service.cc:1467] Processing CreateTablet for tablet f6ab6cf7858348f7a9c6aba1d61ade87 (DEFAULT_TABLE table=default.Table_a93c32ea6b7c49a1941650334c85b5e4 [id=52c66e0eee004fad9ad1a493645a99ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:52.411753 18745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df92eb85261d47d88cbf16d29d9ad30e" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:52.494098 18745 raft_consensus.cc:2454] T df92eb85261d47d88cbf16d29d9ad30e P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 1.
I20240430 20:11:52.368269 18458 tablet_service.cc:1467] Processing CreateTablet for tablet daeef5d233f24602ad5d64f7685a50eb (DEFAULT_TABLE table=default.Table_6ff76cc99d73469da3e9b8522abec263 [id=128df55b93534de6aaddabb25d21fcfe]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:52.369060 18446 tablet_service.cc:1467] Processing CreateTablet for tablet 9a5654fd913f4fbeb9d126f035ac32bf (DEFAULT_TABLE table=default.Table_766e3df58895427b9e5179eb555fe7e4 [id=5c11311047b34a3eb06cc06e8cc10223]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:52.369921 18448 tablet_service.cc:1467] Processing CreateTablet for tablet 2ca98d4245d74c9e8978cd40f2c26342 (DEFAULT_TABLE table=default.Table_066abbbc2a9a4c8f8025b62ceb68ca8a [id=bac68754698f4c298984c39fe9f15853]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:52.370663 18461 tablet_service.cc:1467] Processing CreateTablet for tablet 9743ebed69784fda9d26dc125d205fbf (DEFAULT_TABLE table=default.Table_49d181ca513240d88696579ad5b4d3d8 [id=4de11fd888f645519d9398eb7b556d85]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:52.371779 18457 tablet_service.cc:1467] Processing CreateTablet for tablet 85baf4cd31cb40d281bae432bafd2791 (DEFAULT_TABLE table=default.Table_11ef4a2c1255424bac9dd34fc0313926 [id=1906e2cda0f74f568d0e00f7b3badeba]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:52.380565 18456 tablet_service.cc:1467] Processing CreateTablet for tablet dd1de890e4394db287cf38bb411f5586 (DEFAULT_TABLE table=default.Table_84084a27f7504dd992fce07448c10df3 [id=6257559b0e9545c58848a76af0c5a5f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:52.354950 19940 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: Sending DeleteTablet for 3 replicas of tablet f9bf604e6c234349a9f72f45a4c4aef6
I20240430 20:11:52.495806 19940 catalog_manager.cc:6554] ReplaceTablet: tablet f9bf604e6c234349a9f72f45a4c4aef6 deleted and replaced by tablet 415d99c36b1941e08aa4a16c50483ee4
I20240430 20:11:52.495956 19940 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:50746 (request call id 64) took 1588 ms. Trace:
I20240430 20:11:52.496031 19940 rpcz_store.cc:276] 0430 20:11:50.907548 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:51.975161 (+1067613us) service_pool.cc:225] Handling call
0430 20:11:52.060128 (+ 84967us) spinlock_profiling.cc:243] Waited 84.6 ms on lock 0x616000070654. stack: 00007f6d23ac8756 00007f6d23ac8510 00007f6d22afb48d 0000000000955804 00000000009557cd 00007f6d3260715d 00007f6d32607253 00007f6d2f339d41 00007f6d36f2534d 00007f6d370be05b 00007f6d2dea6be7 00007f6d2dea6915 00007f6d2a7d004e 00007f6d2a7cef2e 00007f6d2a7d5405 00007f6d2a7d5d2f
0430 20:11:52.495895 (+435767us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:52.204805 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:52.204935 (+ 130us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:52.204940 (+ 5us) write_op.cc:427] Acquired schema lock
0430 20:11:52.204943 (+ 3us) tablet.cc:660] Decoding operations
0430 20:11:52.205028 (+ 85us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:52.205037 (+ 9us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:52.205041 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
0430 20:11:52.205107 (+ 66us) tablet.cc:699] Row locks acquired
0430 20:11:52.205109 (+ 2us) write_op.cc:260] PREPARE: finished
0430 20:11:52.205138 (+ 29us) write_op.cc:270] Start()
0430 20:11:52.205163 (+ 25us) write_op.cc:276] Timestamp: P: 1714507912205116 usec, L: 0
0430 20:11:52.205166 (+ 3us) op_driver.cc:331] REPLICATION: starting
0430 20:11:52.205309 (+ 143us) log.cc:842] Serialized 625 byte log entry
0430 20:11:52.248233 (+ 42924us) op_driver.cc:444] REPLICATION: finished
0430 20:11:52.249235 (+ 1002us) write_op.cc:301] APPLY: starting
0430 20:11:52.249249 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:52.249292 (+ 43us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:52.249331 (+ 39us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:52.249429 (+ 98us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:52.249486 (+ 57us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0430 20:11:52.249497 (+ 11us) write_op.cc:312] APPLY: finished
0430 20:11:52.249558 (+ 61us) log.cc:842] Serialized 29 byte log entry
0430 20:11:52.249604 (+ 46us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:52.249625 (+ 21us) write_op.cc:446] Released schema lock
0430 20:11:52.249632 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:52.249689 (+ 57us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:52.249696 (+ 7us) write_op.cc:446] Released schema lock
Metrics: {"spinlock_wait_cycles":194493184,"child_traces":[["op",{"apply.queue_time_us":964,"apply.run_cpu_time_us":508,"apply.run_wall_time_us":508,"num_ops":2,"prepare.queue_time_us":46,"prepare.run_cpu_time_us":573,"prepare.run_wall_time_us":877,"replication_time_us":43048}]]}
I20240430 20:11:52.496448 18759 raft_consensus.cc:2454] T eb6c96047b114ebba73db7a371f2da8e P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
I20240430 20:11:52.498207 18447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06e2dd1f7e8d434d9094f1acdec95e09. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:52.498432 18760 raft_consensus.cc:2454] T 62732af95c6a4c749c5b726f0460dc72 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 0.
I20240430 20:11:52.414223 18699 leader_election.cc:302] T 62732af95c6a4c749c5b726f0460dc72 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710
I20240430 20:11:52.435798 21198 raft_consensus.cc:482] T 35fdb052135541cfb42a8aeb03b64dc8 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:52.491981 18733 tablet_service.cc:1467] Processing CreateTablet for tablet c556f1aa75ac4c908e654e01d72d45aa (DEFAULT_TABLE table=default.Table_c1538175cbb94c18b8fd9e5de5436047 [id=1a4baeb013fb478c96f8802a6f0b419f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:52.404011 18753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89b774967624498fa2be499878b71124" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:52.355832 19944 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: Sending DeleteTablet for 3 replicas of tablet 13c2e69ae91f4957887fd64b90f6e435
I20240430 20:11:52.498790 21198 raft_consensus.cc:504] T 35fdb052135541cfb42a8aeb03b64dc8 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
W20240430 20:11:52.498917 18699 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.177.1:41217, user_credentials={real_user=slave}} blocked reactor thread for 84743.4us
I20240430 20:11:52.499289 20938 raft_consensus.cc:3037] T 62732af95c6a4c749c5b726f0460dc72 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:52.499351 19944 catalog_manager.cc:6554] ReplaceTablet: tablet 13c2e69ae91f4957887fd64b90f6e435 deleted and replaced by tablet eba4bd4665ae4c04a2cab3ab1a2da780
I20240430 20:11:52.383934 18455 tablet_service.cc:1467] Processing CreateTablet for tablet 8abf0b43f8a447dc902135ce94e4fc78 (DEFAULT_TABLE table=default.Table_5572f0acadaa432e8570b159fd9b76bc [id=d0f3b1709ac94aed8ec568c0327eab34]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:52.499492 19944 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:50746 (request call id 63) took 1614 ms. Trace:
I20240430 20:11:52.499553 21198 leader_election.cc:288] T 35fdb052135541cfb42a8aeb03b64dc8 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:52.499574 19944 rpcz_store.cc:276] 0430 20:11:50.884731 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:51.966275 (+1081544us) service_pool.cc:225] Handling call
0430 20:11:52.042194 (+ 75919us) spinlock_profiling.cc:243] Waited 75.4 ms on lock 0x616000070654. stack: 00007f6d23ac8756 00007f6d23ac8510 00007f6d22afb48d 0000000000955804 00000000009557cd 00007f6d3260715d 00007f6d32607253 00007f6d2f339d41 00007f6d36f2534d 00007f6d370be05b 00007f6d2dea6be7 00007f6d2dea6915 00007f6d2a7d004e 00007f6d2a7cef2e 00007f6d2a7d5405 00007f6d2a7d5d2f
0430 20:11:52.499444 (+457250us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:52.202399 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:52.202507 (+ 108us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:52.202513 (+ 6us) write_op.cc:427] Acquired schema lock
0430 20:11:52.202517 (+ 4us) tablet.cc:660] Decoding operations
0430 20:11:52.202648 (+ 131us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:52.202658 (+ 10us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:52.202665 (+ 7us) tablet.cc:683] Acquiring locks for 2 operations
0430 20:11:52.202741 (+ 76us) tablet.cc:699] Row locks acquired
0430 20:11:52.202745 (+ 4us) write_op.cc:260] PREPARE: finished
0430 20:11:52.202788 (+ 43us) write_op.cc:270] Start()
0430 20:11:52.202814 (+ 26us) write_op.cc:276] Timestamp: P: 1714507912202766 usec, L: 0
0430 20:11:52.202818 (+ 4us) op_driver.cc:331] REPLICATION: starting
0430 20:11:52.202955 (+ 137us) log.cc:842] Serialized 625 byte log entry
0430 20:11:52.248144 (+ 45189us) op_driver.cc:444] REPLICATION: finished
0430 20:11:52.248352 (+ 208us) write_op.cc:301] APPLY: starting
0430 20:11:52.248380 (+ 28us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:52.248474 (+ 94us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:52.248523 (+ 49us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:52.248686 (+ 163us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:52.248771 (+ 85us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0430 20:11:52.248783 (+ 12us) write_op.cc:312] APPLY: finished
0430 20:11:52.248901 (+ 118us) log.cc:842] Serialized 29 byte log entry
0430 20:11:52.248992 (+ 91us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:52.249024 (+ 32us) write_op.cc:446] Released schema lock
0430 20:11:52.249035 (+ 11us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:52.249114 (+ 79us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:52.249120 (+ 6us) write_op.cc:446] Released schema lock
Metrics: {"spinlock_wait_cycles":173475584,"child_traces":[["op",{"apply.queue_time_us":130,"apply.run_cpu_time_us":839,"apply.run_wall_time_us":838,"num_ops":2,"prepare.queue_time_us":3303,"prepare.run_cpu_time_us":639,"prepare.run_wall_time_us":1482,"replication_time_us":45283}]]}
I20240430 20:11:52.500411 19944 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50746:
name: "default.Table_663f87c0ffdd472dba96f9c67665e8d6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:52.500989 19944 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_663f87c0ffdd472dba96f9c67665e8d6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:52.501688 20938 raft_consensus.cc:2726] T 62732af95c6a4c749c5b726f0460dc72 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:52.257194 18417 leader_election.cc:302] T 9f0b341a31224bf1be525b84b9500304 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters: 5e1c6397f09747f091534faf93542710
W20240430 20:11:52.502089 18417 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.177.3:44867, user_credentials={real_user=slave}} blocked reactor thread for 244907us
I20240430 20:11:52.502334 18417 leader_election.cc:302] T 898b2c91adf34545808df9c786e218c9 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a; no voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710
I20240430 20:11:52.502775 20630 raft_consensus.cc:2781] T 9f0b341a31224bf1be525b84b9500304 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240430 20:11:52.502835 20630 raft_consensus.cc:482] T 9f0b341a31224bf1be525b84b9500304 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:52.502897 20630 raft_consensus.cc:3037] T 9f0b341a31224bf1be525b84b9500304 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Advancing to term 2
I20240430 20:11:52.398240 19945 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 6b51cbc731794083a88fe4bed1ddeebf (table default.Table_c6c962da746e4fd8a4d7200d09f08e9e [id=8e0f5d8c941d423ea7b30a89513d5170]). Expected version 1 got 0
I20240430 20:11:52.503000 18739 tablet_service.cc:1467] Processing CreateTablet for tablet 2ca98d4245d74c9e8978cd40f2c26342 (DEFAULT_TABLE table=default.Table_066abbbc2a9a4c8f8025b62ceb68ca8a [id=bac68754698f4c298984c39fe9f15853]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:52.503285 19945 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 62732af95c6a4c749c5b726f0460dc72 (table default.Table_e3160f5ced1b4718b479401ca4df675c [id=2d44ec94131148048021b2ee6366ba30]). Expected version 3 got 0
I20240430 20:11:52.503583 19945 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet a0baee91f5b44945b595d0e234af018e (table default.Table_5ed3ea587cf843db9a1ff1ffab40b678 [id=b3d1c2f97c4648518dbea83a46cadd85]). Expected version 2 got 0
I20240430 20:11:52.503751 19945 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 266fd23786bc435db5083370f3330ed0 (table default.Table_cf60b652e9dc49e7a948ce66fd1cbffe [id=9734bb8dd05a400ca4244b3829f01dee]). Expected version 1 got 0
I20240430 20:11:52.504158 19945 catalog_manager.cc:5484] T f90a5c5818c54390a2ae323fed7ebe57 P 2efb542aa55e439b99ccb98d1c5c884a reported cstate change: term changed from 1 to 2, leader changed from 5e1c6397f09747f091534faf93542710 (127.17.177.2) to 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1). New cstate: current_term: 2 leader_uuid: "2efb542aa55e439b99ccb98d1c5c884a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } health_report { overall_health: HEALTHY } } }
I20240430 20:11:52.504650 21203 raft_consensus.cc:2229] T ebf74809b52f47f3b0ed21b6ac0f541a P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:52.504848 21203 raft_consensus.cc:2258] T ebf74809b52f47f3b0ed21b6ac0f541a P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:52.505373 18742 tablet_service.cc:1467] Processing CreateTablet for tablet f6ab6cf7858348f7a9c6aba1d61ade87 (DEFAULT_TABLE table=default.Table_a93c32ea6b7c49a1941650334c85b5e4 [id=52c66e0eee004fad9ad1a493645a99ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:52.505446 21203 ts_tablet_manager.cc:1875] T ebf74809b52f47f3b0ed21b6ac0f541a P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:52.505563 19945 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 89b774967624498fa2be499878b71124 (table default.Table_2ec9e316bc3347b189a13b7a8bbcf829 [id=9fa22833979f4c2f978de651145acef9]). Expected version 1 got 0
I20240430 20:11:52.505640 18750 raft_consensus.cc:2454] T 35fdb052135541cfb42a8aeb03b64dc8 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 0.
I20240430 20:11:52.505805 19945 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 130007acc8ba4e709c3eb85fabe75903 (table default.Table_78987875650940c293247bc43db345ab [id=36f478a20a1f4a53a52fc29ade62b7f6]). Expected version 1 got 0
I20240430 20:11:52.506003 19945 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet dac141d1b70c48d785caddd7182811ee (table default.Table_fc0a4540f98b4eea95099b8ab46ea235 [id=f3c53ccffe64410495cc8c379a05cfae]). Expected version 1 got 0
I20240430 20:11:52.506783 19945 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet ed0c37a71ede4bfc9eb5ad1a4365f25e (table default.Table_a2e85efc6b454524b0b122102ec07002 [id=86d795a82e674d859104741ebd04f4e8]). Expected version 1 got 0
I20240430 20:11:52.507023 19945 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 12795670f21447a8964fe1efc6e92398 (table default.Table_896bbae1f6944005b0272ca8ebc97218 [id=d5cea90cbc2c48df9d8b80da1eccc949]). Expected version 3 got 0
I20240430 20:11:52.507318 21054 raft_consensus.cc:2726] T 898b2c91adf34545808df9c786e218c9 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:52.507859 18730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8991518bfe3a44c9bb85a52b4e06040b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:52.510931 19945 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 724e51ff05ec4baebc2bb55e05c8eddc (table default.Table_19df33614a6c4f94b6419a5b441e3ef8 [id=4dce8a767f0541e1beb5464b2ed95d06]). Expected version 2 got 1
I20240430 20:11:52.511165 19945 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet f1073fc792954c5795a05aa7d0ec24df (table default.Table_ba8b218e936b4bfcb073f8182f474cca [id=aedf11fda8914a3b843a57f8a7b69a53]). Expected version 1 got 0
I20240430 20:11:52.511844 19945 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 66dd767bd3f8494f94061fb89e5e5680 (table default.Table_b064d5659f7243f5b0ec167fb6c3f404 [id=5227206f44d04fa89b4842b899f33afb]). Expected version 2 got 0
I20240430 20:11:52.512156 19945 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 37b2ea4be7494f0985808ade788dbe2b (table default.Table_5f6e0fce999b46de927f9271e5de5071 [id=0baad83a8c0f4aeebc7ddad068625a69]). Expected version 1 got 0
I20240430 20:11:52.512405 19945 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 22fef8bb6b034db098c3372ae992f7a7 (table default.Table_4440df38610340758047115845b42b95 [id=7d186ca20c094cc19aef792fe1c094e9]). Expected version 1 got 0
I20240430 20:11:52.512554 19945 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 330453ddeea6456e8ef93fa92608b0d3 (table default.Table_c1f36f7df5ae4e7c8eb8b4e99b89748b [id=a50abf28790d4284a76ac6ee22c02606]). Expected version 1 got 0
I20240430 20:11:52.490370 18595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet daeef5d233f24602ad5d64f7685a50eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:52.512712 19945 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet acd23f0a63d5448aaa7ed78422e67bad (table default.Table_972f13be4dca4cd19b9a04534dc59f51 [id=69f37db4de634a3185613e40013e0ed7]). Expected version 1 got 0
I20240430 20:11:52.512912 19945 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 2e79df8744e74b5991a79ce978ff0202 (table default.Table_276bbdc7cfe542d1a2936485a2aad1ec [id=f146a15e6e8a4acbb4c734c1c863da57]). Expected version 1 got 0
I20240430 20:11:52.513211 19945 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet ca64de16d40f4a44b8b411f5526ba11a (table default.Table_1cf914acf7fe4e98b8dbc297b6372f21 [id=f4ba9536e9c141deb68f4f62b7e8fa70]). Expected version 1 got 0
I20240430 20:11:52.513383 19945 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 69f88532952a4591bf1577541c3430e6 (table default.Table_1177828cad1f4a24bd397619dad96170 [id=a95fed1bc3fa4865bf9a35dfc51b659c]). Expected version 2 got 0
I20240430 20:11:52.513759 19945 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 086b3dfa84c045a18c8c4f177dce606a (table default.Table_df19cb8a6f4d46699350ec5b69c14e5a [id=197923633fc54a4582207942b51530e2]). Expected version 2 got 0
I20240430 20:11:52.513927 19945 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 1050ab528c7d41559831652a1437d358 (table default.Table_acddc4cbf6334546b5fb3e4752fef2a5 [id=274cb15fd751410e83327ac4e8dc4629]). Expected version 2 got 0
I20240430 20:11:52.514818 18736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a37ff93ed9bf4b70b57fea7c038d2195. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:52.490825 18587 tablet_service.cc:1467] Processing CreateTablet for tablet 2ca98d4245d74c9e8978cd40f2c26342 (DEFAULT_TABLE table=default.Table_066abbbc2a9a4c8f8025b62ceb68ca8a [id=bac68754698f4c298984c39fe9f15853]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:52.528873 18587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ca98d4245d74c9e8978cd40f2c26342. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:52.491014 18585 tablet_service.cc:1467] Processing CreateTablet for tablet 9743ebed69784fda9d26dc125d205fbf (DEFAULT_TABLE table=default.Table_49d181ca513240d88696579ad5b4d3d8 [id=4de11fd888f645519d9398eb7b556d85]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:52.530938 18585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9743ebed69784fda9d26dc125d205fbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:52.491186 18593 tablet_service.cc:1467] Processing CreateTablet for tablet 85baf4cd31cb40d281bae432bafd2791 (DEFAULT_TABLE table=default.Table_11ef4a2c1255424bac9dd34fc0313926 [id=1906e2cda0f74f568d0e00f7b3badeba]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:52.534459 18593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85baf4cd31cb40d281bae432bafd2791. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:52.491372 18599 tablet_service.cc:1467] Processing CreateTablet for tablet 8abf0b43f8a447dc902135ce94e4fc78 (DEFAULT_TABLE table=default.Table_5572f0acadaa432e8570b159fd9b76bc [id=d0f3b1709ac94aed8ec568c0327eab34]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:52.538758 18599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8abf0b43f8a447dc902135ce94e4fc78. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:52.493968 18602 tablet_service.cc:1467] Processing CreateTablet for tablet 8991518bfe3a44c9bb85a52b4e06040b (DEFAULT_TABLE table=default.Table_8bcd1481133a4f45aac58ee0460b7ef8 [id=cb8d130400a1468fb08e0811d07dba95]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:52.494115 18590 tablet_service.cc:1467] Processing CreateTablet for tablet 06e2dd1f7e8d434d9094f1acdec95e09 (DEFAULT_TABLE table=default.Table_31760cb86ebf4666abcffc4217aa2708 [id=f235b4c2340d45a2b348f1488dc3a6c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:52.494240 18603 tablet_service.cc:1467] Processing CreateTablet for tablet a37ff93ed9bf4b70b57fea7c038d2195 (DEFAULT_TABLE table=default.Table_e158c1ec239f48a9be498828ad9a5a72 [id=eb0231e25f204632808706605865d342]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:52.494400 18586 tablet_service.cc:1467] Processing CreateTablet for tablet ca952d9886304b1593160338d7d30458 (DEFAULT_TABLE table=default.Table_45b0755ed3e748f78fe856a4ab1831cc [id=71f2b0943f0f4b4e96617108202740fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:52.494539 18592 tablet_service.cc:1467] Processing CreateTablet for tablet c556f1aa75ac4c908e654e01d72d45aa (DEFAULT_TABLE table=default.Table_c1538175cbb94c18b8fd9e5de5436047 [id=1a4baeb013fb478c96f8802a6f0b419f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:52.542974 18602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8991518bfe3a44c9bb85a52b4e06040b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:52.543274 18603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a37ff93ed9bf4b70b57fea7c038d2195. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:52.494808 18589 tablet_service.cc:1467] Processing CreateTablet for tablet 9a5654fd913f4fbeb9d126f035ac32bf (DEFAULT_TABLE table=default.Table_766e3df58895427b9e5179eb555fe7e4 [id=5c11311047b34a3eb06cc06e8cc10223]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:52.546991 18589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a5654fd913f4fbeb9d126f035ac32bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:52.514825 18988 tablet_bootstrap.cc:492] T 2aa2a489116940dea3173cad16d94fa5 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:52.518350 18732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca952d9886304b1593160338d7d30458. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:52.561401 18739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ca98d4245d74c9e8978cd40f2c26342. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:52.561708 18744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd1de890e4394db287cf38bb411f5586. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:52.561921 18733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c556f1aa75ac4c908e654e01d72d45aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:52.555361 20587 consensus_queue.cc:1035] T 417351cbc9d54f9d84bb7155f39da119 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:52.564878 18742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6ab6cf7858348f7a9c6aba1d61ade87. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:52.566900 18724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8abf0b43f8a447dc902135ce94e4fc78. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:52.567652 18738 tablet_service.cc:1514] Processing DeleteTablet for tablet f9bf604e6c234349a9f72f45a4c4aef6 with delete_type TABLET_DATA_DELETED (replaced by tablet 415d99c36b1941e08aa4a16c50483ee4) from {username='slave'} at 127.0.0.1:56540
I20240430 20:11:52.567879 18731 tablet_service.cc:1514] Processing DeleteTablet for tablet 13c2e69ae91f4957887fd64b90f6e435 with delete_type TABLET_DATA_DELETED (replaced by tablet eba4bd4665ae4c04a2cab3ab1a2da780) from {username='slave'} at 127.0.0.1:56540
I20240430 20:11:52.568017 18988 tablet_bootstrap.cc:654] T 2aa2a489116940dea3173cad16d94fa5 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:52.514412 19945 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet e041756c962b4a9f97c2cb314e7d8c05 (table default.Table_de88cafd92b149f9b545369bff71033c [id=c4bf4806f3d84579b16a3038d8d982c3]). Expected version 1 got 0
I20240430 20:11:52.535317 19944 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:50746 (request call id 66) took 1570 ms. Trace:
I20240430 20:11:52.572029 19944 rpcz_store.cc:276] 0430 20:11:50.965068 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:52.500171 (+1535103us) service_pool.cc:225] Handling call
0430 20:11:52.501088 (+ 917us) catalog_manager.cc:2072] Acquired catalog manager lock
0430 20:11:52.501534 (+ 446us) catalog_manager.cc:2118] Created new table and tablet info
0430 20:11:52.530722 (+ 29188us) catalog_manager.cc:2178] Wrote table and tablets to system table
0430 20:11:52.535119 (+ 4397us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0430 20:11:52.535248 (+ 129us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:52.518147 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:52.518296 (+ 149us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:52.518304 (+ 8us) write_op.cc:427] Acquired schema lock
0430 20:11:52.518306 (+ 2us) tablet.cc:660] Decoding operations
0430 20:11:52.518429 (+ 123us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:52.518445 (+ 16us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:52.518450 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0430 20:11:52.518539 (+ 89us) tablet.cc:699] Row locks acquired
0430 20:11:52.518542 (+ 3us) write_op.cc:260] PREPARE: finished
0430 20:11:52.518588 (+ 46us) write_op.cc:270] Start()
0430 20:11:52.519459 (+ 871us) write_op.cc:276] Timestamp: P: 1714507912518534 usec, L: 0
0430 20:11:52.519466 (+ 7us) op_driver.cc:331] REPLICATION: starting
0430 20:11:52.519721 (+ 255us) log.cc:842] Serialized 433 byte log entry
0430 20:11:52.526243 (+ 6522us) op_driver.cc:444] REPLICATION: finished
0430 20:11:52.526372 (+ 129us) write_op.cc:301] APPLY: starting
0430 20:11:52.526395 (+ 23us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:52.526446 (+ 51us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:52.526483 (+ 37us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:52.526677 (+ 194us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:52.526724 (+ 47us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0430 20:11:52.526731 (+ 7us) write_op.cc:312] APPLY: finished
0430 20:11:52.526826 (+ 95us) log.cc:842] Serialized 29 byte log entry
0430 20:11:52.526897 (+ 71us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:52.526929 (+ 32us) write_op.cc:446] Released schema lock
0430 20:11:52.526944 (+ 15us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:52.527010 (+ 66us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:52.527014 (+ 4us) write_op.cc:446] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":73,"apply.run_cpu_time_us":684,"apply.run_wall_time_us":701,"num_ops":2,"prepare.queue_time_us":16091,"prepare.run_cpu_time_us":913,"prepare.run_wall_time_us":1693,"raft.queue_time_us":450,"raft.run_cpu_time_us":259,"raft.run_wall_time_us":259,"replication_time_us":6750}]]}
I20240430 20:11:52.572957 18729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9743ebed69784fda9d26dc125d205fbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:52.573550 19944 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50746:
name: "default.Table_8c21c0353321455a82097231a06f0957"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:52.574148 19944 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8c21c0353321455a82097231a06f0957 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:52.578534 18737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85baf4cd31cb40d281bae432bafd2791. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:52.583315 18725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06e2dd1f7e8d434d9094f1acdec95e09. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:52.585093 19944 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:50746 (request call id 67) took 1619 ms. Trace:
I20240430 20:11:52.586717 18735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1754ef24e86421f83ca3476dbc3a43a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:52.591475 18472 raft_consensus.cc:3037] T 828ca913e2aa4102a6277bdbc6686ef5 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:52.585824 19945 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet aaef85eea87f4443943fd3692bb2a38f (table default.Table_3da5b4c8668c470483ae2a0c7773eaaf [id=e292d818ee3c45df90522f08eb2d6227]). Expected version 1 got 0
I20240430 20:11:52.597056 19945 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet a59ef6c545dd41a28265a4c1b8ce10e0 (table default.Table_07e40af8f1c546048975c8edf88170d7 [id=86df5d161d8641c89070e7eed9a39360]). Expected version 2 got 0
I20240430 20:11:52.598121 19945 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 81787eccca6c42c1b421377a3bf9ebe6 (table default.Table_0b4ca4cd0af148399dfa21f2d32114e3 [id=7f602584cef9486eb7d9b4389be3f43e]). Expected version 3 got 2
I20240430 20:11:52.598871 19945 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 47bf64ecc41b445da4ff697e141243dd (table default.Table_f1ad29d5986d4b799cfc67afb2b0dfdf [id=838fc8b61a864dea8877a6f144e4cb4f]). Expected version 2 got 0
I20240430 20:11:52.599431 18734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a5654fd913f4fbeb9d126f035ac32bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:52.599460 19945 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 94fbd4a57a1e438891fcb456f6b40be3 (table default.Table_25ddc3adef6b4ccf8b9c00cc6772031c [id=4fa623ffdcf347639150a355806038af]). Expected version 1 got 0
I20240430 20:11:52.599733 18740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a81c2d2790aa4e94809a2edd7cd0555e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:52.600665 19945 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet a482f4622f774d60b4efd6827b8be7d7 (table default.Table_e686cdfcbcc046749fa5b51578a64794 [id=c3dc9f05316d46d6a3ef4d03f40877ec]). Expected version 1 got 0
I20240430 20:11:52.585316 19944 rpcz_store.cc:276] 0430 20:11:50.965148 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:52.573269 (+1608121us) service_pool.cc:225] Handling call
0430 20:11:52.574237 (+ 968us) catalog_manager.cc:2072] Acquired catalog manager lock
0430 20:11:52.574805 (+ 568us) catalog_manager.cc:2118] Created new table and tablet info
0430 20:11:52.584281 (+ 9476us) catalog_manager.cc:2178] Wrote table and tablets to system table
0430 20:11:52.584580 (+ 299us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0430 20:11:52.584854 (+ 274us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:52.575953 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:52.576091 (+ 138us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:52.576099 (+ 8us) write_op.cc:427] Acquired schema lock
0430 20:11:52.576104 (+ 5us) tablet.cc:660] Decoding operations
0430 20:11:52.576239 (+ 135us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:52.576280 (+ 41us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:52.576288 (+ 8us) tablet.cc:683] Acquiring locks for 2 operations
0430 20:11:52.576396 (+ 108us) tablet.cc:699] Row locks acquired
0430 20:11:52.576400 (+ 4us) write_op.cc:260] PREPARE: finished
0430 20:11:52.576443 (+ 43us) write_op.cc:270] Start()
0430 20:11:52.576489 (+ 46us) write_op.cc:276] Timestamp: P: 1714507912576384 usec, L: 0
0430 20:11:52.576493 (+ 4us) op_driver.cc:331] REPLICATION: starting
0430 20:11:52.576733 (+ 240us) log.cc:842] Serialized 433 byte log entry
0430 20:11:52.580782 (+ 4049us) op_driver.cc:444] REPLICATION: finished
0430 20:11:52.580904 (+ 122us) write_op.cc:301] APPLY: starting
0430 20:11:52.580923 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:52.580994 (+ 71us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:52.581030 (+ 36us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:52.581181 (+ 151us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:52.581228 (+ 47us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0430 20:11:52.581241 (+ 13us) write_op.cc:312] APPLY: finished
0430 20:11:52.581337 (+ 96us) log.cc:842] Serialized 29 byte log entry
0430 20:11:52.581409 (+ 72us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:52.581438 (+ 29us) write_op.cc:446] Released schema lock
0430 20:11:52.581446 (+ 8us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:52.581510 (+ 64us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:52.581516 (+ 6us) write_op.cc:446] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":66,"apply.run_cpu_time_us":669,"apply.run_wall_time_us":670,"num_ops":2,"prepare.queue_time_us":94,"prepare.run_cpu_time_us":910,"prepare.run_wall_time_us":911,"raft.queue_time_us":90,"raft.run_cpu_time_us":329,"raft.run_wall_time_us":329,"replication_time_us":4259}]]}
I20240430 20:11:52.602815 18726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet daeef5d233f24602ad5d64f7685a50eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:52.604157 19944 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50746:
name: "default.Table_1a823c2465ce4224a3cf173600f6e32c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:52.605340 19944 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1a823c2465ce4224a3cf173600f6e32c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:52.612185 18988 tablet_bootstrap.cc:492] T 2aa2a489116940dea3173cad16d94fa5 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:52.612715 18988 ts_tablet_manager.cc:1367] T 2aa2a489116940dea3173cad16d94fa5 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.098s user 0.002s sys 0.009s
I20240430 20:11:52.615821 18988 raft_consensus.cc:348] T 2aa2a489116940dea3173cad16d94fa5 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:52.616518 18988 raft_consensus.cc:374] T 2aa2a489116940dea3173cad16d94fa5 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:52.616776 18988 raft_consensus.cc:729] T 2aa2a489116940dea3173cad16d94fa5 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:52.617584 18988 consensus_queue.cc:260] T 2aa2a489116940dea3173cad16d94fa5 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:52.619918 18988 ts_tablet_manager.cc:1398] T 2aa2a489116940dea3173cad16d94fa5 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20240430 20:11:52.620883 18988 tablet_bootstrap.cc:492] T aa9998f4e17147aa8f2b35c01980463a P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:52.624938 21203 ts_tablet_manager.cc:1888] T ebf74809b52f47f3b0ed21b6ac0f541a P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240430 20:11:52.625061 21203 log.cc:1196] T ebf74809b52f47f3b0ed21b6ac0f541a P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/ebf74809b52f47f3b0ed21b6ac0f541a
I20240430 20:11:52.625432 21203 ts_tablet_manager.cc:1909] T ebf74809b52f47f3b0ed21b6ac0f541a P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:52.627389 19900 catalog_manager.cc:4830] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): tablet ebf74809b52f47f3b0ed21b6ac0f541a (table default.Table_b2d07520b9944a89ad62fca70396174e [id=1c8e0a3a44ec4a798770c0165ed3dffa]) successfully deleted
W20240430 20:11:52.627703 19900 catalog_manager.cc:4810] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): delete failed for tablet f9bf604e6c234349a9f72f45a4c4aef6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f9bf604e6c234349a9f72f45a4c4aef6 already in progress: creating tablet
I20240430 20:11:52.628276 19945 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet eb6c96047b114ebba73db7a371f2da8e (table default.Table_7d5f6f18cbd64c869739872a7dff9be2 [id=e67d619e36c0485f8f22b4a81cb4cb6c]). Expected version 1 got 0
W20240430 20:11:52.633023 18699 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.177.1:41217, user_credentials={real_user=slave}} blocked reactor thread for 133752us
I20240430 20:11:52.633428 21198 consensus_queue.cc:1035] T 898b2c91adf34545808df9c786e218c9 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:52.548180 18601 tablet_service.cc:1514] Processing DeleteTablet for tablet f9bf604e6c234349a9f72f45a4c4aef6 with delete_type TABLET_DATA_DELETED (replaced by tablet 415d99c36b1941e08aa4a16c50483ee4) from {username='slave'} at 127.0.0.1:45934
I20240430 20:11:52.512418 18616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35fdb052135541cfb42a8aeb03b64dc8" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:52.494689 18588 tablet_service.cc:1467] Processing CreateTablet for tablet a81c2d2790aa4e94809a2edd7cd0555e (DEFAULT_TABLE table=default.Table_93f5811e02a742e180954822432cacd1 [id=f3912511868f48fb8124ebe82b0c5fa0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:52.494920 18591 tablet_service.cc:1467] Processing CreateTablet for tablet dd1de890e4394db287cf38bb411f5586 (DEFAULT_TABLE table=default.Table_84084a27f7504dd992fce07448c10df3 [id=6257559b0e9545c58848a76af0c5a5f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:52.550824 18586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca952d9886304b1593160338d7d30458. 1 dirs total, 0 dirs full, 0 dirs failed
W20240430 20:11:52.639469 19895 catalog_manager.cc:4810] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): delete failed for tablet f9bf604e6c234349a9f72f45a4c4aef6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f9bf604e6c234349a9f72f45a4c4aef6 already in progress: creating tablet
I20240430 20:11:52.640733 18988 tablet_bootstrap.cc:654] T aa9998f4e17147aa8f2b35c01980463a P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:52.642776 18591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd1de890e4394db287cf38bb411f5586. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:52.647341 18588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a81c2d2790aa4e94809a2edd7cd0555e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:52.647687 18462 tablet_service.cc:1514] Processing DeleteTablet for tablet f9bf604e6c234349a9f72f45a4c4aef6 with delete_type TABLET_DATA_DELETED (replaced by tablet 415d99c36b1941e08aa4a16c50483ee4) from {username='slave'} at 127.0.0.1:41450
I20240430 20:11:52.648818 21226 consensus_queue.cc:1035] T f90a5c5818c54390a2ae323fed7ebe57 P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:52.653801 18461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9743ebed69784fda9d26dc125d205fbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:52.654222 18445 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8991518bfe3a44c9bb85a52b4e06040b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:52.658944 18455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8abf0b43f8a447dc902135ce94e4fc78. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:52.610924 18983 tablet_bootstrap.cc:492] T a692b253db914531b8fc6ffaf3e350b5 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:52.616555 18600 tablet_service.cc:1514] Processing DeleteTablet for tablet 13c2e69ae91f4957887fd64b90f6e435 with delete_type TABLET_DATA_DELETED (replaced by tablet eba4bd4665ae4c04a2cab3ab1a2da780) from {username='slave'} at 127.0.0.1:45934
I20240430 20:11:52.660333 18983 ts_tablet_manager.cc:1367] T a692b253db914531b8fc6ffaf3e350b5 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.180s user 0.004s sys 0.000s
I20240430 20:11:52.660467 21213 tablet_replica.cc:330] T 13c2e69ae91f4957887fd64b90f6e435 P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:52.660699 21213 raft_consensus.cc:2229] T 13c2e69ae91f4957887fd64b90f6e435 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:52.660866 21213 raft_consensus.cc:2258] T 13c2e69ae91f4957887fd64b90f6e435 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:52.661823 21213 ts_tablet_manager.cc:1875] T 13c2e69ae91f4957887fd64b90f6e435 P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:52.667089 18983 raft_consensus.cc:348] T a692b253db914531b8fc6ffaf3e350b5 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:52.667379 18983 raft_consensus.cc:374] T a692b253db914531b8fc6ffaf3e350b5 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:52.667464 18983 raft_consensus.cc:729] T a692b253db914531b8fc6ffaf3e350b5 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:52.667697 18983 consensus_queue.cc:260] T a692b253db914531b8fc6ffaf3e350b5 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:52.668587 18983 ts_tablet_manager.cc:1398] T a692b253db914531b8fc6ffaf3e350b5 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240430 20:11:52.668920 18983 tablet_bootstrap.cc:492] T 69f88532952a4591bf1577541c3430e6 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:52.670305 18452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca952d9886304b1593160338d7d30458. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:52.671995 20630 raft_consensus.cc:504] T 9f0b341a31224bf1be525b84b9500304 P 2efb542aa55e439b99ccb98d1c5c884a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:52.672775 21213 ts_tablet_manager.cc:1888] T 13c2e69ae91f4957887fd64b90f6e435 P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:52.672883 21213 log.cc:1196] T 13c2e69ae91f4957887fd64b90f6e435 P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/13c2e69ae91f4957887fd64b90f6e435
I20240430 20:11:52.673197 21213 ts_tablet_manager.cc:1909] T 13c2e69ae91f4957887fd64b90f6e435 P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:52.673368 18985 raft_consensus.cc:348] T 2e1cac64cefe4336bd3207b5ca58b631 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:52.673547 18985 raft_consensus.cc:374] T 2e1cac64cefe4336bd3207b5ca58b631 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:52.673599 18985 raft_consensus.cc:729] T 2e1cac64cefe4336bd3207b5ca58b631 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:52.673846 19895 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet 13c2e69ae91f4957887fd64b90f6e435 (table default.Table_6b67c0c03ceb4b9491ad5828022b357f [id=368fcf43e863494ca260862d12331c0d]) successfully deleted
I20240430 20:11:52.673802 18985 consensus_queue.cc:260] T 2e1cac64cefe4336bd3207b5ca58b631 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:52.674054 18590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06e2dd1f7e8d434d9094f1acdec95e09. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:52.674332 18985 ts_tablet_manager.cc:1398] T 2e1cac64cefe4336bd3207b5ca58b631 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.195s user 0.000s sys 0.002s
I20240430 20:11:52.674724 21281 tablet.cc:1722] T b285e3eb16d148779c74e057458d8461 P 3c51c33d5616482cafb68e26688820f4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240430 20:11:52.675367 18456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd1de890e4394db287cf38bb411f5586. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:52.676054 18450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a81c2d2790aa4e94809a2edd7cd0555e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:52.678429 18448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ca98d4245d74c9e8978cd40f2c26342. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:52.678627 18457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85baf4cd31cb40d281bae432bafd2791. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:52.678730 18592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c556f1aa75ac4c908e654e01d72d45aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:52.678777 18458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet daeef5d233f24602ad5d64f7685a50eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:52.679342 18446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a5654fd913f4fbeb9d126f035ac32bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:52.679744 18983 tablet_bootstrap.cc:654] T 69f88532952a4591bf1577541c3430e6 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:52.681133 18454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c556f1aa75ac4c908e654e01d72d45aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:52.681936 18449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6ab6cf7858348f7a9c6aba1d61ade87. 1 dirs total, 0 dirs full, 0 dirs failed
W20240430 20:11:52.682482 18557 leader_election.cc:341] T aa9998f4e17147aa8f2b35c01980463a P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:52.681644 18988 tablet_bootstrap.cc:492] T aa9998f4e17147aa8f2b35c01980463a P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:52.682633 18557 leader_election.cc:302] T aa9998f4e17147aa8f2b35c01980463a P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:52.683079 21238 raft_consensus.cc:2726] T aa9998f4e17147aa8f2b35c01980463a P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240430 20:11:52.684418 19900 catalog_manager.cc:4810] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): delete failed for tablet 13c2e69ae91f4957887fd64b90f6e435 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 13c2e69ae91f4957887fd64b90f6e435 already in progress: creating tablet
I20240430 20:11:52.684707 18453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a37ff93ed9bf4b70b57fea7c038d2195. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:52.684124 18988 ts_tablet_manager.cc:1367] T aa9998f4e17147aa8f2b35c01980463a P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.063s user 0.004s sys 0.003s
I20240430 20:11:52.686055 20630 leader_election.cc:288] T 9f0b341a31224bf1be525b84b9500304 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
W20240430 20:11:52.691349 18557 leader_election.cc:341] T 3a03b6997921408cb5a65f4cae223e9c P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
W20240430 20:11:52.692010 18557 leader_election.cc:341] T 2aa2a489116940dea3173cad16d94fa5 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
W20240430 20:11:52.692461 18557 leader_election.cc:341] T 2aa2a489116940dea3173cad16d94fa5 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
W20240430 20:11:52.692864 18557 leader_election.cc:341] T 3a03b6997921408cb5a65f4cae223e9c P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
W20240430 20:11:52.694829 18557 leader_election.cc:341] T 89b774967624498fa2be499878b71124 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:52.694962 18557 leader_election.cc:302] T 89b774967624498fa2be499878b71124 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:52.696812 18988 raft_consensus.cc:348] T aa9998f4e17147aa8f2b35c01980463a P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:52.699002 18988 raft_consensus.cc:374] T aa9998f4e17147aa8f2b35c01980463a P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:52.700639 21221 raft_consensus.cc:2726] T 89b774967624498fa2be499878b71124 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:52.700361 18988 raft_consensus.cc:729] T aa9998f4e17147aa8f2b35c01980463a P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
W20240430 20:11:52.701737 18415 leader_election.cc:341] T 9f0b341a31224bf1be525b84b9500304 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Not found: Tablet not found: 9f0b341a31224bf1be525b84b9500304
I20240430 20:11:52.704298 19945 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 99203cc522de424184c6f97453f166fd (table default.Table_cb2a4b9ceaf4431facfff2c9bcf94c5d [id=3f8690a437ae49a3beb009fa7899d5d0]). Expected version 1 got 0
I20240430 20:11:52.704599 19945 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 821c55edb3414d72b1489eea97d3f73a (table default.Table_4d68c38d9ba7452a829f8318b2b267b8 [id=d06e3e9045b1429aaeebd1fd67d841de]). Expected version 1 got 0
I20240430 20:11:52.704861 19945 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 5df364628561475391c9a5e10ff5a05f (table default.Table_3c8f9f7b22f64fb1bdc3f1550987233a [id=7fd6f8576f4c4cbc8ffcdb4ab95e15dc]). Expected version 1 got 0
I20240430 20:11:52.705286 19945 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 8e9711d0a382424f9f758bfa883866a0 (table default.Table_f1dc481c6d8c43f589fe2b1892a4cb10 [id=db696ce66005401fa73746e55c071f44]). Expected version 1 got 0
I20240430 20:11:52.706234 19945 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet f5eaf2413e1d4e69b56fae61f768f0ce (table default.Table_1280cfe32d5b41abb4d3eb1bd70641bf [id=8489b872074a42b6abb713e420968141]). Expected version 1 got 0
I20240430 20:11:52.711735 21301 tablet.cc:1722] T b285e3eb16d148779c74e057458d8461 P 5e1c6397f09747f091534faf93542710: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240430 20:11:52.726763 18451 tablet_service.cc:1514] Processing DeleteTablet for tablet 13c2e69ae91f4957887fd64b90f6e435 with delete_type TABLET_DATA_DELETED (replaced by tablet eba4bd4665ae4c04a2cab3ab1a2da780) from {username='slave'} at 127.0.0.1:41450
I20240430 20:11:52.703102 18988 consensus_queue.cc:260] T aa9998f4e17147aa8f2b35c01980463a P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:52.732790 19944 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:50746 (request call id 68) took 1767 ms. Trace:
I20240430 20:11:52.733021 19944 rpcz_store.cc:276] 0430 20:11:50.965759 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:52.603682 (+1637923us) service_pool.cc:225] Handling call
0430 20:11:52.717869 (+114187us) catalog_manager.cc:2072] Acquired catalog manager lock
0430 20:11:52.718398 (+ 529us) catalog_manager.cc:2118] Created new table and tablet info
0430 20:11:52.732456 (+ 14058us) catalog_manager.cc:2178] Wrote table and tablets to system table
0430 20:11:52.732599 (+ 143us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0430 20:11:52.732714 (+ 115us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:52.719084 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:52.719221 (+ 137us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:52.719228 (+ 7us) write_op.cc:427] Acquired schema lock
0430 20:11:52.719232 (+ 4us) tablet.cc:660] Decoding operations
0430 20:11:52.719335 (+ 103us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:52.719352 (+ 17us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:52.719358 (+ 6us) tablet.cc:683] Acquiring locks for 2 operations
0430 20:11:52.719447 (+ 89us) tablet.cc:699] Row locks acquired
0430 20:11:52.719450 (+ 3us) write_op.cc:260] PREPARE: finished
0430 20:11:52.719489 (+ 39us) write_op.cc:270] Start()
0430 20:11:52.719526 (+ 37us) write_op.cc:276] Timestamp: P: 1714507912719468 usec, L: 0
0430 20:11:52.719530 (+ 4us) op_driver.cc:331] REPLICATION: starting
0430 20:11:52.719730 (+ 200us) log.cc:842] Serialized 433 byte log entry
0430 20:11:52.724069 (+ 4339us) op_driver.cc:444] REPLICATION: finished
0430 20:11:52.724182 (+ 113us) write_op.cc:301] APPLY: starting
0430 20:11:52.724199 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:52.724255 (+ 56us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:52.724300 (+ 45us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:52.724403 (+ 103us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:52.724447 (+ 44us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0430 20:11:52.724455 (+ 8us) write_op.cc:312] APPLY: finished
0430 20:11:52.724547 (+ 92us) log.cc:842] Serialized 29 byte log entry
0430 20:11:52.724620 (+ 73us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:52.724643 (+ 23us) write_op.cc:446] Released schema lock
0430 20:11:52.724651 (+ 8us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:52.724702 (+ 51us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:52.724705 (+ 3us) write_op.cc:446] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":67,"apply.run_cpu_time_us":572,"apply.run_wall_time_us":571,"num_ops":2,"prepare.queue_time_us":121,"prepare.run_cpu_time_us":752,"prepare.run_wall_time_us":752,"raft.queue_time_us":472,"raft.run_cpu_time_us":265,"raft.run_wall_time_us":266,"replication_time_us":4515}]]}
I20240430 20:11:52.734864 19944 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50746:
name: "default.Table_7b102da111ef451aba55ec04c1215cc7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:52.735689 19944 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7b102da111ef451aba55ec04c1215cc7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:52.740865 18983 tablet_bootstrap.cc:492] T 69f88532952a4591bf1577541c3430e6 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:52.741148 18983 ts_tablet_manager.cc:1367] T 69f88532952a4591bf1577541c3430e6 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.072s user 0.003s sys 0.000s
I20240430 20:11:52.741951 18983 raft_consensus.cc:348] T 69f88532952a4591bf1577541c3430e6 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:52.742233 18983 raft_consensus.cc:374] T 69f88532952a4591bf1577541c3430e6 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:52.742338 18983 raft_consensus.cc:729] T 69f88532952a4591bf1577541c3430e6 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:52.742674 18983 consensus_queue.cc:260] T 69f88532952a4591bf1577541c3430e6 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:52.743854 18983 ts_tablet_manager.cc:1398] T 69f88532952a4591bf1577541c3430e6 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240430 20:11:52.744190 18983 tablet_bootstrap.cc:492] T 2797eed593fb4b06a3c4d6a28e8075a2 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:52.745874 18983 tablet_bootstrap.cc:654] T 2797eed593fb4b06a3c4d6a28e8075a2 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:52.752940 18474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35fdb052135541cfb42a8aeb03b64dc8" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:52.753329 18474 raft_consensus.cc:2379] T 35fdb052135541cfb42a8aeb03b64dc8 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c51c33d5616482cafb68e26688820f4 in current term 1: Already voted for candidate 5e1c6397f09747f091534faf93542710 in this term.
I20240430 20:11:52.753739 20198 raft_consensus.cc:482] T 2aa2a489116940dea3173cad16d94fa5 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:52.753962 20198 raft_consensus.cc:504] T 2aa2a489116940dea3173cad16d94fa5 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:52.754678 20198 leader_election.cc:288] T 2aa2a489116940dea3173cad16d94fa5 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:52.755677 18810 heartbeater.cc:503] Master 127.17.177.60:37225 was elected leader, sending a full tablet report...
I20240430 20:11:52.756011 21049 raft_consensus.cc:482] T aa9998f4e17147aa8f2b35c01980463a P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:52.756145 21049 raft_consensus.cc:504] T aa9998f4e17147aa8f2b35c01980463a P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:52.756698 21049 leader_election.cc:288] T aa9998f4e17147aa8f2b35c01980463a P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:52.759411 18668 heartbeater.cc:503] Master 127.17.177.62:36441 was elected leader, sending a full tablet report...
I20240430 20:11:52.761996 18608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aa2a489116940dea3173cad16d94fa5" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:52.765422 18988 ts_tablet_manager.cc:1398] T aa9998f4e17147aa8f2b35c01980463a P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.080s user 0.001s sys 0.001s
I20240430 20:11:52.765755 18988 tablet_bootstrap.cc:492] T 3a03b6997921408cb5a65f4cae223e9c P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:52.767386 18988 tablet_bootstrap.cc:654] T 3a03b6997921408cb5a65f4cae223e9c P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:52.772325 18623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa9998f4e17147aa8f2b35c01980463a" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:52.772570 18623 raft_consensus.cc:2454] T aa9998f4e17147aa8f2b35c01980463a P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 0.
I20240430 20:11:52.772955 18415 leader_election.cc:302] T aa9998f4e17147aa8f2b35c01980463a P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:52.780390 21049 raft_consensus.cc:2781] T aa9998f4e17147aa8f2b35c01980463a P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:52.780547 21049 raft_consensus.cc:482] T aa9998f4e17147aa8f2b35c01980463a P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:52.780618 21049 raft_consensus.cc:3037] T aa9998f4e17147aa8f2b35c01980463a P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:52.788576 21198 raft_consensus.cc:482] T 50486644ec234933b9a0b45ee39a6685 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:52.788830 21198 raft_consensus.cc:504] T 50486644ec234933b9a0b45ee39a6685 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:52.789703 21198 leader_election.cc:288] T 50486644ec234933b9a0b45ee39a6685 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:52.790225 18988 tablet_bootstrap.cc:492] T 3a03b6997921408cb5a65f4cae223e9c P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:52.790400 18988 ts_tablet_manager.cc:1367] T 3a03b6997921408cb5a65f4cae223e9c P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.025s user 0.004s sys 0.000s
W20240430 20:11:52.791464 18696 leader_election.cc:341] T 50486644ec234933b9a0b45ee39a6685 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Not found: Tablet not found: 50486644ec234933b9a0b45ee39a6685
I20240430 20:11:52.782953 19945 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet fbdeb4714b8c4b93ab72822c43329aae (table default.Table_c12ca6938d424fbfa2e1df5cdb00fdbc [id=0fb93f36bc17408ea56bc4cafcfd8580]). Expected version 1 got 0
I20240430 20:11:52.799429 18988 raft_consensus.cc:348] T 3a03b6997921408cb5a65f4cae223e9c P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:52.799767 18988 raft_consensus.cc:374] T 3a03b6997921408cb5a65f4cae223e9c P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:52.799875 18988 raft_consensus.cc:729] T 3a03b6997921408cb5a65f4cae223e9c P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:52.800185 18988 consensus_queue.cc:260] T 3a03b6997921408cb5a65f4cae223e9c P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:52.801923 18988 ts_tablet_manager.cc:1398] T 3a03b6997921408cb5a65f4cae223e9c P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.011s user 0.003s sys 0.000s
I20240430 20:11:52.803179 18988 tablet_bootstrap.cc:492] T 89b774967624498fa2be499878b71124 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:52.808715 21049 raft_consensus.cc:504] T aa9998f4e17147aa8f2b35c01980463a P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:52.809823 21049 leader_election.cc:288] T aa9998f4e17147aa8f2b35c01980463a P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:52.809948 18623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa9998f4e17147aa8f2b35c01980463a" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:52.810178 18623 raft_consensus.cc:3037] T aa9998f4e17147aa8f2b35c01980463a P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:52.813161 18623 raft_consensus.cc:2454] T aa9998f4e17147aa8f2b35c01980463a P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 1.
I20240430 20:11:52.813364 21221 consensus_queue.cc:1035] T 828ca913e2aa4102a6277bdbc6686ef5 P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:52.813997 18415 leader_election.cc:302] T aa9998f4e17147aa8f2b35c01980463a P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:52.814368 21049 raft_consensus.cc:2781] T aa9998f4e17147aa8f2b35c01980463a P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:52.814656 21049 raft_consensus.cc:686] T aa9998f4e17147aa8f2b35c01980463a P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Becoming Leader. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Running, Role: LEADER
I20240430 20:11:52.814947 21049 consensus_queue.cc:237] T aa9998f4e17147aa8f2b35c01980463a P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:52.816378 19944 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:50746 (request call id 69) took 1817 ms. Trace:
I20240430 20:11:52.819823 18988 tablet_bootstrap.cc:654] T 89b774967624498fa2be499878b71124 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:52.829862 21221 consensus_queue.cc:1035] T 828ca913e2aa4102a6277bdbc6686ef5 P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:52.799429 19945 catalog_manager.cc:5484] T 488d0e609b224674932f890939664df2 P 2efb542aa55e439b99ccb98d1c5c884a reported cstate change: term changed from 0 to 2, leader changed from <none> to 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1). New cstate: current_term: 2 leader_uuid: "2efb542aa55e439b99ccb98d1c5c884a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } health_report { overall_health: HEALTHY } } }
I20240430 20:11:52.831605 19945 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 488d0e609b224674932f890939664df2 (table default.Table_58f35a2120874472978ca0e5f816e86c [id=ff18001c6d814afc8c1768d0c84a6bac]). Expected version 1 got 0
I20240430 20:11:52.836611 19945 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 78ba48dadef548708153878810cc46f8 (table default.Table_b9983e101a6547649eb44ee04b5e1147 [id=29769be2b9ec4ecba9de5c0a65692f85]). Expected version 1 got 0
I20240430 20:11:52.836763 18988 tablet_bootstrap.cc:492] T 89b774967624498fa2be499878b71124 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:52.837909 18983 tablet_bootstrap.cc:492] T 2797eed593fb4b06a3c4d6a28e8075a2 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:52.838364 18988 ts_tablet_manager.cc:1367] T 89b774967624498fa2be499878b71124 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.036s user 0.005s sys 0.002s
I20240430 20:11:52.849812 18472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aa2a489116940dea3173cad16d94fa5" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:52.851164 18988 raft_consensus.cc:348] T 89b774967624498fa2be499878b71124 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:52.851411 18988 raft_consensus.cc:374] T 89b774967624498fa2be499878b71124 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:52.851475 18988 raft_consensus.cc:729] T 89b774967624498fa2be499878b71124 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:52.851714 18988 consensus_queue.cc:260] T 89b774967624498fa2be499878b71124 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:52.852351 18988 ts_tablet_manager.cc:1398] T 89b774967624498fa2be499878b71124 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.014s user 0.001s sys 0.000s
I20240430 20:11:52.852672 18988 tablet_bootstrap.cc:492] T d4d6c37b578c4fe1aa666b0a6b0bc7c9 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:52.854853 18985 tablet_bootstrap.cc:492] T 1fe7c6f1ab54499facd4d01573d3e7bc P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:52.854977 18988 tablet_bootstrap.cc:654] T d4d6c37b578c4fe1aa666b0a6b0bc7c9 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:52.859001 21198 consensus_queue.cc:1035] T 417351cbc9d54f9d84bb7155f39da119 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:52.859504 18472 raft_consensus.cc:2379] T 2aa2a489116940dea3173cad16d94fa5 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c51c33d5616482cafb68e26688820f4 in current term 1: Already voted for candidate 5e1c6397f09747f091534faf93542710 in this term.
I20240430 20:11:52.851264 21293 raft_consensus.cc:482] T 1fe7c6f1ab54499facd4d01573d3e7bc P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:52.860524 21293 raft_consensus.cc:504] T 1fe7c6f1ab54499facd4d01573d3e7bc P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:52.860958 19945 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 772eba5f24c549c791cad75565858285 (table default.Table_5fc77d5f60ae44f4bb4ca9f7f183672c [id=6ca223ca6354449c959d85ed2ac6a9c8]). Expected version 1 got 0
I20240430 20:11:52.861308 21293 leader_election.cc:288] T 1fe7c6f1ab54499facd4d01573d3e7bc P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:52.861600 19945 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet c6e3d09402574f1e8fe9e4160c70a3ee (table default.Table_656db97355f64d7c8aa29b1352dfe9c8 [id=450f5d1c7e084a1bb88d035c0ccfcf82]). Expected version 1 got 0
I20240430 20:11:52.861946 19945 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 1faa2f2c84184b129ba6e4d2857822f0 (table default.Table_0dd55b391be94213b4af60ea6e5a979f [id=1a382ad8009b4522a50d55c307db8f96]). Expected version 1 got 0
I20240430 20:11:52.862123 19945 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 3a03b6997921408cb5a65f4cae223e9c (table default.Table_744b5c96ea014bdb8feae740356a790e [id=83b1ac2e9c3d4a10b02f96ebb17faf44]). Expected version 1 got 0
I20240430 20:11:52.862280 19945 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet ae20f44df50343289b0a512a4ea53692 (table default.Table_1844f6b1bda84812825384e14a2fd94e [id=71da9acc3cff4d95bbac4a686b708fd5]). Expected version 1 got 0
I20240430 20:11:52.830711 19944 rpcz_store.cc:276] 0430 20:11:50.998988 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:52.734522 (+1735534us) service_pool.cc:225] Handling call
0430 20:11:52.791127 (+ 56605us) catalog_manager.cc:2072] Acquired catalog manager lock
0430 20:11:52.792711 (+ 1584us) catalog_manager.cc:2118] Created new table and tablet info
0430 20:11:52.815888 (+ 23177us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:52.815898 (+ 10us) write_op.cc:446] Released schema lock
0430 20:11:52.816027 (+ 129us) catalog_manager.cc:2178] Wrote table and tablets to system table
0430 20:11:52.816181 (+ 154us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0430 20:11:52.816300 (+ 119us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:52.802828 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:52.803002 (+ 174us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:52.803010 (+ 8us) write_op.cc:427] Acquired schema lock
0430 20:11:52.803013 (+ 3us) tablet.cc:660] Decoding operations
0430 20:11:52.803138 (+ 125us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:52.803155 (+ 17us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:52.803164 (+ 9us) tablet.cc:683] Acquiring locks for 2 operations
0430 20:11:52.803276 (+ 112us) tablet.cc:699] Row locks acquired
0430 20:11:52.803278 (+ 2us) write_op.cc:260] PREPARE: finished
0430 20:11:52.803322 (+ 44us) write_op.cc:270] Start()
0430 20:11:52.803364 (+ 42us) write_op.cc:276] Timestamp: P: 1714507912803294 usec, L: 0
0430 20:11:52.803384 (+ 20us) op_driver.cc:331] REPLICATION: starting
0430 20:11:52.803637 (+ 253us) log.cc:842] Serialized 433 byte log entry
0430 20:11:52.808773 (+ 5136us) op_driver.cc:444] REPLICATION: finished
0430 20:11:52.808916 (+ 143us) write_op.cc:301] APPLY: starting
0430 20:11:52.808945 (+ 29us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:52.809004 (+ 59us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:52.809048 (+ 44us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:52.809175 (+ 127us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:52.809221 (+ 46us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0430 20:11:52.809228 (+ 7us) write_op.cc:312] APPLY: finished
0430 20:11:52.809332 (+ 104us) log.cc:842] Serialized 29 byte log entry
0430 20:11:52.809411 (+ 79us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:52.809437 (+ 26us) write_op.cc:446] Released schema lock
0430 20:11:52.809445 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":84,"apply.run_cpu_time_us":603,"apply.run_wall_time_us":602,"num_ops":2,"prepare.queue_time_us":9591,"prepare.run_cpu_time_us":956,"prepare.run_wall_time_us":955,"raft.queue_time_us":1205,"raft.run_cpu_time_us":275,"raft.run_wall_time_us":275,"replication_time_us":5358}]]}
I20240430 20:11:52.860541 18983 ts_tablet_manager.cc:1367] T 2797eed593fb4b06a3c4d6a28e8075a2 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.116s user 0.001s sys 0.002s
I20240430 20:11:52.865002 18983 raft_consensus.cc:348] T 2797eed593fb4b06a3c4d6a28e8075a2 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:52.865204 18983 raft_consensus.cc:374] T 2797eed593fb4b06a3c4d6a28e8075a2 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:52.865272 18983 raft_consensus.cc:729] T 2797eed593fb4b06a3c4d6a28e8075a2 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:52.865481 18983 consensus_queue.cc:260] T 2797eed593fb4b06a3c4d6a28e8075a2 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:52.866044 18983 ts_tablet_manager.cc:1398] T 2797eed593fb4b06a3c4d6a28e8075a2 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240430 20:11:52.866376 18983 tablet_bootstrap.cc:492] T 2e1cac64cefe4336bd3207b5ca58b631 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:52.870869 18699 leader_election.cc:302] T 35fdb052135541cfb42a8aeb03b64dc8 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710
I20240430 20:11:52.871429 21198 raft_consensus.cc:3037] T 35fdb052135541cfb42a8aeb03b64dc8 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:52.873966 21198 raft_consensus.cc:2726] T 35fdb052135541cfb42a8aeb03b64dc8 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:52.881040 18760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fe7c6f1ab54499facd4d01573d3e7bc" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:52.884200 18988 tablet_bootstrap.cc:492] T d4d6c37b578c4fe1aa666b0a6b0bc7c9 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:52.884434 18988 ts_tablet_manager.cc:1367] T d4d6c37b578c4fe1aa666b0a6b0bc7c9 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.032s user 0.004s sys 0.000s
I20240430 20:11:52.885453 18988 raft_consensus.cc:348] T d4d6c37b578c4fe1aa666b0a6b0bc7c9 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:52.885728 18988 raft_consensus.cc:374] T d4d6c37b578c4fe1aa666b0a6b0bc7c9 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:52.885829 18988 raft_consensus.cc:729] T d4d6c37b578c4fe1aa666b0a6b0bc7c9 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:52.886096 18988 consensus_queue.cc:260] T d4d6c37b578c4fe1aa666b0a6b0bc7c9 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:52.886757 18988 ts_tablet_manager.cc:1398] T d4d6c37b578c4fe1aa666b0a6b0bc7c9 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240430 20:11:52.887091 18988 tablet_bootstrap.cc:492] T 2e1cac64cefe4336bd3207b5ca58b631 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
W20240430 20:11:52.887972 18417 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.177.3:44867, user_credentials={real_user=slave}} blocked reactor thread for 239802us
I20240430 20:11:52.888729 18988 tablet_bootstrap.cc:654] T 2e1cac64cefe4336bd3207b5ca58b631 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
W20240430 20:11:52.889845 18557 leader_election.cc:341] T 1fe7c6f1ab54499facd4d01573d3e7bc P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:52.894766 18699 leader_election.cc:302] T 2aa2a489116940dea3173cad16d94fa5 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710
I20240430 20:11:52.897305 18983 tablet_bootstrap.cc:654] T 2e1cac64cefe4336bd3207b5ca58b631 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:52.898259 21306 raft_consensus.cc:3037] T 2aa2a489116940dea3173cad16d94fa5 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:52.900828 21306 raft_consensus.cc:2726] T 2aa2a489116940dea3173cad16d94fa5 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240430 20:11:52.901913 18699 leader_election.cc:341] T 50486644ec234933b9a0b45ee39a6685 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Not found: Tablet not found: 50486644ec234933b9a0b45ee39a6685
I20240430 20:11:52.902905 18699 leader_election.cc:302] T 50486644ec234933b9a0b45ee39a6685 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710
W20240430 20:11:52.890617 18417 leader_election.cc:341] T d4d6c37b578c4fe1aa666b0a6b0bc7c9 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:52.893796 18476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fe7c6f1ab54499facd4d01573d3e7bc" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:52.907507 18985 tablet_bootstrap.cc:654] T 1fe7c6f1ab54499facd4d01573d3e7bc P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:52.909873 21306 raft_consensus.cc:2726] T 50486644ec234933b9a0b45ee39a6685 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240430 20:11:52.911387 19902 catalog_manager.cc:4810] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): delete failed for tablet f9bf604e6c234349a9f72f45a4c4aef6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f9bf604e6c234349a9f72f45a4c4aef6 already in progress: creating tablet
W20240430 20:11:52.912274 19902 catalog_manager.cc:4810] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): delete failed for tablet 13c2e69ae91f4957887fd64b90f6e435 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 13c2e69ae91f4957887fd64b90f6e435 already in progress: creating tablet
I20240430 20:11:52.913656 19945 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 7ce09fcbe1ac457a9c05efaf000c3263 (table default.Table_edc4c27ed846454288f73f6a0d88cb69 [id=78fd3f516083489ab3b42d90dcef7e4d]). Expected version 1 got 0
I20240430 20:11:52.914208 19945 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 331c031d44854d679a27d10d9151d0ea (table default.Table_763f0800e8bb4fa69976886416250f42 [id=61c8a5b99ef34a0baeeb343fef7cfd05]). Expected version 1 got 0
I20240430 20:11:52.922989 19945 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 17ddcaf971154a52b606bbf29a539bf0 (table default.Table_f2f38f5d69e34eaa9e3dbf24cfd1cdd6 [id=872c2faa4a464edd98a51df2e1b3f817]). Expected version 1 got 0
I20240430 20:11:52.923467 19945 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 2da900b6bbfa4fa99fe8494fd2e68da8 (table default.Table_19a91a7f17ff456ebb145e8dd89c0940 [id=67283dd3995a44c6a7a7bc8722e4f955]). Expected version 1 got 0
I20240430 20:11:52.923761 19945 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 8e91920e7e36418d8c27d1f0bfd9b981 (table default.Table_c8b25b14d955429cbd79515081deceb1 [id=f393bcc9669c4ea985a2d40deb161f0c]). Expected version 2 got 0
I20240430 20:11:52.924044 19945 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 0368d37dd9e2452982fb81f539c6805b (table default.Table_404b186c544e4b30bb36cb77d1a14120 [id=4330626e342a48699ea939add5c91b11]). Expected version 1 got 0
I20240430 20:11:52.924371 19945 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 677481aee5fa4eb4b189322d7a3f8f2e (table default.Table_1ce204e15fce4c469816cf4e7f80cac7 [id=d268c31d82df4871bc1a78aea5871b33]). Expected version 1 got 0
I20240430 20:11:52.924733 19945 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 5b26319488644d6a8b6be60460a5e7af (table default.Table_69892a8264a4477db48ee6248758e046 [id=5a1c4a7be2fc465fac89a50854eb3322]). Expected version 3 got 0
I20240430 20:11:52.925189 19945 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet b285e3eb16d148779c74e057458d8461 (table default.Table_9df4a0b4170f4f28a6c17b0cb052ead6 [id=d1655c507b434f129ad7ae0b04365e51]). Expected version 2 got 0
W20240430 20:11:52.908530 18558 leader_election.cc:341] T 1fe7c6f1ab54499facd4d01573d3e7bc P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:52.938472 18558 leader_election.cc:302] T 1fe7c6f1ab54499facd4d01573d3e7bc P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:52.939333 21221 raft_consensus.cc:2726] T 1fe7c6f1ab54499facd4d01573d3e7bc P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240430 20:11:52.939448 18417 leader_election.cc:341] T d4d6c37b578c4fe1aa666b0a6b0bc7c9 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:52.946898 21299 consensus_queue.cc:1035] T 8901327b3efb4201adce61a9266142ea P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:52.948321 18988 tablet_bootstrap.cc:492] T 2e1cac64cefe4336bd3207b5ca58b631 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:52.948489 18988 ts_tablet_manager.cc:1367] T 2e1cac64cefe4336bd3207b5ca58b631 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.061s user 0.003s sys 0.000s
I20240430 20:11:52.949260 18988 raft_consensus.cc:348] T 2e1cac64cefe4336bd3207b5ca58b631 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:52.949460 18988 raft_consensus.cc:374] T 2e1cac64cefe4336bd3207b5ca58b631 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:52.949522 18988 raft_consensus.cc:729] T 2e1cac64cefe4336bd3207b5ca58b631 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:52.949757 18988 consensus_queue.cc:260] T 2e1cac64cefe4336bd3207b5ca58b631 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:52.950415 18988 ts_tablet_manager.cc:1398] T 2e1cac64cefe4336bd3207b5ca58b631 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240430 20:11:52.950701 18988 tablet_bootstrap.cc:492] T 2797eed593fb4b06a3c4d6a28e8075a2 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:52.955304 18988 tablet_bootstrap.cc:654] T 2797eed593fb4b06a3c4d6a28e8075a2 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:52.965332 21306 raft_consensus.cc:482] T 3a03b6997921408cb5a65f4cae223e9c P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:52.965524 21306 raft_consensus.cc:504] T 3a03b6997921408cb5a65f4cae223e9c P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:52.966639 18623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a03b6997921408cb5a65f4cae223e9c" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:52.966825 21306 leader_election.cc:288] T 3a03b6997921408cb5a65f4cae223e9c P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:52.968976 18477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a03b6997921408cb5a65f4cae223e9c" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:52.969281 18477 raft_consensus.cc:2379] T 3a03b6997921408cb5a65f4cae223e9c P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c51c33d5616482cafb68e26688820f4 in current term 1: Already voted for candidate 5e1c6397f09747f091534faf93542710 in this term.
I20240430 20:11:52.969991 18417 leader_election.cc:302] T 828ca913e2aa4102a6277bdbc6686ef5 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a; no voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710
I20240430 20:11:52.970347 21054 raft_consensus.cc:2726] T 828ca913e2aa4102a6277bdbc6686ef5 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240430 20:11:52.972591 18417 leader_election.cc:341] T 89b774967624498fa2be499878b71124 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:52.976016 18758 raft_consensus.cc:3037] T 489dc838744a49eba542009c86688aa5 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:52.975544 18417 leader_election.cc:302] T df92eb85261d47d88cbf16d29d9ad30e P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters: 5e1c6397f09747f091534faf93542710
I20240430 20:11:52.979015 21226 raft_consensus.cc:2781] T df92eb85261d47d88cbf16d29d9ad30e P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240430 20:11:52.979092 21226 raft_consensus.cc:482] T df92eb85261d47d88cbf16d29d9ad30e P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:52.979152 21226 raft_consensus.cc:3037] T df92eb85261d47d88cbf16d29d9ad30e P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Advancing to term 2
I20240430 20:11:52.979307 18699 leader_election.cc:302] T 3a03b6997921408cb5a65f4cae223e9c P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710
I20240430 20:11:52.979562 21306 raft_consensus.cc:3037] T 3a03b6997921408cb5a65f4cae223e9c P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
W20240430 20:11:52.979776 18417 leader_election.cc:341] T 89b774967624498fa2be499878b71124 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:52.982080 21306 raft_consensus.cc:2726] T 3a03b6997921408cb5a65f4cae223e9c P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:52.990204 21238 consensus_queue.cc:1035] T 489dc838744a49eba542009c86688aa5 P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:52.993902 18983 tablet_bootstrap.cc:492] T 2e1cac64cefe4336bd3207b5ca58b631 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:52.994304 18983 ts_tablet_manager.cc:1367] T 2e1cac64cefe4336bd3207b5ca58b631 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.128s user 0.004s sys 0.000s
I20240430 20:11:52.995486 18983 raft_consensus.cc:348] T 2e1cac64cefe4336bd3207b5ca58b631 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:52.995790 18983 raft_consensus.cc:374] T 2e1cac64cefe4336bd3207b5ca58b631 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:52.995893 18983 raft_consensus.cc:729] T 2e1cac64cefe4336bd3207b5ca58b631 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:52.996241 18983 consensus_queue.cc:260] T 2e1cac64cefe4336bd3207b5ca58b631 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:52.997189 18983 ts_tablet_manager.cc:1398] T 2e1cac64cefe4336bd3207b5ca58b631 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240430 20:11:52.997525 18983 tablet_bootstrap.cc:492] T a663030d37614053a7c0369841403422 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:52.979485 21055 consensus_queue.cc:1035] T d4d6c37b578c4fe1aa666b0a6b0bc7c9 P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.019s
I20240430 20:11:52.999290 18983 tablet_bootstrap.cc:654] T a663030d37614053a7c0369841403422 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:53.002743 19937 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 8e91920e7e36418d8c27d1f0bfd9b981 (table default.Table_c8b25b14d955429cbd79515081deceb1 [id=f393bcc9669c4ea985a2d40deb161f0c]). Expected version 2 got 0
I20240430 20:11:53.003051 19937 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 0368d37dd9e2452982fb81f539c6805b (table default.Table_404b186c544e4b30bb36cb77d1a14120 [id=4330626e342a48699ea939add5c91b11]). Expected version 1 got 0
I20240430 20:11:53.003418 21306 raft_consensus.cc:482] T 89b774967624498fa2be499878b71124 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:53.003451 19937 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 12795670f21447a8964fe1efc6e92398 (table default.Table_896bbae1f6944005b0272ca8ebc97218 [id=d5cea90cbc2c48df9d8b80da1eccc949]). Expected version 3 got 0
I20240430 20:11:53.003562 21306 raft_consensus.cc:504] T 89b774967624498fa2be499878b71124 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:53.003801 19937 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 62732af95c6a4c749c5b726f0460dc72 (table default.Table_e3160f5ced1b4718b479401ca4df675c [id=2d44ec94131148048021b2ee6366ba30]). Expected version 3 got 0
I20240430 20:11:53.004033 19937 catalog_manager.cc:5484] T 417351cbc9d54f9d84bb7155f39da119 P 3c51c33d5616482cafb68e26688820f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c51c33d5616482cafb68e26688820f4 (127.17.177.3). New cstate: current_term: 1 leader_uuid: "3c51c33d5616482cafb68e26688820f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } health_report { overall_health: UNKNOWN } } }
I20240430 20:11:53.004612 19937 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 8e9711d0a382424f9f758bfa883866a0 (table default.Table_f1dc481c6d8c43f589fe2b1892a4cb10 [id=db696ce66005401fa73746e55c071f44]). Expected version 1 got 0
I20240430 20:11:53.004637 21306 leader_election.cc:288] T 89b774967624498fa2be499878b71124 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:53.004840 19937 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet c6e3d09402574f1e8fe9e4160c70a3ee (table default.Table_656db97355f64d7c8aa29b1352dfe9c8 [id=450f5d1c7e084a1bb88d035c0ccfcf82]). Expected version 1 got 0
I20240430 20:11:53.005079 19937 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet eb6c96047b114ebba73db7a371f2da8e (table default.Table_7d5f6f18cbd64c869739872a7dff9be2 [id=e67d619e36c0485f8f22b4a81cb4cb6c]). Expected version 1 got 0
I20240430 20:11:53.005120 18616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89b774967624498fa2be499878b71124" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:53.005338 18616 raft_consensus.cc:2379] T 89b774967624498fa2be499878b71124 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c51c33d5616482cafb68e26688820f4 in current term 1: Already voted for candidate 2efb542aa55e439b99ccb98d1c5c884a in this term.
I20240430 20:11:53.005314 19937 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 22fef8bb6b034db098c3372ae992f7a7 (table default.Table_4440df38610340758047115845b42b95 [id=7d186ca20c094cc19aef792fe1c094e9]). Expected version 1 got 0
I20240430 20:11:53.005576 19937 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet a482f4622f774d60b4efd6827b8be7d7 (table default.Table_e686cdfcbcc046749fa5b51578a64794 [id=c3dc9f05316d46d6a3ef4d03f40877ec]). Expected version 1 got 0
I20240430 20:11:53.006177 19937 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet ed0c37a71ede4bfc9eb5ad1a4365f25e (table default.Table_a2e85efc6b454524b0b122102ec07002 [id=86d795a82e674d859104741ebd04f4e8]). Expected version 1 got 0
I20240430 20:11:53.006531 19937 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 130007acc8ba4e709c3eb85fabe75903 (table default.Table_78987875650940c293247bc43db345ab [id=36f478a20a1f4a53a52fc29ade62b7f6]). Expected version 1 got 0
I20240430 20:11:53.008838 19937 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet b285e3eb16d148779c74e057458d8461 (table default.Table_9df4a0b4170f4f28a6c17b0cb052ead6 [id=d1655c507b434f129ad7ae0b04365e51]). Expected version 2 got 0
I20240430 20:11:53.009291 19937 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 6b51cbc731794083a88fe4bed1ddeebf (table default.Table_c6c962da746e4fd8a4d7200d09f08e9e [id=8e0f5d8c941d423ea7b30a89513d5170]). Expected version 1 got 0
I20240430 20:11:53.014178 21226 raft_consensus.cc:504] T df92eb85261d47d88cbf16d29d9ad30e P 2efb542aa55e439b99ccb98d1c5c884a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:53.020560 21226 leader_election.cc:288] T df92eb85261d47d88cbf16d29d9ad30e P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
W20240430 20:11:53.021644 18415 leader_election.cc:341] T df92eb85261d47d88cbf16d29d9ad30e P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Not found: Tablet not found: df92eb85261d47d88cbf16d29d9ad30e
I20240430 20:11:53.026700 18985 tablet_bootstrap.cc:492] T 1fe7c6f1ab54499facd4d01573d3e7bc P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:53.026736 21226 consensus_queue.cc:1035] T 89b774967624498fa2be499878b71124 P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:53.032750 21226 raft_consensus.cc:482] T 7683caba80594627900a0185f772c131 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5e1c6397f09747f091534faf93542710)
I20240430 20:11:53.032919 21226 raft_consensus.cc:504] T 7683caba80594627900a0185f772c131 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:53.033512 21226 leader_election.cc:288] T 7683caba80594627900a0185f772c131 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
W20240430 20:11:53.038055 18415 leader_election.cc:341] T 7683caba80594627900a0185f772c131 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Not found: Tablet not found: 7683caba80594627900a0185f772c131
I20240430 20:11:53.026865 18985 ts_tablet_manager.cc:1367] T 1fe7c6f1ab54499facd4d01573d3e7bc P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.172s user 0.008s sys 0.012s
I20240430 20:11:53.040479 18985 raft_consensus.cc:348] T 1fe7c6f1ab54499facd4d01573d3e7bc P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:53.041855 18988 tablet_bootstrap.cc:492] T 2797eed593fb4b06a3c4d6a28e8075a2 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:53.046828 18759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f0b341a31224bf1be525b84b9500304" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:53.048755 18759 raft_consensus.cc:3037] T 9f0b341a31224bf1be525b84b9500304 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Advancing to term 2
I20240430 20:11:53.051832 18759 raft_consensus.cc:2454] T 9f0b341a31224bf1be525b84b9500304 P 3c51c33d5616482cafb68e26688820f4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 2.
I20240430 20:11:53.047521 18753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa9998f4e17147aa8f2b35c01980463a" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:53.052171 18753 raft_consensus.cc:2454] T aa9998f4e17147aa8f2b35c01980463a P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 0.
I20240430 20:11:53.048252 18750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa9998f4e17147aa8f2b35c01980463a" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:53.052589 18750 raft_consensus.cc:3037] T aa9998f4e17147aa8f2b35c01980463a P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:53.055136 18750 raft_consensus.cc:2454] T aa9998f4e17147aa8f2b35c01980463a P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 1.
I20240430 20:11:53.055665 18988 ts_tablet_manager.cc:1367] T 2797eed593fb4b06a3c4d6a28e8075a2 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.105s user 0.003s sys 0.000s
I20240430 20:11:53.056744 18988 raft_consensus.cc:348] T 2797eed593fb4b06a3c4d6a28e8075a2 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:53.044320 21326 tablet.cc:1722] T b285e3eb16d148779c74e057458d8461 P 2efb542aa55e439b99ccb98d1c5c884a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240430 20:11:53.057219 18985 raft_consensus.cc:374] T 1fe7c6f1ab54499facd4d01573d3e7bc P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:53.058404 18985 raft_consensus.cc:729] T 1fe7c6f1ab54499facd4d01573d3e7bc P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:53.063230 18985 consensus_queue.cc:260] T 1fe7c6f1ab54499facd4d01573d3e7bc P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:53.064105 18985 ts_tablet_manager.cc:1398] T 1fe7c6f1ab54499facd4d01573d3e7bc P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.024s user 0.002s sys 0.000s
I20240430 20:11:53.064464 18985 tablet_bootstrap.cc:492] T a692b253db914531b8fc6ffaf3e350b5 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:53.066102 18985 tablet_bootstrap.cc:654] T a692b253db914531b8fc6ffaf3e350b5 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:53.071807 19938 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 3a03b6997921408cb5a65f4cae223e9c (table default.Table_744b5c96ea014bdb8feae740356a790e [id=83b1ac2e9c3d4a10b02f96ebb17faf44]). Expected version 1 got 0
I20240430 20:11:53.076639 19938 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet ae20f44df50343289b0a512a4ea53692 (table default.Table_1844f6b1bda84812825384e14a2fd94e [id=71da9acc3cff4d95bbac4a686b708fd5]). Expected version 1 got 0
I20240430 20:11:53.077045 19938 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 1faa2f2c84184b129ba6e4d2857822f0 (table default.Table_0dd55b391be94213b4af60ea6e5a979f [id=1a382ad8009b4522a50d55c307db8f96]). Expected version 1 got 0
I20240430 20:11:53.078621 19938 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet eb6c96047b114ebba73db7a371f2da8e (table default.Table_7d5f6f18cbd64c869739872a7dff9be2 [id=e67d619e36c0485f8f22b4a81cb4cb6c]). Expected version 1 got 0
I20240430 20:11:53.078991 19938 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 6b51cbc731794083a88fe4bed1ddeebf (table default.Table_c6c962da746e4fd8a4d7200d09f08e9e [id=8e0f5d8c941d423ea7b30a89513d5170]). Expected version 1 got 0
I20240430 20:11:53.079249 19938 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 62732af95c6a4c749c5b726f0460dc72 (table default.Table_e3160f5ced1b4718b479401ca4df675c [id=2d44ec94131148048021b2ee6366ba30]). Expected version 3 got 0
I20240430 20:11:53.079614 19938 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet a0baee91f5b44945b595d0e234af018e (table default.Table_5ed3ea587cf843db9a1ff1ffab40b678 [id=b3d1c2f97c4648518dbea83a46cadd85]). Expected version 2 got 0
I20240430 20:11:53.079943 19938 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 266fd23786bc435db5083370f3330ed0 (table default.Table_cf60b652e9dc49e7a948ce66fd1cbffe [id=9734bb8dd05a400ca4244b3829f01dee]). Expected version 1 got 0
I20240430 20:11:53.079885 21299 consensus_queue.cc:1035] T 8e9711d0a382424f9f758bfa883866a0 P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:53.081533 19938 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 89b774967624498fa2be499878b71124 (table default.Table_2ec9e316bc3347b189a13b7a8bbcf829 [id=9fa22833979f4c2f978de651145acef9]). Expected version 1 got 0
I20240430 20:11:53.082760 19938 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet ed0c37a71ede4bfc9eb5ad1a4365f25e (table default.Table_a2e85efc6b454524b0b122102ec07002 [id=86d795a82e674d859104741ebd04f4e8]). Expected version 1 got 0
I20240430 20:11:53.083088 19938 catalog_manager.cc:5484] T e900f62a5aa143e791c7539660b60cd9 P 5e1c6397f09747f091534faf93542710 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e1c6397f09747f091534faf93542710 (127.17.177.2). New cstate: current_term: 1 leader_uuid: "5e1c6397f09747f091534faf93542710" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } health_report { overall_health: UNKNOWN } } }
I20240430 20:11:53.084393 19938 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 12795670f21447a8964fe1efc6e92398 (table default.Table_896bbae1f6944005b0272ca8ebc97218 [id=d5cea90cbc2c48df9d8b80da1eccc949]). Expected version 3 got 0
I20240430 20:11:53.085453 19938 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 2e79df8744e74b5991a79ce978ff0202 (table default.Table_276bbdc7cfe542d1a2936485a2aad1ec [id=f146a15e6e8a4acbb4c734c1c863da57]). Expected version 1 got 0
I20240430 20:11:53.085776 18983 tablet_bootstrap.cc:492] T a663030d37614053a7c0369841403422 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:53.085999 18983 ts_tablet_manager.cc:1367] T a663030d37614053a7c0369841403422 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.089s user 0.003s sys 0.000s
I20240430 20:11:53.077284 18761 raft_consensus.cc:3037] T d4d6c37b578c4fe1aa666b0a6b0bc7c9 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:53.077554 18760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df92eb85261d47d88cbf16d29d9ad30e" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:53.085073 18810 heartbeater.cc:503] Master 127.17.177.60:37225 was elected leader, sending a full tablet report...
I20240430 20:11:53.085215 18529 heartbeater.cc:347] Connected to a master server at 127.17.177.60:37225
I20240430 20:11:53.090663 18760 raft_consensus.cc:3037] T df92eb85261d47d88cbf16d29d9ad30e P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Advancing to term 2
I20240430 20:11:53.092546 19938 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 22fef8bb6b034db098c3372ae992f7a7 (table default.Table_4440df38610340758047115845b42b95 [id=7d186ca20c094cc19aef792fe1c094e9]). Expected version 1 got 0
I20240430 20:11:53.092944 19938 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 330453ddeea6456e8ef93fa92608b0d3 (table default.Table_c1f36f7df5ae4e7c8eb8b4e99b89748b [id=a50abf28790d4284a76ac6ee22c02606]). Expected version 1 got 0
I20240430 20:11:53.090765 18988 raft_consensus.cc:374] T 2797eed593fb4b06a3c4d6a28e8075a2 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:53.093845 18988 raft_consensus.cc:729] T 2797eed593fb4b06a3c4d6a28e8075a2 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:53.094143 19938 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet acd23f0a63d5448aaa7ed78422e67bad (table default.Table_972f13be4dca4cd19b9a04534dc59f51 [id=69f37db4de634a3185613e40013e0ed7]). Expected version 1 got 0
I20240430 20:11:53.094081 18988 consensus_queue.cc:260] T 2797eed593fb4b06a3c4d6a28e8075a2 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:53.094781 18760 raft_consensus.cc:2454] T df92eb85261d47d88cbf16d29d9ad30e P 3c51c33d5616482cafb68e26688820f4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 2.
I20240430 20:11:53.094784 19938 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 69f88532952a4591bf1577541c3430e6 (table default.Table_1177828cad1f4a24bd397619dad96170 [id=a95fed1bc3fa4865bf9a35dfc51b659c]). Expected version 2 got 0
I20240430 20:11:53.095208 19938 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 1050ab528c7d41559831652a1437d358 (table default.Table_acddc4cbf6334546b5fb3e4752fef2a5 [id=274cb15fd751410e83327ac4e8dc4629]). Expected version 2 got 0
I20240430 20:11:53.097427 18988 ts_tablet_manager.cc:1398] T 2797eed593fb4b06a3c4d6a28e8075a2 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.041s user 0.001s sys 0.000s
I20240430 20:11:53.097743 18988 tablet_bootstrap.cc:492] T a692b253db914531b8fc6ffaf3e350b5 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:53.099684 19938 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 086b3dfa84c045a18c8c4f177dce606a (table default.Table_df19cb8a6f4d46699350ec5b69c14e5a [id=197923633fc54a4582207942b51530e2]). Expected version 2 got 0
I20240430 20:11:53.099937 18988 tablet_bootstrap.cc:654] T a692b253db914531b8fc6ffaf3e350b5 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:53.101020 18983 raft_consensus.cc:348] T a663030d37614053a7c0369841403422 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:53.101387 18983 raft_consensus.cc:374] T a663030d37614053a7c0369841403422 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:53.101500 18983 raft_consensus.cc:729] T a663030d37614053a7c0369841403422 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:53.101768 18983 consensus_queue.cc:260] T a663030d37614053a7c0369841403422 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:53.102526 18983 ts_tablet_manager.cc:1398] T a663030d37614053a7c0369841403422 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.016s user 0.001s sys 0.000s
I20240430 20:11:53.102896 18983 tablet_bootstrap.cc:492] T 436579f473844c6ca1038b496201a550 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:53.103333 19938 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet e041756c962b4a9f97c2cb314e7d8c05 (table default.Table_de88cafd92b149f9b545369bff71033c [id=c4bf4806f3d84579b16a3038d8d982c3]). Expected version 1 got 0
I20240430 20:11:53.104455 21221 consensus_queue.cc:1035] T 62732af95c6a4c749c5b726f0460dc72 P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:53.104734 18983 tablet_bootstrap.cc:654] T 436579f473844c6ca1038b496201a550 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:53.103785 19938 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 94fbd4a57a1e438891fcb456f6b40be3 (table default.Table_25ddc3adef6b4ccf8b9c00cc6772031c [id=4fa623ffdcf347639150a355806038af]). Expected version 1 got 0
I20240430 20:11:53.105333 19938 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 8e9711d0a382424f9f758bfa883866a0 (table default.Table_f1dc481c6d8c43f589fe2b1892a4cb10 [id=db696ce66005401fa73746e55c071f44]). Expected version 1 got 0
I20240430 20:11:53.105547 19938 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 7a0176f8a74c47efbbe3d3343b3dc896 (table default.Table_144e5402c1864d818436015b26648024 [id=73e5c9f0470d47849109819c063e1ce4]). Expected version 2 got 0
I20240430 20:11:53.106258 18472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89b774967624498fa2be499878b71124" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:53.107621 21276 consensus_queue.cc:1035] T 8901327b3efb4201adce61a9266142ea P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:53.110843 18699 leader_election.cc:302] T 89b774967624498fa2be499878b71124 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710
I20240430 20:11:53.106910 19938 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet aaef85eea87f4443943fd3692bb2a38f (table default.Table_3da5b4c8668c470483ae2a0c7773eaaf [id=e292d818ee3c45df90522f08eb2d6227]). Expected version 1 got 0
I20240430 20:11:53.111267 21198 raft_consensus.cc:3037] T 89b774967624498fa2be499878b71124 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:53.111645 19938 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet a59ef6c545dd41a28265a4c1b8ce10e0 (table default.Table_07e40af8f1c546048975c8edf88170d7 [id=86df5d161d8641c89070e7eed9a39360]). Expected version 2 got 0
I20240430 20:11:53.112406 19938 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet a482f4622f774d60b4efd6827b8be7d7 (table default.Table_e686cdfcbcc046749fa5b51578a64794 [id=c3dc9f05316d46d6a3ef4d03f40877ec]). Expected version 1 got 0
I20240430 20:11:53.112663 19938 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 37b2ea4be7494f0985808ade788dbe2b (table default.Table_5f6e0fce999b46de927f9271e5de5071 [id=0baad83a8c0f4aeebc7ddad068625a69]). Expected version 1 got 0
I20240430 20:11:53.115700 19938 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 99203cc522de424184c6f97453f166fd (table default.Table_cb2a4b9ceaf4431facfff2c9bcf94c5d [id=3f8690a437ae49a3beb009fa7899d5d0]). Expected version 1 got 0
I20240430 20:11:53.115773 21198 raft_consensus.cc:2726] T 89b774967624498fa2be499878b71124 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:53.116024 19938 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 47bf64ecc41b445da4ff697e141243dd (table default.Table_f1ad29d5986d4b799cfc67afb2b0dfdf [id=838fc8b61a864dea8877a6f144e4cb4f]). Expected version 2 got 0
I20240430 20:11:53.116324 19938 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 821c55edb3414d72b1489eea97d3f73a (table default.Table_4d68c38d9ba7452a829f8318b2b267b8 [id=d06e3e9045b1429aaeebd1fd67d841de]). Expected version 1 got 0
I20240430 20:11:53.116662 19938 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 130007acc8ba4e709c3eb85fabe75903 (table default.Table_78987875650940c293247bc43db345ab [id=36f478a20a1f4a53a52fc29ade62b7f6]). Expected version 1 got 0
I20240430 20:11:53.116874 19938 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet dac141d1b70c48d785caddd7182811ee (table default.Table_fc0a4540f98b4eea95099b8ab46ea235 [id=f3c53ccffe64410495cc8c379a05cfae]). Expected version 1 got 0
I20240430 20:11:53.118156 19938 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 5df364628561475391c9a5e10ff5a05f (table default.Table_3c8f9f7b22f64fb1bdc3f1550987233a [id=7fd6f8576f4c4cbc8ffcdb4ab95e15dc]). Expected version 1 got 0
I20240430 20:11:53.118494 19938 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet f5eaf2413e1d4e69b56fae61f768f0ce (table default.Table_1280cfe32d5b41abb4d3eb1bd70641bf [id=8489b872074a42b6abb713e420968141]). Expected version 1 got 0
I20240430 20:11:53.118906 19938 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 724e51ff05ec4baebc2bb55e05c8eddc (table default.Table_19df33614a6c4f94b6419a5b441e3ef8 [id=4dce8a767f0541e1beb5464b2ed95d06]). Expected version 2 got 1
I20240430 20:11:53.119619 19938 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet ca64de16d40f4a44b8b411f5526ba11a (table default.Table_1cf914acf7fe4e98b8dbc297b6372f21 [id=f4ba9536e9c141deb68f4f62b7e8fa70]). Expected version 1 got 0
I20240430 20:11:53.119741 18985 tablet_bootstrap.cc:492] T a692b253db914531b8fc6ffaf3e350b5 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:53.119902 18985 ts_tablet_manager.cc:1367] T a692b253db914531b8fc6ffaf3e350b5 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.056s user 0.002s sys 0.000s
I20240430 20:11:53.120584 18985 raft_consensus.cc:348] T a692b253db914531b8fc6ffaf3e350b5 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:53.120859 18985 raft_consensus.cc:374] T a692b253db914531b8fc6ffaf3e350b5 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:53.120927 18985 raft_consensus.cc:729] T a692b253db914531b8fc6ffaf3e350b5 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:53.121178 18985 consensus_queue.cc:260] T a692b253db914531b8fc6ffaf3e350b5 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:53.121702 18985 ts_tablet_manager.cc:1398] T a692b253db914531b8fc6ffaf3e350b5 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240430 20:11:53.122109 18985 tablet_bootstrap.cc:492] T 13c2e69ae91f4957887fd64b90f6e435 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:53.123535 18985 tablet_bootstrap.cc:654] T 13c2e69ae91f4957887fd64b90f6e435 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:53.125242 19938 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet b285e3eb16d148779c74e057458d8461 (table default.Table_9df4a0b4170f4f28a6c17b0cb052ead6 [id=d1655c507b434f129ad7ae0b04365e51]). Expected version 2 got 0
I20240430 20:11:53.131232 19938 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet fbdeb4714b8c4b93ab72822c43329aae (table default.Table_c12ca6938d424fbfa2e1df5cdb00fdbc [id=0fb93f36bc17408ea56bc4cafcfd8580]). Expected version 1 got 0
I20240430 20:11:53.137032 21221 consensus_queue.cc:1035] T 3a03b6997921408cb5a65f4cae223e9c P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:53.138047 19938 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 772eba5f24c549c791cad75565858285 (table default.Table_5fc77d5f60ae44f4bb4ca9f7f183672c [id=6ca223ca6354449c959d85ed2ac6a9c8]). Expected version 1 got 0
I20240430 20:11:53.138334 19938 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet c6e3d09402574f1e8fe9e4160c70a3ee (table default.Table_656db97355f64d7c8aa29b1352dfe9c8 [id=450f5d1c7e084a1bb88d035c0ccfcf82]). Expected version 1 got 0
I20240430 20:11:53.138718 19938 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 78ba48dadef548708153878810cc46f8 (table default.Table_b9983e101a6547649eb44ee04b5e1147 [id=29769be2b9ec4ecba9de5c0a65692f85]). Expected version 1 got 0
I20240430 20:11:53.139103 19938 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 2cd41b407d6b444d844c4c7c0ebf82e5 (table default.Table_6f8e0eff8f5a484dbd6a9c3fdb5d03f6 [id=3813b5e9778c4bd683a06598cef0d86f]). Expected version 1 got 0
I20240430 20:11:53.139750 19938 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 488d0e609b224674932f890939664df2 (table default.Table_58f35a2120874472978ca0e5f816e86c [id=ff18001c6d814afc8c1768d0c84a6bac]). Expected version 1 got 0
I20240430 20:11:53.140169 19938 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 17ddcaf971154a52b606bbf29a539bf0 (table default.Table_f2f38f5d69e34eaa9e3dbf24cfd1cdd6 [id=872c2faa4a464edd98a51df2e1b3f817]). Expected version 1 got 0
I20240430 20:11:53.140480 19938 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 2da900b6bbfa4fa99fe8494fd2e68da8 (table default.Table_19a91a7f17ff456ebb145e8dd89c0940 [id=67283dd3995a44c6a7a7bc8722e4f955]). Expected version 1 got 0
I20240430 20:11:53.140797 19938 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 8e91920e7e36418d8c27d1f0bfd9b981 (table default.Table_c8b25b14d955429cbd79515081deceb1 [id=f393bcc9669c4ea985a2d40deb161f0c]). Expected version 2 got 0
I20240430 20:11:53.141079 19938 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 0368d37dd9e2452982fb81f539c6805b (table default.Table_404b186c544e4b30bb36cb77d1a14120 [id=4330626e342a48699ea939add5c91b11]). Expected version 1 got 0
I20240430 20:11:53.142235 19938 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 7ce09fcbe1ac457a9c05efaf000c3263 (table default.Table_edc4c27ed846454288f73f6a0d88cb69 [id=78fd3f516083489ab3b42d90dcef7e4d]). Expected version 1 got 0
I20240430 20:11:53.142551 19938 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 331c031d44854d679a27d10d9151d0ea (table default.Table_763f0800e8bb4fa69976886416250f42 [id=61c8a5b99ef34a0baeeb343fef7cfd05]). Expected version 1 got 0
I20240430 20:11:53.143990 19938 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 677481aee5fa4eb4b189322d7a3f8f2e (table default.Table_1ce204e15fce4c469816cf4e7f80cac7 [id=d268c31d82df4871bc1a78aea5871b33]). Expected version 1 got 0
I20240430 20:11:53.144405 19938 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 5b26319488644d6a8b6be60460a5e7af (table default.Table_69892a8264a4477db48ee6248758e046 [id=5a1c4a7be2fc465fac89a50854eb3322]). Expected version 3 got 0
I20240430 20:11:53.144734 19938 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet f1073fc792954c5795a05aa7d0ec24df (table default.Table_ba8b218e936b4bfcb073f8182f474cca [id=aedf11fda8914a3b843a57f8a7b69a53]). Expected version 1 got 0
I20240430 20:11:53.144989 19938 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 66dd767bd3f8494f94061fb89e5e5680 (table default.Table_b064d5659f7243f5b0ec167fb6c3f404 [id=5227206f44d04fa89b4842b899f33afb]). Expected version 2 got 0
I20240430 20:11:53.145360 19938 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 81787eccca6c42c1b421377a3bf9ebe6 (table default.Table_0b4ca4cd0af148399dfa21f2d32114e3 [id=7f602584cef9486eb7d9b4389be3f43e]). Expected version 3 got 2
I20240430 20:11:53.152030 21293 consensus_queue.cc:1035] T 3a03b6997921408cb5a65f4cae223e9c P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:53.152931 21293 consensus_queue.cc:1035] T 35fdb052135541cfb42a8aeb03b64dc8 P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:53.156534 18988 tablet_bootstrap.cc:492] T a692b253db914531b8fc6ffaf3e350b5 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:53.156690 18988 ts_tablet_manager.cc:1367] T a692b253db914531b8fc6ffaf3e350b5 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.059s user 0.003s sys 0.000s
I20240430 20:11:53.157349 18988 raft_consensus.cc:348] T a692b253db914531b8fc6ffaf3e350b5 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:53.157541 18988 raft_consensus.cc:374] T a692b253db914531b8fc6ffaf3e350b5 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:53.157598 18988 raft_consensus.cc:729] T a692b253db914531b8fc6ffaf3e350b5 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:53.157793 18988 consensus_queue.cc:260] T a692b253db914531b8fc6ffaf3e350b5 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:53.158564 18988 ts_tablet_manager.cc:1398] T a692b253db914531b8fc6ffaf3e350b5 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240430 20:11:53.158927 18988 tablet_bootstrap.cc:492] T 3191aa54a9644783a2403991c3be27c8 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
W20240430 20:11:53.161136 18417 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.177.3:44867, user_credentials={real_user=slave}} blocked reactor thread for 53940.8us
I20240430 20:11:53.165652 18417 leader_election.cc:302] T 9f0b341a31224bf1be525b84b9500304 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters: 5e1c6397f09747f091534faf93542710
I20240430 20:11:53.171197 21054 raft_consensus.cc:2781] T 9f0b341a31224bf1be525b84b9500304 P 2efb542aa55e439b99ccb98d1c5c884a [term 2 FOLLOWER]: Leader election won for term 2
I20240430 20:11:53.171342 21054 raft_consensus.cc:686] T 9f0b341a31224bf1be525b84b9500304 P 2efb542aa55e439b99ccb98d1c5c884a [term 2 LEADER]: Becoming Leader. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Running, Role: LEADER
I20240430 20:11:53.171587 21054 consensus_queue.cc:237] T 9f0b341a31224bf1be525b84b9500304 P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:53.172418 21198 consensus_queue.cc:1035] T 6b51cbc731794083a88fe4bed1ddeebf P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:53.174964 18988 tablet_bootstrap.cc:654] T 3191aa54a9644783a2403991c3be27c8 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:53.176210 21221 consensus_queue.cc:1035] T 489dc838744a49eba542009c86688aa5 P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240430 20:11:53.216130 18417 leader_election.cc:341] T 7683caba80594627900a0185f772c131 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Not found: Tablet not found: 7683caba80594627900a0185f772c131
I20240430 20:11:53.217006 18417 leader_election.cc:302] T 7683caba80594627900a0185f772c131 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a; no voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710
I20240430 20:11:53.218750 21049 raft_consensus.cc:2726] T 7683caba80594627900a0185f772c131 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240430 20:11:53.220083 21238 raft_consensus.cc:482] T a692b253db914531b8fc6ffaf3e350b5 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:53.220279 21238 raft_consensus.cc:504] T a692b253db914531b8fc6ffaf3e350b5 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:53.221738 21238 leader_election.cc:288] T a692b253db914531b8fc6ffaf3e350b5 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:53.221899 18482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a692b253db914531b8fc6ffaf3e350b5" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:53.222151 18482 raft_consensus.cc:2454] T a692b253db914531b8fc6ffaf3e350b5 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 0.
I20240430 20:11:53.222318 21049 consensus_queue.cc:1035] T d4d6c37b578c4fe1aa666b0a6b0bc7c9 P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:53.222654 18558 leader_election.cc:302] T a692b253db914531b8fc6ffaf3e350b5 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:53.222977 21293 raft_consensus.cc:2781] T a692b253db914531b8fc6ffaf3e350b5 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:53.223001 18417 leader_election.cc:302] T df92eb85261d47d88cbf16d29d9ad30e P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters: 5e1c6397f09747f091534faf93542710
I20240430 20:11:53.223109 21293 raft_consensus.cc:482] T a692b253db914531b8fc6ffaf3e350b5 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:53.223218 21293 raft_consensus.cc:3037] T a692b253db914531b8fc6ffaf3e350b5 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:53.223371 21049 raft_consensus.cc:2781] T df92eb85261d47d88cbf16d29d9ad30e P 2efb542aa55e439b99ccb98d1c5c884a [term 2 FOLLOWER]: Leader election won for term 2
I20240430 20:11:53.223484 21049 raft_consensus.cc:686] T df92eb85261d47d88cbf16d29d9ad30e P 2efb542aa55e439b99ccb98d1c5c884a [term 2 LEADER]: Becoming Leader. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Running, Role: LEADER
I20240430 20:11:53.223703 21049 consensus_queue.cc:237] T df92eb85261d47d88cbf16d29d9ad30e P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:53.226233 21293 raft_consensus.cc:504] T a692b253db914531b8fc6ffaf3e350b5 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:53.220000 18983 tablet_bootstrap.cc:492] T 436579f473844c6ca1038b496201a550 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:53.228049 18983 ts_tablet_manager.cc:1367] T 436579f473844c6ca1038b496201a550 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.125s user 0.006s sys 0.000s
I20240430 20:11:53.232885 18985 tablet_bootstrap.cc:492] T 13c2e69ae91f4957887fd64b90f6e435 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:53.234417 18725 tablet_service.cc:1467] Processing CreateTablet for tablet d84b399395ee4456b76117768ca626b4 (DEFAULT_TABLE table=default.Table_5ab8136f886143f2a4a7afb5d64a3690 [id=adc53037af0a416c9a8acb46ee50f24d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:53.234736 21198 consensus_queue.cc:1035] T 6b51cbc731794083a88fe4bed1ddeebf P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:53.234875 18725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d84b399395ee4456b76117768ca626b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:53.236749 18983 raft_consensus.cc:348] T 436579f473844c6ca1038b496201a550 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:53.239624 18985 ts_tablet_manager.cc:1367] T 13c2e69ae91f4957887fd64b90f6e435 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.118s user 0.003s sys 0.000s
I20240430 20:11:53.242794 18988 tablet_bootstrap.cc:492] T 3191aa54a9644783a2403991c3be27c8 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:53.242969 18988 ts_tablet_manager.cc:1367] T 3191aa54a9644783a2403991c3be27c8 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.084s user 0.003s sys 0.001s
I20240430 20:11:53.250114 18988 raft_consensus.cc:348] T 3191aa54a9644783a2403991c3be27c8 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:53.250358 18988 raft_consensus.cc:374] T 3191aa54a9644783a2403991c3be27c8 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:53.250423 18988 raft_consensus.cc:729] T 3191aa54a9644783a2403991c3be27c8 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:53.250658 18988 consensus_queue.cc:260] T 3191aa54a9644783a2403991c3be27c8 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:53.251195 18988 ts_tablet_manager.cc:1398] T 3191aa54a9644783a2403991c3be27c8 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240430 20:11:53.251953 18810 heartbeater.cc:503] Master 127.17.177.60:37225 was elected leader, sending a full tablet report...
I20240430 20:11:53.253141 18725 tablet_service.cc:1467] Processing CreateTablet for tablet 991d73fe46bf4faaaac03d190bf27e6c (DEFAULT_TABLE table=default.Table_22d93bfc850f42bab0f67f0e0545601f [id=568d7291829346cb9b421788d4cc84c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:53.254753 18725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 991d73fe46bf4faaaac03d190bf27e6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:53.238219 18482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a692b253db914531b8fc6ffaf3e350b5" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:53.255465 18482 raft_consensus.cc:3037] T a692b253db914531b8fc6ffaf3e350b5 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:53.258173 18482 raft_consensus.cc:2454] T a692b253db914531b8fc6ffaf3e350b5 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
I20240430 20:11:53.232821 21054 consensus_queue.cc:1035] T 89b774967624498fa2be499878b71124 P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:53.252535 18456 tablet_service.cc:1467] Processing CreateTablet for tablet d84b399395ee4456b76117768ca626b4 (DEFAULT_TABLE table=default.Table_5ab8136f886143f2a4a7afb5d64a3690 [id=adc53037af0a416c9a8acb46ee50f24d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:53.260143 18456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d84b399395ee4456b76117768ca626b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:53.262202 18735 tablet_service.cc:1467] Processing CreateTablet for tablet eba4bd4665ae4c04a2cab3ab1a2da780 (DEFAULT_TABLE table=default.Table_6b67c0c03ceb4b9491ad5828022b357f [id=368fcf43e863494ca260862d12331c0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:53.262559 18735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eba4bd4665ae4c04a2cab3ab1a2da780. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:53.263031 18988 tablet_bootstrap.cc:492] T 13c2e69ae91f4957887fd64b90f6e435 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:53.263552 18450 tablet_service.cc:1467] Processing CreateTablet for tablet 991d73fe46bf4faaaac03d190bf27e6c (DEFAULT_TABLE table=default.Table_22d93bfc850f42bab0f67f0e0545601f [id=568d7291829346cb9b421788d4cc84c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:53.263877 18450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 991d73fe46bf4faaaac03d190bf27e6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:53.264561 18988 tablet_bootstrap.cc:654] T 13c2e69ae91f4957887fd64b90f6e435 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:53.267055 18445 tablet_service.cc:1467] Processing CreateTablet for tablet eba4bd4665ae4c04a2cab3ab1a2da780 (DEFAULT_TABLE table=default.Table_6b67c0c03ceb4b9491ad5828022b357f [id=368fcf43e863494ca260862d12331c0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:53.267422 18445 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eba4bd4665ae4c04a2cab3ab1a2da780. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:53.238301 20489 consensus_queue.cc:1035] T 62732af95c6a4c749c5b726f0460dc72 P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:53.270906 18724 tablet_service.cc:1467] Processing CreateTablet for tablet 415d99c36b1941e08aa4a16c50483ee4 (DEFAULT_TABLE table=default.Table_9049c9eeb10540d3815bab49caa1e82e [id=619746ec6b3244ee9ba511745c38bc25]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:53.271258 18724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 415d99c36b1941e08aa4a16c50483ee4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:53.271675 18586 tablet_service.cc:1467] Processing CreateTablet for tablet 9e8ab75e7d9c4b64a7b7dd2678569490 (DEFAULT_TABLE table=default.Table_b2d07520b9944a89ad62fca70396174e [id=1c8e0a3a44ec4a798770c0165ed3dffa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:53.272040 18586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e8ab75e7d9c4b64a7b7dd2678569490. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:53.273591 18983 raft_consensus.cc:374] T 436579f473844c6ca1038b496201a550 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:53.273665 18983 raft_consensus.cc:729] T 436579f473844c6ca1038b496201a550 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:53.273901 18983 consensus_queue.cc:260] T 436579f473844c6ca1038b496201a550 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:53.274581 18983 ts_tablet_manager.cc:1398] T 436579f473844c6ca1038b496201a550 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.038s user 0.001s sys 0.000s
I20240430 20:11:53.274799 20587 raft_consensus.cc:482] T 2e1cac64cefe4336bd3207b5ca58b631 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:53.274869 18737 tablet_service.cc:1467] Processing CreateTablet for tablet 9e8ab75e7d9c4b64a7b7dd2678569490 (DEFAULT_TABLE table=default.Table_b2d07520b9944a89ad62fca70396174e [id=1c8e0a3a44ec4a798770c0165ed3dffa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:53.274914 20587 raft_consensus.cc:504] T 2e1cac64cefe4336bd3207b5ca58b631 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:53.275159 18737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e8ab75e7d9c4b64a7b7dd2678569490. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:53.275534 20587 leader_election.cc:288] T 2e1cac64cefe4336bd3207b5ca58b631 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:53.276253 18983 tablet_bootstrap.cc:492] T 3191aa54a9644783a2403991c3be27c8 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:53.239830 21293 leader_election.cc:288] T a692b253db914531b8fc6ffaf3e350b5 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:53.249433 18592 tablet_service.cc:1467] Processing CreateTablet for tablet d84b399395ee4456b76117768ca626b4 (DEFAULT_TABLE table=default.Table_5ab8136f886143f2a4a7afb5d64a3690 [id=adc53037af0a416c9a8acb46ee50f24d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:53.278369 18592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d84b399395ee4456b76117768ca626b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:53.278779 18985 raft_consensus.cc:348] T 13c2e69ae91f4957887fd64b90f6e435 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:53.278978 18985 raft_consensus.cc:374] T 13c2e69ae91f4957887fd64b90f6e435 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:53.279031 18985 raft_consensus.cc:729] T 13c2e69ae91f4957887fd64b90f6e435 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:53.279167 18452 tablet_service.cc:1467] Processing CreateTablet for tablet 415d99c36b1941e08aa4a16c50483ee4 (DEFAULT_TABLE table=default.Table_9049c9eeb10540d3815bab49caa1e82e [id=619746ec6b3244ee9ba511745c38bc25]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:53.279193 18985 consensus_queue.cc:260] T 13c2e69ae91f4957887fd64b90f6e435 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:53.279814 18985 ts_tablet_manager.cc:1398] T 13c2e69ae91f4957887fd64b90f6e435 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.024s user 0.001s sys 0.000s
I20240430 20:11:53.279892 18453 tablet_service.cc:1467] Processing CreateTablet for tablet 9e8ab75e7d9c4b64a7b7dd2678569490 (DEFAULT_TABLE table=default.Table_b2d07520b9944a89ad62fca70396174e [id=1c8e0a3a44ec4a798770c0165ed3dffa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:53.281566 18452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 415d99c36b1941e08aa4a16c50483ee4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:53.253471 18590 tablet_service.cc:1467] Processing CreateTablet for tablet 991d73fe46bf4faaaac03d190bf27e6c (DEFAULT_TABLE table=default.Table_22d93bfc850f42bab0f67f0e0545601f [id=568d7291829346cb9b421788d4cc84c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:53.267611 18595 tablet_service.cc:1467] Processing CreateTablet for tablet eba4bd4665ae4c04a2cab3ab1a2da780 (DEFAULT_TABLE table=default.Table_6b67c0c03ceb4b9491ad5828022b357f [id=368fcf43e863494ca260862d12331c0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:53.268391 18600 tablet_service.cc:1467] Processing CreateTablet for tablet 415d99c36b1941e08aa4a16c50483ee4 (DEFAULT_TABLE table=default.Table_9049c9eeb10540d3815bab49caa1e82e [id=619746ec6b3244ee9ba511745c38bc25]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:53.282878 18590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 991d73fe46bf4faaaac03d190bf27e6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:53.283066 18595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eba4bd4665ae4c04a2cab3ab1a2da780. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:53.286172 18600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 415d99c36b1941e08aa4a16c50483ee4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:53.287111 18558 leader_election.cc:302] T a692b253db914531b8fc6ffaf3e350b5 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:53.287417 18619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e1cac64cefe4336bd3207b5ca58b631" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:53.287593 18619 raft_consensus.cc:2454] T 2e1cac64cefe4336bd3207b5ca58b631 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:53.291273 21221 raft_consensus.cc:2781] T a692b253db914531b8fc6ffaf3e350b5 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:53.291395 21221 raft_consensus.cc:686] T a692b253db914531b8fc6ffaf3e350b5 P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Becoming Leader. State: Replica: 5e1c6397f09747f091534faf93542710, State: Running, Role: LEADER
I20240430 20:11:53.291600 21221 consensus_queue.cc:237] T a692b253db914531b8fc6ffaf3e350b5 P 5e1c6397f09747f091534faf93542710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:53.292825 18696 leader_election.cc:302] T 2e1cac64cefe4336bd3207b5ca58b631 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:53.296460 20587 raft_consensus.cc:2781] T 2e1cac64cefe4336bd3207b5ca58b631 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:53.296555 20587 raft_consensus.cc:482] T 2e1cac64cefe4336bd3207b5ca58b631 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:53.296608 20587 raft_consensus.cc:3037] T 2e1cac64cefe4336bd3207b5ca58b631 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:53.299597 20587 raft_consensus.cc:504] T 2e1cac64cefe4336bd3207b5ca58b631 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:53.300742 20587 leader_election.cc:288] T 2e1cac64cefe4336bd3207b5ca58b631 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Requested vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
W20240430 20:11:53.304023 19895 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 2cd41b407d6b444d844c4c7c0ebf82e5 (table default.Table_6f8e0eff8f5a484dbd6a9c3fdb5d03f6 [id=3813b5e9778c4bd683a06598cef0d86f], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 2cd41b407d6b444d844c4c7c0ebf82e5 consensus state has no leader
I20240430 20:11:53.306773 18619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e1cac64cefe4336bd3207b5ca58b631" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:53.307011 18619 raft_consensus.cc:3037] T 2e1cac64cefe4336bd3207b5ca58b631 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:53.308072 20198 raft_consensus.cc:482] T a692b253db914531b8fc6ffaf3e350b5 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:53.308200 20198 raft_consensus.cc:504] T a692b253db914531b8fc6ffaf3e350b5 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:53.308769 20198 leader_election.cc:288] T a692b253db914531b8fc6ffaf3e350b5 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:53.309551 18619 raft_consensus.cc:2454] T 2e1cac64cefe4336bd3207b5ca58b631 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:53.310195 18696 leader_election.cc:302] T 2e1cac64cefe4336bd3207b5ca58b631 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:53.310441 21198 raft_consensus.cc:2781] T 2e1cac64cefe4336bd3207b5ca58b631 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:53.310549 21198 raft_consensus.cc:686] T 2e1cac64cefe4336bd3207b5ca58b631 P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Becoming Leader. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Running, Role: LEADER
I20240430 20:11:53.310742 18528 heartbeater.cc:503] Master 127.17.177.62:36441 was elected leader, sending a full tablet report...
I20240430 20:11:53.310801 21198 consensus_queue.cc:237] T 2e1cac64cefe4336bd3207b5ca58b631 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:53.311908 18619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a692b253db914531b8fc6ffaf3e350b5" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:53.316463 20489 consensus_queue.cc:1035] T 2aa2a489116940dea3173cad16d94fa5 P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:53.317227 20489 consensus_queue.cc:1035] T 35fdb052135541cfb42a8aeb03b64dc8 P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:53.319159 19940 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 6b51cbc731794083a88fe4bed1ddeebf (table default.Table_c6c962da746e4fd8a4d7200d09f08e9e [id=8e0f5d8c941d423ea7b30a89513d5170]). Expected version 1 got 0
I20240430 20:11:53.323482 18760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a692b253db914531b8fc6ffaf3e350b5" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:53.323776 18760 raft_consensus.cc:2454] T a692b253db914531b8fc6ffaf3e350b5 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 0.
I20240430 20:11:53.324179 18760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a692b253db914531b8fc6ffaf3e350b5" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:53.324331 18760 raft_consensus.cc:3037] T a692b253db914531b8fc6ffaf3e350b5 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:53.327987 18760 raft_consensus.cc:2454] T a692b253db914531b8fc6ffaf3e350b5 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
I20240430 20:11:53.330160 19940 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 62732af95c6a4c749c5b726f0460dc72 (table default.Table_e3160f5ced1b4718b479401ca4df675c [id=2d44ec94131148048021b2ee6366ba30]). Expected version 3 got 0
I20240430 20:11:53.331125 19940 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet a0baee91f5b44945b595d0e234af018e (table default.Table_5ed3ea587cf843db9a1ff1ffab40b678 [id=b3d1c2f97c4648518dbea83a46cadd85]). Expected version 2 got 0
I20240430 20:11:53.332280 19940 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 266fd23786bc435db5083370f3330ed0 (table default.Table_cf60b652e9dc49e7a948ce66fd1cbffe [id=9734bb8dd05a400ca4244b3829f01dee]). Expected version 1 got 0
I20240430 20:11:53.333956 18472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e1cac64cefe4336bd3207b5ca58b631" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:53.334175 18472 raft_consensus.cc:2454] T 2e1cac64cefe4336bd3207b5ca58b631 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:53.335474 19940 catalog_manager.cc:5484] T 89b774967624498fa2be499878b71124 P 2efb542aa55e439b99ccb98d1c5c884a reported cstate change: term changed from 0 to 1, leader changed from <none> to 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1). New cstate: current_term: 1 leader_uuid: "2efb542aa55e439b99ccb98d1c5c884a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } health_report { overall_health: HEALTHY } } }
I20240430 20:11:53.338119 19940 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 89b774967624498fa2be499878b71124 (table default.Table_2ec9e316bc3347b189a13b7a8bbcf829 [id=9fa22833979f4c2f978de651145acef9]). Expected version 1 got 0
I20240430 20:11:53.339064 19940 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 130007acc8ba4e709c3eb85fabe75903 (table default.Table_78987875650940c293247bc43db345ab [id=36f478a20a1f4a53a52fc29ade62b7f6]). Expected version 1 got 0
I20240430 20:11:53.339143 18472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e1cac64cefe4336bd3207b5ca58b631" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:53.339329 18472 raft_consensus.cc:3037] T 2e1cac64cefe4336bd3207b5ca58b631 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:53.339946 19940 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet dac141d1b70c48d785caddd7182811ee (table default.Table_fc0a4540f98b4eea95099b8ab46ea235 [id=f3c53ccffe64410495cc8c379a05cfae]). Expected version 1 got 0
I20240430 20:11:53.340135 19940 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet ed0c37a71ede4bfc9eb5ad1a4365f25e (table default.Table_a2e85efc6b454524b0b122102ec07002 [id=86d795a82e674d859104741ebd04f4e8]). Expected version 1 got 0
I20240430 20:11:53.340282 19940 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 12795670f21447a8964fe1efc6e92398 (table default.Table_896bbae1f6944005b0272ca8ebc97218 [id=d5cea90cbc2c48df9d8b80da1eccc949]). Expected version 3 got 0
I20240430 20:11:53.340727 19940 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet b285e3eb16d148779c74e057458d8461 (table default.Table_9df4a0b4170f4f28a6c17b0cb052ead6 [id=d1655c507b434f129ad7ae0b04365e51]). Expected version 2 got 0
I20240430 20:11:53.340970 19940 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet a482f4622f774d60b4efd6827b8be7d7 (table default.Table_e686cdfcbcc046749fa5b51578a64794 [id=c3dc9f05316d46d6a3ef4d03f40877ec]). Expected version 1 got 0
I20240430 20:11:53.341395 19940 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet acd23f0a63d5448aaa7ed78422e67bad (table default.Table_972f13be4dca4cd19b9a04534dc59f51 [id=69f37db4de634a3185613e40013e0ed7]). Expected version 1 got 0
I20240430 20:11:53.341614 19940 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 2e79df8744e74b5991a79ce978ff0202 (table default.Table_276bbdc7cfe542d1a2936485a2aad1ec [id=f146a15e6e8a4acbb4c734c1c863da57]). Expected version 1 got 0
I20240430 20:11:53.341825 19940 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 69f88532952a4591bf1577541c3430e6 (table default.Table_1177828cad1f4a24bd397619dad96170 [id=a95fed1bc3fa4865bf9a35dfc51b659c]). Expected version 2 got 0
I20240430 20:11:53.342105 19940 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 330453ddeea6456e8ef93fa92608b0d3 (table default.Table_c1f36f7df5ae4e7c8eb8b4e99b89748b [id=a50abf28790d4284a76ac6ee22c02606]). Expected version 1 got 0
I20240430 20:11:53.342227 18472 raft_consensus.cc:2454] T 2e1cac64cefe4336bd3207b5ca58b631 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:53.342342 18453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e8ab75e7d9c4b64a7b7dd2678569490. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:53.343502 18472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a692b253db914531b8fc6ffaf3e350b5" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:53.343791 18472 raft_consensus.cc:2379] T a692b253db914531b8fc6ffaf3e350b5 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c51c33d5616482cafb68e26688820f4 in current term 1: Already voted for candidate 5e1c6397f09747f091534faf93542710 in this term.
I20240430 20:11:53.354835 21293 raft_consensus.cc:482] T 69f88532952a4591bf1577541c3430e6 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:53.355178 21293 raft_consensus.cc:504] T 69f88532952a4591bf1577541c3430e6 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:53.355882 21293 leader_election.cc:288] T 69f88532952a4591bf1577541c3430e6 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:53.359273 18985 tablet_bootstrap.cc:492] T 2797eed593fb4b06a3c4d6a28e8075a2 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:53.362707 19940 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 22fef8bb6b034db098c3372ae992f7a7 (table default.Table_4440df38610340758047115845b42b95 [id=7d186ca20c094cc19aef792fe1c094e9]). Expected version 1 got 0
I20240430 20:11:53.366804 19940 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 086b3dfa84c045a18c8c4f177dce606a (table default.Table_df19cb8a6f4d46699350ec5b69c14e5a [id=197923633fc54a4582207942b51530e2]). Expected version 2 got 0
I20240430 20:11:53.368350 19940 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 1050ab528c7d41559831652a1437d358 (table default.Table_acddc4cbf6334546b5fb3e4752fef2a5 [id=274cb15fd751410e83327ac4e8dc4629]). Expected version 2 got 0
I20240430 20:11:53.369282 19940 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet e041756c962b4a9f97c2cb314e7d8c05 (table default.Table_de88cafd92b149f9b545369bff71033c [id=c4bf4806f3d84579b16a3038d8d982c3]). Expected version 1 got 0
I20240430 20:11:53.369968 19940 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 94fbd4a57a1e438891fcb456f6b40be3 (table default.Table_25ddc3adef6b4ccf8b9c00cc6772031c [id=4fa623ffdcf347639150a355806038af]). Expected version 1 got 0
I20240430 20:11:53.370895 19940 catalog_manager.cc:5484] T 8e9711d0a382424f9f758bfa883866a0 P 2efb542aa55e439b99ccb98d1c5c884a reported cstate change: term changed from 0 to 1, leader changed from <none> to 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1). New cstate: current_term: 1 leader_uuid: "2efb542aa55e439b99ccb98d1c5c884a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } health_report { overall_health: HEALTHY } } }
I20240430 20:11:53.375942 19940 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 8e9711d0a382424f9f758bfa883866a0 (table default.Table_f1dc481c6d8c43f589fe2b1892a4cb10 [id=db696ce66005401fa73746e55c071f44]). Expected version 1 got 0
I20240430 20:11:53.376650 18477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69f88532952a4591bf1577541c3430e6" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:53.376853 19940 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 7a0176f8a74c47efbbe3d3343b3dc896 (table default.Table_144e5402c1864d818436015b26648024 [id=73e5c9f0470d47849109819c063e1ce4]). Expected version 2 got 0
I20240430 20:11:53.377264 19940 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 47bf64ecc41b445da4ff697e141243dd (table default.Table_f1ad29d5986d4b799cfc67afb2b0dfdf [id=838fc8b61a864dea8877a6f144e4cb4f]). Expected version 2 got 0
I20240430 20:11:53.378269 19940 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet f5eaf2413e1d4e69b56fae61f768f0ce (table default.Table_1280cfe32d5b41abb4d3eb1bd70641bf [id=8489b872074a42b6abb713e420968141]). Expected version 1 got 0
I20240430 20:11:53.379076 19940 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet eb6c96047b114ebba73db7a371f2da8e (table default.Table_7d5f6f18cbd64c869739872a7dff9be2 [id=e67d619e36c0485f8f22b4a81cb4cb6c]). Expected version 1 got 0
I20240430 20:11:53.383632 18983 tablet_bootstrap.cc:654] T 3191aa54a9644783a2403991c3be27c8 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:53.386283 19940 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet aaef85eea87f4443943fd3692bb2a38f (table default.Table_3da5b4c8668c470483ae2a0c7773eaaf [id=e292d818ee3c45df90522f08eb2d6227]). Expected version 1 got 0
I20240430 20:11:53.389672 18988 tablet_bootstrap.cc:492] T 13c2e69ae91f4957887fd64b90f6e435 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:53.389904 18988 ts_tablet_manager.cc:1367] T 13c2e69ae91f4957887fd64b90f6e435 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.138s user 0.008s sys 0.006s
I20240430 20:11:53.390718 18988 raft_consensus.cc:348] T 13c2e69ae91f4957887fd64b90f6e435 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:53.391016 18988 raft_consensus.cc:374] T 13c2e69ae91f4957887fd64b90f6e435 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:53.391114 18988 raft_consensus.cc:729] T 13c2e69ae91f4957887fd64b90f6e435 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:53.391356 18988 consensus_queue.cc:260] T 13c2e69ae91f4957887fd64b90f6e435 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:53.391944 18988 ts_tablet_manager.cc:1398] T 13c2e69ae91f4957887fd64b90f6e435 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240430 20:11:53.392266 18988 tablet_bootstrap.cc:492] T 1fe7c6f1ab54499facd4d01573d3e7bc P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:53.395571 18985 tablet_bootstrap.cc:654] T 2797eed593fb4b06a3c4d6a28e8075a2 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:53.397583 19940 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet a59ef6c545dd41a28265a4c1b8ce10e0 (table default.Table_07e40af8f1c546048975c8edf88170d7 [id=86df5d161d8641c89070e7eed9a39360]). Expected version 2 got 0
I20240430 20:11:53.397970 18758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69f88532952a4591bf1577541c3430e6" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:53.398054 19940 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 37b2ea4be7494f0985808ade788dbe2b (table default.Table_5f6e0fce999b46de927f9271e5de5071 [id=0baad83a8c0f4aeebc7ddad068625a69]). Expected version 1 got 0
I20240430 20:11:53.399091 19940 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 99203cc522de424184c6f97453f166fd (table default.Table_cb2a4b9ceaf4431facfff2c9bcf94c5d [id=3f8690a437ae49a3beb009fa7899d5d0]). Expected version 1 got 0
I20240430 20:11:53.401206 18983 tablet_bootstrap.cc:492] T 3191aa54a9644783a2403991c3be27c8 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:53.401548 18983 ts_tablet_manager.cc:1367] T 3191aa54a9644783a2403991c3be27c8 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.125s user 0.004s sys 0.000s
I20240430 20:11:53.401695 18988 tablet_bootstrap.cc:654] T 1fe7c6f1ab54499facd4d01573d3e7bc P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:53.402570 18983 raft_consensus.cc:348] T 3191aa54a9644783a2403991c3be27c8 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:53.402778 19940 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 821c55edb3414d72b1489eea97d3f73a (table default.Table_4d68c38d9ba7452a829f8318b2b267b8 [id=d06e3e9045b1429aaeebd1fd67d841de]). Expected version 1 got 0
I20240430 20:11:53.402940 18983 raft_consensus.cc:374] T 3191aa54a9644783a2403991c3be27c8 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:53.403012 19940 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 5df364628561475391c9a5e10ff5a05f (table default.Table_3c8f9f7b22f64fb1bdc3f1550987233a [id=7fd6f8576f4c4cbc8ffcdb4ab95e15dc]). Expected version 1 got 0
I20240430 20:11:53.403079 18983 raft_consensus.cc:729] T 3191aa54a9644783a2403991c3be27c8 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:53.403304 19940 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 724e51ff05ec4baebc2bb55e05c8eddc (table default.Table_19df33614a6c4f94b6419a5b441e3ef8 [id=4dce8a767f0541e1beb5464b2ed95d06]). Expected version 2 got 1
I20240430 20:11:53.403412 18983 consensus_queue.cc:260] T 3191aa54a9644783a2403991c3be27c8 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:53.403613 19940 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet ca64de16d40f4a44b8b411f5526ba11a (table default.Table_1cf914acf7fe4e98b8dbc297b6372f21 [id=f4ba9536e9c141deb68f4f62b7e8fa70]). Expected version 1 got 0
I20240430 20:11:53.403851 19940 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 81787eccca6c42c1b421377a3bf9ebe6 (table default.Table_0b4ca4cd0af148399dfa21f2d32114e3 [id=7f602584cef9486eb7d9b4389be3f43e]). Expected version 3 got 2
I20240430 20:11:53.406528 18699 leader_election.cc:302] T a692b253db914531b8fc6ffaf3e350b5 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710
I20240430 20:11:53.406896 20198 raft_consensus.cc:2726] T a692b253db914531b8fc6ffaf3e350b5 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:53.407080 19940 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 66dd767bd3f8494f94061fb89e5e5680 (table default.Table_b064d5659f7243f5b0ec167fb6c3f404 [id=5227206f44d04fa89b4842b899f33afb]). Expected version 2 got 0
I20240430 20:11:53.407349 19940 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet f1073fc792954c5795a05aa7d0ec24df (table default.Table_ba8b218e936b4bfcb073f8182f474cca [id=aedf11fda8914a3b843a57f8a7b69a53]). Expected version 1 got 0
I20240430 20:11:53.408047 19940 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet fbdeb4714b8c4b93ab72822c43329aae (table default.Table_c12ca6938d424fbfa2e1df5cdb00fdbc [id=0fb93f36bc17408ea56bc4cafcfd8580]). Expected version 1 got 0
I20240430 20:11:53.412891 19940 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet c6e3d09402574f1e8fe9e4160c70a3ee (table default.Table_656db97355f64d7c8aa29b1352dfe9c8 [id=450f5d1c7e084a1bb88d035c0ccfcf82]). Expected version 1 got 0
I20240430 20:11:53.413312 19940 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 488d0e609b224674932f890939664df2 (table default.Table_58f35a2120874472978ca0e5f816e86c [id=ff18001c6d814afc8c1768d0c84a6bac]). Expected version 1 got 0
I20240430 20:11:53.413910 19940 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 78ba48dadef548708153878810cc46f8 (table default.Table_b9983e101a6547649eb44ee04b5e1147 [id=29769be2b9ec4ecba9de5c0a65692f85]). Expected version 1 got 0
I20240430 20:11:53.418974 21276 raft_consensus.cc:482] T 4f6f9ae722da49e58c9b72107f6da2fa P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5e1c6397f09747f091534faf93542710)
I20240430 20:11:53.419126 21276 raft_consensus.cc:504] T 4f6f9ae722da49e58c9b72107f6da2fa P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:53.419781 21276 leader_election.cc:288] T 4f6f9ae722da49e58c9b72107f6da2fa P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
W20240430 20:11:53.420692 18415 leader_election.cc:341] T 4f6f9ae722da49e58c9b72107f6da2fa P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Not found: Tablet not found: 4f6f9ae722da49e58c9b72107f6da2fa
I20240430 20:11:53.417151 21221 raft_consensus.cc:482] T a663030d37614053a7c0369841403422 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240430 20:11:53.417544 18557 leader_election.cc:341] T 69f88532952a4591bf1577541c3430e6 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:53.423512 21221 raft_consensus.cc:504] T a663030d37614053a7c0369841403422 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:53.423614 19940 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 2cd41b407d6b444d844c4c7c0ebf82e5 (table default.Table_6f8e0eff8f5a484dbd6a9c3fdb5d03f6 [id=3813b5e9778c4bd683a06598cef0d86f]). Expected version 1 got 0
I20240430 20:11:53.424280 21221 leader_election.cc:288] T a663030d37614053a7c0369841403422 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:53.424851 19940 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 772eba5f24c549c791cad75565858285 (table default.Table_5fc77d5f60ae44f4bb4ca9f7f183672c [id=6ca223ca6354449c959d85ed2ac6a9c8]). Expected version 1 got 0
I20240430 20:11:53.425863 18983 ts_tablet_manager.cc:1398] T 3191aa54a9644783a2403991c3be27c8 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.024s user 0.000s sys 0.002s
W20240430 20:11:53.426375 18558 leader_election.cc:341] T 69f88532952a4591bf1577541c3430e6 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:53.426478 18558 leader_election.cc:302] T 69f88532952a4591bf1577541c3430e6 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:53.427315 21221 raft_consensus.cc:2726] T 69f88532952a4591bf1577541c3430e6 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:53.430042 19940 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 1faa2f2c84184b129ba6e4d2857822f0 (table default.Table_0dd55b391be94213b4af60ea6e5a979f [id=1a382ad8009b4522a50d55c307db8f96]). Expected version 1 got 0
I20240430 20:11:53.432700 18477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a663030d37614053a7c0369841403422" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:53.433641 18988 tablet_bootstrap.cc:492] T 1fe7c6f1ab54499facd4d01573d3e7bc P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:53.433833 18988 ts_tablet_manager.cc:1367] T 1fe7c6f1ab54499facd4d01573d3e7bc P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.042s user 0.003s sys 0.001s
I20240430 20:11:53.434675 18988 raft_consensus.cc:348] T 1fe7c6f1ab54499facd4d01573d3e7bc P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:53.435021 18988 raft_consensus.cc:374] T 1fe7c6f1ab54499facd4d01573d3e7bc P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:53.435058 18985 tablet_bootstrap.cc:492] T 2797eed593fb4b06a3c4d6a28e8075a2 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
W20240430 20:11:53.435104 18558 leader_election.cc:341] T a663030d37614053a7c0369841403422 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:53.435211 18985 ts_tablet_manager.cc:1367] T 2797eed593fb4b06a3c4d6a28e8075a2 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.076s user 0.004s sys 0.000s
I20240430 20:11:53.435158 18988 raft_consensus.cc:729] T 1fe7c6f1ab54499facd4d01573d3e7bc P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:53.435496 18988 consensus_queue.cc:260] T 1fe7c6f1ab54499facd4d01573d3e7bc P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:53.436079 18988 ts_tablet_manager.cc:1398] T 1fe7c6f1ab54499facd4d01573d3e7bc P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240430 20:11:53.436378 18988 tablet_bootstrap.cc:492] T 436579f473844c6ca1038b496201a550 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:53.438694 19940 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 3a03b6997921408cb5a65f4cae223e9c (table default.Table_744b5c96ea014bdb8feae740356a790e [id=83b1ac2e9c3d4a10b02f96ebb17faf44]). Expected version 1 got 0
I20240430 20:11:53.439040 19940 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet ae20f44df50343289b0a512a4ea53692 (table default.Table_1844f6b1bda84812825384e14a2fd94e [id=71da9acc3cff4d95bbac4a686b708fd5]). Expected version 1 got 0
I20240430 20:11:53.439430 19940 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 7ce09fcbe1ac457a9c05efaf000c3263 (table default.Table_edc4c27ed846454288f73f6a0d88cb69 [id=78fd3f516083489ab3b42d90dcef7e4d]). Expected version 1 got 0
I20240430 20:11:53.439741 19940 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 331c031d44854d679a27d10d9151d0ea (table default.Table_763f0800e8bb4fa69976886416250f42 [id=61c8a5b99ef34a0baeeb343fef7cfd05]). Expected version 1 got 0
I20240430 20:11:53.440137 19940 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 17ddcaf971154a52b606bbf29a539bf0 (table default.Table_f2f38f5d69e34eaa9e3dbf24cfd1cdd6 [id=872c2faa4a464edd98a51df2e1b3f817]). Expected version 1 got 0
I20240430 20:11:53.440524 18988 tablet_bootstrap.cc:654] T 436579f473844c6ca1038b496201a550 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:53.440815 19940 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 2da900b6bbfa4fa99fe8494fd2e68da8 (table default.Table_19a91a7f17ff456ebb145e8dd89c0940 [id=67283dd3995a44c6a7a7bc8722e4f955]). Expected version 1 got 0
I20240430 20:11:53.441282 18760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a663030d37614053a7c0369841403422" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:53.435873 18985 raft_consensus.cc:348] T 2797eed593fb4b06a3c4d6a28e8075a2 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:53.441746 18985 raft_consensus.cc:374] T 2797eed593fb4b06a3c4d6a28e8075a2 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:53.441680 19940 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 0368d37dd9e2452982fb81f539c6805b (table default.Table_404b186c544e4b30bb36cb77d1a14120 [id=4330626e342a48699ea939add5c91b11]). Expected version 1 got 0
I20240430 20:11:53.441831 18985 raft_consensus.cc:729] T 2797eed593fb4b06a3c4d6a28e8075a2 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:53.442116 19940 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 677481aee5fa4eb4b189322d7a3f8f2e (table default.Table_1ce204e15fce4c469816cf4e7f80cac7 [id=d268c31d82df4871bc1a78aea5871b33]). Expected version 1 got 0
I20240430 20:11:53.442323 18983 tablet_bootstrap.cc:492] T 4dac8b3838954d839094021f1c0503da P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:53.443087 19940 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 5b26319488644d6a8b6be60460a5e7af (table default.Table_69892a8264a4477db48ee6248758e046 [id=5a1c4a7be2fc465fac89a50854eb3322]). Expected version 3 got 0
W20240430 20:11:53.444273 18417 leader_election.cc:341] T 4f6f9ae722da49e58c9b72107f6da2fa P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Not found: Tablet not found: 4f6f9ae722da49e58c9b72107f6da2fa
I20240430 20:11:53.444407 18417 leader_election.cc:302] T 4f6f9ae722da49e58c9b72107f6da2fa P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a; no voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710
I20240430 20:11:53.444805 21276 raft_consensus.cc:2726] T 4f6f9ae722da49e58c9b72107f6da2fa P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240430 20:11:53.446163 18557 leader_election.cc:341] T a663030d37614053a7c0369841403422 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:53.446297 18557 leader_election.cc:302] T a663030d37614053a7c0369841403422 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:53.446903 20489 raft_consensus.cc:2726] T a663030d37614053a7c0369841403422 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:53.442057 18985 consensus_queue.cc:260] T 2797eed593fb4b06a3c4d6a28e8075a2 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:53.448793 18985 ts_tablet_manager.cc:1398] T 2797eed593fb4b06a3c4d6a28e8075a2 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20240430 20:11:53.449409 18983 tablet_bootstrap.cc:654] T 4dac8b3838954d839094021f1c0503da P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:53.450071 18985 tablet_bootstrap.cc:492] T 69f88532952a4591bf1577541c3430e6 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:53.455071 18985 tablet_bootstrap.cc:654] T 69f88532952a4591bf1577541c3430e6 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:53.461810 18669 heartbeater.cc:347] Connected to a master server at 127.17.177.60:37225
I20240430 20:11:53.462906 21365 consensus_queue.cc:1035] T 2aa2a489116940dea3173cad16d94fa5 P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:53.469349 18988 tablet_bootstrap.cc:492] T 436579f473844c6ca1038b496201a550 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:53.469514 18988 ts_tablet_manager.cc:1367] T 436579f473844c6ca1038b496201a550 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.033s user 0.005s sys 0.000s
I20240430 20:11:53.470338 18988 raft_consensus.cc:348] T 436579f473844c6ca1038b496201a550 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:53.470567 18988 raft_consensus.cc:374] T 436579f473844c6ca1038b496201a550 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:53.470669 18988 raft_consensus.cc:729] T 436579f473844c6ca1038b496201a550 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:53.470892 18988 consensus_queue.cc:260] T 436579f473844c6ca1038b496201a550 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:53.471400 18988 ts_tablet_manager.cc:1398] T 436579f473844c6ca1038b496201a550 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240430 20:11:53.471683 18988 tablet_bootstrap.cc:492] T 4dac8b3838954d839094021f1c0503da P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:53.473429 18988 tablet_bootstrap.cc:654] T 4dac8b3838954d839094021f1c0503da P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:53.479712 21365 consensus_queue.cc:1035] T eb6c96047b114ebba73db7a371f2da8e P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:53.495988 18988 tablet_bootstrap.cc:492] T 4dac8b3838954d839094021f1c0503da P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:53.496148 18988 ts_tablet_manager.cc:1367] T 4dac8b3838954d839094021f1c0503da P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.002s
I20240430 20:11:53.496968 18988 raft_consensus.cc:348] T 4dac8b3838954d839094021f1c0503da P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:53.497169 18988 raft_consensus.cc:374] T 4dac8b3838954d839094021f1c0503da P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:53.497223 18988 raft_consensus.cc:729] T 4dac8b3838954d839094021f1c0503da P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:53.497433 18988 consensus_queue.cc:260] T 4dac8b3838954d839094021f1c0503da P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:53.498051 18988 ts_tablet_manager.cc:1398] T 4dac8b3838954d839094021f1c0503da P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240430 20:11:53.506090 18983 tablet_bootstrap.cc:492] T 4dac8b3838954d839094021f1c0503da P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:53.506341 18983 ts_tablet_manager.cc:1367] T 4dac8b3838954d839094021f1c0503da P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.064s user 0.003s sys 0.004s
I20240430 20:11:53.507114 18988 tablet_bootstrap.cc:492] T a663030d37614053a7c0369841403422 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:53.507751 18983 raft_consensus.cc:348] T 4dac8b3838954d839094021f1c0503da P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:53.508074 18983 raft_consensus.cc:374] T 4dac8b3838954d839094021f1c0503da P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:53.508198 18983 raft_consensus.cc:729] T 4dac8b3838954d839094021f1c0503da P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:53.508489 18983 consensus_queue.cc:260] T 4dac8b3838954d839094021f1c0503da P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:53.508708 18988 tablet_bootstrap.cc:654] T a663030d37614053a7c0369841403422 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:53.509303 18983 ts_tablet_manager.cc:1398] T 4dac8b3838954d839094021f1c0503da P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240430 20:11:53.509634 18983 tablet_bootstrap.cc:492] T 330453ddeea6456e8ef93fa92608b0d3 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:53.511091 18985 tablet_bootstrap.cc:492] T 69f88532952a4591bf1577541c3430e6 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:53.511233 18985 ts_tablet_manager.cc:1367] T 69f88532952a4591bf1577541c3430e6 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.061s user 0.005s sys 0.000s
I20240430 20:11:53.511873 18985 raft_consensus.cc:348] T 69f88532952a4591bf1577541c3430e6 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:53.512065 18985 raft_consensus.cc:374] T 69f88532952a4591bf1577541c3430e6 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:53.512111 18985 raft_consensus.cc:729] T 69f88532952a4591bf1577541c3430e6 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:53.512306 18985 consensus_queue.cc:260] T 69f88532952a4591bf1577541c3430e6 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:53.512807 18985 ts_tablet_manager.cc:1398] T 69f88532952a4591bf1577541c3430e6 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240430 20:11:53.515645 18983 tablet_bootstrap.cc:654] T 330453ddeea6456e8ef93fa92608b0d3 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:53.516165 18985 tablet_bootstrap.cc:492] T 436579f473844c6ca1038b496201a550 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:53.518290 18985 tablet_bootstrap.cc:654] T 436579f473844c6ca1038b496201a550 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:53.534003 21365 consensus_queue.cc:1035] T eb6c96047b114ebba73db7a371f2da8e P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:53.534335 18988 tablet_bootstrap.cc:492] T a663030d37614053a7c0369841403422 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:53.534476 18988 ts_tablet_manager.cc:1367] T a663030d37614053a7c0369841403422 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.027s user 0.002s sys 0.002s
I20240430 20:11:53.535396 18988 raft_consensus.cc:348] T a663030d37614053a7c0369841403422 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:53.535643 18988 raft_consensus.cc:374] T a663030d37614053a7c0369841403422 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:53.535776 18988 raft_consensus.cc:729] T a663030d37614053a7c0369841403422 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:53.540712 18988 consensus_queue.cc:260] T a663030d37614053a7c0369841403422 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:53.542176 18988 ts_tablet_manager.cc:1398] T a663030d37614053a7c0369841403422 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.008s user 0.003s sys 0.000s
I20240430 20:11:53.542495 18988 tablet_bootstrap.cc:492] T 69f88532952a4591bf1577541c3430e6 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:53.546027 18988 tablet_bootstrap.cc:654] T 69f88532952a4591bf1577541c3430e6 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:53.548254 18983 tablet_bootstrap.cc:492] T 330453ddeea6456e8ef93fa92608b0d3 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:53.548770 18983 ts_tablet_manager.cc:1367] T 330453ddeea6456e8ef93fa92608b0d3 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.039s user 0.003s sys 0.001s
I20240430 20:11:53.549662 18983 raft_consensus.cc:348] T 330453ddeea6456e8ef93fa92608b0d3 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:53.551728 18983 raft_consensus.cc:374] T 330453ddeea6456e8ef93fa92608b0d3 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:53.551955 18983 raft_consensus.cc:729] T 330453ddeea6456e8ef93fa92608b0d3 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:53.552455 18983 consensus_queue.cc:260] T 330453ddeea6456e8ef93fa92608b0d3 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:53.556814 18983 ts_tablet_manager.cc:1398] T 330453ddeea6456e8ef93fa92608b0d3 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20240430 20:11:53.558840 18988 tablet_bootstrap.cc:492] T 69f88532952a4591bf1577541c3430e6 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:53.559612 18983 tablet_bootstrap.cc:492] T 1050ab528c7d41559831652a1437d358 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:53.560016 18988 ts_tablet_manager.cc:1367] T 69f88532952a4591bf1577541c3430e6 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.002s
I20240430 20:11:53.565985 18988 raft_consensus.cc:348] T 69f88532952a4591bf1577541c3430e6 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:53.567975 18983 tablet_bootstrap.cc:654] T 1050ab528c7d41559831652a1437d358 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:53.569631 19944 catalog_manager.cc:5484] T 3a03b6997921408cb5a65f4cae223e9c P 5e1c6397f09747f091534faf93542710 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e1c6397f09747f091534faf93542710 (127.17.177.2). New cstate: current_term: 1 leader_uuid: "5e1c6397f09747f091534faf93542710" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } health_report { overall_health: UNKNOWN } } }
I20240430 20:11:53.570324 18988 raft_consensus.cc:374] T 69f88532952a4591bf1577541c3430e6 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:53.570478 18988 raft_consensus.cc:729] T 69f88532952a4591bf1577541c3430e6 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:53.570771 18988 consensus_queue.cc:260] T 69f88532952a4591bf1577541c3430e6 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:53.571632 18985 tablet_bootstrap.cc:492] T 436579f473844c6ca1038b496201a550 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:53.571831 18985 ts_tablet_manager.cc:1367] T 436579f473844c6ca1038b496201a550 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.056s user 0.009s sys 0.007s
I20240430 20:11:53.572662 18985 raft_consensus.cc:348] T 436579f473844c6ca1038b496201a550 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:53.572921 18985 raft_consensus.cc:374] T 436579f473844c6ca1038b496201a550 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:53.573029 18985 raft_consensus.cc:729] T 436579f473844c6ca1038b496201a550 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:53.573300 18985 consensus_queue.cc:260] T 436579f473844c6ca1038b496201a550 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:53.573884 18985 ts_tablet_manager.cc:1398] T 436579f473844c6ca1038b496201a550 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240430 20:11:53.577205 18985 tablet_bootstrap.cc:492] T a663030d37614053a7c0369841403422 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:53.568207 21365 raft_consensus.cc:482] T 330453ddeea6456e8ef93fa92608b0d3 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:53.577719 21365 raft_consensus.cc:504] T 330453ddeea6456e8ef93fa92608b0d3 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:53.578477 21365 leader_election.cc:288] T 330453ddeea6456e8ef93fa92608b0d3 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:53.579466 18750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "330453ddeea6456e8ef93fa92608b0d3" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:53.572429 18988 ts_tablet_manager.cc:1398] T 69f88532952a4591bf1577541c3430e6 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
W20240430 20:11:53.580049 18557 leader_election.cc:341] T 330453ddeea6456e8ef93fa92608b0d3 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:53.580492 18988 tablet_bootstrap.cc:492] T 330453ddeea6456e8ef93fa92608b0d3 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:53.580402 18477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "330453ddeea6456e8ef93fa92608b0d3" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
W20240430 20:11:53.582154 18558 leader_election.cc:341] T 330453ddeea6456e8ef93fa92608b0d3 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:53.582274 18558 leader_election.cc:302] T 330453ddeea6456e8ef93fa92608b0d3 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:53.583252 21365 raft_consensus.cc:2726] T 330453ddeea6456e8ef93fa92608b0d3 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:53.583444 18810 heartbeater.cc:503] Master 127.17.177.60:37225 was elected leader, sending a full tablet report...
I20240430 20:11:53.585188 19944 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 3a03b6997921408cb5a65f4cae223e9c (table default.Table_744b5c96ea014bdb8feae740356a790e [id=83b1ac2e9c3d4a10b02f96ebb17faf44]). Expected version 1 got 0
I20240430 20:11:53.585873 19944 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet ae20f44df50343289b0a512a4ea53692 (table default.Table_1844f6b1bda84812825384e14a2fd94e [id=71da9acc3cff4d95bbac4a686b708fd5]). Expected version 1 got 0
I20240430 20:11:53.583148 18985 tablet_bootstrap.cc:654] T a663030d37614053a7c0369841403422 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:53.587702 19944 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 1faa2f2c84184b129ba6e4d2857822f0 (table default.Table_0dd55b391be94213b4af60ea6e5a979f [id=1a382ad8009b4522a50d55c307db8f96]). Expected version 1 got 0
I20240430 20:11:53.588076 19944 catalog_manager.cc:5484] T eb6c96047b114ebba73db7a371f2da8e P 5e1c6397f09747f091534faf93542710 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e1c6397f09747f091534faf93542710 (127.17.177.2). New cstate: current_term: 1 leader_uuid: "5e1c6397f09747f091534faf93542710" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } health_report { overall_health: UNKNOWN } } }
I20240430 20:11:53.589192 19944 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet eb6c96047b114ebba73db7a371f2da8e (table default.Table_7d5f6f18cbd64c869739872a7dff9be2 [id=e67d619e36c0485f8f22b4a81cb4cb6c]). Expected version 1 got 0
I20240430 20:11:53.589609 18988 tablet_bootstrap.cc:654] T 330453ddeea6456e8ef93fa92608b0d3 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:53.589604 19944 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 6b51cbc731794083a88fe4bed1ddeebf (table default.Table_c6c962da746e4fd8a4d7200d09f08e9e [id=8e0f5d8c941d423ea7b30a89513d5170]). Expected version 1 got 0
I20240430 20:11:53.591253 19944 catalog_manager.cc:5484] T 62732af95c6a4c749c5b726f0460dc72 P 5e1c6397f09747f091534faf93542710 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e1c6397f09747f091534faf93542710 (127.17.177.2). New cstate: current_term: 1 leader_uuid: "5e1c6397f09747f091534faf93542710" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } health_report { overall_health: UNKNOWN } } }
I20240430 20:11:53.591817 19944 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 62732af95c6a4c749c5b726f0460dc72 (table default.Table_e3160f5ced1b4718b479401ca4df675c [id=2d44ec94131148048021b2ee6366ba30]). Expected version 3 got 0
I20240430 20:11:53.592855 19944 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet a0baee91f5b44945b595d0e234af018e (table default.Table_5ed3ea587cf843db9a1ff1ffab40b678 [id=b3d1c2f97c4648518dbea83a46cadd85]). Expected version 2 got 0
I20240430 20:11:53.595563 18988 tablet_bootstrap.cc:492] T 330453ddeea6456e8ef93fa92608b0d3 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:53.595712 18988 ts_tablet_manager.cc:1367] T 330453ddeea6456e8ef93fa92608b0d3 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.003s
I20240430 20:11:53.596222 19944 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 266fd23786bc435db5083370f3330ed0 (table default.Table_cf60b652e9dc49e7a948ce66fd1cbffe [id=9734bb8dd05a400ca4244b3829f01dee]). Expected version 1 got 0
I20240430 20:11:53.596462 18988 raft_consensus.cc:348] T 330453ddeea6456e8ef93fa92608b0d3 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:53.596664 18988 raft_consensus.cc:374] T 330453ddeea6456e8ef93fa92608b0d3 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:53.596729 18988 raft_consensus.cc:729] T 330453ddeea6456e8ef93fa92608b0d3 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:53.596932 18988 consensus_queue.cc:260] T 330453ddeea6456e8ef93fa92608b0d3 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:53.599203 19944 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 89b774967624498fa2be499878b71124 (table default.Table_2ec9e316bc3347b189a13b7a8bbcf829 [id=9fa22833979f4c2f978de651145acef9]). Expected version 1 got 0
I20240430 20:11:53.599578 19944 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet ed0c37a71ede4bfc9eb5ad1a4365f25e (table default.Table_a2e85efc6b454524b0b122102ec07002 [id=86d795a82e674d859104741ebd04f4e8]). Expected version 1 got 0
I20240430 20:11:53.600067 19944 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 12795670f21447a8964fe1efc6e92398 (table default.Table_896bbae1f6944005b0272ca8ebc97218 [id=d5cea90cbc2c48df9d8b80da1eccc949]). Expected version 3 got 0
I20240430 20:11:53.600420 19944 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet dd1de890e4394db287cf38bb411f5586 (table default.Table_84084a27f7504dd992fce07448c10df3 [id=6257559b0e9545c58848a76af0c5a5f7]). Expected version 2 got 0
I20240430 20:11:53.600731 18983 tablet_bootstrap.cc:492] T 1050ab528c7d41559831652a1437d358 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:53.600922 18983 ts_tablet_manager.cc:1367] T 1050ab528c7d41559831652a1437d358 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.041s user 0.002s sys 0.002s
I20240430 20:11:53.601642 18983 raft_consensus.cc:348] T 1050ab528c7d41559831652a1437d358 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:53.601850 18983 raft_consensus.cc:374] T 1050ab528c7d41559831652a1437d358 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:53.601953 18983 raft_consensus.cc:729] T 1050ab528c7d41559831652a1437d358 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:53.602221 18983 consensus_queue.cc:260] T 1050ab528c7d41559831652a1437d358 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:53.603415 18983 ts_tablet_manager.cc:1398] T 1050ab528c7d41559831652a1437d358 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240430 20:11:53.607055 19944 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 130007acc8ba4e709c3eb85fabe75903 (table default.Table_78987875650940c293247bc43db345ab [id=36f478a20a1f4a53a52fc29ade62b7f6]). Expected version 1 got 0
I20240430 20:11:53.607291 19944 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet dac141d1b70c48d785caddd7182811ee (table default.Table_fc0a4540f98b4eea95099b8ab46ea235 [id=f3c53ccffe64410495cc8c379a05cfae]). Expected version 1 got 0
I20240430 20:11:53.607492 19944 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 22fef8bb6b034db098c3372ae992f7a7 (table default.Table_4440df38610340758047115845b42b95 [id=7d186ca20c094cc19aef792fe1c094e9]). Expected version 1 got 0
I20240430 20:11:53.607677 18983 tablet_bootstrap.cc:492] T 086b3dfa84c045a18c8c4f177dce606a P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:53.607671 19944 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 330453ddeea6456e8ef93fa92608b0d3 (table default.Table_c1f36f7df5ae4e7c8eb8b4e99b89748b [id=a50abf28790d4284a76ac6ee22c02606]). Expected version 1 got 0
I20240430 20:11:53.607882 19944 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet acd23f0a63d5448aaa7ed78422e67bad (table default.Table_972f13be4dca4cd19b9a04534dc59f51 [id=69f37db4de634a3185613e40013e0ed7]). Expected version 1 got 0
I20240430 20:11:53.608084 19944 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 69f88532952a4591bf1577541c3430e6 (table default.Table_1177828cad1f4a24bd397619dad96170 [id=a95fed1bc3fa4865bf9a35dfc51b659c]). Expected version 2 got 0
I20240430 20:11:53.608438 19944 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 1050ab528c7d41559831652a1437d358 (table default.Table_acddc4cbf6334546b5fb3e4752fef2a5 [id=274cb15fd751410e83327ac4e8dc4629]). Expected version 2 got 0
I20240430 20:11:53.608647 19944 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 086b3dfa84c045a18c8c4f177dce606a (table default.Table_df19cb8a6f4d46699350ec5b69c14e5a [id=197923633fc54a4582207942b51530e2]). Expected version 2 got 0
I20240430 20:11:53.609164 19944 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet e041756c962b4a9f97c2cb314e7d8c05 (table default.Table_de88cafd92b149f9b545369bff71033c [id=c4bf4806f3d84579b16a3038d8d982c3]). Expected version 1 got 0
I20240430 20:11:53.609436 19944 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 8e9711d0a382424f9f758bfa883866a0 (table default.Table_f1dc481c6d8c43f589fe2b1892a4cb10 [id=db696ce66005401fa73746e55c071f44]). Expected version 1 got 0
I20240430 20:11:53.610011 18983 tablet_bootstrap.cc:654] T 086b3dfa84c045a18c8c4f177dce606a P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:53.610633 18988 ts_tablet_manager.cc:1398] T 330453ddeea6456e8ef93fa92608b0d3 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.015s user 0.001s sys 0.002s
I20240430 20:11:53.612973 18985 tablet_bootstrap.cc:492] T a663030d37614053a7c0369841403422 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:53.613135 18985 ts_tablet_manager.cc:1367] T a663030d37614053a7c0369841403422 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.036s user 0.003s sys 0.000s
I20240430 20:11:53.613878 18985 raft_consensus.cc:348] T a663030d37614053a7c0369841403422 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:53.614099 18985 raft_consensus.cc:374] T a663030d37614053a7c0369841403422 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:53.614153 18985 raft_consensus.cc:729] T a663030d37614053a7c0369841403422 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:53.614367 18985 consensus_queue.cc:260] T a663030d37614053a7c0369841403422 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:53.614984 18985 ts_tablet_manager.cc:1398] T a663030d37614053a7c0369841403422 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240430 20:11:53.615760 18985 tablet_bootstrap.cc:492] T 3191aa54a9644783a2403991c3be27c8 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:53.616047 18988 tablet_bootstrap.cc:492] T 086b3dfa84c045a18c8c4f177dce606a P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:53.617358 18985 tablet_bootstrap.cc:654] T 3191aa54a9644783a2403991c3be27c8 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:53.623723 18988 tablet_bootstrap.cc:654] T 086b3dfa84c045a18c8c4f177dce606a P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:53.629829 18988 tablet_bootstrap.cc:492] T 086b3dfa84c045a18c8c4f177dce606a P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:53.630110 19944 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet f5eaf2413e1d4e69b56fae61f768f0ce (table default.Table_1280cfe32d5b41abb4d3eb1bd70641bf [id=8489b872074a42b6abb713e420968141]). Expected version 1 got 0
I20240430 20:11:53.630484 19944 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 5df364628561475391c9a5e10ff5a05f (table default.Table_3c8f9f7b22f64fb1bdc3f1550987233a [id=7fd6f8576f4c4cbc8ffcdb4ab95e15dc]). Expected version 1 got 0
I20240430 20:11:53.630734 19944 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 821c55edb3414d72b1489eea97d3f73a (table default.Table_4d68c38d9ba7452a829f8318b2b267b8 [id=d06e3e9045b1429aaeebd1fd67d841de]). Expected version 1 got 0
I20240430 20:11:53.631083 19944 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 47bf64ecc41b445da4ff697e141243dd (table default.Table_f1ad29d5986d4b799cfc67afb2b0dfdf [id=838fc8b61a864dea8877a6f144e4cb4f]). Expected version 2 got 0
I20240430 20:11:53.631574 19944 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 99203cc522de424184c6f97453f166fd (table default.Table_cb2a4b9ceaf4431facfff2c9bcf94c5d [id=3f8690a437ae49a3beb009fa7899d5d0]). Expected version 1 got 0
I20240430 20:11:53.631636 18988 ts_tablet_manager.cc:1367] T 086b3dfa84c045a18c8c4f177dce606a P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.000s
I20240430 20:11:53.632632 19944 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet a482f4622f774d60b4efd6827b8be7d7 (table default.Table_e686cdfcbcc046749fa5b51578a64794 [id=c3dc9f05316d46d6a3ef4d03f40877ec]). Expected version 1 got 0
I20240430 20:11:53.632891 19944 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 37b2ea4be7494f0985808ade788dbe2b (table default.Table_5f6e0fce999b46de927f9271e5de5071 [id=0baad83a8c0f4aeebc7ddad068625a69]). Expected version 1 got 0
I20240430 20:11:53.637420 18985 tablet_bootstrap.cc:492] T 3191aa54a9644783a2403991c3be27c8 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:53.637590 18985 ts_tablet_manager.cc:1367] T 3191aa54a9644783a2403991c3be27c8 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.002s
I20240430 20:11:53.638339 18985 raft_consensus.cc:348] T 3191aa54a9644783a2403991c3be27c8 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:53.638546 18985 raft_consensus.cc:374] T 3191aa54a9644783a2403991c3be27c8 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:53.638621 18985 raft_consensus.cc:729] T 3191aa54a9644783a2403991c3be27c8 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:53.633867 19944 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet aaef85eea87f4443943fd3692bb2a38f (table default.Table_3da5b4c8668c470483ae2a0c7773eaaf [id=e292d818ee3c45df90522f08eb2d6227]). Expected version 1 got 0
I20240430 20:11:53.638865 19944 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet a59ef6c545dd41a28265a4c1b8ce10e0 (table default.Table_07e40af8f1c546048975c8edf88170d7 [id=86df5d161d8641c89070e7eed9a39360]). Expected version 2 got 0
I20240430 20:11:53.638830 18985 consensus_queue.cc:260] T 3191aa54a9644783a2403991c3be27c8 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:53.639389 18985 ts_tablet_manager.cc:1398] T 3191aa54a9644783a2403991c3be27c8 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240430 20:11:53.640412 18988 raft_consensus.cc:348] T 086b3dfa84c045a18c8c4f177dce606a P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:53.641499 18983 tablet_bootstrap.cc:492] T 086b3dfa84c045a18c8c4f177dce606a P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:53.641706 18983 ts_tablet_manager.cc:1367] T 086b3dfa84c045a18c8c4f177dce606a P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.034s user 0.003s sys 0.002s
I20240430 20:11:53.642352 18985 tablet_bootstrap.cc:492] T 4dac8b3838954d839094021f1c0503da P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:53.639945 19944 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 2e79df8744e74b5991a79ce978ff0202 (table default.Table_276bbdc7cfe542d1a2936485a2aad1ec [id=f146a15e6e8a4acbb4c734c1c863da57]). Expected version 1 got 0
I20240430 20:11:53.642836 19944 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 7a0176f8a74c47efbbe3d3343b3dc896 (table default.Table_144e5402c1864d818436015b26648024 [id=73e5c9f0470d47849109819c063e1ce4]). Expected version 2 got 0
I20240430 20:11:53.644431 19944 catalog_manager.cc:5484] T 828ca913e2aa4102a6277bdbc6686ef5 P 5e1c6397f09747f091534faf93542710 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e1c6397f09747f091534faf93542710 (127.17.177.2). New cstate: current_term: 1 leader_uuid: "5e1c6397f09747f091534faf93542710" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } health_report { overall_health: UNKNOWN } } }
I20240430 20:11:53.645156 19944 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 94fbd4a57a1e438891fcb456f6b40be3 (table default.Table_25ddc3adef6b4ccf8b9c00cc6772031c [id=4fa623ffdcf347639150a355806038af]). Expected version 1 got 0
I20240430 20:11:53.645721 18985 tablet_bootstrap.cc:654] T 4dac8b3838954d839094021f1c0503da P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:53.646618 19944 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 724e51ff05ec4baebc2bb55e05c8eddc (table default.Table_19df33614a6c4f94b6419a5b441e3ef8 [id=4dce8a767f0541e1beb5464b2ed95d06]). Expected version 2 got 1
I20240430 20:11:53.647516 18988 raft_consensus.cc:374] T 086b3dfa84c045a18c8c4f177dce606a P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:53.647607 18988 raft_consensus.cc:729] T 086b3dfa84c045a18c8c4f177dce606a P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:53.647825 18988 consensus_queue.cc:260] T 086b3dfa84c045a18c8c4f177dce606a P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:53.648895 18988 ts_tablet_manager.cc:1398] T 086b3dfa84c045a18c8c4f177dce606a P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.017s user 0.001s sys 0.001s
I20240430 20:11:53.649215 18988 tablet_bootstrap.cc:492] T 1050ab528c7d41559831652a1437d358 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:53.649335 18983 raft_consensus.cc:348] T 086b3dfa84c045a18c8c4f177dce606a P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:53.649607 18983 raft_consensus.cc:374] T 086b3dfa84c045a18c8c4f177dce606a P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:53.649714 18983 raft_consensus.cc:729] T 086b3dfa84c045a18c8c4f177dce606a P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:53.650118 18983 consensus_queue.cc:260] T 086b3dfa84c045a18c8c4f177dce606a P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:53.651415 21365 raft_consensus.cc:482] T 1050ab528c7d41559831652a1437d358 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:53.651619 21365 raft_consensus.cc:504] T 1050ab528c7d41559831652a1437d358 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:53.652316 21365 leader_election.cc:288] T 1050ab528c7d41559831652a1437d358 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:53.653100 18750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1050ab528c7d41559831652a1437d358" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
W20240430 20:11:53.653640 18557 leader_election.cc:341] T 1050ab528c7d41559831652a1437d358 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:53.650955 18983 ts_tablet_manager.cc:1398] T 086b3dfa84c045a18c8c4f177dce606a P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20240430 20:11:53.654381 19944 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet ca64de16d40f4a44b8b411f5526ba11a (table default.Table_1cf914acf7fe4e98b8dbc297b6372f21 [id=f4ba9536e9c141deb68f4f62b7e8fa70]). Expected version 1 got 0
I20240430 20:11:53.655124 18473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1050ab528c7d41559831652a1437d358" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:53.656852 18988 tablet_bootstrap.cc:654] T 1050ab528c7d41559831652a1437d358 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:53.660668 18985 tablet_bootstrap.cc:492] T 4dac8b3838954d839094021f1c0503da P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:53.660933 18985 ts_tablet_manager.cc:1367] T 4dac8b3838954d839094021f1c0503da P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.002s
I20240430 20:11:53.660843 18983 tablet_bootstrap.cc:492] T e5b85a066cdc46d48ccf8201e5423c29 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:53.661758 18985 raft_consensus.cc:348] T 4dac8b3838954d839094021f1c0503da P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:53.662073 18985 raft_consensus.cc:374] T 4dac8b3838954d839094021f1c0503da P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:53.662194 18985 raft_consensus.cc:729] T 4dac8b3838954d839094021f1c0503da P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:53.662487 18985 consensus_queue.cc:260] T 4dac8b3838954d839094021f1c0503da P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:53.663342 18985 ts_tablet_manager.cc:1398] T 4dac8b3838954d839094021f1c0503da P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240430 20:11:53.663694 18985 tablet_bootstrap.cc:492] T 330453ddeea6456e8ef93fa92608b0d3 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:53.667544 18985 tablet_bootstrap.cc:654] T 330453ddeea6456e8ef93fa92608b0d3 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
W20240430 20:11:53.668653 18558 leader_election.cc:341] T 1050ab528c7d41559831652a1437d358 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:53.668792 18558 leader_election.cc:302] T 1050ab528c7d41559831652a1437d358 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:53.669684 21366 raft_consensus.cc:2726] T 1050ab528c7d41559831652a1437d358 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:53.676252 19944 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet fbdeb4714b8c4b93ab72822c43329aae (table default.Table_c12ca6938d424fbfa2e1df5cdb00fdbc [id=0fb93f36bc17408ea56bc4cafcfd8580]). Expected version 1 got 0
I20240430 20:11:53.678260 19944 catalog_manager.cc:5484] T 35fdb052135541cfb42a8aeb03b64dc8 P 5e1c6397f09747f091534faf93542710 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e1c6397f09747f091534faf93542710 (127.17.177.2). New cstate: current_term: 1 leader_uuid: "5e1c6397f09747f091534faf93542710" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } health_report { overall_health: UNKNOWN } } }
I20240430 20:11:53.680837 19944 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 772eba5f24c549c791cad75565858285 (table default.Table_5fc77d5f60ae44f4bb4ca9f7f183672c [id=6ca223ca6354449c959d85ed2ac6a9c8]). Expected version 1 got 0
I20240430 20:11:53.682657 18983 tablet_bootstrap.cc:654] T e5b85a066cdc46d48ccf8201e5423c29 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:53.683600 18985 tablet_bootstrap.cc:492] T 330453ddeea6456e8ef93fa92608b0d3 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:53.683779 18985 ts_tablet_manager.cc:1367] T 330453ddeea6456e8ef93fa92608b0d3 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.004s
I20240430 20:11:53.684885 18985 raft_consensus.cc:348] T 330453ddeea6456e8ef93fa92608b0d3 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:53.685580 18985 raft_consensus.cc:374] T 330453ddeea6456e8ef93fa92608b0d3 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:53.685791 18985 raft_consensus.cc:729] T 330453ddeea6456e8ef93fa92608b0d3 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:53.686879 19944 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet c6e3d09402574f1e8fe9e4160c70a3ee (table default.Table_656db97355f64d7c8aa29b1352dfe9c8 [id=450f5d1c7e084a1bb88d035c0ccfcf82]). Expected version 1 got 0
I20240430 20:11:53.687835 21198 raft_consensus.cc:482] T 2797eed593fb4b06a3c4d6a28e8075a2 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:53.687848 19944 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 78ba48dadef548708153878810cc46f8 (table default.Table_b9983e101a6547649eb44ee04b5e1147 [id=29769be2b9ec4ecba9de5c0a65692f85]). Expected version 1 got 0
I20240430 20:11:53.688071 19944 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 2cd41b407d6b444d844c4c7c0ebf82e5 (table default.Table_6f8e0eff8f5a484dbd6a9c3fdb5d03f6 [id=3813b5e9778c4bd683a06598cef0d86f]). Expected version 1 got 0
I20240430 20:11:53.687992 21198 raft_consensus.cc:504] T 2797eed593fb4b06a3c4d6a28e8075a2 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:53.688473 19944 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 488d0e609b224674932f890939664df2 (table default.Table_58f35a2120874472978ca0e5f816e86c [id=ff18001c6d814afc8c1768d0c84a6bac]). Expected version 1 got 0
I20240430 20:11:53.688560 18985 consensus_queue.cc:260] T 330453ddeea6456e8ef93fa92608b0d3 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:53.688676 21198 leader_election.cc:288] T 2797eed593fb4b06a3c4d6a28e8075a2 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:53.689293 18473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2797eed593fb4b06a3c4d6a28e8075a2" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:53.689445 18623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2797eed593fb4b06a3c4d6a28e8075a2" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:53.689574 18473 raft_consensus.cc:2454] T 2797eed593fb4b06a3c4d6a28e8075a2 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:53.689639 18623 raft_consensus.cc:2454] T 2797eed593fb4b06a3c4d6a28e8075a2 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:53.689133 18985 ts_tablet_manager.cc:1398] T 330453ddeea6456e8ef93fa92608b0d3 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240430 20:11:53.689998 18699 leader_election.cc:302] T 2797eed593fb4b06a3c4d6a28e8075a2 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters:
I20240430 20:11:53.690268 21198 raft_consensus.cc:2781] T 2797eed593fb4b06a3c4d6a28e8075a2 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:53.690299 18985 tablet_bootstrap.cc:492] T 086b3dfa84c045a18c8c4f177dce606a P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:53.690364 21198 raft_consensus.cc:482] T 2797eed593fb4b06a3c4d6a28e8075a2 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:53.690433 21198 raft_consensus.cc:3037] T 2797eed593fb4b06a3c4d6a28e8075a2 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:53.692538 18985 tablet_bootstrap.cc:654] T 086b3dfa84c045a18c8c4f177dce606a P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:53.693181 21198 raft_consensus.cc:504] T 2797eed593fb4b06a3c4d6a28e8075a2 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:53.694700 21198 leader_election.cc:288] T 2797eed593fb4b06a3c4d6a28e8075a2 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Requested vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:53.695304 18473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2797eed593fb4b06a3c4d6a28e8075a2" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:53.695492 18473 raft_consensus.cc:3037] T 2797eed593fb4b06a3c4d6a28e8075a2 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:53.696484 18623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2797eed593fb4b06a3c4d6a28e8075a2" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:53.696691 18623 raft_consensus.cc:3037] T 2797eed593fb4b06a3c4d6a28e8075a2 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:53.697702 18988 tablet_bootstrap.cc:492] T 1050ab528c7d41559831652a1437d358 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:53.697841 18988 ts_tablet_manager.cc:1367] T 1050ab528c7d41559831652a1437d358 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.049s user 0.006s sys 0.001s
I20240430 20:11:53.698026 18473 raft_consensus.cc:2454] T 2797eed593fb4b06a3c4d6a28e8075a2 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:53.698488 18699 leader_election.cc:302] T 2797eed593fb4b06a3c4d6a28e8075a2 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters:
I20240430 20:11:53.699167 18988 raft_consensus.cc:348] T 1050ab528c7d41559831652a1437d358 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:53.699309 21198 raft_consensus.cc:2781] T 2797eed593fb4b06a3c4d6a28e8075a2 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:53.699368 18988 raft_consensus.cc:374] T 1050ab528c7d41559831652a1437d358 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:53.699481 18988 raft_consensus.cc:729] T 1050ab528c7d41559831652a1437d358 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:53.699733 18988 consensus_queue.cc:260] T 1050ab528c7d41559831652a1437d358 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:53.700286 18988 ts_tablet_manager.cc:1398] T 1050ab528c7d41559831652a1437d358 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240430 20:11:53.700582 18988 tablet_bootstrap.cc:492] T a0d3c4ba64024bccad95ecc909d03ae0 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:53.701478 18623 raft_consensus.cc:2454] T 2797eed593fb4b06a3c4d6a28e8075a2 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:53.702240 18988 tablet_bootstrap.cc:654] T a0d3c4ba64024bccad95ecc909d03ae0 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:53.690498 19944 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 17ddcaf971154a52b606bbf29a539bf0 (table default.Table_f2f38f5d69e34eaa9e3dbf24cfd1cdd6 [id=872c2faa4a464edd98a51df2e1b3f817]). Expected version 1 got 0
I20240430 20:11:53.703294 21198 raft_consensus.cc:686] T 2797eed593fb4b06a3c4d6a28e8075a2 P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Becoming Leader. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Running, Role: LEADER
I20240430 20:11:53.703640 19944 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 2da900b6bbfa4fa99fe8494fd2e68da8 (table default.Table_19a91a7f17ff456ebb145e8dd89c0940 [id=67283dd3995a44c6a7a7bc8722e4f955]). Expected version 1 got 0
I20240430 20:11:53.703589 21198 consensus_queue.cc:237] T 2797eed593fb4b06a3c4d6a28e8075a2 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:53.706388 19944 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 0368d37dd9e2452982fb81f539c6805b (table default.Table_404b186c544e4b30bb36cb77d1a14120 [id=4330626e342a48699ea939add5c91b11]). Expected version 1 got 0
I20240430 20:11:53.706748 19944 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 7ce09fcbe1ac457a9c05efaf000c3263 (table default.Table_edc4c27ed846454288f73f6a0d88cb69 [id=78fd3f516083489ab3b42d90dcef7e4d]). Expected version 1 got 0
I20240430 20:11:53.707068 19944 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 331c031d44854d679a27d10d9151d0ea (table default.Table_763f0800e8bb4fa69976886416250f42 [id=61c8a5b99ef34a0baeeb343fef7cfd05]). Expected version 1 got 0
I20240430 20:11:53.708304 19944 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 677481aee5fa4eb4b189322d7a3f8f2e (table default.Table_1ce204e15fce4c469816cf4e7f80cac7 [id=d268c31d82df4871bc1a78aea5871b33]). Expected version 1 got 0
I20240430 20:11:53.709111 19944 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 5b26319488644d6a8b6be60460a5e7af (table default.Table_69892a8264a4477db48ee6248758e046 [id=5a1c4a7be2fc465fac89a50854eb3322]). Expected version 3 got 0
I20240430 20:11:53.709393 19944 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet f1073fc792954c5795a05aa7d0ec24df (table default.Table_ba8b218e936b4bfcb073f8182f474cca [id=aedf11fda8914a3b843a57f8a7b69a53]). Expected version 1 got 0
I20240430 20:11:53.709604 19944 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 66dd767bd3f8494f94061fb89e5e5680 (table default.Table_b064d5659f7243f5b0ec167fb6c3f404 [id=5227206f44d04fa89b4842b899f33afb]). Expected version 2 got 0
I20240430 20:11:53.709810 19944 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 81787eccca6c42c1b421377a3bf9ebe6 (table default.Table_0b4ca4cd0af148399dfa21f2d32114e3 [id=7f602584cef9486eb7d9b4389be3f43e]). Expected version 3 got 2
I20240430 20:11:53.710809 19941 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: Sending DeleteTablet for 3 replicas of tablet a692b253db914531b8fc6ffaf3e350b5
I20240430 20:11:53.711563 19941 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:50746 (request call id 58) took 3124 ms. Trace:
I20240430 20:11:53.711702 19941 rpcz_store.cc:276] 0430 20:11:50.587184 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:51.572648 (+985464us) service_pool.cc:225] Handling call
0430 20:11:51.929219 (+356571us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0430 20:11:51.929368 (+ 149us) catalog_manager.cc:2749] Modifying in-memory table state
0430 20:11:51.929434 (+ 66us) catalog_manager.cc:2758] Locking tablets
0430 20:11:52.196949 (+267515us) catalog_manager.cc:2772] Removing table and tablets from system table
0430 20:11:52.210074 (+ 13125us) catalog_manager.cc:2790] Removing table from by-name map
0430 20:11:52.354724 (+144650us) catalog_manager.cc:2806] Committing in-memory state
0430 20:11:53.710719 (+1355995us) catalog_manager.cc:2810] Aborting table tasks
0430 20:11:53.711513 (+ 794us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:52.200298 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:52.200403 (+ 105us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:52.200409 (+ 6us) write_op.cc:427] Acquired schema lock
0430 20:11:52.200412 (+ 3us) tablet.cc:660] Decoding operations
0430 20:11:52.200543 (+ 131us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:52.200555 (+ 12us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:52.200559 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
0430 20:11:52.200667 (+ 108us) tablet.cc:699] Row locks acquired
0430 20:11:52.200670 (+ 3us) write_op.cc:260] PREPARE: finished
0430 20:11:52.200705 (+ 35us) write_op.cc:270] Start()
0430 20:11:52.200737 (+ 32us) write_op.cc:276] Timestamp: P: 1714507912200681 usec, L: 0
0430 20:11:52.200741 (+ 4us) op_driver.cc:331] REPLICATION: starting
0430 20:11:52.200925 (+ 184us) log.cc:842] Serialized 728 byte log entry
0430 20:11:52.208833 (+ 7908us) op_driver.cc:444] REPLICATION: finished
0430 20:11:52.209469 (+ 636us) write_op.cc:301] APPLY: starting
0430 20:11:52.209479 (+ 10us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:52.209514 (+ 35us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:52.209548 (+ 34us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:52.209628 (+ 80us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:52.209674 (+ 46us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0430 20:11:52.209682 (+ 8us) write_op.cc:312] APPLY: finished
0430 20:11:52.209732 (+ 50us) log.cc:842] Serialized 29 byte log entry
0430 20:11:52.209769 (+ 37us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:52.209782 (+ 13us) write_op.cc:446] Released schema lock
0430 20:11:52.209788 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:52.209833 (+ 45us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:52.209837 (+ 4us) write_op.cc:446] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":605,"apply.run_cpu_time_us":408,"apply.run_wall_time_us":408,"num_ops":2,"prepare.queue_time_us":2440,"prepare.run_cpu_time_us":739,"prepare.run_wall_time_us":2058,"raft.queue_time_us":1185,"raft.run_cpu_time_us":4,"raft.run_wall_time_us":4,"replication_time_us":8077}]]}
I20240430 20:11:53.718921 18453 tablet_service.cc:1514] Processing DeleteTablet for tablet a692b253db914531b8fc6ffaf3e350b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:51 UTC) from {username='slave'} at 127.0.0.1:41450
I20240430 20:11:53.720207 18600 tablet_service.cc:1514] Processing DeleteTablet for tablet a692b253db914531b8fc6ffaf3e350b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:51 UTC) from {username='slave'} at 127.0.0.1:45934
I20240430 20:11:53.724210 20630 raft_consensus.cc:482] T 1fe7c6f1ab54499facd4d01573d3e7bc P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:53.724387 20630 raft_consensus.cc:504] T 1fe7c6f1ab54499facd4d01573d3e7bc P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:53.725236 20630 leader_election.cc:288] T 1fe7c6f1ab54499facd4d01573d3e7bc P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:53.726101 18760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fe7c6f1ab54499facd4d01573d3e7bc" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:53.726341 18760 raft_consensus.cc:2454] T 1fe7c6f1ab54499facd4d01573d3e7bc P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 0.
I20240430 20:11:53.726871 18417 leader_election.cc:302] T 1fe7c6f1ab54499facd4d01573d3e7bc P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters:
I20240430 20:11:53.727136 20630 raft_consensus.cc:2781] T 1fe7c6f1ab54499facd4d01573d3e7bc P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:53.727202 20630 raft_consensus.cc:482] T 1fe7c6f1ab54499facd4d01573d3e7bc P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:53.727257 20630 raft_consensus.cc:3037] T 1fe7c6f1ab54499facd4d01573d3e7bc P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:53.729892 20630 raft_consensus.cc:504] T 1fe7c6f1ab54499facd4d01573d3e7bc P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:53.730698 18725 tablet_service.cc:1514] Processing DeleteTablet for tablet a692b253db914531b8fc6ffaf3e350b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:51 UTC) from {username='slave'} at 127.0.0.1:56540
I20240430 20:11:53.731508 21306 raft_consensus.cc:482] T 3191aa54a9644783a2403991c3be27c8 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:53.731637 21306 raft_consensus.cc:504] T 3191aa54a9644783a2403991c3be27c8 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:53.732300 21306 leader_election.cc:288] T 3191aa54a9644783a2403991c3be27c8 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:53.733314 18616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fe7c6f1ab54499facd4d01573d3e7bc" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:53.733546 18616 raft_consensus.cc:2454] T 1fe7c6f1ab54499facd4d01573d3e7bc P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 0.
I20240430 20:11:53.734558 21049 consensus_queue.cc:1035] T aa9998f4e17147aa8f2b35c01980463a P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:53.734836 21365 raft_consensus.cc:482] T 436579f473844c6ca1038b496201a550 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:53.734953 21365 raft_consensus.cc:504] T 436579f473844c6ca1038b496201a550 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:53.735569 18616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3191aa54a9644783a2403991c3be27c8" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:53.735706 18724 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f4e860484784aa9a2d28af2e9fb611c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:39 UTC) from {username='slave'} at 127.0.0.1:56540
I20240430 20:11:53.735745 18616 raft_consensus.cc:2454] T 3191aa54a9644783a2403991c3be27c8 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:53.735845 18735 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f0b341a31224bf1be525b84b9500304 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:44 UTC) from {username='slave'} at 127.0.0.1:56540
I20240430 20:11:53.735904 18737 tablet_service.cc:1514] Processing DeleteTablet for tablet dad16ae02b8b4e29a724d8eec67bb4b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:42 UTC) from {username='slave'} at 127.0.0.1:56540
I20240430 20:11:53.736030 18724 tablet_service.cc:1514] Processing DeleteTablet for tablet 6db55bb510b24b0cae897936211e15f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:34 UTC) from {username='slave'} at 127.0.0.1:56540
I20240430 20:11:53.736037 18735 tablet_service.cc:1514] Processing DeleteTablet for tablet 90a694bd5db649afa50f3c4e752939c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:39 UTC) from {username='slave'} at 127.0.0.1:56540
I20240430 20:11:53.736191 18737 tablet_service.cc:1514] Processing DeleteTablet for tablet c1017a50d1d646c486c0b9a75cb88ce4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:43 UTC) from {username='slave'} at 127.0.0.1:56540
I20240430 20:11:53.736362 18737 tablet_service.cc:1514] Processing DeleteTablet for tablet 898b2c91adf34545808df9c786e218c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:40 UTC) from {username='slave'} at 127.0.0.1:56540
I20240430 20:11:53.736527 18737 tablet_service.cc:1514] Processing DeleteTablet for tablet 558e71ec370d46ca8fbdcec5b57eb2ec with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:37 UTC) from {username='slave'} at 127.0.0.1:56540
I20240430 20:11:53.736940 18737 tablet_service.cc:1514] Processing DeleteTablet for tablet f7d3dea34cf54fe2b949d1b7ca68577a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:40 UTC) from {username='slave'} at 127.0.0.1:56540
I20240430 20:11:53.737107 18726 tablet_service.cc:1514] Processing DeleteTablet for tablet df92eb85261d47d88cbf16d29d9ad30e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:45 UTC) from {username='slave'} at 127.0.0.1:56540
I20240430 20:11:53.737571 18744 tablet_service.cc:1514] Processing DeleteTablet for tablet a445fdfde01c4371a5e6b3daea5b00ab with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:44 UTC) from {username='slave'} at 127.0.0.1:56540
I20240430 20:11:53.737771 18730 tablet_service.cc:1514] Processing DeleteTablet for tablet 8901327b3efb4201adce61a9266142ea with delete_type TABLET_DATA_DELETED (replaced by tablet b1277899f0c040cea7dc2b12f3fd68de) from {username='slave'} at 127.0.0.1:56540
I20240430 20:11:53.737879 18733 tablet_service.cc:1514] Processing DeleteTablet for tablet 47e98935eb21449abd3d4d24327778d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:45 UTC) from {username='slave'} at 127.0.0.1:56540
I20240430 20:11:53.738025 18726 tablet_service.cc:1514] Processing DeleteTablet for tablet 06d28b6f48a34b17b0d7fb3102df2f4c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:33 UTC) from {username='slave'} at 127.0.0.1:56540
I20240430 20:11:53.738245 18737 tablet_service.cc:1514] Processing DeleteTablet for tablet 1701a15b115741aea13f8327e3d26b34 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:50 UTC) from {username='slave'} at 127.0.0.1:56540
I20240430 20:11:53.738966 18696 leader_election.cc:302] T 3191aa54a9644783a2403991c3be27c8 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:53.739334 21306 raft_consensus.cc:2781] T 3191aa54a9644783a2403991c3be27c8 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:53.739419 21306 raft_consensus.cc:482] T 3191aa54a9644783a2403991c3be27c8 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:53.739492 21306 raft_consensus.cc:3037] T 3191aa54a9644783a2403991c3be27c8 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:53.739785 18477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3191aa54a9644783a2403991c3be27c8" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:53.739977 18477 raft_consensus.cc:2454] T 3191aa54a9644783a2403991c3be27c8 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:53.740252 18724 tablet_service.cc:1514] Processing DeleteTablet for tablet 283e5e73cbc04daba0de19f4541f9fd0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:40 UTC) from {username='slave'} at 127.0.0.1:56540
I20240430 20:11:53.741079 18760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fe7c6f1ab54499facd4d01573d3e7bc" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:53.741259 18760 raft_consensus.cc:3037] T 1fe7c6f1ab54499facd4d01573d3e7bc P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:53.735589 21365 leader_election.cc:288] T 436579f473844c6ca1038b496201a550 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:53.743155 18473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "436579f473844c6ca1038b496201a550" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:53.743412 18473 raft_consensus.cc:2454] T 436579f473844c6ca1038b496201a550 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 0.
I20240430 20:11:53.743811 18558 leader_election.cc:302] T 436579f473844c6ca1038b496201a550 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:53.743816 18750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "436579f473844c6ca1038b496201a550" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:53.744012 18750 raft_consensus.cc:2454] T 436579f473844c6ca1038b496201a550 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 0.
I20240430 20:11:53.744186 21365 raft_consensus.cc:2781] T 436579f473844c6ca1038b496201a550 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:53.744273 21365 raft_consensus.cc:482] T 436579f473844c6ca1038b496201a550 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:53.744329 21365 raft_consensus.cc:3037] T 436579f473844c6ca1038b496201a550 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:53.744655 18760 raft_consensus.cc:2454] T 1fe7c6f1ab54499facd4d01573d3e7bc P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 1.
I20240430 20:11:53.746574 18417 leader_election.cc:302] T 1fe7c6f1ab54499facd4d01573d3e7bc P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters:
I20240430 20:11:53.746923 21054 raft_consensus.cc:2781] T 1fe7c6f1ab54499facd4d01573d3e7bc P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:53.747049 21054 raft_consensus.cc:686] T 1fe7c6f1ab54499facd4d01573d3e7bc P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Becoming Leader. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Running, Role: LEADER
I20240430 20:11:53.747378 21365 raft_consensus.cc:504] T 436579f473844c6ca1038b496201a550 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:53.747339 21054 consensus_queue.cc:237] T 1fe7c6f1ab54499facd4d01573d3e7bc P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:53.748466 18473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "436579f473844c6ca1038b496201a550" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:53.748629 18473 raft_consensus.cc:3037] T 436579f473844c6ca1038b496201a550 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:53.749394 21365 leader_election.cc:288] T 436579f473844c6ca1038b496201a550 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:53.752223 18750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "436579f473844c6ca1038b496201a550" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:53.752351 21276 consensus_queue.cc:1035] T aa9998f4e17147aa8f2b35c01980463a P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:53.752427 18750 raft_consensus.cc:3037] T 436579f473844c6ca1038b496201a550 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:53.755761 18750 raft_consensus.cc:2454] T 436579f473844c6ca1038b496201a550 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
I20240430 20:11:53.756462 18557 leader_election.cc:302] T 436579f473844c6ca1038b496201a550 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:53.718127 19946 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: Sending DeleteTablet for 3 replicas of tablet 3e5225f1f5f544ecbf8dfa470af3daca
I20240430 20:11:53.757004 21366 raft_consensus.cc:2781] T 436579f473844c6ca1038b496201a550 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:53.757117 21366 raft_consensus.cc:686] T 436579f473844c6ca1038b496201a550 P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Becoming Leader. State: Replica: 5e1c6397f09747f091534faf93542710, State: Running, Role: LEADER
I20240430 20:11:53.757431 19946 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:50746 (request call id 59) took 3164 ms. Trace:
I20240430 20:11:53.757380 21366 consensus_queue.cc:237] T 436579f473844c6ca1038b496201a550 P 5e1c6397f09747f091534faf93542710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:53.757544 19946 rpcz_store.cc:276] 0430 20:11:50.593292 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:51.587078 (+993786us) service_pool.cc:225] Handling call
0430 20:11:51.921770 (+334692us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0430 20:11:51.921920 (+ 150us) catalog_manager.cc:2749] Modifying in-memory table state
0430 20:11:51.921998 (+ 78us) catalog_manager.cc:2758] Locking tablets
0430 20:11:52.195760 (+273762us) catalog_manager.cc:2772] Removing table and tablets from system table
0430 20:11:52.210524 (+ 14764us) catalog_manager.cc:2790] Removing table from by-name map
0430 20:11:52.355643 (+145119us) catalog_manager.cc:2806] Committing in-memory state
0430 20:11:53.718032 (+1362389us) catalog_manager.cc:2810] Aborting table tasks
0430 20:11:53.757391 (+ 39359us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:52.199154 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:52.199316 (+ 162us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:52.199327 (+ 11us) write_op.cc:427] Acquired schema lock
0430 20:11:52.199331 (+ 4us) tablet.cc:660] Decoding operations
0430 20:11:52.199549 (+ 218us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:52.199569 (+ 20us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:52.199577 (+ 8us) tablet.cc:683] Acquiring locks for 2 operations
0430 20:11:52.199757 (+ 180us) tablet.cc:699] Row locks acquired
0430 20:11:52.199761 (+ 4us) write_op.cc:260] PREPARE: finished
0430 20:11:52.199832 (+ 71us) write_op.cc:270] Start()
0430 20:11:52.199878 (+ 46us) write_op.cc:276] Timestamp: P: 1714507912199799 usec, L: 0
0430 20:11:52.199885 (+ 7us) op_driver.cc:331] REPLICATION: starting
0430 20:11:52.200158 (+ 273us) log.cc:842] Serialized 793 byte log entry
0430 20:11:52.208772 (+ 8614us) op_driver.cc:444] REPLICATION: finished
0430 20:11:52.208919 (+ 147us) write_op.cc:301] APPLY: starting
0430 20:11:52.208935 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:52.208984 (+ 49us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:52.209011 (+ 27us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:52.209093 (+ 82us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:52.209130 (+ 37us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0430 20:11:52.209139 (+ 9us) write_op.cc:312] APPLY: finished
0430 20:11:52.209233 (+ 94us) log.cc:842] Serialized 29 byte log entry
0430 20:11:52.209295 (+ 62us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:52.209313 (+ 18us) write_op.cc:446] Released schema lock
0430 20:11:52.209320 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:52.209371 (+ 51us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:52.209375 (+ 4us) write_op.cc:446] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":99,"apply.run_cpu_time_us":518,"apply.run_wall_time_us":517,"num_ops":2,"prepare.queue_time_us":2438,"prepare.run_cpu_time_us":1119,"prepare.run_wall_time_us":1121,"raft.queue_time_us":1520,"raft.run_cpu_time_us":343,"raft.run_wall_time_us":343,"replication_time_us":8862}]]}
I20240430 20:11:53.758127 18473 raft_consensus.cc:2454] T 436579f473844c6ca1038b496201a550 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
W20240430 20:11:53.725387 19930 catalog_manager.cc:5676] Failed to send AlterTable RPC for tablet 3a03b6997921408cb5a65f4cae223e9c (table default.Table_744b5c96ea014bdb8feae740356a790e [id=83b1ac2e9c3d4a10b02f96ebb17faf44], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 3a03b6997921408cb5a65f4cae223e9c consensus state has no leader
W20240430 20:11:53.759285 19930 catalog_manager.cc:5676] Failed to send AlterTable RPC for tablet eb6c96047b114ebba73db7a371f2da8e (table default.Table_7d5f6f18cbd64c869739872a7dff9be2 [id=e67d619e36c0485f8f22b4a81cb4cb6c], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet eb6c96047b114ebba73db7a371f2da8e consensus state has no leader
W20240430 20:11:53.760242 19930 catalog_manager.cc:5676] Failed to send AlterTable RPC for tablet 266fd23786bc435db5083370f3330ed0 (table default.Table_cf60b652e9dc49e7a948ce66fd1cbffe [id=9734bb8dd05a400ca4244b3829f01dee], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 266fd23786bc435db5083370f3330ed0 consensus state has no leader
I20240430 20:11:53.761317 18456 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e5225f1f5f544ecbf8dfa470af3daca with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:51 UTC) from {username='slave'} at 127.0.0.1:41450
W20240430 20:11:53.761426 19930 catalog_manager.cc:5676] Failed to send AlterTable RPC for tablet 37b2ea4be7494f0985808ade788dbe2b (table default.Table_5f6e0fce999b46de927f9271e5de5071 [id=0baad83a8c0f4aeebc7ddad068625a69], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 37b2ea4be7494f0985808ade788dbe2b consensus state has no leader
I20240430 20:11:53.761884 18595 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e5225f1f5f544ecbf8dfa470af3daca with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:51 UTC) from {username='slave'} at 127.0.0.1:45934
I20240430 20:11:53.762137 18595 tablet_service.cc:1514] Processing DeleteTablet for tablet 2aa2a489116940dea3173cad16d94fa5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:40 UTC) from {username='slave'} at 127.0.0.1:45934
I20240430 20:11:53.762395 18595 tablet_service.cc:1514] Processing DeleteTablet for tablet 6db55bb510b24b0cae897936211e15f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:34 UTC) from {username='slave'} at 127.0.0.1:45934
I20240430 20:11:53.762785 18601 tablet_service.cc:1514] Processing DeleteTablet for tablet c1017a50d1d646c486c0b9a75cb88ce4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:43 UTC) from {username='slave'} at 127.0.0.1:45934
I20240430 20:11:53.763025 18601 tablet_service.cc:1514] Processing DeleteTablet for tablet f940d17b02b541289121463e05b0bb5f with delete_type TABLET_DATA_DELETED (replaced by tablet 53da4b3855344b6d92211af767195dc8) from {username='slave'} at 127.0.0.1:45934
I20240430 20:11:53.763216 18601 tablet_service.cc:1514] Processing DeleteTablet for tablet 04b3dff86d5c48a09d03da58bd48f360 with delete_type TABLET_DATA_DELETED (replaced by tablet 89b774967624498fa2be499878b71124) from {username='slave'} at 127.0.0.1:45934
I20240430 20:11:53.763404 18590 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b7fdc6ec7e046c5af894f9a6eda17b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:40 UTC) from {username='slave'} at 127.0.0.1:45934
I20240430 20:11:53.763500 18601 tablet_service.cc:1514] Processing DeleteTablet for tablet d7c1fc7e81364fb890bbc13d7fb6848b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:40 UTC) from {username='slave'} at 127.0.0.1:45934
I20240430 20:11:53.763599 18588 tablet_service.cc:1514] Processing DeleteTablet for tablet 489dc838744a49eba542009c86688aa5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:43 UTC) from {username='slave'} at 127.0.0.1:45934
I20240430 20:11:53.763751 18591 tablet_service.cc:1514] Processing DeleteTablet for tablet 47e98935eb21449abd3d4d24327778d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:45 UTC) from {username='slave'} at 127.0.0.1:45934
W20240430 20:11:53.731962 19937 catalog_manager.cc:5676] Failed to send AlterTable RPC for tablet 62732af95c6a4c749c5b726f0460dc72 (table default.Table_e3160f5ced1b4718b479401ca4df675c [id=2d44ec94131148048021b2ee6366ba30], current schema version=3): Not found: failed to reset TS proxy: No leader found: Tablet 62732af95c6a4c749c5b726f0460dc72 consensus state has no leader
I20240430 20:11:53.769228 21306 raft_consensus.cc:504] T 3191aa54a9644783a2403991c3be27c8 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
W20240430 20:11:53.770200 19930 catalog_manager.cc:5676] Failed to send AlterTable RPC for tablet 330453ddeea6456e8ef93fa92608b0d3 (table default.Table_c1f36f7df5ae4e7c8eb8b4e99b89748b [id=a50abf28790d4284a76ac6ee22c02606], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 330453ddeea6456e8ef93fa92608b0d3 consensus state has no leader
I20240430 20:11:53.770386 21306 leader_election.cc:288] T 3191aa54a9644783a2403991c3be27c8 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Requested vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
W20240430 20:11:53.770730 19930 catalog_manager.cc:5676] Failed to send AlterTable RPC for tablet 69f88532952a4591bf1577541c3430e6 (table default.Table_1177828cad1f4a24bd397619dad96170 [id=a95fed1bc3fa4865bf9a35dfc51b659c], current schema version=2): Not found: failed to reset TS proxy: No leader found: Tablet 69f88532952a4591bf1577541c3430e6 consensus state has no leader
W20240430 20:11:53.771203 19930 catalog_manager.cc:5676] Failed to send AlterTable RPC for tablet 1050ab528c7d41559831652a1437d358 (table default.Table_acddc4cbf6334546b5fb3e4752fef2a5 [id=274cb15fd751410e83327ac4e8dc4629], current schema version=2): Not found: failed to reset TS proxy: No leader found: Tablet 1050ab528c7d41559831652a1437d358 consensus state has no leader
W20240430 20:11:53.771384 19930 catalog_manager.cc:5676] Failed to send AlterTable RPC for tablet 086b3dfa84c045a18c8c4f177dce606a (table default.Table_df19cb8a6f4d46699350ec5b69c14e5a [id=197923633fc54a4582207942b51530e2], current schema version=2): Not found: failed to reset TS proxy: No leader found: Tablet 086b3dfa84c045a18c8c4f177dce606a consensus state has no leader
I20240430 20:11:53.771673 18477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3191aa54a9644783a2403991c3be27c8" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:53.771957 18477 raft_consensus.cc:3037] T 3191aa54a9644783a2403991c3be27c8 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
W20240430 20:11:53.774335 19930 catalog_manager.cc:5676] Failed to send AlterTable RPC for tablet 821c55edb3414d72b1489eea97d3f73a (table default.Table_4d68c38d9ba7452a829f8318b2b267b8 [id=d06e3e9045b1429aaeebd1fd67d841de], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 821c55edb3414d72b1489eea97d3f73a consensus state has no leader
I20240430 20:11:53.776106 18623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3191aa54a9644783a2403991c3be27c8" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:53.776345 18623 raft_consensus.cc:3037] T 3191aa54a9644783a2403991c3be27c8 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:53.778522 18588 tablet_service.cc:1514] Processing DeleteTablet for tablet a663030d37614053a7c0369841403422 with delete_type TABLET_DATA_DELETED (Sending initial creation of tablet) from {username='slave'} at 127.0.0.1:45934
I20240430 20:11:53.778990 18601 tablet_service.cc:1514] Processing DeleteTablet for tablet 436579f473844c6ca1038b496201a550 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:40 UTC) from {username='slave'} at 127.0.0.1:45934
I20240430 20:11:53.779690 18601 tablet_service.cc:1514] Processing DeleteTablet for tablet dad16ae02b8b4e29a724d8eec67bb4b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:42 UTC) from {username='slave'} at 127.0.0.1:45934
I20240430 20:11:53.780503 18601 tablet_service.cc:1514] Processing DeleteTablet for tablet fd1b8407071e4e4287fc172009777414 with delete_type TABLET_DATA_DELETED (replaced by tablet 8abf0b43f8a447dc902135ce94e4fc78) from {username='slave'} at 127.0.0.1:45934
I20240430 20:11:53.782470 18726 tablet_service.cc:1514] Processing DeleteTablet for tablet fdf53061efd74fa29a9d8208d1fd210e with delete_type TABLET_DATA_DELETED (replaced by tablet 5df364628561475391c9a5e10ff5a05f) from {username='slave'} at 127.0.0.1:56540
I20240430 20:11:53.783978 18726 tablet_service.cc:1514] Processing DeleteTablet for tablet f3711610610542128c4a84e17bce4a3e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:40 UTC) from {username='slave'} at 127.0.0.1:56540
I20240430 20:11:53.784938 18735 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e5225f1f5f544ecbf8dfa470af3daca with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:51 UTC) from {username='slave'} at 127.0.0.1:56540
I20240430 20:11:53.785120 18726 tablet_service.cc:1514] Processing DeleteTablet for tablet 246fb63a5e9842df8263800e28a5935c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:32 UTC) from {username='slave'} at 127.0.0.1:56540
I20240430 20:11:53.786149 18726 tablet_service.cc:1514] Processing DeleteTablet for tablet e4cffa295d8c401aa8415ce7e8f56526 with delete_type TABLET_DATA_DELETED (replaced by tablet e5b85a066cdc46d48ccf8201e5423c29) from {username='slave'} at 127.0.0.1:56540
W20240430 20:11:53.786693 19942 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.17.177.3:39881 (request call id 136) took 2864 ms (client timeout 1000 ms). Trace:
I20240430 20:11:53.786726 18619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fe7c6f1ab54499facd4d01573d3e7bc" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:53.786983 18619 raft_consensus.cc:3037] T 1fe7c6f1ab54499facd4d01573d3e7bc P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
W20240430 20:11:53.786831 19942 rpcz_store.cc:269] 0430 20:11:50.921728 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:51.439729 (+518001us) service_pool.cc:225] Handling call
0430 20:11:51.891857 (+452128us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:51.891867 (+ 10us) write_op.cc:446] Released schema lock
0430 20:11:53.786646 (+1894779us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:51.791159 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:51.791275 (+ 116us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:51.791282 (+ 7us) write_op.cc:427] Acquired schema lock
0430 20:11:51.791285 (+ 3us) tablet.cc:660] Decoding operations
0430 20:11:51.791507 (+ 222us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:51.791521 (+ 14us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:51.791527 (+ 6us) tablet.cc:683] Acquiring locks for 6 operations
0430 20:11:51.791668 (+ 141us) tablet.cc:699] Row locks acquired
0430 20:11:51.791670 (+ 2us) write_op.cc:260] PREPARE: finished
0430 20:11:51.791709 (+ 39us) write_op.cc:270] Start()
0430 20:11:51.791744 (+ 35us) write_op.cc:276] Timestamp: P: 1714507911791682 usec, L: 0
0430 20:11:51.791747 (+ 3us) op_driver.cc:331] REPLICATION: starting
0430 20:11:51.791939 (+ 192us) log.cc:842] Serialized 2416 byte log entry
0430 20:11:51.796567 (+ 4628us) op_driver.cc:444] REPLICATION: finished
0430 20:11:51.796692 (+ 125us) write_op.cc:301] APPLY: starting
0430 20:11:51.796711 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:51.796772 (+ 61us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:51.796802 (+ 30us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:51.796926 (+ 124us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:51.796968 (+ 42us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=6
0430 20:11:51.796976 (+ 8us) write_op.cc:312] APPLY: finished
0430 20:11:51.797064 (+ 88us) log.cc:842] Serialized 53 byte log entry
0430 20:11:51.797133 (+ 69us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:51.797161 (+ 28us) write_op.cc:446] Released schema lock
0430 20:11:51.797169 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"reported_tablets":33,"child_traces":[["op",{"apply.queue_time_us":74,"apply.run_cpu_time_us":555,"apply.run_wall_time_us":556,"num_ops":6,"prepare.queue_time_us":154,"prepare.run_cpu_time_us":887,"prepare.run_wall_time_us":888,"raft.queue_time_us":471,"raft.run_cpu_time_us":286,"raft.run_wall_time_us":286,"replication_time_us":4793}]]}
I20240430 20:11:53.788084 19942 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50746:
name: "default.Table_10a24c59b9bc473da9f773530f720f1d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:53.788836 19942 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_10a24c59b9bc473da9f773530f720f1d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240430 20:11:53.717825 19929 catalog_manager.cc:5676] Failed to send AlterTable RPC for tablet 6b51cbc731794083a88fe4bed1ddeebf (table default.Table_c6c962da746e4fd8a4d7200d09f08e9e [id=8e0f5d8c941d423ea7b30a89513d5170], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 6b51cbc731794083a88fe4bed1ddeebf consensus state has no leader
I20240430 20:11:53.793682 18619 raft_consensus.cc:2454] T 1fe7c6f1ab54499facd4d01573d3e7bc P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 1.
W20240430 20:11:53.794816 19929 catalog_manager.cc:5676] Failed to send AlterTable RPC for tablet 1701a15b115741aea13f8327e3d26b34 (table default.Table_3776a4b9d4f24aa6b7f882bb4d0116ef [id=7127dbd368864f7ba80a755549fe7ede], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 1701a15b115741aea13f8327e3d26b34 consensus state has no leader
W20240430 20:11:53.795575 19929 catalog_manager.cc:5676] Failed to send AlterTable RPC for tablet f1073fc792954c5795a05aa7d0ec24df (table default.Table_ba8b218e936b4bfcb073f8182f474cca [id=aedf11fda8914a3b843a57f8a7b69a53], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet f1073fc792954c5795a05aa7d0ec24df consensus state has no leader
W20240430 20:11:53.796813 19930 catalog_manager.cc:5676] Failed to send AlterTable RPC for tablet 94fbd4a57a1e438891fcb456f6b40be3 (table default.Table_25ddc3adef6b4ccf8b9c00cc6772031c [id=4fa623ffdcf347639150a355806038af], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 94fbd4a57a1e438891fcb456f6b40be3 consensus state has no leader
W20240430 20:11:53.797386 19930 catalog_manager.cc:5676] Failed to send AlterTable RPC for tablet aaef85eea87f4443943fd3692bb2a38f (table default.Table_3da5b4c8668c470483ae2a0c7773eaaf [id=e292d818ee3c45df90522f08eb2d6227], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet aaef85eea87f4443943fd3692bb2a38f consensus state has no leader
W20240430 20:11:53.798043 19930 catalog_manager.cc:5676] Failed to send AlterTable RPC for tablet e041756c962b4a9f97c2cb314e7d8c05 (table default.Table_de88cafd92b149f9b545369bff71033c [id=c4bf4806f3d84579b16a3038d8d982c3], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet e041756c962b4a9f97c2cb314e7d8c05 consensus state has no leader
I20240430 20:11:53.802505 18445 tablet_service.cc:1514] Processing DeleteTablet for tablet 6db55bb510b24b0cae897936211e15f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:34 UTC) from {username='slave'} at 127.0.0.1:41450
I20240430 20:11:53.803072 18445 tablet_service.cc:1514] Processing DeleteTablet for tablet 47e98935eb21449abd3d4d24327778d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:45 UTC) from {username='slave'} at 127.0.0.1:41450
I20240430 20:11:53.803509 18445 tablet_service.cc:1514] Processing DeleteTablet for tablet f3711610610542128c4a84e17bce4a3e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:40 UTC) from {username='slave'} at 127.0.0.1:41450
I20240430 20:11:53.803962 18445 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f4e860484784aa9a2d28af2e9fb611c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:39 UTC) from {username='slave'} at 127.0.0.1:41450
I20240430 20:11:53.804334 18445 tablet_service.cc:1514] Processing DeleteTablet for tablet f7d3dea34cf54fe2b949d1b7ca68577a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:40 UTC) from {username='slave'} at 127.0.0.1:41450
I20240430 20:11:53.804697 18445 tablet_service.cc:1514] Processing DeleteTablet for tablet a445fdfde01c4371a5e6b3daea5b00ab with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:44 UTC) from {username='slave'} at 127.0.0.1:41450
I20240430 20:11:53.805097 18445 tablet_service.cc:1514] Processing DeleteTablet for tablet 0eb4f3e7ece6496293e01b63620d644b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:49 UTC) from {username='slave'} at 127.0.0.1:41450
I20240430 20:11:53.808641 18477 raft_consensus.cc:2454] T 3191aa54a9644783a2403991c3be27c8 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:53.811198 18601 tablet_service.cc:1514] Processing DeleteTablet for tablet a37d917bcc45468f9e3ee805640fba98 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:44 UTC) from {username='slave'} at 127.0.0.1:45934
I20240430 20:11:53.814280 21276 raft_consensus.cc:482] T 4dac8b3838954d839094021f1c0503da P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:53.814486 21276 raft_consensus.cc:504] T 4dac8b3838954d839094021f1c0503da P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:53.814757 18585 tablet_service.cc:1514] Processing DeleteTablet for tablet 7bde1d6934784c9aa5911181e8df9eeb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:50 UTC) from {username='slave'} at 127.0.0.1:45934
I20240430 20:11:53.815428 21276 leader_election.cc:288] T 4dac8b3838954d839094021f1c0503da P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:53.815630 18590 tablet_service.cc:1514] Processing DeleteTablet for tablet 024069ab786749028c020abd1c836cf5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:44 UTC) from {username='slave'} at 127.0.0.1:45934
I20240430 20:11:53.815836 18588 tablet_service.cc:1514] Processing DeleteTablet for tablet 2fc6908ecf954020a676b9d10a2b00e7 with delete_type TABLET_DATA_DELETED (replaced by tablet 9a5654fd913f4fbeb9d126f035ac32bf) from {username='slave'} at 127.0.0.1:45934
I20240430 20:11:53.815855 18608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dac8b3838954d839094021f1c0503da" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:53.816074 18608 raft_consensus.cc:2454] T 4dac8b3838954d839094021f1c0503da P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 0.
I20240430 20:11:53.816540 18415 leader_election.cc:302] T 4dac8b3838954d839094021f1c0503da P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:53.816617 18588 tablet_service.cc:1514] Processing DeleteTablet for tablet 2797eed593fb4b06a3c4d6a28e8075a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:42 UTC) from {username='slave'} at 127.0.0.1:45934
I20240430 20:11:53.816848 21049 raft_consensus.cc:2781] T 4dac8b3838954d839094021f1c0503da P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:53.816929 21049 raft_consensus.cc:482] T 4dac8b3838954d839094021f1c0503da P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:53.816999 21049 raft_consensus.cc:3037] T 4dac8b3838954d839094021f1c0503da P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
W20240430 20:11:53.798552 19930 catalog_manager.cc:5676] Failed to send AlterTable RPC for tablet 5df364628561475391c9a5e10ff5a05f (table default.Table_3c8f9f7b22f64fb1bdc3f1550987233a [id=7fd6f8576f4c4cbc8ffcdb4ab95e15dc], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 5df364628561475391c9a5e10ff5a05f consensus state has no leader
I20240430 20:11:53.818648 18760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dac8b3838954d839094021f1c0503da" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:53.818930 18760 raft_consensus.cc:2454] T 4dac8b3838954d839094021f1c0503da P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 0.
W20240430 20:11:53.820449 19930 catalog_manager.cc:5676] Failed to send AlterTable RPC for tablet 772eba5f24c549c791cad75565858285 (table default.Table_5fc77d5f60ae44f4bb4ca9f7f183672c [id=6ca223ca6354449c959d85ed2ac6a9c8], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 772eba5f24c549c791cad75565858285 consensus state has no leader
I20240430 20:11:53.828040 18668 heartbeater.cc:503] Master 127.17.177.62:36441 was elected leader, sending a full tablet report...
W20240430 20:11:53.830999 19937 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.17.177.3:39881 (request call id 137) took 1709 ms (client timeout 1000 ms). Trace:
W20240430 20:11:53.831194 19937 rpcz_store.cc:269] 0430 20:11:52.121069 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:52.486169 (+365100us) service_pool.cc:225] Handling call
0430 20:11:53.830928 (+1344759us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:53.010163 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:53.010297 (+ 134us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:53.010304 (+ 7us) write_op.cc:427] Acquired schema lock
0430 20:11:53.010324 (+ 20us) tablet.cc:660] Decoding operations
0430 20:11:53.010438 (+ 114us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:53.010452 (+ 14us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:53.010457 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0430 20:11:53.010529 (+ 72us) tablet.cc:699] Row locks acquired
0430 20:11:53.010531 (+ 2us) write_op.cc:260] PREPARE: finished
0430 20:11:53.010566 (+ 35us) write_op.cc:270] Start()
0430 20:11:53.010633 (+ 67us) write_op.cc:276] Timestamp: P: 1714507913010520 usec, L: 0
0430 20:11:53.010637 (+ 4us) op_driver.cc:331] REPLICATION: starting
0430 20:11:53.010831 (+ 194us) log.cc:842] Serialized 530 byte log entry
0430 20:11:53.015078 (+ 4247us) op_driver.cc:444] REPLICATION: finished
0430 20:11:53.015210 (+ 132us) write_op.cc:301] APPLY: starting
0430 20:11:53.015228 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:53.015272 (+ 44us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:53.015312 (+ 40us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:53.015373 (+ 61us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:53.015410 (+ 37us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0430 20:11:53.015422 (+ 12us) write_op.cc:312] APPLY: finished
0430 20:11:53.015527 (+ 105us) log.cc:842] Serialized 23 byte log entry
0430 20:11:53.015598 (+ 71us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:53.015622 (+ 24us) write_op.cc:446] Released schema lock
0430 20:11:53.015630 (+ 8us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:53.015702 (+ 72us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:53.015706 (+ 4us) write_op.cc:446] Released schema lock
Metrics: {"reported_tablets":47,"child_traces":[["op",{"apply.queue_time_us":78,"apply.run_cpu_time_us":549,"apply.run_wall_time_us":549,"num_ops":1,"prepare.queue_time_us":54,"prepare.run_cpu_time_us":769,"prepare.run_wall_time_us":769,"raft.queue_time_us":465,"raft.run_cpu_time_us":237,"raft.run_wall_time_us":237,"replication_time_us":4412}]]}
I20240430 20:11:53.832233 19937 master_service.cc:946] ReplaceTablet: received request to replace tablet f9bf604e6c234349a9f72f45a4c4aef6 from {username='slave'} at 127.0.0.1:50746
I20240430 20:11:53.833529 18623 raft_consensus.cc:2454] T 3191aa54a9644783a2403991c3be27c8 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:53.834117 18595 tablet_service.cc:1514] Processing DeleteTablet for tablet b7141e215d6d44bc8406c80508190874 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:43 UTC) from {username='slave'} at 127.0.0.1:45934
I20240430 20:11:53.834254 18696 leader_election.cc:302] T 3191aa54a9644783a2403991c3be27c8 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:53.835274 18588 tablet_service.cc:1514] Processing DeleteTablet for tablet dbb427aefa6b47b0805d1751fea4efcb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:45 UTC) from {username='slave'} at 127.0.0.1:45934
I20240430 20:11:53.835407 18585 tablet_service.cc:1514] Processing DeleteTablet for tablet 558e71ec370d46ca8fbdcec5b57eb2ec with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:37 UTC) from {username='slave'} at 127.0.0.1:45934
I20240430 20:11:53.835479 18603 tablet_service.cc:1514] Processing DeleteTablet for tablet 898b2c91adf34545808df9c786e218c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:40 UTC) from {username='slave'} at 127.0.0.1:45934
I20240430 20:11:53.835777 18603 tablet_service.cc:1514] Processing DeleteTablet for tablet d34a450a91d542c98261456d1f635e9c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:49 UTC) from {username='slave'} at 127.0.0.1:45934
I20240430 20:11:53.836187 18603 tablet_service.cc:1514] Processing DeleteTablet for tablet 8668f7a285de43f48d83abecb4b2e96c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:50 UTC) from {username='slave'} at 127.0.0.1:45934
I20240430 20:11:53.836544 18603 tablet_service.cc:1514] Processing DeleteTablet for tablet aa9998f4e17147aa8f2b35c01980463a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:42 UTC) from {username='slave'} at 127.0.0.1:45934
I20240430 20:11:53.836900 18603 tablet_service.cc:1514] Processing DeleteTablet for tablet 1462ecf1176c4e66be42abddbe36b00c with delete_type TABLET_DATA_DELETED (replaced by tablet dd1de890e4394db287cf38bb411f5586) from {username='slave'} at 127.0.0.1:45934
I20240430 20:11:53.837922 18603 tablet_service.cc:1514] Processing DeleteTablet for tablet fcc16db4d6be42edb7bd1ce18fe0190d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:45 UTC) from {username='slave'} at 127.0.0.1:45934
I20240430 20:11:53.837898 18595 tablet_service.cc:1514] Processing DeleteTablet for tablet 4dac8b3838954d839094021f1c0503da with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:40 UTC) from {username='slave'} at 127.0.0.1:45934
I20240430 20:11:53.838349 18595 tablet_service.cc:1514] Processing DeleteTablet for tablet d4d6c37b578c4fe1aa666b0a6b0bc7c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:40 UTC) from {username='slave'} at 127.0.0.1:45934
I20240430 20:11:53.838656 21306 raft_consensus.cc:2781] T 3191aa54a9644783a2403991c3be27c8 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:53.838938 21306 raft_consensus.cc:686] T 3191aa54a9644783a2403991c3be27c8 P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Becoming Leader. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Running, Role: LEADER
I20240430 20:11:53.861650 18603 tablet_service.cc:1514] Processing DeleteTablet for tablet 5bd0362ff41b4faca0205d136d596737 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:31 UTC) from {username='slave'} at 127.0.0.1:45934
I20240430 20:11:53.862274 18603 tablet_service.cc:1514] Processing DeleteTablet for tablet eb76f60448774fe1967fffe7c6873cfc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:32 UTC) from {username='slave'} at 127.0.0.1:45934
I20240430 20:11:53.865926 18983 tablet_bootstrap.cc:492] T e5b85a066cdc46d48ccf8201e5423c29 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:53.866251 18983 ts_tablet_manager.cc:1367] T e5b85a066cdc46d48ccf8201e5423c29 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.206s user 0.010s sys 0.003s
I20240430 20:11:53.867355 18983 raft_consensus.cc:348] T e5b85a066cdc46d48ccf8201e5423c29 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:53.867806 18983 raft_consensus.cc:374] T e5b85a066cdc46d48ccf8201e5423c29 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:53.883788 20630 leader_election.cc:288] T 1fe7c6f1ab54499facd4d01573d3e7bc P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
W20240430 20:11:53.898317 19930 catalog_manager.cc:5676] Failed to send AlterTable RPC for tablet 5b26319488644d6a8b6be60460a5e7af (table default.Table_69892a8264a4477db48ee6248758e046 [id=5a1c4a7be2fc465fac89a50854eb3322], current schema version=3): Not found: failed to reset TS proxy: No leader found: Tablet 5b26319488644d6a8b6be60460a5e7af consensus state has no leader
W20240430 20:11:53.902290 19930 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.17.177.2:50593 (request call id 148) took 3425 ms (client timeout 1000 ms). Trace:
I20240430 20:11:53.887830 21049 raft_consensus.cc:504] T 4dac8b3838954d839094021f1c0503da P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:53.910615 21049 leader_election.cc:288] T 4dac8b3838954d839094021f1c0503da P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:53.911302 18983 raft_consensus.cc:729] T e5b85a066cdc46d48ccf8201e5423c29 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:53.911751 18983 consensus_queue.cc:260] T e5b85a066cdc46d48ccf8201e5423c29 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:53.912642 18983 ts_tablet_manager.cc:1398] T e5b85a066cdc46d48ccf8201e5423c29 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.046s user 0.002s sys 0.000s
I20240430 20:11:53.914929 18456 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c73d0a6bc414d3b9900570848d19a51 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:41 UTC) from {username='slave'} at 127.0.0.1:41450
I20240430 20:11:53.869547 21396 tablet_replica.cc:330] T a692b253db914531b8fc6ffaf3e350b5 P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:53.915262 21396 raft_consensus.cc:2229] T a692b253db914531b8fc6ffaf3e350b5 P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Raft consensus shutting down.
I20240430 20:11:53.915541 21396 pending_rounds.cc:62] T a692b253db914531b8fc6ffaf3e350b5 P 5e1c6397f09747f091534faf93542710: Trying to abort 1 pending ops.
I20240430 20:11:53.915596 21396 pending_rounds.cc:68] T a692b253db914531b8fc6ffaf3e350b5 P 5e1c6397f09747f091534faf93542710: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7022624412844929024 op_type: NO_OP noop_request { }
I20240430 20:11:53.915740 21396 raft_consensus.cc:2866] T a692b253db914531b8fc6ffaf3e350b5 P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240430 20:11:53.915818 21396 raft_consensus.cc:2258] T a692b253db914531b8fc6ffaf3e350b5 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:53.918517 18985 tablet_bootstrap.cc:492] T 086b3dfa84c045a18c8c4f177dce606a P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:53.918778 18985 ts_tablet_manager.cc:1367] T 086b3dfa84c045a18c8c4f177dce606a P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.229s user 0.005s sys 0.009s
I20240430 20:11:53.919021 21396 ts_tablet_manager.cc:1875] T a692b253db914531b8fc6ffaf3e350b5 P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:53.927309 18454 tablet_service.cc:1514] Processing DeleteTablet for tablet 06d28b6f48a34b17b0d7fb3102df2f4c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:33 UTC) from {username='slave'} at 127.0.0.1:41450
I20240430 20:11:53.927898 18454 tablet_service.cc:1514] Processing DeleteTablet for tablet f940d17b02b541289121463e05b0bb5f with delete_type TABLET_DATA_DELETED (replaced by tablet 53da4b3855344b6d92211af767195dc8) from {username='slave'} at 127.0.0.1:41450
I20240430 20:11:53.928289 18456 tablet_service.cc:1514] Processing DeleteTablet for tablet 04b3dff86d5c48a09d03da58bd48f360 with delete_type TABLET_DATA_DELETED (replaced by tablet 89b774967624498fa2be499878b71124) from {username='slave'} at 127.0.0.1:41450
I20240430 20:11:53.931706 18983 tablet_bootstrap.cc:492] T 347a8224114b49e5b7dcb75d9b726e0a P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:53.934094 18983 tablet_bootstrap.cc:654] T 347a8224114b49e5b7dcb75d9b726e0a P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:53.937772 18590 tablet_service.cc:1514] Processing DeleteTablet for tablet de1dfe6c82d34a4d8d02e466c55327ff with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:39 UTC) from {username='slave'} at 127.0.0.1:45934
I20240430 20:11:53.938097 18591 tablet_service.cc:1514] Processing DeleteTablet for tablet f7d3dea34cf54fe2b949d1b7ca68577a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:40 UTC) from {username='slave'} at 127.0.0.1:45934
I20240430 20:11:53.938520 18599 tablet_service.cc:1514] Processing DeleteTablet for tablet 1701a15b115741aea13f8327e3d26b34 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:50 UTC) from {username='slave'} at 127.0.0.1:45934
I20240430 20:11:53.938647 18589 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c73d0a6bc414d3b9900570848d19a51 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:41 UTC) from {username='slave'} at 127.0.0.1:45934
I20240430 20:11:53.938203 18595 tablet_service.cc:1514] Processing DeleteTablet for tablet e4cffa295d8c401aa8415ce7e8f56526 with delete_type TABLET_DATA_DELETED (replaced by tablet e5b85a066cdc46d48ccf8201e5423c29) from {username='slave'} at 127.0.0.1:45934
I20240430 20:11:53.938278 18585 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f4e860484784aa9a2d28af2e9fb611c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:39 UTC) from {username='slave'} at 127.0.0.1:45934
I20240430 20:11:53.938412 18588 tablet_service.cc:1514] Processing DeleteTablet for tablet 0eb4f3e7ece6496293e01b63620d644b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:49 UTC) from {username='slave'} at 127.0.0.1:45934
I20240430 20:11:53.938247 18586 tablet_service.cc:1514] Processing DeleteTablet for tablet 2873af6da928463f9809864b8eaf0c4a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:32 UTC) from {username='slave'} at 127.0.0.1:45934
I20240430 20:11:53.939028 18600 tablet_service.cc:1514] Processing DeleteTablet for tablet 8901327b3efb4201adce61a9266142ea with delete_type TABLET_DATA_DELETED (replaced by tablet b1277899f0c040cea7dc2b12f3fd68de) from {username='slave'} at 127.0.0.1:45934
I20240430 20:11:53.938282 18592 tablet_service.cc:1514] Processing DeleteTablet for tablet 246fb63a5e9842df8263800e28a5935c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:32 UTC) from {username='slave'} at 127.0.0.1:45934
I20240430 20:11:53.939291 21365 raft_consensus.cc:482] T 086b3dfa84c045a18c8c4f177dce606a P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:53.939456 21365 raft_consensus.cc:504] T 086b3dfa84c045a18c8c4f177dce606a P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:53.940753 21365 leader_election.cc:288] T 086b3dfa84c045a18c8c4f177dce606a P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:53.939220 18601 tablet_service.cc:1514] Processing DeleteTablet for tablet 90a694bd5db649afa50f3c4e752939c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:39 UTC) from {username='slave'} at 127.0.0.1:45934
I20240430 20:11:53.944392 18988 tablet_bootstrap.cc:492] T a0d3c4ba64024bccad95ecc909d03ae0 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:53.944564 18988 ts_tablet_manager.cc:1367] T a0d3c4ba64024bccad95ecc909d03ae0 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.244s user 0.000s sys 0.004s
I20240430 20:11:53.956626 19937 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:50746 (request call id 71) took 2920 ms. Trace:
I20240430 20:11:53.956912 19937 rpcz_store.cc:276] 0430 20:11:51.035923 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:53.832136 (+2796213us) service_pool.cc:225] Handling call
0430 20:11:53.956555 (+124419us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240430 20:11:53.957630 19937 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50746:
name: "default.Table_04cfbe148d8b4fc8aca451328df01be6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:53.958556 19937 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_04cfbe148d8b4fc8aca451328df01be6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:53.967497 18595 tablet_service.cc:1514] Processing DeleteTablet for tablet f3711610610542128c4a84e17bce4a3e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:40 UTC) from {username='slave'} at 127.0.0.1:45934
I20240430 20:11:53.968001 18595 tablet_service.cc:1514] Processing DeleteTablet for tablet a445fdfde01c4371a5e6b3daea5b00ab with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:44 UTC) from {username='slave'} at 127.0.0.1:45934
I20240430 20:11:53.974081 18985 raft_consensus.cc:348] T 086b3dfa84c045a18c8c4f177dce606a P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:53.974495 18985 raft_consensus.cc:374] T 086b3dfa84c045a18c8c4f177dce606a P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:53.974718 18985 raft_consensus.cc:729] T 086b3dfa84c045a18c8c4f177dce606a P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
W20240430 20:11:53.902506 19930 rpcz_store.cc:269] 0430 20:11:50.476350 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:50.758985 (+282635us) service_pool.cc:225] Handling call
0430 20:11:50.796405 (+ 37420us) spinlock_profiling.cc:243] Waited 29.5 ms on lock 0x616000070654. stack: 00007f6d23ac8756 00007f6d23ac8510 00007f6d22afb48d 0000000000955804 00000000009557cd 00007f6d3260715d 00007f6d32607253 00007f6d2f339d41 00007f6d36f2534d 00007f6d370ac668 00007f6d2de97786 00007f6d2de974b5 00007f6d2a7d004e 00007f6d2a7cef2e 00007f6d2a7d5405 00007f6d2a7d5d2f
0430 20:11:53.902219 (+3105814us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:51.670724 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:51.670869 (+ 145us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:51.670876 (+ 7us) write_op.cc:427] Acquired schema lock
0430 20:11:51.670880 (+ 4us) tablet.cc:660] Decoding operations
0430 20:11:51.671091 (+ 211us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:51.671105 (+ 14us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:51.671112 (+ 7us) tablet.cc:683] Acquiring locks for 4 operations
0430 20:11:51.671216 (+ 104us) tablet.cc:699] Row locks acquired
0430 20:11:51.671219 (+ 3us) write_op.cc:260] PREPARE: finished
0430 20:11:51.671255 (+ 36us) write_op.cc:270] Start()
0430 20:11:51.671292 (+ 37us) write_op.cc:276] Timestamp: P: 1714507911671241 usec, L: 0
0430 20:11:51.671295 (+ 3us) op_driver.cc:331] REPLICATION: starting
0430 20:11:51.671507 (+ 212us) log.cc:842] Serialized 1662 byte log entry
0430 20:11:51.675313 (+ 3806us) op_driver.cc:444] REPLICATION: finished
0430 20:11:51.675438 (+ 125us) write_op.cc:301] APPLY: starting
0430 20:11:51.675453 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:51.675524 (+ 71us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:51.675559 (+ 35us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:51.675648 (+ 89us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:51.675686 (+ 38us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=4
0430 20:11:51.675694 (+ 8us) write_op.cc:312] APPLY: finished
0430 20:11:51.675793 (+ 99us) log.cc:842] Serialized 41 byte log entry
0430 20:11:51.675855 (+ 62us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:51.675881 (+ 26us) write_op.cc:446] Released schema lock
0430 20:11:51.675888 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:51.675943 (+ 55us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:51.675947 (+ 4us) write_op.cc:446] Released schema lock
Metrics: {"mutex_wait_us":737,"reported_tablets":183,"spinlock_wait_cycles":67822336,"child_traces":[["op",{"apply.queue_time_us":76,"apply.run_cpu_time_us":561,"apply.run_wall_time_us":559,"num_ops":4,"prepare.queue_time_us":371,"prepare.run_cpu_time_us":904,"prepare.run_wall_time_us":903,"raft.queue_time_us":502,"raft.run_cpu_time_us":305,"raft.run_wall_time_us":306,"replication_time_us":3995}]]}
I20240430 20:11:53.975136 18985 consensus_queue.cc:260] T 086b3dfa84c045a18c8c4f177dce606a P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:53.975766 19930 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50746:
name: "default.Table_2c97168cc0e24d8fbece3e129cb15067"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:53.976505 19930 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2c97168cc0e24d8fbece3e129cb15067 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:53.980643 19942 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:50746 (request call id 70) took 2981 ms. Trace:
I20240430 20:11:53.980873 19942 rpcz_store.cc:276] 0430 20:11:50.999117 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:53.787733 (+2788616us) service_pool.cc:225] Handling call
0430 20:11:53.788941 (+ 1208us) catalog_manager.cc:2072] Acquired catalog manager lock
0430 20:11:53.789578 (+ 637us) catalog_manager.cc:2118] Created new table and tablet info
0430 20:11:53.836951 (+ 47373us) catalog_manager.cc:2178] Wrote table and tablets to system table
0430 20:11:53.980144 (+143193us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0430 20:11:53.980581 (+ 437us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:53.790277 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:53.790471 (+ 194us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:53.790482 (+ 11us) write_op.cc:427] Acquired schema lock
0430 20:11:53.790486 (+ 4us) tablet.cc:660] Decoding operations
0430 20:11:53.790647 (+ 161us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:53.790668 (+ 21us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:53.790674 (+ 6us) tablet.cc:683] Acquiring locks for 2 operations
0430 20:11:53.790786 (+ 112us) tablet.cc:699] Row locks acquired
0430 20:11:53.790790 (+ 4us) write_op.cc:260] PREPARE: finished
0430 20:11:53.790838 (+ 48us) write_op.cc:270] Start()
0430 20:11:53.790884 (+ 46us) write_op.cc:276] Timestamp: P: 1714507913790802 usec, L: 0
0430 20:11:53.790889 (+ 5us) op_driver.cc:331] REPLICATION: starting
0430 20:11:53.791171 (+ 282us) log.cc:842] Serialized 433 byte log entry
0430 20:11:53.797559 (+ 6388us) op_driver.cc:444] REPLICATION: finished
0430 20:11:53.797686 (+ 127us) write_op.cc:301] APPLY: starting
0430 20:11:53.797706 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:53.797760 (+ 54us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:53.797822 (+ 62us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:53.797943 (+ 121us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:53.797981 (+ 38us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0430 20:11:53.797990 (+ 9us) write_op.cc:312] APPLY: finished
0430 20:11:53.798094 (+ 104us) log.cc:842] Serialized 29 byte log entry
0430 20:11:53.798170 (+ 76us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:53.798197 (+ 27us) write_op.cc:446] Released schema lock
0430 20:11:53.798206 (+ 9us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:53.798276 (+ 70us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:53.798280 (+ 4us) write_op.cc:446] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":72,"apply.run_cpu_time_us":655,"apply.run_wall_time_us":654,"num_ops":2,"prepare.queue_time_us":58,"prepare.run_cpu_time_us":1035,"prepare.run_wall_time_us":1035,"raft.queue_time_us":81,"raft.run_cpu_time_us":313,"raft.run_wall_time_us":313,"replication_time_us":6633}]]}
I20240430 20:11:53.981859 18810 heartbeater.cc:503] Master 127.17.177.60:37225 was elected leader, sending a full tablet report...
I20240430 20:11:53.982241 19942 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50746:
name: "default.Table_dac5789e49e24b798431be43e39edb7c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:53.982976 19942 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dac5789e49e24b798431be43e39edb7c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:53.987222 19937 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:50746 (request call id 72) took 2941 ms. Trace:
I20240430 20:11:53.987426 19937 rpcz_store.cc:276] 0430 20:11:51.045746 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:53.957298 (+2911552us) service_pool.cc:225] Handling call
0430 20:11:53.980333 (+ 23035us) catalog_manager.cc:2072] Acquired catalog manager lock
0430 20:11:53.980878 (+ 545us) catalog_manager.cc:2118] Created new table and tablet info
0430 20:11:53.986712 (+ 5834us) catalog_manager.cc:2178] Wrote table and tablets to system table
0430 20:11:53.987075 (+ 363us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0430 20:11:53.987170 (+ 95us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:53.981452 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:53.981566 (+ 114us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:53.981573 (+ 7us) write_op.cc:427] Acquired schema lock
0430 20:11:53.981575 (+ 2us) tablet.cc:660] Decoding operations
0430 20:11:53.981672 (+ 97us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:53.981687 (+ 15us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:53.981692 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0430 20:11:53.981772 (+ 80us) tablet.cc:699] Row locks acquired
0430 20:11:53.981775 (+ 3us) write_op.cc:260] PREPARE: finished
0430 20:11:53.981811 (+ 36us) write_op.cc:270] Start()
0430 20:11:53.981845 (+ 34us) write_op.cc:276] Timestamp: P: 1714507913981762 usec, L: 0
0430 20:11:53.981859 (+ 14us) op_driver.cc:331] REPLICATION: starting
0430 20:11:53.982045 (+ 186us) log.cc:842] Serialized 433 byte log entry
0430 20:11:53.985722 (+ 3677us) op_driver.cc:444] REPLICATION: finished
0430 20:11:53.986001 (+ 279us) write_op.cc:301] APPLY: starting
0430 20:11:53.986021 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:53.986080 (+ 59us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:53.986125 (+ 45us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:53.986280 (+ 155us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:53.986330 (+ 50us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0430 20:11:53.986340 (+ 10us) write_op.cc:312] APPLY: finished
0430 20:11:53.986437 (+ 97us) log.cc:842] Serialized 29 byte log entry
0430 20:11:53.986517 (+ 80us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:53.986546 (+ 29us) write_op.cc:446] Released schema lock
0430 20:11:53.986555 (+ 9us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:53.986639 (+ 84us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:53.986643 (+ 4us) write_op.cc:446] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":211,"apply.run_cpu_time_us":705,"apply.run_wall_time_us":704,"num_ops":2,"prepare.queue_time_us":53,"prepare.run_cpu_time_us":690,"prepare.run_wall_time_us":690,"raft.queue_time_us":406,"raft.run_cpu_time_us":267,"raft.run_wall_time_us":267,"replication_time_us":3835}]]}
I20240430 20:11:53.988442 19937 master_service.cc:946] ReplaceTablet: received request to replace tablet cefce2f2a392495bb71399ed75db938c from {username='slave'} at 127.0.0.1:50746
I20240430 20:11:53.990531 18452 tablet_service.cc:1514] Processing DeleteTablet for tablet d7c1fc7e81364fb890bbc13d7fb6848b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:40 UTC) from {username='slave'} at 127.0.0.1:41450
I20240430 20:11:53.990854 18985 ts_tablet_manager.cc:1398] T 086b3dfa84c045a18c8c4f177dce606a P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.072s user 0.002s sys 0.000s
I20240430 20:11:53.991204 18985 tablet_bootstrap.cc:492] T 1050ab528c7d41559831652a1437d358 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
W20240430 20:11:54.007164 19929 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.17.177.1:47345 (request call id 133) took 3878 ms (client timeout 1000 ms). Trace:
W20240430 20:11:54.007696 19929 rpcz_store.cc:269] 0430 20:11:50.129067 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:50.159446 (+ 30379us) service_pool.cc:225] Handling call
0430 20:11:50.203195 (+ 43749us) spinlock_profiling.cc:243] Waited 26.4 ms on lock 0x616000070654. stack: 00007f6d23ac8756 00007f6d23ac8510 00007f6d22afb48d 0000000000955804 00000000009557cd 00007f6d3260715d 00007f6d32607253 00007f6d2f339d41 00007f6d36f2534d 00007f6d370ac668 00007f6d2de97786 00007f6d2de974b5 00007f6d2a7d004e 00007f6d2a7cef2e 00007f6d2a7d5405 00007f6d2a7d5d2f
0430 20:11:54.007099 (+3803904us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:51.381859 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:51.381989 (+ 130us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:51.381996 (+ 7us) write_op.cc:427] Acquired schema lock
0430 20:11:51.381999 (+ 3us) tablet.cc:660] Decoding operations
0430 20:11:51.382514 (+ 515us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:51.382529 (+ 15us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:51.382535 (+ 6us) tablet.cc:683] Acquiring locks for 29 operations
0430 20:11:51.383181 (+ 646us) tablet.cc:699] Row locks acquired
0430 20:11:51.383184 (+ 3us) write_op.cc:260] PREPARE: finished
0430 20:11:51.383245 (+ 61us) write_op.cc:270] Start()
0430 20:11:51.383285 (+ 40us) write_op.cc:276] Timestamp: P: 1714507911383208 usec, L: 0
0430 20:11:51.383288 (+ 3us) op_driver.cc:331] REPLICATION: starting
0430 20:11:51.383528 (+ 240us) log.cc:842] Serialized 11087 byte log entry
0430 20:11:51.408552 (+ 25024us) op_driver.cc:444] REPLICATION: finished
0430 20:11:51.408686 (+ 134us) write_op.cc:301] APPLY: starting
0430 20:11:51.408708 (+ 22us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:51.408826 (+ 118us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:51.408861 (+ 35us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:51.409251 (+ 390us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:51.409308 (+ 57us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=29
0430 20:11:51.409316 (+ 8us) write_op.cc:312] APPLY: finished
0430 20:11:51.409464 (+ 148us) log.cc:842] Serialized 194 byte log entry
0430 20:11:51.409544 (+ 80us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:51.409599 (+ 55us) write_op.cc:446] Released schema lock
0430 20:11:51.409608 (+ 9us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:51.409671 (+ 63us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:51.409675 (+ 4us) write_op.cc:446] Released schema lock
Metrics: {"reported_tablets":198,"spinlock_wait_cycles":60612864,"child_traces":[["op",{"apply.queue_time_us":79,"apply.run_cpu_time_us":1056,"apply.run_wall_time_us":1055,"num_ops":29,"prepare.queue_time_us":1210,"prepare.run_cpu_time_us":1771,"prepare.run_wall_time_us":1779,"raft.queue_time_us":706,"raft.run_cpu_time_us":279,"raft.run_wall_time_us":279,"replication_time_us":25233}]]}
I20240430 20:11:54.010449 19929 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50746:
table { table_name: "default.Table_ba8b218e936b4bfcb073f8182f474cca" } new_table_name: "default.Table_91417a5439c8457caa8bb0a12221a5dd" modify_external_catalogs: true
I20240430 20:11:54.013269 18452 tablet_service.cc:1514] Processing DeleteTablet for tablet c1017a50d1d646c486c0b9a75cb88ce4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:43 UTC) from {username='slave'} at 127.0.0.1:41450
I20240430 20:11:54.014045 19930 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:50746 (request call id 73) took 2968 ms. Trace:
I20240430 20:11:54.015283 18452 tablet_service.cc:1514] Processing DeleteTablet for tablet 436579f473844c6ca1038b496201a550 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:40 UTC) from {username='slave'} at 127.0.0.1:41450
I20240430 20:11:54.019851 18983 tablet_bootstrap.cc:492] T 347a8224114b49e5b7dcb75d9b726e0a P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:54.020123 18983 ts_tablet_manager.cc:1367] T 347a8224114b49e5b7dcb75d9b726e0a P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.088s user 0.001s sys 0.004s
I20240430 20:11:54.021116 18983 raft_consensus.cc:348] T 347a8224114b49e5b7dcb75d9b726e0a P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:54.021488 18983 raft_consensus.cc:374] T 347a8224114b49e5b7dcb75d9b726e0a P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:54.021600 18983 raft_consensus.cc:729] T 347a8224114b49e5b7dcb75d9b726e0a P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:54.021967 18983 consensus_queue.cc:260] T 347a8224114b49e5b7dcb75d9b726e0a P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:54.023000 18758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086b3dfa84c045a18c8c4f177dce606a" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:54.023222 18758 raft_consensus.cc:2454] T 086b3dfa84c045a18c8c4f177dce606a P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 0.
I20240430 20:11:54.024945 18985 tablet_bootstrap.cc:654] T 1050ab528c7d41559831652a1437d358 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:54.014227 19930 rpcz_store.cc:276] 0430 20:11:51.045883 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:53.975483 (+2929600us) service_pool.cc:225] Handling call
0430 20:11:53.996210 (+ 20727us) catalog_manager.cc:2072] Acquired catalog manager lock
0430 20:11:53.996695 (+ 485us) catalog_manager.cc:2118] Created new table and tablet info
0430 20:11:54.013110 (+ 16415us) catalog_manager.cc:2178] Wrote table and tablets to system table
0430 20:11:54.013892 (+ 782us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0430 20:11:54.014005 (+ 113us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:53.997246 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:53.997400 (+ 154us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:53.997410 (+ 10us) write_op.cc:427] Acquired schema lock
0430 20:11:53.997415 (+ 5us) tablet.cc:660] Decoding operations
0430 20:11:53.997551 (+ 136us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:53.997570 (+ 19us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:53.997577 (+ 7us) tablet.cc:683] Acquiring locks for 2 operations
0430 20:11:53.997689 (+ 112us) tablet.cc:699] Row locks acquired
0430 20:11:53.997693 (+ 4us) write_op.cc:260] PREPARE: finished
0430 20:11:53.997738 (+ 45us) write_op.cc:270] Start()
0430 20:11:53.997783 (+ 45us) write_op.cc:276] Timestamp: P: 1714507913997685 usec, L: 0
0430 20:11:53.997789 (+ 6us) op_driver.cc:331] REPLICATION: starting
0430 20:11:53.998051 (+ 262us) log.cc:842] Serialized 433 byte log entry
0430 20:11:54.002351 (+ 4300us) op_driver.cc:444] REPLICATION: finished
0430 20:11:54.002494 (+ 143us) write_op.cc:301] APPLY: starting
0430 20:11:54.002517 (+ 23us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:54.002582 (+ 65us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:54.002637 (+ 55us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:54.002814 (+ 177us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:54.002876 (+ 62us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0430 20:11:54.002889 (+ 13us) write_op.cc:312] APPLY: finished
0430 20:11:54.002992 (+ 103us) log.cc:842] Serialized 29 byte log entry
0430 20:11:54.003072 (+ 80us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:54.003102 (+ 30us) write_op.cc:446] Released schema lock
0430 20:11:54.003112 (+ 10us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:54.003184 (+ 72us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:54.003190 (+ 6us) write_op.cc:446] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":83,"apply.run_cpu_time_us":757,"apply.run_wall_time_us":755,"num_ops":2,"prepare.queue_time_us":58,"prepare.run_cpu_time_us":930,"prepare.run_wall_time_us":932,"raft.queue_time_us":536,"raft.run_cpu_time_us":307,"raft.run_wall_time_us":307,"replication_time_us":4529}]]}
I20240430 20:11:54.027628 19930 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50746:
table { table_name: "default.Table_2e4e483bcd6c4dd6bdfcc938cb0e2da6" } modify_external_catalogs: true
I20240430 20:11:54.027907 19930 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50746:
table { table_name: "default.Table_2e4e483bcd6c4dd6bdfcc938cb0e2da6" } modify_external_catalogs: true
I20240430 20:11:54.041319 21396 ts_tablet_manager.cc:1888] T a692b253db914531b8fc6ffaf3e350b5 P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:54.049139 19941 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 0368d37dd9e2452982fb81f539c6805b (table default.Table_404b186c544e4b30bb36cb77d1a14120 [id=4330626e342a48699ea939add5c91b11]). Expected version 1 got 0
I20240430 20:11:54.049888 19941 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 12795670f21447a8964fe1efc6e92398 (table default.Table_896bbae1f6944005b0272ca8ebc97218 [id=d5cea90cbc2c48df9d8b80da1eccc949]). Expected version 3 got 0
I20240430 20:11:54.050755 19941 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet a482f4622f774d60b4efd6827b8be7d7 (table default.Table_e686cdfcbcc046749fa5b51578a64794 [id=c3dc9f05316d46d6a3ef4d03f40877ec]). Expected version 1 got 0
I20240430 20:11:54.051156 19941 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 488d0e609b224674932f890939664df2 (table default.Table_58f35a2120874472978ca0e5f816e86c [id=ff18001c6d814afc8c1768d0c84a6bac]). Expected version 1 got 0
I20240430 20:11:54.051815 19941 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 22fef8bb6b034db098c3372ae992f7a7 (table default.Table_4440df38610340758047115845b42b95 [id=7d186ca20c094cc19aef792fe1c094e9]). Expected version 1 got 0
I20240430 20:11:54.043303 18454 tablet_service.cc:1514] Processing DeleteTablet for tablet a663030d37614053a7c0369841403422 with delete_type TABLET_DATA_DELETED (Sending initial creation of tablet) from {username='slave'} at 127.0.0.1:41450
I20240430 20:11:54.053068 18454 tablet_service.cc:1514] Processing DeleteTablet for tablet fd1b8407071e4e4287fc172009777414 with delete_type TABLET_DATA_DELETED (replaced by tablet 8abf0b43f8a447dc902135ce94e4fc78) from {username='slave'} at 127.0.0.1:41450
I20240430 20:11:54.054636 18454 tablet_service.cc:1514] Processing DeleteTablet for tablet 2fc6908ecf954020a676b9d10a2b00e7 with delete_type TABLET_DATA_DELETED (replaced by tablet 9a5654fd913f4fbeb9d126f035ac32bf) from {username='slave'} at 127.0.0.1:41450
I20240430 20:11:54.055063 19941 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 3a03b6997921408cb5a65f4cae223e9c (table default.Table_744b5c96ea014bdb8feae740356a790e [id=83b1ac2e9c3d4a10b02f96ebb17faf44]). Expected version 1 got 0
I20240430 20:11:54.055394 18454 tablet_service.cc:1514] Processing DeleteTablet for tablet df92eb85261d47d88cbf16d29d9ad30e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:45 UTC) from {username='slave'} at 127.0.0.1:41450
I20240430 20:11:54.055931 19941 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 89b774967624498fa2be499878b71124 (table default.Table_2ec9e316bc3347b189a13b7a8bbcf829 [id=9fa22833979f4c2f978de651145acef9]). Expected version 1 got 0
I20240430 20:11:54.056625 19941 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 62732af95c6a4c749c5b726f0460dc72 (table default.Table_e3160f5ced1b4718b479401ca4df675c [id=2d44ec94131148048021b2ee6366ba30]). Expected version 3 got 0
I20240430 20:11:54.058504 18452 tablet_service.cc:1514] Processing DeleteTablet for tablet dad16ae02b8b4e29a724d8eec67bb4b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:42 UTC) from {username='slave'} at 127.0.0.1:41450
I20240430 20:11:54.059597 18452 tablet_service.cc:1514] Processing DeleteTablet for tablet aa9998f4e17147aa8f2b35c01980463a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:42 UTC) from {username='slave'} at 127.0.0.1:41450
W20240430 20:11:54.061138 18417 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.177.3:44867, user_credentials={real_user=slave}} blocked reactor thread for 242739us
I20240430 20:11:54.061861 18452 tablet_service.cc:1514] Processing DeleteTablet for tablet 90a694bd5db649afa50f3c4e752939c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:39 UTC) from {username='slave'} at 127.0.0.1:41450
I20240430 20:11:54.063079 18459 tablet_service.cc:1514] Processing DeleteTablet for tablet 7bde1d6934784c9aa5911181e8df9eeb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:50 UTC) from {username='slave'} at 127.0.0.1:41450
I20240430 20:11:54.063159 18455 tablet_service.cc:1514] Processing DeleteTablet for tablet fdf53061efd74fa29a9d8208d1fd210e with delete_type TABLET_DATA_DELETED (replaced by tablet 5df364628561475391c9a5e10ff5a05f) from {username='slave'} at 127.0.0.1:41450
I20240430 20:11:54.063463 18455 tablet_service.cc:1514] Processing DeleteTablet for tablet a37d917bcc45468f9e3ee805640fba98 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:44 UTC) from {username='slave'} at 127.0.0.1:41450
I20240430 20:11:54.063771 18455 tablet_service.cc:1514] Processing DeleteTablet for tablet 2797eed593fb4b06a3c4d6a28e8075a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:42 UTC) from {username='slave'} at 127.0.0.1:41450
I20240430 20:11:54.064071 18455 tablet_service.cc:1514] Processing DeleteTablet for tablet b7141e215d6d44bc8406c80508190874 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:43 UTC) from {username='slave'} at 127.0.0.1:41450
I20240430 20:11:54.064405 18455 tablet_service.cc:1514] Processing DeleteTablet for tablet 95383d9d5a234a4d9d841a8b114428b0 with delete_type TABLET_DATA_DELETED (replaced by tablet 6f3996b7e29941e391ea141dd6cc44b2) from {username='slave'} at 127.0.0.1:41450
I20240430 20:11:54.064751 18455 tablet_service.cc:1514] Processing DeleteTablet for tablet dbb427aefa6b47b0805d1751fea4efcb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:45 UTC) from {username='slave'} at 127.0.0.1:41450
I20240430 20:11:54.065160 18455 tablet_service.cc:1514] Processing DeleteTablet for tablet 558e71ec370d46ca8fbdcec5b57eb2ec with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:37 UTC) from {username='slave'} at 127.0.0.1:41450
I20240430 20:11:54.065531 18455 tablet_service.cc:1514] Processing DeleteTablet for tablet 898b2c91adf34545808df9c786e218c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:40 UTC) from {username='slave'} at 127.0.0.1:41450
I20240430 20:11:54.065987 18455 tablet_service.cc:1514] Processing DeleteTablet for tablet d34a450a91d542c98261456d1f635e9c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:49 UTC) from {username='slave'} at 127.0.0.1:41450
I20240430 20:11:54.066320 18455 tablet_service.cc:1514] Processing DeleteTablet for tablet 4dac8b3838954d839094021f1c0503da with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:40 UTC) from {username='slave'} at 127.0.0.1:41450
I20240430 20:11:54.066727 18455 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f6f9ae722da49e58c9b72107f6da2fa with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:42 UTC) from {username='slave'} at 127.0.0.1:41450
I20240430 20:11:54.067095 18455 tablet_service.cc:1514] Processing DeleteTablet for tablet d4d6c37b578c4fe1aa666b0a6b0bc7c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:40 UTC) from {username='slave'} at 127.0.0.1:41450
I20240430 20:11:54.067461 18455 tablet_service.cc:1514] Processing DeleteTablet for tablet 8668f7a285de43f48d83abecb4b2e96c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:50 UTC) from {username='slave'} at 127.0.0.1:41450
I20240430 20:11:54.067891 18455 tablet_service.cc:1514] Processing DeleteTablet for tablet e4cffa295d8c401aa8415ce7e8f56526 with delete_type TABLET_DATA_DELETED (replaced by tablet e5b85a066cdc46d48ccf8201e5423c29) from {username='slave'} at 127.0.0.1:41450
I20240430 20:11:54.068248 18455 tablet_service.cc:1514] Processing DeleteTablet for tablet 489dc838744a49eba542009c86688aa5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:43 UTC) from {username='slave'} at 127.0.0.1:41450
I20240430 20:11:54.068621 18455 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ec3993042f5498b818bdf32d5087928 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:43 UTC) from {username='slave'} at 127.0.0.1:41450
I20240430 20:11:54.069139 18459 tablet_service.cc:1514] Processing DeleteTablet for tablet a9f37f0f7ce8425f9b24699513d41d61 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:40 UTC) from {username='slave'} at 127.0.0.1:41450
I20240430 20:11:54.069223 18455 tablet_service.cc:1514] Processing DeleteTablet for tablet 1462ecf1176c4e66be42abddbe36b00c with delete_type TABLET_DATA_DELETED (replaced by tablet dd1de890e4394db287cf38bb411f5586) from {username='slave'} at 127.0.0.1:41450
I20240430 20:11:54.069460 18448 tablet_service.cc:1514] Processing DeleteTablet for tablet e87fb2f0361d4aa3ada27bbdfa88e94c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:36 UTC) from {username='slave'} at 127.0.0.1:41450
I20240430 20:11:54.069792 18448 tablet_service.cc:1514] Processing DeleteTablet for tablet fcc16db4d6be42edb7bd1ce18fe0190d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:45 UTC) from {username='slave'} at 127.0.0.1:41450
I20240430 20:11:54.070119 18448 tablet_service.cc:1514] Processing DeleteTablet for tablet d61c279d20b0471a967942525ceb4fcb with delete_type TABLET_DATA_DELETED (replaced by tablet 69f88532952a4591bf1577541c3430e6) from {username='slave'} at 127.0.0.1:41450
I20240430 20:11:54.070474 18448 tablet_service.cc:1514] Processing DeleteTablet for tablet eb76f60448774fe1967fffe7c6873cfc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:32 UTC) from {username='slave'} at 127.0.0.1:41450
I20240430 20:11:54.070873 18448 tablet_service.cc:1514] Processing DeleteTablet for tablet c6378f1495144c458af49b3c19a2b67c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:45 UTC) from {username='slave'} at 127.0.0.1:41450
I20240430 20:11:54.071269 18448 tablet_service.cc:1514] Processing DeleteTablet for tablet 7683caba80594627900a0185f772c131 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:42 UTC) from {username='slave'} at 127.0.0.1:41450
I20240430 20:11:54.071779 18448 tablet_service.cc:1514] Processing DeleteTablet for tablet f6cfd089f08b47409fb2a6cf4f6b0385 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:44 UTC) from {username='slave'} at 127.0.0.1:41450
I20240430 20:11:54.072209 18448 tablet_service.cc:1514] Processing DeleteTablet for tablet de1dfe6c82d34a4d8d02e466c55327ff with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:39 UTC) from {username='slave'} at 127.0.0.1:41450
I20240430 20:11:54.072715 18448 tablet_service.cc:1514] Processing DeleteTablet for tablet 2873af6da928463f9809864b8eaf0c4a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:32 UTC) from {username='slave'} at 127.0.0.1:41450
I20240430 20:11:54.073064 18448 tablet_service.cc:1514] Processing DeleteTablet for tablet 246fb63a5e9842df8263800e28a5935c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:32 UTC) from {username='slave'} at 127.0.0.1:41450
I20240430 20:11:54.073472 18448 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b7fdc6ec7e046c5af894f9a6eda17b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:40 UTC) from {username='slave'} at 127.0.0.1:41450
I20240430 20:11:54.073860 18448 tablet_service.cc:1514] Processing DeleteTablet for tablet 2aa2a489116940dea3173cad16d94fa5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:40 UTC) from {username='slave'} at 127.0.0.1:41450
I20240430 20:11:54.074229 18448 tablet_service.cc:1514] Processing DeleteTablet for tablet 283e5e73cbc04daba0de19f4541f9fd0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:40 UTC) from {username='slave'} at 127.0.0.1:41450
I20240430 20:11:54.074678 18448 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d1d2193fd424c91b143f5256df5fcd0 with delete_type TABLET_DATA_DELETED (replaced by tablet bbdbb8a3a4b444f58a2761466577a88f) from {username='slave'} at 127.0.0.1:41450
I20240430 20:11:54.075070 18448 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f0b341a31224bf1be525b84b9500304 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:44 UTC) from {username='slave'} at 127.0.0.1:41450
I20240430 20:11:54.077320 21395 tablet_replica.cc:330] T a692b253db914531b8fc6ffaf3e350b5 P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:54.079699 21457 raft_consensus.cc:482] T 1050ab528c7d41559831652a1437d358 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:54.079859 21457 raft_consensus.cc:504] T 1050ab528c7d41559831652a1437d358 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:54.080730 21457 leader_election.cc:288] T 1050ab528c7d41559831652a1437d358 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:54.082231 19941 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet eb6c96047b114ebba73db7a371f2da8e (table default.Table_7d5f6f18cbd64c869739872a7dff9be2 [id=e67d619e36c0485f8f22b4a81cb4cb6c]). Expected version 1 got 0
I20240430 20:11:54.082572 19941 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet c6e3d09402574f1e8fe9e4160c70a3ee (table default.Table_656db97355f64d7c8aa29b1352dfe9c8 [id=450f5d1c7e084a1bb88d035c0ccfcf82]). Expected version 1 got 0
I20240430 20:11:54.082876 19941 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 8e9711d0a382424f9f758bfa883866a0 (table default.Table_f1dc481c6d8c43f589fe2b1892a4cb10 [id=db696ce66005401fa73746e55c071f44]). Expected version 1 got 0
I20240430 20:11:54.079468 18983 ts_tablet_manager.cc:1398] T 347a8224114b49e5b7dcb75d9b726e0a P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.059s user 0.001s sys 0.001s
I20240430 20:11:54.085945 21396 log.cc:1196] T a692b253db914531b8fc6ffaf3e350b5 P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/a692b253db914531b8fc6ffaf3e350b5
I20240430 20:11:54.086426 21396 ts_tablet_manager.cc:1909] T a692b253db914531b8fc6ffaf3e350b5 P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:54.079147 18448 tablet_service.cc:1514] Processing DeleteTablet for tablet 8901327b3efb4201adce61a9266142ea with delete_type TABLET_DATA_DELETED (replaced by tablet b1277899f0c040cea7dc2b12f3fd68de) from {username='slave'} at 127.0.0.1:41450
I20240430 20:11:54.078122 21395 raft_consensus.cc:2229] T a692b253db914531b8fc6ffaf3e350b5 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:54.087617 18595 tablet_service.cc:1467] Processing CreateTablet for tablet 3aa5853f572c4989919f3b9ccb7f56ab (DEFAULT_TABLE table=default.Table_7b102da111ef451aba55ec04c1215cc7 [id=699c3d4ff0334398821693f3434ec15b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:54.087785 21395 raft_consensus.cc:2258] T a692b253db914531b8fc6ffaf3e350b5 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:54.088325 21395 ts_tablet_manager.cc:1875] T a692b253db914531b8fc6ffaf3e350b5 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:54.102406 21397 tablet_replica.cc:330] T a692b253db914531b8fc6ffaf3e350b5 P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:54.103018 21054 raft_consensus.cc:482] T 330453ddeea6456e8ef93fa92608b0d3 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:54.103300 21054 raft_consensus.cc:504] T 330453ddeea6456e8ef93fa92608b0d3 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:54.104205 21054 leader_election.cc:288] T 330453ddeea6456e8ef93fa92608b0d3 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:54.104662 18988 raft_consensus.cc:348] T a0d3c4ba64024bccad95ecc909d03ae0 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:54.104925 18988 raft_consensus.cc:374] T a0d3c4ba64024bccad95ecc909d03ae0 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:54.104977 18988 raft_consensus.cc:729] T a0d3c4ba64024bccad95ecc909d03ae0 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:54.105206 18988 consensus_queue.cc:260] T a0d3c4ba64024bccad95ecc909d03ae0 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:54.105734 18988 ts_tablet_manager.cc:1398] T a0d3c4ba64024bccad95ecc909d03ae0 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.161s user 0.001s sys 0.001s
I20240430 20:11:54.106014 18988 tablet_bootstrap.cc:492] T 3b7fdc6ec7e046c5af894f9a6eda17b0 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:54.107012 18453 tablet_service.cc:1514] Processing DeleteTablet for tablet 024069ab786749028c020abd1c836cf5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:44 UTC) from {username='slave'} at 127.0.0.1:41450
W20240430 20:11:54.109021 18696 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.177.2:46027, user_credentials={real_user=slave}} blocked reactor thread for 176952us
I20240430 20:11:54.112419 21397 raft_consensus.cc:2229] T a692b253db914531b8fc6ffaf3e350b5 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:54.113251 21397 raft_consensus.cc:2258] T a692b253db914531b8fc6ffaf3e350b5 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:54.113842 21397 ts_tablet_manager.cc:1875] T a692b253db914531b8fc6ffaf3e350b5 P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:54.080842 19937 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: Sending DeleteTablet for 3 replicas of tablet cefce2f2a392495bb71399ed75db938c
I20240430 20:11:54.072134 19942 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:50746 (request call id 74) took 3020 ms. Trace:
I20240430 20:11:54.121132 18623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1050ab528c7d41559831652a1437d358" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:54.121529 18601 tablet_service.cc:1467] Processing CreateTablet for tablet 92956889b2284975ba22523fe10d4ded (DEFAULT_TABLE table=default.Table_663f87c0ffdd472dba96f9c67665e8d6 [id=c902110d321b4286b161cbf9a03e47f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:54.121894 18592 tablet_service.cc:1467] Processing CreateTablet for tablet a06572a8df774c508923a9cb160f77d7 (DEFAULT_TABLE table=default.Table_8c21c0353321455a82097231a06f0957 [id=3b5c663f10a14933bac74b1cb9722e42]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:54.122042 18623 raft_consensus.cc:2454] T 1050ab528c7d41559831652a1437d358 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:54.121887 18600 tablet_service.cc:1514] Processing DeleteTablet for tablet cefce2f2a392495bb71399ed75db938c with delete_type TABLET_DATA_DELETED (replaced by tablet 9e1fed3070794d8da6e31692adfcb8cc) from {username='slave'} at 127.0.0.1:45934
I20240430 20:11:54.121551 18603 tablet_service.cc:1467] Processing CreateTablet for tablet 293283469725407993ec9351d8b29054 (DEFAULT_TABLE table=default.Table_1a823c2465ce4224a3cf173600f6e32c [id=3ad80fb32e2c44d7800834b09d1ace7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:54.126375 21458 raft_consensus.cc:482] T 13c2e69ae91f4957887fd64b90f6e435 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:54.103600 21437 raft_consensus.cc:482] T 69f88532952a4591bf1577541c3430e6 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:54.126762 21437 raft_consensus.cc:504] T 69f88532952a4591bf1577541c3430e6 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:54.127516 21437 leader_election.cc:288] T 69f88532952a4591bf1577541c3430e6 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:54.129043 21439 raft_consensus.cc:482] T 13c2e69ae91f4957887fd64b90f6e435 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:54.129212 21439 raft_consensus.cc:504] T 13c2e69ae91f4957887fd64b90f6e435 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:54.129762 21439 leader_election.cc:288] T 13c2e69ae91f4957887fd64b90f6e435 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:54.134960 18696 leader_election.cc:302] T 1050ab528c7d41559831652a1437d358 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:54.145217 18453 tablet_service.cc:1514] Processing DeleteTablet for tablet b22aaedb0a6c4fd9ad0ea8fa019001e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:42 UTC) from {username='slave'} at 127.0.0.1:41450
W20240430 20:11:54.106556 19945 catalog_manager.cc:5676] Failed to send AlterTable RPC for tablet dac141d1b70c48d785caddd7182811ee (table default.Table_fc0a4540f98b4eea95099b8ab46ea235 [id=f3c53ccffe64410495cc8c379a05cfae], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet dac141d1b70c48d785caddd7182811ee consensus state has no leader
I20240430 20:11:54.149197 21454 raft_consensus.cc:2781] T 1050ab528c7d41559831652a1437d358 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:54.149621 21454 raft_consensus.cc:482] T 1050ab528c7d41559831652a1437d358 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:54.149957 21454 raft_consensus.cc:3037] T 1050ab528c7d41559831652a1437d358 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:54.151664 21481 tablet.cc:1722] T 22fef8bb6b034db098c3372ae992f7a7 P 5e1c6397f09747f091534faf93542710: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240430 20:11:54.119931 19941 catalog_manager.cc:5484] T 6b51cbc731794083a88fe4bed1ddeebf P 3c51c33d5616482cafb68e26688820f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c51c33d5616482cafb68e26688820f4 (127.17.177.3). New cstate: current_term: 1 leader_uuid: "3c51c33d5616482cafb68e26688820f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } health_report { overall_health: UNKNOWN } } }
I20240430 20:11:54.155231 19941 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 6b51cbc731794083a88fe4bed1ddeebf (table default.Table_c6c962da746e4fd8a4d7200d09f08e9e [id=8e0f5d8c941d423ea7b30a89513d5170]). Expected version 1 got 0
I20240430 20:11:54.155745 19941 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet ed0c37a71ede4bfc9eb5ad1a4365f25e (table default.Table_a2e85efc6b454524b0b122102ec07002 [id=86d795a82e674d859104741ebd04f4e8]). Expected version 1 got 0
I20240430 20:11:54.156212 19941 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 130007acc8ba4e709c3eb85fabe75903 (table default.Table_78987875650940c293247bc43db345ab [id=36f478a20a1f4a53a52fc29ade62b7f6]). Expected version 1 got 0
I20240430 20:11:54.120512 19937 catalog_manager.cc:6554] ReplaceTablet: tablet cefce2f2a392495bb71399ed75db938c deleted and replaced by tablet 9e1fed3070794d8da6e31692adfcb8cc
I20240430 20:11:54.162376 19937 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:50746 (request call id 75) took 3080 ms. Trace:
I20240430 20:11:54.162497 19937 rpcz_store.cc:276] 0430 20:11:51.081841 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:53.988355 (+2906514us) service_pool.cc:225] Handling call
0430 20:11:54.162302 (+173947us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:54.062565 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:54.062717 (+ 152us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:54.062723 (+ 6us) write_op.cc:427] Acquired schema lock
0430 20:11:54.062726 (+ 3us) tablet.cc:660] Decoding operations
0430 20:11:54.062873 (+ 147us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:54.062889 (+ 16us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:54.062895 (+ 6us) tablet.cc:683] Acquiring locks for 2 operations
0430 20:11:54.063244 (+ 349us) tablet.cc:699] Row locks acquired
0430 20:11:54.063247 (+ 3us) write_op.cc:260] PREPARE: finished
0430 20:11:54.063290 (+ 43us) write_op.cc:270] Start()
0430 20:11:54.063330 (+ 40us) write_op.cc:276] Timestamp: P: 1714507914063233 usec, L: 0
0430 20:11:54.063334 (+ 4us) op_driver.cc:331] REPLICATION: starting
0430 20:11:54.063549 (+ 215us) log.cc:842] Serialized 625 byte log entry
0430 20:11:54.068820 (+ 5271us) op_driver.cc:444] REPLICATION: finished
0430 20:11:54.069328 (+ 508us) write_op.cc:301] APPLY: starting
0430 20:11:54.069348 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:54.069400 (+ 52us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:54.069429 (+ 29us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:54.069542 (+ 113us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:54.069614 (+ 72us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0430 20:11:54.069622 (+ 8us) write_op.cc:312] APPLY: finished
0430 20:11:54.069702 (+ 80us) log.cc:842] Serialized 29 byte log entry
0430 20:11:54.069771 (+ 69us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:54.069792 (+ 21us) write_op.cc:446] Released schema lock
0430 20:11:54.069799 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:54.069872 (+ 73us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:54.069876 (+ 4us) write_op.cc:446] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":443,"apply.run_cpu_time_us":614,"apply.run_wall_time_us":614,"num_ops":2,"prepare.queue_time_us":8218,"prepare.run_cpu_time_us":860,"prepare.run_wall_time_us":1102,"raft.queue_time_us":1072,"raft.run_cpu_time_us":318,"raft.run_wall_time_us":316,"replication_time_us":5462}]]}
I20240430 20:11:54.120658 19942 rpcz_store.cc:276] 0430 20:11:51.051664 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:53.981995 (+2930331us) service_pool.cc:225] Handling call
0430 20:11:53.983129 (+ 1134us) catalog_manager.cc:2072] Acquired catalog manager lock
0430 20:11:53.983588 (+ 459us) catalog_manager.cc:2118] Created new table and tablet info
0430 20:11:54.071784 (+ 88196us) catalog_manager.cc:2178] Wrote table and tablets to system table
0430 20:11:54.071958 (+ 174us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0430 20:11:54.072081 (+ 123us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:54.063687 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:54.063783 (+ 96us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:54.064142 (+ 359us) write_op.cc:427] Acquired schema lock
0430 20:11:54.064148 (+ 6us) tablet.cc:660] Decoding operations
0430 20:11:54.064252 (+ 104us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:54.064261 (+ 9us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:54.064268 (+ 7us) tablet.cc:683] Acquiring locks for 2 operations
0430 20:11:54.064346 (+ 78us) tablet.cc:699] Row locks acquired
0430 20:11:54.064349 (+ 3us) write_op.cc:260] PREPARE: finished
0430 20:11:54.064381 (+ 32us) write_op.cc:270] Start()
0430 20:11:54.064411 (+ 30us) write_op.cc:276] Timestamp: P: 1714507914064325 usec, L: 0
0430 20:11:54.064415 (+ 4us) op_driver.cc:331] REPLICATION: starting
0430 20:11:54.064581 (+ 166us) log.cc:842] Serialized 433 byte log entry
0430 20:11:54.068907 (+ 4326us) op_driver.cc:444] REPLICATION: finished
0430 20:11:54.069967 (+ 1060us) write_op.cc:301] APPLY: starting
0430 20:11:54.069979 (+ 12us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:54.070016 (+ 37us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:54.070037 (+ 21us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:54.070133 (+ 96us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:54.070163 (+ 30us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0430 20:11:54.070171 (+ 8us) write_op.cc:312] APPLY: finished
0430 20:11:54.070261 (+ 90us) log.cc:842] Serialized 29 byte log entry
0430 20:11:54.070304 (+ 43us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:54.070321 (+ 17us) write_op.cc:446] Released schema lock
0430 20:11:54.070326 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:54.070372 (+ 46us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:54.070376 (+ 4us) write_op.cc:446] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1028,"apply.run_cpu_time_us":447,"apply.run_wall_time_us":446,"mutex_wait_us":7521,"num_ops":2,"prepare.queue_time_us":8664,"prepare.run_cpu_time_us":639,"prepare.run_wall_time_us":971,"raft.queue_time_us":422,"raft.run_cpu_time_us":4,"raft.run_wall_time_us":4,"replication_time_us":4472,"threads_started":1}]]}
I20240430 20:11:54.166623 21483 tablet.cc:1722] T a482f4622f774d60b4efd6827b8be7d7 P 5e1c6397f09747f091534faf93542710: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240430 20:11:54.168886 18988 tablet_bootstrap.cc:654] T 3b7fdc6ec7e046c5af894f9a6eda17b0 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:54.126654 21458 raft_consensus.cc:504] T 13c2e69ae91f4957887fd64b90f6e435 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
W20240430 20:11:54.180449 19945 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.17.177.1:47345 (request call id 134) took 2976 ms (client timeout 1000 ms). Trace:
I20240430 20:11:54.188917 18616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dac8b3838954d839094021f1c0503da" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:54.189363 18616 raft_consensus.cc:3037] T 4dac8b3838954d839094021f1c0503da P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:54.193131 18616 raft_consensus.cc:2454] T 4dac8b3838954d839094021f1c0503da P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 1.
I20240430 20:11:54.198258 21395 ts_tablet_manager.cc:1888] T a692b253db914531b8fc6ffaf3e350b5 P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:54.194768 18725 tablet_service.cc:1467] Processing CreateTablet for tablet 3aa5853f572c4989919f3b9ccb7f56ab (DEFAULT_TABLE table=default.Table_7b102da111ef451aba55ec04c1215cc7 [id=699c3d4ff0334398821693f3434ec15b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:54.203240 18725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3aa5853f572c4989919f3b9ccb7f56ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:54.206018 18724 tablet_service.cc:1467] Processing CreateTablet for tablet 92956889b2284975ba22523fe10d4ded (DEFAULT_TABLE table=default.Table_663f87c0ffdd472dba96f9c67665e8d6 [id=c902110d321b4286b161cbf9a03e47f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:54.206494 18724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92956889b2284975ba22523fe10d4ded. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:54.209630 19937 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50746:
table { table_name: "default.Table_dfb59f35018441fd9a72ea860cdd01f6" } modify_external_catalogs: true
I20240430 20:11:54.209964 19937 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50746:
table { table_name: "default.Table_dfb59f35018441fd9a72ea860cdd01f6" } modify_external_catalogs: true
I20240430 20:11:54.213297 19938 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: default.Table_c8b25b14d955429cbd79515081deceb1 [id=f393bcc9669c4ea985a2d40deb161f0c] alter complete (version 2)
W20240430 20:11:54.234328 19940 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.17.177.1:47345 (request call id 135) took 1932 ms (client timeout 1000 ms). Trace:
W20240430 20:11:54.236557 19940 rpcz_store.cc:269] 0430 20:11:52.302296 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:52.496585 (+194289us) service_pool.cc:225] Handling call
0430 20:11:53.457182 (+960597us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:53.457192 (+ 10us) write_op.cc:446] Released schema lock
0430 20:11:54.234256 (+777064us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:53.446994 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:53.447119 (+ 125us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:53.447126 (+ 7us) write_op.cc:427] Acquired schema lock
0430 20:11:53.447131 (+ 5us) tablet.cc:660] Decoding operations
0430 20:11:53.447257 (+ 126us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:53.447271 (+ 14us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:53.447276 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0430 20:11:53.447396 (+ 120us) tablet.cc:699] Row locks acquired
0430 20:11:53.447400 (+ 4us) write_op.cc:260] PREPARE: finished
0430 20:11:53.447447 (+ 47us) write_op.cc:270] Start()
0430 20:11:53.447486 (+ 39us) write_op.cc:276] Timestamp: P: 1714507913447401 usec, L: 0
0430 20:11:53.447491 (+ 5us) op_driver.cc:331] REPLICATION: starting
0430 20:11:53.447699 (+ 208us) log.cc:842] Serialized 907 byte log entry
0430 20:11:53.451507 (+ 3808us) op_driver.cc:444] REPLICATION: finished
0430 20:11:53.451627 (+ 120us) write_op.cc:301] APPLY: starting
0430 20:11:53.451645 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:53.451694 (+ 49us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:53.451732 (+ 38us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:53.451806 (+ 74us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:53.451844 (+ 38us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0430 20:11:53.451852 (+ 8us) write_op.cc:312] APPLY: finished
0430 20:11:53.451929 (+ 77us) log.cc:842] Serialized 29 byte log entry
0430 20:11:53.451993 (+ 64us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:53.452019 (+ 26us) write_op.cc:446] Released schema lock
0430 20:11:53.452029 (+ 10us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":184474,"reported_tablets":215,"child_traces":[["op",{"apply.queue_time_us":69,"apply.run_cpu_time_us":465,"apply.run_wall_time_us":465,"num_ops":2,"prepare.queue_time_us":49,"prepare.run_cpu_time_us":841,"prepare.run_wall_time_us":842,"raft.queue_time_us":554,"raft.run_cpu_time_us":269,"raft.run_wall_time_us":270,"replication_time_us":3992}]]}
I20240430 20:11:54.241458 18607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "330453ddeea6456e8ef93fa92608b0d3" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:54.241777 18607 raft_consensus.cc:2454] T 330453ddeea6456e8ef93fa92608b0d3 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 0.
W20240430 20:11:54.192072 19945 rpcz_store.cc:269] 0430 20:11:51.203722 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:51.561595 (+357873us) service_pool.cc:225] Handling call
0430 20:11:54.180391 (+2618796us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:52.926566 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:52.926729 (+ 163us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:52.926736 (+ 7us) write_op.cc:427] Acquired schema lock
0430 20:11:52.926739 (+ 3us) tablet.cc:660] Decoding operations
0430 20:11:52.926874 (+ 135us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:52.926892 (+ 18us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:52.926898 (+ 6us) tablet.cc:683] Acquiring locks for 2 operations
0430 20:11:52.927026 (+ 128us) tablet.cc:699] Row locks acquired
0430 20:11:52.927030 (+ 4us) write_op.cc:260] PREPARE: finished
0430 20:11:52.927080 (+ 50us) write_op.cc:270] Start()
0430 20:11:52.927118 (+ 38us) write_op.cc:276] Timestamp: P: 1714507912927038 usec, L: 0
0430 20:11:52.927122 (+ 4us) op_driver.cc:331] REPLICATION: starting
0430 20:11:52.927321 (+ 199us) log.cc:842] Serialized 907 byte log entry
0430 20:11:52.931018 (+ 3697us) op_driver.cc:444] REPLICATION: finished
0430 20:11:52.931146 (+ 128us) write_op.cc:301] APPLY: starting
0430 20:11:52.931164 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:52.931213 (+ 49us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:52.931240 (+ 27us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:52.931314 (+ 74us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:52.931351 (+ 37us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0430 20:11:52.931361 (+ 10us) write_op.cc:312] APPLY: finished
0430 20:11:52.931462 (+ 101us) log.cc:842] Serialized 29 byte log entry
0430 20:11:52.931532 (+ 70us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:52.931556 (+ 24us) write_op.cc:446] Released schema lock
0430 20:11:52.931563 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:52.931620 (+ 57us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:52.931624 (+ 4us) write_op.cc:446] Released schema lock
Metrics: {"mutex_wait_us":576615,"reported_tablets":201,"child_traces":[["op",{"apply.queue_time_us":74,"apply.run_cpu_time_us":530,"apply.run_wall_time_us":530,"num_ops":2,"prepare.queue_time_us":54,"prepare.run_cpu_time_us":854,"prepare.run_wall_time_us":855,"raft.queue_time_us":429,"raft.run_cpu_time_us":270,"raft.run_wall_time_us":270,"replication_time_us":3871}]]}
I20240430 20:11:54.273772 19945 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50746:
table { table_name: "default.Table_9110c0cb4db441dbb8bdce422eadc189" } new_table_name: "default.Table_e3cea4c635274d37a44427ca00897fb9" modify_external_catalogs: true
I20240430 20:11:54.275120 21489 tablet.cc:1722] T 12795670f21447a8964fe1efc6e92398 P 5e1c6397f09747f091534faf93542710: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240430 20:11:54.275185 19940 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50746:
table { table_name: "default.Table_656db97355f64d7c8aa29b1352dfe9c8" } modify_external_catalogs: true
I20240430 20:11:54.280638 21481 tablet.cc:1722] T 7ce09fcbe1ac457a9c05efaf000c3263 P 5e1c6397f09747f091534faf93542710: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240430 20:11:54.286362 21487 tablet.cc:1722] T 0368d37dd9e2452982fb81f539c6805b P 5e1c6397f09747f091534faf93542710: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240430 20:11:54.292090 18726 tablet_service.cc:1467] Processing CreateTablet for tablet 293283469725407993ec9351d8b29054 (DEFAULT_TABLE table=default.Table_1a823c2465ce4224a3cf173600f6e32c [id=3ad80fb32e2c44d7800834b09d1ace7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:54.292528 18726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 293283469725407993ec9351d8b29054. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:54.301231 18609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69f88532952a4591bf1577541c3430e6" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:54.303370 18609 raft_consensus.cc:2454] T 69f88532952a4591bf1577541c3430e6 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 0.
I20240430 20:11:54.309176 21458 leader_election.cc:288] T 13c2e69ae91f4957887fd64b90f6e435 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:54.313030 18733 tablet_service.cc:1467] Processing CreateTablet for tablet a06572a8df774c508923a9cb160f77d7 (DEFAULT_TABLE table=default.Table_8c21c0353321455a82097231a06f0957 [id=3b5c663f10a14933bac74b1cb9722e42]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:54.313432 18733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a06572a8df774c508923a9cb160f77d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:54.316290 18729 tablet_service.cc:1514] Processing DeleteTablet for tablet cefce2f2a392495bb71399ed75db938c with delete_type TABLET_DATA_DELETED (replaced by tablet 9e1fed3070794d8da6e31692adfcb8cc) from {username='slave'} at 127.0.0.1:56540
W20240430 20:11:54.317221 18696 leader_election.cc:341] T 13c2e69ae91f4957887fd64b90f6e435 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Not found: Tablet not found: 13c2e69ae91f4957887fd64b90f6e435
I20240430 20:11:54.322263 21481 tablet.cc:1722] T 81787eccca6c42c1b421377a3bf9ebe6 P 5e1c6397f09747f091534faf93542710: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
W20240430 20:11:54.330134 19895 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 37b2ea4be7494f0985808ade788dbe2b (table default.Table_5f6e0fce999b46de927f9271e5de5071 [id=0baad83a8c0f4aeebc7ddad068625a69], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 37b2ea4be7494f0985808ade788dbe2b consensus state has no leader
I20240430 20:11:54.342151 18983 tablet_bootstrap.cc:492] T a0d3c4ba64024bccad95ecc909d03ae0 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:54.342154 21396 tablet_replica.cc:330] T 3e5225f1f5f544ecbf8dfa470af3daca P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:54.344024 18983 tablet_bootstrap.cc:654] T a0d3c4ba64024bccad95ecc909d03ae0 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:54.345710 18595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3aa5853f572c4989919f3b9ccb7f56ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:54.347584 18601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92956889b2284975ba22523fe10d4ded. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:54.350788 18592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a06572a8df774c508923a9cb160f77d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:54.354218 21396 raft_consensus.cc:2229] T 3e5225f1f5f544ecbf8dfa470af3daca P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Raft consensus shutting down.
I20240430 20:11:54.354490 21396 pending_rounds.cc:62] T 3e5225f1f5f544ecbf8dfa470af3daca P 5e1c6397f09747f091534faf93542710: Trying to abort 1 pending ops.
I20240430 20:11:54.354549 21396 pending_rounds.cc:68] T 3e5225f1f5f544ecbf8dfa470af3daca P 5e1c6397f09747f091534faf93542710: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7022624414983913472 op_type: ALTER_SCHEMA_OP alter_schema_request { tablet_id: "3e5225f1f5f544ecbf8dfa470af3daca" schema { columns { id: 10 name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } schema_version: 1 new_table_name: "default.Table_c7c87a8e019944cab3ec99e9563ac01a" dest_uuid: "5e1c6397f09747f091534faf93542710" new_extra_config { } }
W20240430 20:11:54.354990 21396 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45934: Aborted: Op aborted
I20240430 20:11:54.355139 18603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 293283469725407993ec9351d8b29054. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:54.355270 21396 raft_consensus.cc:2258] T 3e5225f1f5f544ecbf8dfa470af3daca P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:54.356325 21396 ts_tablet_manager.cc:1875] T 3e5225f1f5f544ecbf8dfa470af3daca P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:54.366611 21396 ts_tablet_manager.cc:1888] T 3e5225f1f5f544ecbf8dfa470af3daca P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240430 20:11:54.368125 21396 log.cc:1196] T 3e5225f1f5f544ecbf8dfa470af3daca P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/3e5225f1f5f544ecbf8dfa470af3daca
I20240430 20:11:54.368997 21396 ts_tablet_manager.cc:1909] T 3e5225f1f5f544ecbf8dfa470af3daca P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:54.371443 21396 tablet_replica.cc:330] T 2aa2a489116940dea3173cad16d94fa5 P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:54.372754 21476 tablet.cc:1722] T a482f4622f774d60b4efd6827b8be7d7 P 3c51c33d5616482cafb68e26688820f4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240430 20:11:54.373701 21458 raft_consensus.cc:482] T a0d3c4ba64024bccad95ecc909d03ae0 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:54.373858 21463 raft_consensus.cc:482] T 330453ddeea6456e8ef93fa92608b0d3 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:54.373980 21463 raft_consensus.cc:504] T 330453ddeea6456e8ef93fa92608b0d3 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:54.373865 21458 raft_consensus.cc:504] T a0d3c4ba64024bccad95ecc909d03ae0 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:54.377928 21475 tablet.cc:1722] T 130007acc8ba4e709c3eb85fabe75903 P 3c51c33d5616482cafb68e26688820f4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240430 20:11:54.378806 21458 leader_election.cc:288] T a0d3c4ba64024bccad95ecc909d03ae0 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:54.382755 19895 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet a692b253db914531b8fc6ffaf3e350b5 (table default.Table_96aea8c4ff80428da8a0394f1bf7b493 [id=03ccb721140c4e0193e2e48d509430f9]) successfully deleted
W20240430 20:11:54.385813 19895 catalog_manager.cc:4909] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): alter failed for tablet 3e5225f1f5f544ecbf8dfa470af3daca (table default.Table_c7c87a8e019944cab3ec99e9563ac01a [id=abf3765074eb4d7d9d2bcc4ab721e6bd]): Aborted: Op aborted
I20240430 20:11:54.386312 21454 raft_consensus.cc:504] T 1050ab528c7d41559831652a1437d358 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:54.386421 19895 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet 3e5225f1f5f544ecbf8dfa470af3daca (table default.Table_c7c87a8e019944cab3ec99e9563ac01a [id=abf3765074eb4d7d9d2bcc4ab721e6bd]) successfully deleted
I20240430 20:11:54.397354 21463 leader_election.cc:288] T 330453ddeea6456e8ef93fa92608b0d3 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:54.397766 21454 leader_election.cc:288] T 1050ab528c7d41559831652a1437d358 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Requested vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:54.404086 21396 raft_consensus.cc:2229] T 2aa2a489116940dea3173cad16d94fa5 P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Raft consensus shutting down.
I20240430 20:11:54.404502 21396 raft_consensus.cc:2258] T 2aa2a489116940dea3173cad16d94fa5 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:54.405284 21396 ts_tablet_manager.cc:1875] T 2aa2a489116940dea3173cad16d94fa5 P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:54.222923 18477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086b3dfa84c045a18c8c4f177dce606a" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:54.406672 18477 raft_consensus.cc:2454] T 086b3dfa84c045a18c8c4f177dce606a P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 0.
I20240430 20:11:54.295569 21054 raft_consensus.cc:482] T 95383d9d5a234a4d9d841a8b114428b0 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:54.407138 21054 raft_consensus.cc:504] T 95383d9d5a234a4d9d841a8b114428b0 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:54.407883 21054 leader_election.cc:288] T 95383d9d5a234a4d9d841a8b114428b0 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:54.316481 18985 tablet_bootstrap.cc:492] T 1050ab528c7d41559831652a1437d358 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:54.408186 18985 ts_tablet_manager.cc:1367] T 1050ab528c7d41559831652a1437d358 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.417s user 0.002s sys 0.006s
I20240430 20:11:54.409001 18985 raft_consensus.cc:348] T 1050ab528c7d41559831652a1437d358 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:54.409292 18985 raft_consensus.cc:374] T 1050ab528c7d41559831652a1437d358 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:54.409392 18985 raft_consensus.cc:729] T 1050ab528c7d41559831652a1437d358 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:54.409649 18985 consensus_queue.cc:260] T 1050ab528c7d41559831652a1437d358 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:54.410409 18985 ts_tablet_manager.cc:1398] T 1050ab528c7d41559831652a1437d358 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240430 20:11:54.410754 18985 tablet_bootstrap.cc:492] T 3b7fdc6ec7e046c5af894f9a6eda17b0 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:54.411324 21306 consensus_queue.cc:237] T 3191aa54a9644783a2403991c3be27c8 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:54.412382 18985 tablet_bootstrap.cc:654] T 3b7fdc6ec7e046c5af894f9a6eda17b0 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:54.401633 18415 leader_election.cc:302] T 4dac8b3838954d839094021f1c0503da P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:54.415969 21054 raft_consensus.cc:2781] T 4dac8b3838954d839094021f1c0503da P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:54.416191 21054 raft_consensus.cc:686] T 4dac8b3838954d839094021f1c0503da P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Becoming Leader. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Running, Role: LEADER
I20240430 20:11:54.416491 21054 consensus_queue.cc:237] T 4dac8b3838954d839094021f1c0503da P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:54.416577 21476 tablet.cc:1722] T 12795670f21447a8964fe1efc6e92398 P 3c51c33d5616482cafb68e26688820f4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240430 20:11:54.418673 21397 ts_tablet_manager.cc:1888] T a692b253db914531b8fc6ffaf3e350b5 P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:54.418880 21397 log.cc:1196] T a692b253db914531b8fc6ffaf3e350b5 P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/a692b253db914531b8fc6ffaf3e350b5
I20240430 20:11:54.419265 21397 ts_tablet_manager.cc:1909] T a692b253db914531b8fc6ffaf3e350b5 P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:54.419993 21397 tablet_replica.cc:330] T 1f4e860484784aa9a2d28af2e9fb611c P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:54.419962 19900 catalog_manager.cc:4830] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): tablet a692b253db914531b8fc6ffaf3e350b5 (table default.Table_96aea8c4ff80428da8a0394f1bf7b493 [id=03ccb721140c4e0193e2e48d509430f9]) successfully deleted
I20240430 20:11:54.420362 21397 raft_consensus.cc:2229] T 1f4e860484784aa9a2d28af2e9fb611c P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:54.420606 21397 raft_consensus.cc:2258] T 1f4e860484784aa9a2d28af2e9fb611c P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:54.421326 21395 log.cc:1196] T a692b253db914531b8fc6ffaf3e350b5 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/a692b253db914531b8fc6ffaf3e350b5
I20240430 20:11:54.421779 21395 ts_tablet_manager.cc:1909] T a692b253db914531b8fc6ffaf3e350b5 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:54.374488 21489 tablet.cc:1722] T 130007acc8ba4e709c3eb85fabe75903 P 5e1c6397f09747f091534faf93542710: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240430 20:11:54.381659 21467 raft_consensus.cc:482] T 347a8224114b49e5b7dcb75d9b726e0a P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:54.423467 21467 raft_consensus.cc:504] T 347a8224114b49e5b7dcb75d9b726e0a P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:54.426975 21467 leader_election.cc:288] T 347a8224114b49e5b7dcb75d9b726e0a P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:54.429082 18557 leader_election.cc:302] T 086b3dfa84c045a18c8c4f177dce606a P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:54.429661 20489 raft_consensus.cc:2781] T 086b3dfa84c045a18c8c4f177dce606a P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:54.429759 20489 raft_consensus.cc:482] T 086b3dfa84c045a18c8c4f177dce606a P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:54.429836 20489 raft_consensus.cc:3037] T 086b3dfa84c045a18c8c4f177dce606a P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:54.431851 18617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1050ab528c7d41559831652a1437d358" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:54.432018 18617 raft_consensus.cc:3037] T 1050ab528c7d41559831652a1437d358 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:54.432303 20489 raft_consensus.cc:504] T 086b3dfa84c045a18c8c4f177dce606a P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:54.432945 20489 leader_election.cc:288] T 086b3dfa84c045a18c8c4f177dce606a P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:54.434247 21487 tablet.cc:1722] T c6e3d09402574f1e8fe9e4160c70a3ee P 5e1c6397f09747f091534faf93542710: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240430 20:11:54.420902 21496 tablet.cc:1722] T 22fef8bb6b034db098c3372ae992f7a7 P 3c51c33d5616482cafb68e26688820f4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240430 20:11:54.421756 21475 tablet.cc:1722] T c6e3d09402574f1e8fe9e4160c70a3ee P 3c51c33d5616482cafb68e26688820f4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240430 20:11:54.432009 21454 raft_consensus.cc:482] T 2cdccf4be7094ca5b863dd5c3432d55d P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:54.443413 21454 raft_consensus.cc:504] T 2cdccf4be7094ca5b863dd5c3432d55d P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:54.453366 21397 ts_tablet_manager.cc:1875] T 1f4e860484784aa9a2d28af2e9fb611c P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:54.455710 21463 raft_consensus.cc:482] T 69f88532952a4591bf1577541c3430e6 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:54.471674 21395 tablet_replica.cc:330] T 3e5225f1f5f544ecbf8dfa470af3daca P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:54.471969 21395 raft_consensus.cc:2229] T 3e5225f1f5f544ecbf8dfa470af3daca P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:54.472121 21395 pending_rounds.cc:62] T 3e5225f1f5f544ecbf8dfa470af3daca P 2efb542aa55e439b99ccb98d1c5c884a: Trying to abort 1 pending ops.
I20240430 20:11:54.472173 21395 pending_rounds.cc:68] T 3e5225f1f5f544ecbf8dfa470af3daca P 2efb542aa55e439b99ccb98d1c5c884a: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7022624414983913472 op_type: ALTER_SCHEMA_OP alter_schema_request { tablet_id: "3e5225f1f5f544ecbf8dfa470af3daca" schema { columns { id: 10 name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } schema_version: 1 new_table_name: "default.Table_c7c87a8e019944cab3ec99e9563ac01a" dest_uuid: "5e1c6397f09747f091534faf93542710" new_extra_config { } }
I20240430 20:11:54.472716 21395 raft_consensus.cc:2258] T 3e5225f1f5f544ecbf8dfa470af3daca P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:54.422724 18415 leader_election.cc:302] T 330453ddeea6456e8ef93fa92608b0d3 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
W20240430 20:11:54.473798 18415 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.177.2:46027, user_credentials={real_user=slave}} blocked reactor thread for 51099us
I20240430 20:11:54.474164 21439 raft_consensus.cc:2781] T 330453ddeea6456e8ef93fa92608b0d3 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:54.474249 21439 raft_consensus.cc:482] T 330453ddeea6456e8ef93fa92608b0d3 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:54.474308 21439 raft_consensus.cc:3037] T 330453ddeea6456e8ef93fa92608b0d3 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:54.476884 21439 raft_consensus.cc:504] T 330453ddeea6456e8ef93fa92608b0d3 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:54.478051 21439 leader_election.cc:288] T 330453ddeea6456e8ef93fa92608b0d3 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:54.455917 21463 raft_consensus.cc:504] T 69f88532952a4591bf1577541c3430e6 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:54.434410 18617 raft_consensus.cc:2454] T 1050ab528c7d41559831652a1437d358 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:54.479036 21463 leader_election.cc:288] T 69f88532952a4591bf1577541c3430e6 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:54.434774 18616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0d3c4ba64024bccad95ecc909d03ae0" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:54.480034 18696 leader_election.cc:302] T 1050ab528c7d41559831652a1437d358 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:54.480409 21463 raft_consensus.cc:2781] T 1050ab528c7d41559831652a1437d358 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:54.480547 21463 raft_consensus.cc:686] T 1050ab528c7d41559831652a1437d358 P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Becoming Leader. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Running, Role: LEADER
W20240430 20:11:54.480938 18696 leader_election.cc:341] T a0d3c4ba64024bccad95ecc909d03ae0 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:54.481428 18616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69f88532952a4591bf1577541c3430e6" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:54.481745 18616 raft_consensus.cc:2454] T 69f88532952a4591bf1577541c3430e6 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
W20240430 20:11:54.482263 18557 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.177.3:44867, user_credentials={real_user=slave}} blocked reactor thread for 48366.5us
I20240430 20:11:54.483017 21487 tablet.cc:1722] T eb6c96047b114ebba73db7a371f2da8e P 5e1c6397f09747f091534faf93542710: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240430 20:11:54.436022 18607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "330453ddeea6456e8ef93fa92608b0d3" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:54.485200 18607 raft_consensus.cc:2454] T 330453ddeea6456e8ef93fa92608b0d3 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:54.436241 21481 tablet.cc:1722] T 62732af95c6a4c749c5b726f0460dc72 P 5e1c6397f09747f091534faf93542710: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240430 20:11:54.488426 18471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1050ab528c7d41559831652a1437d358" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:54.488726 18471 raft_consensus.cc:2454] T 1050ab528c7d41559831652a1437d358 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:54.468416 21467 raft_consensus.cc:482] T e5b85a066cdc46d48ccf8201e5423c29 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:54.437376 18983 tablet_bootstrap.cc:492] T a0d3c4ba64024bccad95ecc909d03ae0 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:54.490685 18983 ts_tablet_manager.cc:1367] T a0d3c4ba64024bccad95ecc909d03ae0 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.149s user 0.006s sys 0.011s
I20240430 20:11:54.490612 21467 raft_consensus.cc:504] T e5b85a066cdc46d48ccf8201e5423c29 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:54.491418 21467 leader_election.cc:288] T e5b85a066cdc46d48ccf8201e5423c29 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:54.478204 21396 ts_tablet_manager.cc:1888] T 2aa2a489116940dea3173cad16d94fa5 P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:54.491714 21396 log.cc:1196] T 2aa2a489116940dea3173cad16d94fa5 P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/2aa2a489116940dea3173cad16d94fa5
I20240430 20:11:54.492260 21396 ts_tablet_manager.cc:1909] T 2aa2a489116940dea3173cad16d94fa5 P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:54.492835 21396 tablet_replica.cc:330] T 6db55bb510b24b0cae897936211e15f0 P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:54.493196 21396 raft_consensus.cc:2229] T 6db55bb510b24b0cae897936211e15f0 P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Raft consensus shutting down.
I20240430 20:11:54.493505 21396 raft_consensus.cc:2258] T 6db55bb510b24b0cae897936211e15f0 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:54.496016 21436 raft_consensus.cc:482] T a663030d37614053a7c0369841403422 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:54.496296 21436 raft_consensus.cc:504] T a663030d37614053a7c0369841403422 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:54.500622 19940 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50746:
table { table_name: "default.Table_656db97355f64d7c8aa29b1352dfe9c8" } modify_external_catalogs: true
I20240430 20:11:54.502190 20630 raft_consensus.cc:482] T 9c73d0a6bc414d3b9900570848d19a51 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:54.506779 18983 raft_consensus.cc:348] T a0d3c4ba64024bccad95ecc909d03ae0 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:54.507020 18983 raft_consensus.cc:374] T a0d3c4ba64024bccad95ecc909d03ae0 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:54.507081 18983 raft_consensus.cc:729] T a0d3c4ba64024bccad95ecc909d03ae0 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:54.507318 18983 consensus_queue.cc:260] T a0d3c4ba64024bccad95ecc909d03ae0 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:54.507695 21487 tablet.cc:1722] T ed0c37a71ede4bfc9eb5ad1a4365f25e P 5e1c6397f09747f091534faf93542710: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240430 20:11:54.507921 18983 ts_tablet_manager.cc:1398] T a0d3c4ba64024bccad95ecc909d03ae0 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.017s user 0.002s sys 0.000s
I20240430 20:11:54.508208 18983 tablet_bootstrap.cc:492] T f8bfbf673ea346529d1b2b376a3da496 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:54.509883 18983 tablet_bootstrap.cc:654] T f8bfbf673ea346529d1b2b376a3da496 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:54.506783 21396 ts_tablet_manager.cc:1875] T 6db55bb510b24b0cae897936211e15f0 P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:54.519759 21487 tablet.cc:1722] T acd23f0a63d5448aaa7ed78422e67bad P 5e1c6397f09747f091534faf93542710: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240430 20:11:54.522840 18745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dac8b3838954d839094021f1c0503da" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:54.523080 18745 raft_consensus.cc:3037] T 4dac8b3838954d839094021f1c0503da P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:54.524766 21454 leader_election.cc:288] T 2cdccf4be7094ca5b863dd5c3432d55d P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:54.527473 21476 tablet.cc:1722] T 7ce09fcbe1ac457a9c05efaf000c3263 P 3c51c33d5616482cafb68e26688820f4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240430 20:11:54.527834 18415 leader_election.cc:302] T 69f88532952a4591bf1577541c3430e6 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
W20240430 20:11:54.529672 18696 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.177.2:46027, user_credentials={real_user=slave}} blocked reactor thread for 48735.5us
I20240430 20:11:54.530617 21463 consensus_queue.cc:237] T 1050ab528c7d41559831652a1437d358 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:54.531708 18696 leader_election.cc:302] T 69f88532952a4591bf1577541c3430e6 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:54.531996 21454 raft_consensus.cc:2781] T 69f88532952a4591bf1577541c3430e6 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:54.532071 21454 raft_consensus.cc:482] T 69f88532952a4591bf1577541c3430e6 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:54.532143 21454 raft_consensus.cc:3037] T 69f88532952a4591bf1577541c3430e6 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:54.533147 18696 leader_election.cc:302] T 330453ddeea6456e8ef93fa92608b0d3 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:54.535099 21395 ts_tablet_manager.cc:1875] T 3e5225f1f5f544ecbf8dfa470af3daca P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:54.540427 21054 raft_consensus.cc:482] T a663030d37614053a7c0369841403422 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:54.540670 21054 raft_consensus.cc:504] T a663030d37614053a7c0369841403422 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
W20240430 20:11:54.547165 18696 leader_election.cc:341] T 2cdccf4be7094ca5b863dd5c3432d55d P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Not found: Tablet not found: 2cdccf4be7094ca5b863dd5c3432d55d
I20240430 20:11:54.548704 20587 raft_consensus.cc:2781] T 330453ddeea6456e8ef93fa92608b0d3 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:54.548818 20587 raft_consensus.cc:482] T 330453ddeea6456e8ef93fa92608b0d3 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:54.548899 20587 raft_consensus.cc:3037] T 330453ddeea6456e8ef93fa92608b0d3 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:54.553525 18745 raft_consensus.cc:2454] T 4dac8b3838954d839094021f1c0503da P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 1.
I20240430 20:11:54.555333 21396 ts_tablet_manager.cc:1888] T 6db55bb510b24b0cae897936211e15f0 P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:54.557013 21436 leader_election.cc:288] T a663030d37614053a7c0369841403422 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:54.563472 20630 raft_consensus.cc:504] T 9c73d0a6bc414d3b9900570848d19a51 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:54.564194 20630 leader_election.cc:288] T 9c73d0a6bc414d3b9900570848d19a51 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
W20240430 20:11:54.564440 18415 leader_election.cc:341] T 13c2e69ae91f4957887fd64b90f6e435 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Not found: Tablet not found: 13c2e69ae91f4957887fd64b90f6e435
I20240430 20:11:54.564726 21439 raft_consensus.cc:2781] T 69f88532952a4591bf1577541c3430e6 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:54.564797 21439 raft_consensus.cc:482] T 69f88532952a4591bf1577541c3430e6 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:54.564884 21439 raft_consensus.cc:3037] T 69f88532952a4591bf1577541c3430e6 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:54.567170 21054 leader_election.cc:288] T a663030d37614053a7c0369841403422 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:54.575716 21496 tablet.cc:1722] T 0368d37dd9e2452982fb81f539c6805b P 3c51c33d5616482cafb68e26688820f4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20240430 20:11:54.576735 21488 log.cc:924] Time spent T ae20f44df50343289b0a512a4ea53692 P 3c51c33d5616482cafb68e26688820f4: Append to log took a long time: real 0.164s user 0.000s sys 0.000s
I20240430 20:11:54.578691 21436 consensus_queue.cc:1035] T 2797eed593fb4b06a3c4d6a28e8075a2 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:54.587745 21454 raft_consensus.cc:504] T 69f88532952a4591bf1577541c3430e6 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:54.591605 21454 leader_election.cc:288] T 69f88532952a4591bf1577541c3430e6 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Requested vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:54.603556 21475 tablet.cc:1722] T 81787eccca6c42c1b421377a3bf9ebe6 P 3c51c33d5616482cafb68e26688820f4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240430 20:11:54.609169 18761 raft_consensus.cc:1261] T 9f0b341a31224bf1be525b84b9500304 P 3c51c33d5616482cafb68e26688820f4 [term 2 FOLLOWER]: Refusing update from remote peer 2efb542aa55e439b99ccb98d1c5c884a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20240430 20:11:54.607434 20587 raft_consensus.cc:504] T 330453ddeea6456e8ef93fa92608b0d3 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:54.610188 20587 leader_election.cc:288] T 330453ddeea6456e8ef93fa92608b0d3 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Requested vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:54.615777 21454 raft_consensus.cc:482] T 9c73d0a6bc414d3b9900570848d19a51 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:54.616040 21454 raft_consensus.cc:504] T 9c73d0a6bc414d3b9900570848d19a51 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:54.568912 21535 tablet.cc:1722] T 724e51ff05ec4baebc2bb55e05c8eddc P 2efb542aa55e439b99ccb98d1c5c884a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240430 20:11:54.633826 21454 leader_election.cc:288] T 9c73d0a6bc414d3b9900570848d19a51 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
W20240430 20:11:54.635394 18696 leader_election.cc:341] T 9c73d0a6bc414d3b9900570848d19a51 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Not found: Tablet not found: 9c73d0a6bc414d3b9900570848d19a51
I20240430 20:11:54.636530 18760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69f88532952a4591bf1577541c3430e6" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:54.636837 18760 raft_consensus.cc:2379] T 69f88532952a4591bf1577541c3430e6 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2efb542aa55e439b99ccb98d1c5c884a in current term 1: Already voted for candidate 3c51c33d5616482cafb68e26688820f4 in this term.
I20240430 20:11:54.640826 21439 raft_consensus.cc:504] T 69f88532952a4591bf1577541c3430e6 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:54.640982 21486 tablet.cc:1722] T 6b51cbc731794083a88fe4bed1ddeebf P 3c51c33d5616482cafb68e26688820f4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240430 20:11:54.641575 18764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "330453ddeea6456e8ef93fa92608b0d3" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:54.641822 18764 raft_consensus.cc:2379] T 330453ddeea6456e8ef93fa92608b0d3 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2efb542aa55e439b99ccb98d1c5c884a in current term 1: Already voted for candidate 3c51c33d5616482cafb68e26688820f4 in this term.
I20240430 20:11:54.644145 21439 leader_election.cc:288] T 69f88532952a4591bf1577541c3430e6 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Requested vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:54.645956 18760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13c2e69ae91f4957887fd64b90f6e435" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:54.646279 18760 raft_consensus.cc:2454] T 13c2e69ae91f4957887fd64b90f6e435 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 0.
I20240430 20:11:54.655417 19895 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet 2aa2a489116940dea3173cad16d94fa5 (table default.Table_a39fa485d1e6488b951a9681aa5f3f00 [id=f02e978b8b7b49a38dc7e9d0cb406ad8]) successfully deleted
W20240430 20:11:54.655638 19895 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.17.177.2:46027, user_credentials={real_user=slave}} blocked reactor thread for 125261us
W20240430 20:11:54.656046 19895 catalog_manager.cc:4903] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): alter failed for tablet 3e5225f1f5f544ecbf8dfa470af3daca (table default.Table_c7c87a8e019944cab3ec99e9563ac01a [id=abf3765074eb4d7d9d2bcc4ab721e6bd]),no further retry: Not found: Tablet not found: 3e5225f1f5f544ecbf8dfa470af3daca
I20240430 20:11:54.662346 21535 tablet.cc:1722] T ca64de16d40f4a44b8b411f5526ba11a P 2efb542aa55e439b99ccb98d1c5c884a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20240430 20:11:54.666105 21450 thread.cc:602] apply [worker] (thread pool) Time spent starting thread: real 0.530s user 0.006s sys 0.014s
I20240430 20:11:54.643564 18473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13c2e69ae91f4957887fd64b90f6e435" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:54.666427 18473 raft_consensus.cc:2454] T 13c2e69ae91f4957887fd64b90f6e435 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:54.668080 21397 ts_tablet_manager.cc:1888] T 1f4e860484784aa9a2d28af2e9fb611c P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:54.668210 21397 log.cc:1196] T 1f4e860484784aa9a2d28af2e9fb611c P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/1f4e860484784aa9a2d28af2e9fb611c
I20240430 20:11:54.670869 21539 tablet.cc:1722] T 66dd767bd3f8494f94061fb89e5e5680 P 2efb542aa55e439b99ccb98d1c5c884a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240430 20:11:54.672153 18473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0d3c4ba64024bccad95ecc909d03ae0" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:54.557938 18607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a663030d37614053a7c0369841403422" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:54.577435 18983 tablet_bootstrap.cc:492] T f8bfbf673ea346529d1b2b376a3da496 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:54.592716 18616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69f88532952a4591bf1577541c3430e6" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:54.672834 18983 ts_tablet_manager.cc:1367] T f8bfbf673ea346529d1b2b376a3da496 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.165s user 0.008s sys 0.003s
I20240430 20:11:54.672812 18607 raft_consensus.cc:2454] T a663030d37614053a7c0369841403422 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:54.610729 18617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "330453ddeea6456e8ef93fa92608b0d3" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:54.673192 18617 raft_consensus.cc:3037] T 330453ddeea6456e8ef93fa92608b0d3 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:54.673489 18696 leader_election.cc:302] T a663030d37614053a7c0369841403422 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:54.673609 18983 raft_consensus.cc:348] T f8bfbf673ea346529d1b2b376a3da496 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:54.673745 21396 log.cc:1196] T 6db55bb510b24b0cae897936211e15f0 P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/6db55bb510b24b0cae897936211e15f0
I20240430 20:11:54.673777 18983 raft_consensus.cc:374] T f8bfbf673ea346529d1b2b376a3da496 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:54.673820 18983 raft_consensus.cc:729] T f8bfbf673ea346529d1b2b376a3da496 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:54.674016 18983 consensus_queue.cc:260] T f8bfbf673ea346529d1b2b376a3da496 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:54.674203 21396 ts_tablet_manager.cc:1909] T 6db55bb510b24b0cae897936211e15f0 P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:54.674541 18983 ts_tablet_manager.cc:1398] T f8bfbf673ea346529d1b2b376a3da496 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240430 20:11:54.670668 21487 tablet.cc:1722] T 6b51cbc731794083a88fe4bed1ddeebf P 5e1c6397f09747f091534faf93542710: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240430 20:11:54.675963 18617 raft_consensus.cc:2454] T 330453ddeea6456e8ef93fa92608b0d3 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:54.672803 18616 raft_consensus.cc:3037] T 69f88532952a4591bf1577541c3430e6 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:54.676905 18456 tablet_service.cc:1467] Processing CreateTablet for tablet 3aa5853f572c4989919f3b9ccb7f56ab (DEFAULT_TABLE table=default.Table_7b102da111ef451aba55ec04c1215cc7 [id=699c3d4ff0334398821693f3434ec15b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:54.677294 18456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3aa5853f572c4989919f3b9ccb7f56ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:54.675994 18473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1050ab528c7d41559831652a1437d358" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:54.679628 18473 raft_consensus.cc:3037] T 1050ab528c7d41559831652a1437d358 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:54.682349 18616 raft_consensus.cc:2454] T 69f88532952a4591bf1577541c3430e6 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:54.684528 18447 tablet_service.cc:1514] Processing DeleteTablet for tablet 1701a15b115741aea13f8327e3d26b34 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:50 UTC) from {username='slave'} at 127.0.0.1:41450
I20240430 20:11:54.686837 18983 tablet_bootstrap.cc:492] T 8668f7a285de43f48d83abecb4b2e96c P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:54.688962 18983 tablet_bootstrap.cc:654] T 8668f7a285de43f48d83abecb4b2e96c P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:54.688952 21395 ts_tablet_manager.cc:1888] T 3e5225f1f5f544ecbf8dfa470af3daca P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240430 20:11:54.689627 18985 tablet_bootstrap.cc:492] T 3b7fdc6ec7e046c5af894f9a6eda17b0 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
W20240430 20:11:54.689679 21177 log.cc:924] Time spent T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: Append to log took a long time: real 0.102s user 0.002s sys 0.000s
I20240430 20:11:54.689841 18985 ts_tablet_manager.cc:1367] T 3b7fdc6ec7e046c5af894f9a6eda17b0 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.279s user 0.005s sys 0.002s
I20240430 20:11:54.691742 18985 raft_consensus.cc:348] T 3b7fdc6ec7e046c5af894f9a6eda17b0 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:54.692250 18985 raft_consensus.cc:374] T 3b7fdc6ec7e046c5af894f9a6eda17b0 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:54.692487 18985 raft_consensus.cc:729] T 3b7fdc6ec7e046c5af894f9a6eda17b0 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:54.693105 18985 consensus_queue.cc:260] T 3b7fdc6ec7e046c5af894f9a6eda17b0 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:54.695714 21397 ts_tablet_manager.cc:1909] T 1f4e860484784aa9a2d28af2e9fb611c P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:54.696394 19900 catalog_manager.cc:4830] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): tablet 1f4e860484784aa9a2d28af2e9fb611c (table default.Table_45d50fc81325443097375ae6e65a1633 [id=9c861883de1d49c78146d65e9d4dd39f]) successfully deleted
I20240430 20:11:54.697661 21397 tablet_replica.cc:330] T 9f0b341a31224bf1be525b84b9500304 P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:54.697870 21397 raft_consensus.cc:2229] T 9f0b341a31224bf1be525b84b9500304 P 3c51c33d5616482cafb68e26688820f4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:54.698029 21397 raft_consensus.cc:2258] T 9f0b341a31224bf1be525b84b9500304 P 3c51c33d5616482cafb68e26688820f4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:54.699837 21397 ts_tablet_manager.cc:1875] T 9f0b341a31224bf1be525b84b9500304 P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:54.704248 20587 raft_consensus.cc:2781] T a663030d37614053a7c0369841403422 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:54.704535 18696 leader_election.cc:302] T 330453ddeea6456e8ef93fa92608b0d3 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:54.704795 18696 leader_election.cc:302] T 69f88532952a4591bf1577541c3430e6 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:54.704896 21450 raft_consensus.cc:2781] T 330453ddeea6456e8ef93fa92608b0d3 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:54.705415 18447 tablet_service.cc:1467] Processing CreateTablet for tablet 92956889b2284975ba22523fe10d4ded (DEFAULT_TABLE table=default.Table_663f87c0ffdd472dba96f9c67665e8d6 [id=c902110d321b4286b161cbf9a03e47f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:54.705456 18453 tablet_service.cc:1467] Processing CreateTablet for tablet 293283469725407993ec9351d8b29054 (DEFAULT_TABLE table=default.Table_1a823c2465ce4224a3cf173600f6e32c [id=3ad80fb32e2c44d7800834b09d1ace7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:54.705591 21493 raft_consensus.cc:2781] T 69f88532952a4591bf1577541c3430e6 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:54.705703 21493 raft_consensus.cc:686] T 69f88532952a4591bf1577541c3430e6 P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Becoming Leader. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Running, Role: LEADER
I20240430 20:11:54.705897 18447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92956889b2284975ba22523fe10d4ded. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:54.706137 18453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 293283469725407993ec9351d8b29054. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:54.711095 18458 tablet_service.cc:1514] Processing DeleteTablet for tablet cefce2f2a392495bb71399ed75db938c with delete_type TABLET_DATA_DELETED (replaced by tablet 9e1fed3070794d8da6e31692adfcb8cc) from {username='slave'} at 127.0.0.1:41450
I20240430 20:11:54.711118 18450 tablet_service.cc:1467] Processing CreateTablet for tablet a06572a8df774c508923a9cb160f77d7 (DEFAULT_TABLE table=default.Table_8c21c0353321455a82097231a06f0957 [id=3b5c663f10a14933bac74b1cb9722e42]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:54.711539 18458 tablet_service.cc:1514] Processing DeleteTablet for tablet 13c2e69ae91f4957887fd64b90f6e435 with delete_type TABLET_DATA_DELETED (replaced by tablet eba4bd4665ae4c04a2cab3ab1a2da780) from {username='slave'} at 127.0.0.1:41450
I20240430 20:11:54.711645 18450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a06572a8df774c508923a9cb160f77d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:54.711812 18458 tablet_service.cc:1514] Processing DeleteTablet for tablet f9bf604e6c234349a9f72f45a4c4aef6 with delete_type TABLET_DATA_DELETED (replaced by tablet 415d99c36b1941e08aa4a16c50483ee4) from {username='slave'} at 127.0.0.1:41450
I20240430 20:11:54.714470 18473 raft_consensus.cc:2454] T 1050ab528c7d41559831652a1437d358 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:54.704841 20587 raft_consensus.cc:482] T a663030d37614053a7c0369841403422 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:54.718158 21450 raft_consensus.cc:686] T 330453ddeea6456e8ef93fa92608b0d3 P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Becoming Leader. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Running, Role: LEADER
I20240430 20:11:54.719902 19895 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet 6db55bb510b24b0cae897936211e15f0 (table default.Table_3471b34c1c0d4d829bb27a8520eff502 [id=f3498e7e44974f65bbd2e92c6c8c119f]) successfully deleted
I20240430 20:11:54.720355 21548 tablet.cc:1722] T 2e79df8744e74b5991a79ce978ff0202 P 2efb542aa55e439b99ccb98d1c5c884a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240430 20:11:54.722620 20587 raft_consensus.cc:3037] T a663030d37614053a7c0369841403422 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:54.726963 21450 consensus_queue.cc:237] T 330453ddeea6456e8ef93fa92608b0d3 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:54.730314 18756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "330453ddeea6456e8ef93fa92608b0d3" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:54.730551 18748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a663030d37614053a7c0369841403422" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:54.730787 18762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c73d0a6bc414d3b9900570848d19a51" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:54.730930 18762 raft_consensus.cc:2454] T 9c73d0a6bc414d3b9900570848d19a51 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 1.
I20240430 20:11:54.731159 18747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69f88532952a4591bf1577541c3430e6" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:54.731798 18756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086b3dfa84c045a18c8c4f177dce606a" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:54.731958 18756 raft_consensus.cc:3037] T 086b3dfa84c045a18c8c4f177dce606a P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:54.732523 18747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "347a8224114b49e5b7dcb75d9b726e0a" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:54.732868 18747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5b85a066cdc46d48ccf8201e5423c29" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:54.733791 21496 tablet.cc:1722] T 62732af95c6a4c749c5b726f0460dc72 P 3c51c33d5616482cafb68e26688820f4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240430 20:11:54.735023 21475 tablet.cc:1722] T eb6c96047b114ebba73db7a371f2da8e P 3c51c33d5616482cafb68e26688820f4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240430 20:11:54.736975 21476 tablet.cc:1722] T ed0c37a71ede4bfc9eb5ad1a4365f25e P 3c51c33d5616482cafb68e26688820f4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240430 20:11:54.737686 21496 tablet.cc:1722] T acd23f0a63d5448aaa7ed78422e67bad P 3c51c33d5616482cafb68e26688820f4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240430 20:11:54.741889 21542 tablet.cc:1722] T 89b774967624498fa2be499878b71124 P 2efb542aa55e439b99ccb98d1c5c884a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240430 20:11:54.747449 21396 tablet_replica.cc:330] T c1017a50d1d646c486c0b9a75cb88ce4 P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:54.747864 21396 raft_consensus.cc:2229] T c1017a50d1d646c486c0b9a75cb88ce4 P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Raft consensus shutting down.
I20240430 20:11:54.748494 21396 raft_consensus.cc:2258] T c1017a50d1d646c486c0b9a75cb88ce4 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:54.749003 18753 raft_consensus.cc:1261] T df92eb85261d47d88cbf16d29d9ad30e P 3c51c33d5616482cafb68e26688820f4 [term 2 FOLLOWER]: Refusing update from remote peer 2efb542aa55e439b99ccb98d1c5c884a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20240430 20:11:54.749459 21396 ts_tablet_manager.cc:1875] T c1017a50d1d646c486c0b9a75cb88ce4 P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:54.754650 21487 tablet.cc:1722] T 3a03b6997921408cb5a65f4cae223e9c P 5e1c6397f09747f091534faf93542710: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20240430 20:11:54.755266 18417 consensus_peers.cc:485] T 1f4e860484784aa9a2d28af2e9fb611c P 2efb542aa55e439b99ccb98d1c5c884a -> Peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Couldn't send request to peer 3c51c33d5616482cafb68e26688820f4. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240430 20:11:54.755596 21481 tablet.cc:1722] T a0baee91f5b44945b595d0e234af018e P 5e1c6397f09747f091534faf93542710: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240430 20:11:54.756385 21438 consensus_queue.cc:1035] T 9f0b341a31224bf1be525b84b9500304 P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:54.756814 21489 tablet.cc:1722] T ae20f44df50343289b0a512a4ea53692 P 5e1c6397f09747f091534faf93542710: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240430 20:11:54.704555 18988 tablet_bootstrap.cc:492] T 3b7fdc6ec7e046c5af894f9a6eda17b0 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:54.757357 18988 ts_tablet_manager.cc:1367] T 3b7fdc6ec7e046c5af894f9a6eda17b0 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.651s user 0.002s sys 0.011s
I20240430 20:11:54.728588 21493 consensus_queue.cc:237] T 69f88532952a4591bf1577541c3430e6 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:54.757818 21395 log.cc:1196] T 3e5225f1f5f544ecbf8dfa470af3daca P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/3e5225f1f5f544ecbf8dfa470af3daca
I20240430 20:11:54.758222 18483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "330453ddeea6456e8ef93fa92608b0d3" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:54.758473 18483 raft_consensus.cc:2379] T 330453ddeea6456e8ef93fa92608b0d3 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c51c33d5616482cafb68e26688820f4 in current term 1: Already voted for candidate 2efb542aa55e439b99ccb98d1c5c884a in this term.
I20240430 20:11:54.765471 21489 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:45934 (request call id 59) took 1003 ms. Trace:
I20240430 20:11:54.765717 21489 rpcz_store.cc:276] 0430 20:11:53.761929 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:53.762249 (+ 320us) service_pool.cc:225] Handling call
0430 20:11:54.765441 (+1003192us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:53.868993 (+ 0us) alter_schema_op.cc:96] PREPARE ALTER-SCHEMA: Starting
0430 20:11:53.869137 (+ 144us) alter_schema_op.cc:57] Acquiring schema lock in exclusive mode
0430 20:11:53.869150 (+ 13us) alter_schema_op.cc:59] Acquired schema lock
0430 20:11:53.869160 (+ 10us) alter_schema_op.cc:111] PREPARE ALTER-SCHEMA: finished
0430 20:11:53.869210 (+ 50us) alter_schema_op.cc:119] START. Timestamp: 1714507913869169
0430 20:11:53.869215 (+ 5us) op_driver.cc:331] REPLICATION: starting
0430 20:11:53.869410 (+ 195us) log.cc:842] Serialized 173 byte log entry
0430 20:11:54.756508 (+887098us) op_driver.cc:444] REPLICATION: finished
0430 20:11:54.756786 (+ 278us) alter_schema_op.cc:124] APPLY ALTER-SCHEMA: Starting
0430 20:11:54.757213 (+ 427us) log.cc:842] Serialized 5 byte log entry
0430 20:11:54.764755 (+ 7542us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:54.765107 (+ 352us) log.cc:842] Serialized 14 byte log entry
0430 20:11:54.765396 (+ 289us) alter_schema_op.cc:65] Released schema lock
0430 20:11:54.765403 (+ 7us) alter_schema_op.cc:167] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":217,"prepare.queue_time_us":106411,"prepare.run_cpu_time_us":3780,"prepare.run_wall_time_us":84514,"raft.queue_time_us":78,"raft.run_cpu_time_us":649,"raft.run_wall_time_us":647,"replication_time_us":887270,"thread_start_us":247206,"threads_started":2,"wal-append.queue_time_us":83182}]]}
I20240430 20:11:54.769291 18756 raft_consensus.cc:2454] T 086b3dfa84c045a18c8c4f177dce606a P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
I20240430 20:11:54.770239 21481 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:45934 (request call id 65) took 1007 ms. Trace:
I20240430 20:11:54.770658 21481 rpcz_store.cc:276] 0430 20:11:53.762545 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:53.762607 (+ 62us) service_pool.cc:225] Handling call
0430 20:11:54.770212 (+1007605us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:53.874499 (+ 0us) alter_schema_op.cc:96] PREPARE ALTER-SCHEMA: Starting
0430 20:11:53.874668 (+ 169us) alter_schema_op.cc:57] Acquiring schema lock in exclusive mode
0430 20:11:53.874679 (+ 11us) alter_schema_op.cc:59] Acquired schema lock
0430 20:11:53.874688 (+ 9us) alter_schema_op.cc:111] PREPARE ALTER-SCHEMA: finished
0430 20:11:53.874736 (+ 48us) alter_schema_op.cc:119] START. Timestamp: 1714507913874701
0430 20:11:53.874740 (+ 4us) op_driver.cc:331] REPLICATION: starting
0430 20:11:53.874944 (+ 204us) log.cc:842] Serialized 173 byte log entry
0430 20:11:54.755439 (+880495us) op_driver.cc:444] REPLICATION: finished
0430 20:11:54.755573 (+ 134us) alter_schema_op.cc:124] APPLY ALTER-SCHEMA: Starting
0430 20:11:54.755986 (+ 413us) log.cc:842] Serialized 5 byte log entry
0430 20:11:54.760647 (+ 4661us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:54.761016 (+ 369us) log.cc:842] Serialized 14 byte log entry
0430 20:11:54.770171 (+ 9155us) alter_schema_op.cc:65] Released schema lock
0430 20:11:54.770178 (+ 7us) alter_schema_op.cc:167] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":86,"prepare.queue_time_us":111008,"prepare.run_cpu_time_us":3395,"prepare.run_wall_time_us":93095,"raft.queue_time_us":87,"raft.run_cpu_time_us":712,"raft.run_wall_time_us":712,"replication_time_us":880675,"thread_start_us":145168,"threads_started":2,"wal-append.queue_time_us":51206}]]}
I20240430 20:11:54.773965 21481 tablet.cc:1722] T a59ef6c545dd41a28265a4c1b8ce10e0 P 5e1c6397f09747f091534faf93542710: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240430 20:11:54.774677 21489 tablet.cc:1722] T 331c031d44854d679a27d10d9151d0ea P 5e1c6397f09747f091534faf93542710: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20240430 20:11:54.772462 18415 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.177.2:46027, user_credentials={real_user=slave}} blocked reactor thread for 206595us
I20240430 20:11:54.776173 21395 ts_tablet_manager.cc:1909] T 3e5225f1f5f544ecbf8dfa470af3daca P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:54.775230 18456 tablet_service.cc:1514] Processing DeleteTablet for tablet ebf74809b52f47f3b0ed21b6ac0f541a with delete_type TABLET_DATA_DELETED (replaced by tablet 9e8ab75e7d9c4b64a7b7dd2678569490) from {username='slave'} at 127.0.0.1:41450
I20240430 20:11:54.777604 18417 leader_election.cc:302] T 13c2e69ae91f4957887fd64b90f6e435 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters: 5e1c6397f09747f091534faf93542710
I20240430 20:11:54.778024 21054 raft_consensus.cc:2781] T 13c2e69ae91f4957887fd64b90f6e435 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:54.778115 21054 raft_consensus.cc:482] T 13c2e69ae91f4957887fd64b90f6e435 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:54.778204 21054 raft_consensus.cc:3037] T 13c2e69ae91f4957887fd64b90f6e435 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:54.781185 21054 raft_consensus.cc:504] T 13c2e69ae91f4957887fd64b90f6e435 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:54.782349 21054 leader_election.cc:288] T 13c2e69ae91f4957887fd64b90f6e435 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:54.784386 21548 tablet.cc:1722] T 99203cc522de424184c6f97453f166fd P 2efb542aa55e439b99ccb98d1c5c884a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240430 20:11:54.789496 21395 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41450 (request call id 54) took 1028 ms. Trace:
I20240430 20:11:54.789637 21395 rpcz_store.cc:276] 0430 20:11:53.761217 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:53.761269 (+ 52us) service_pool.cc:225] Handling call
0430 20:11:54.471601 (+710332us) ts_tablet_manager.cc:1120] Deleting tablet 3e5225f1f5f544ecbf8dfa470af3daca
0430 20:11:54.471615 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:54.640331 (+168716us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:54.688766 (+ 48435us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:54.789466 (+100700us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":710133}
I20240430 20:11:54.790098 21395 tablet_replica.cc:330] T 6db55bb510b24b0cae897936211e15f0 P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:54.790292 21395 raft_consensus.cc:2229] T 6db55bb510b24b0cae897936211e15f0 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:54.790870 21395 raft_consensus.cc:2258] T 6db55bb510b24b0cae897936211e15f0 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:54.792105 21395 ts_tablet_manager.cc:1875] T 6db55bb510b24b0cae897936211e15f0 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:54.795388 18985 ts_tablet_manager.cc:1398] T 3b7fdc6ec7e046c5af894f9a6eda17b0 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.105s user 0.003s sys 0.000s
I20240430 20:11:54.795840 18985 tablet_bootstrap.cc:492] T a0d3c4ba64024bccad95ecc909d03ae0 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
W20240430 20:11:54.796066 18415 consensus_peers.cc:485] T df92eb85261d47d88cbf16d29d9ad30e P 2efb542aa55e439b99ccb98d1c5c884a -> Peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Couldn't send request to peer 5e1c6397f09747f091534faf93542710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: df92eb85261d47d88cbf16d29d9ad30e. This is attempt 1: this message will repeat every 5th retry.
I20240430 20:11:54.797124 21539 tablet.cc:1722] T 78ba48dadef548708153878810cc46f8 P 2efb542aa55e439b99ccb98d1c5c884a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20240430 20:11:54.798700 18417 leader_election.cc:341] T 95383d9d5a234a4d9d841a8b114428b0 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Not found: Tablet not found: 95383d9d5a234a4d9d841a8b114428b0
I20240430 20:11:54.799891 18417 leader_election.cc:302] T 9c73d0a6bc414d3b9900570848d19a51 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters:
I20240430 20:11:54.800335 21054 raft_consensus.cc:2781] T 9c73d0a6bc414d3b9900570848d19a51 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240430 20:11:54.800427 21054 raft_consensus.cc:482] T 9c73d0a6bc414d3b9900570848d19a51 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:54.800503 21054 raft_consensus.cc:3037] T 9c73d0a6bc414d3b9900570848d19a51 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Advancing to term 2
I20240430 20:11:54.800686 21487 tablet.cc:1722] T f5eaf2413e1d4e69b56fae61f768f0ce P 5e1c6397f09747f091534faf93542710: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240430 20:11:54.801023 18983 tablet_bootstrap.cc:492] T 8668f7a285de43f48d83abecb4b2e96c P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:54.801162 18983 ts_tablet_manager.cc:1367] T 8668f7a285de43f48d83abecb4b2e96c P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.114s user 0.005s sys 0.005s
I20240430 20:11:54.801793 18983 raft_consensus.cc:348] T 8668f7a285de43f48d83abecb4b2e96c P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:54.801991 18983 raft_consensus.cc:374] T 8668f7a285de43f48d83abecb4b2e96c P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:54.802049 18983 raft_consensus.cc:729] T 8668f7a285de43f48d83abecb4b2e96c P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:54.802296 18983 consensus_queue.cc:260] T 8668f7a285de43f48d83abecb4b2e96c P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:54.802898 18983 ts_tablet_manager.cc:1398] T 8668f7a285de43f48d83abecb4b2e96c P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240430 20:11:54.802944 21539 tablet.cc:1722] T 8e9711d0a382424f9f758bfa883866a0 P 2efb542aa55e439b99ccb98d1c5c884a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240430 20:11:54.803200 18983 tablet_bootstrap.cc:492] T 4391b5227fb64ef481542a962a5e21a5 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
W20240430 20:11:54.803691 18557 leader_election.cc:341] T 347a8224114b49e5b7dcb75d9b726e0a P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:54.803679 21535 tablet.cc:1722] T 47bf64ecc41b445da4ff697e141243dd P 2efb542aa55e439b99ccb98d1c5c884a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20240430 20:11:54.803886 18557 leader_election.cc:341] T e5b85a066cdc46d48ccf8201e5423c29 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:54.806751 18983 tablet_bootstrap.cc:654] T 4391b5227fb64ef481542a962a5e21a5 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:54.806807 21054 raft_consensus.cc:504] T 9c73d0a6bc414d3b9900570848d19a51 P 2efb542aa55e439b99ccb98d1c5c884a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:54.813045 21539 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:41450 (request call id 58) took 1037 ms. Trace:
I20240430 20:11:54.816166 18988 raft_consensus.cc:348] T 3b7fdc6ec7e046c5af894f9a6eda17b0 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:54.816402 18988 raft_consensus.cc:374] T 3b7fdc6ec7e046c5af894f9a6eda17b0 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:54.816473 18988 raft_consensus.cc:729] T 3b7fdc6ec7e046c5af894f9a6eda17b0 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:54.818969 18988 consensus_queue.cc:260] T 3b7fdc6ec7e046c5af894f9a6eda17b0 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:54.851853 21397 ts_tablet_manager.cc:1888] T 9f0b341a31224bf1be525b84b9500304 P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:54.852025 21397 log.cc:1196] T 9f0b341a31224bf1be525b84b9500304 P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/9f0b341a31224bf1be525b84b9500304
I20240430 20:11:54.852380 21397 ts_tablet_manager.cc:1909] T 9f0b341a31224bf1be525b84b9500304 P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:54.852746 21397 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56540 (request call id 58) took 1117 ms. Trace:
I20240430 20:11:54.856030 21539 rpcz_store.cc:276] 0430 20:11:53.776023 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:53.776081 (+ 58us) service_pool.cc:225] Handling call
0430 20:11:54.813015 (+1036934us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:53.885787 (+ 0us) alter_schema_op.cc:96] PREPARE ALTER-SCHEMA: Starting
0430 20:11:53.885970 (+ 183us) alter_schema_op.cc:57] Acquiring schema lock in exclusive mode
0430 20:11:53.885982 (+ 12us) alter_schema_op.cc:59] Acquired schema lock
0430 20:11:53.885991 (+ 9us) alter_schema_op.cc:111] PREPARE ALTER-SCHEMA: finished
0430 20:11:53.886163 (+ 172us) alter_schema_op.cc:119] START. Timestamp: 1714507913886109
0430 20:11:53.886168 (+ 5us) op_driver.cc:331] REPLICATION: starting
0430 20:11:53.886472 (+ 304us) log.cc:842] Serialized 173 byte log entry
0430 20:11:54.796726 (+910254us) op_driver.cc:444] REPLICATION: finished
0430 20:11:54.802919 (+ 6193us) alter_schema_op.cc:124] APPLY ALTER-SCHEMA: Starting
0430 20:11:54.803545 (+ 626us) log.cc:842] Serialized 5 byte log entry
0430 20:11:54.812543 (+ 8998us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:54.812905 (+ 362us) log.cc:842] Serialized 14 byte log entry
0430 20:11:54.812964 (+ 59us) alter_schema_op.cc:65] Released schema lock
0430 20:11:54.812970 (+ 6us) alter_schema_op.cc:167] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":6144,"mutex_wait_us":22228,"prepare.queue_time_us":109281,"prepare.run_cpu_time_us":8940,"prepare.run_wall_time_us":200664,"raft.queue_time_us":23062,"raft.run_cpu_time_us":403,"raft.run_wall_time_us":400,"replication_time_us":910533,"thread_start_us":314777,"threads_started":2,"wal-append.queue_time_us":173647}]]}
I20240430 20:11:54.831669 18985 tablet_bootstrap.cc:654] T a0d3c4ba64024bccad95ecc909d03ae0 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:54.857177 20587 raft_consensus.cc:504] T a663030d37614053a7c0369841403422 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:54.857750 18748 raft_consensus.cc:2379] T a663030d37614053a7c0369841403422 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2efb542aa55e439b99ccb98d1c5c884a in current term 1: Already voted for candidate 3c51c33d5616482cafb68e26688820f4 in this term.
I20240430 20:11:54.863242 18988 ts_tablet_manager.cc:1398] T 3b7fdc6ec7e046c5af894f9a6eda17b0 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.106s user 0.001s sys 0.001s
I20240430 20:11:54.863123 19946 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 6b51cbc731794083a88fe4bed1ddeebf (table default.Table_c6c962da746e4fd8a4d7200d09f08e9e [id=8e0f5d8c941d423ea7b30a89513d5170]). Expected version 1 got 0
I20240430 20:11:54.864130 19946 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 62732af95c6a4c749c5b726f0460dc72 (table default.Table_e3160f5ced1b4718b479401ca4df675c [id=2d44ec94131148048021b2ee6366ba30]). Expected version 3 got 0
I20240430 20:11:54.865043 19946 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet a0baee91f5b44945b595d0e234af018e (table default.Table_5ed3ea587cf843db9a1ff1ffab40b678 [id=b3d1c2f97c4648518dbea83a46cadd85]). Expected version 2 got 0
I20240430 20:11:54.838992 21535 tablet.cc:1722] T 677481aee5fa4eb4b189322d7a3f8f2e P 2efb542aa55e439b99ccb98d1c5c884a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240430 20:11:54.848074 21542 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:41450 (request call id 66) took 1042 ms. Trace:
I20240430 20:11:54.866704 21054 leader_election.cc:288] T 9c73d0a6bc414d3b9900570848d19a51 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:54.866487 21542 rpcz_store.cc:276] 0430 20:11:53.805637 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:53.805695 (+ 58us) service_pool.cc:225] Handling call
0430 20:11:54.848039 (+1042344us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:54.025103 (+ 0us) alter_schema_op.cc:96] PREPARE ALTER-SCHEMA: Starting
0430 20:11:54.025231 (+ 128us) alter_schema_op.cc:57] Acquiring schema lock in exclusive mode
0430 20:11:54.025246 (+ 15us) alter_schema_op.cc:59] Acquired schema lock
0430 20:11:54.025255 (+ 9us) alter_schema_op.cc:111] PREPARE ALTER-SCHEMA: finished
0430 20:11:54.025303 (+ 48us) alter_schema_op.cc:119] START. Timestamp: 1714507914025242
0430 20:11:54.025307 (+ 4us) op_driver.cc:331] REPLICATION: starting
0430 20:11:54.025499 (+ 192us) log.cc:842] Serialized 173 byte log entry
0430 20:11:54.571546 (+546047us) op_driver.cc:444] REPLICATION: finished
0430 20:11:54.741848 (+170302us) alter_schema_op.cc:124] APPLY ALTER-SCHEMA: Starting
0430 20:11:54.743060 (+ 1212us) log.cc:842] Serialized 5 byte log entry
0430 20:11:54.826331 (+ 83271us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:54.847606 (+ 21275us) log.cc:842] Serialized 14 byte log entry
0430 20:11:54.847975 (+ 369us) alter_schema_op.cc:65] Released schema lock
0430 20:11:54.847981 (+ 6us) alter_schema_op.cc:167] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":117788,"prepare.queue_time_us":218948,"prepare.run_cpu_time_us":512,"prepare.run_wall_time_us":1083,"raft.queue_time_us":2215,"raft.run_cpu_time_us":361,"raft.run_wall_time_us":359,"replication_time_us":546214,"thread_start_us":274834,"threads_started":1}]]}
I20240430 20:11:54.873189 20587 leader_election.cc:288] T a663030d37614053a7c0369841403422 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Requested vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:54.873384 18988 tablet_bootstrap.cc:492] T f8bfbf673ea346529d1b2b376a3da496 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:54.876663 21544 consensus_queue.cc:1035] T df92eb85261d47d88cbf16d29d9ad30e P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240430 20:11:54.878778 21177 log.cc:924] Time spent T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: Append to log took a long time: real 0.062s user 0.002s sys 0.000s
I20240430 20:11:54.884755 20587 consensus_queue.cc:1035] T 2e1cac64cefe4336bd3207b5ca58b631 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:54.900228 18528 heartbeater.cc:503] Master 127.17.177.62:36441 was elected leader, sending a full tablet report...
I20240430 20:11:54.852850 21397 rpcz_store.cc:276] 0430 20:11:53.735736 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:53.735786 (+ 50us) service_pool.cc:225] Handling call
0430 20:11:54.697608 (+961822us) ts_tablet_manager.cc:1120] Deleting tablet 9f0b341a31224bf1be525b84b9500304
0430 20:11:54.697618 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:54.717586 (+ 19968us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:54.851719 (+134133us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:54.852719 (+ 1000us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":961667}
I20240430 20:11:54.911914 21397 tablet_replica.cc:330] T dad16ae02b8b4e29a724d8eec67bb4b8 P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:54.912154 21397 raft_consensus.cc:2229] T dad16ae02b8b4e29a724d8eec67bb4b8 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:54.912351 21397 raft_consensus.cc:2258] T dad16ae02b8b4e29a724d8eec67bb4b8 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:54.912674 19900 catalog_manager.cc:4830] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): tablet 9f0b341a31224bf1be525b84b9500304 (table default.Table_5b6a4bb3832e497888af8e8db7b96c12 [id=883dc5b0bfa349e9947d4c5dfe84a970]) successfully deleted
I20240430 20:11:54.914096 18985 tablet_bootstrap.cc:492] T a0d3c4ba64024bccad95ecc909d03ae0 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:54.914321 18985 ts_tablet_manager.cc:1367] T a0d3c4ba64024bccad95ecc909d03ae0 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.119s user 0.002s sys 0.002s
I20240430 20:11:54.915185 18985 raft_consensus.cc:348] T a0d3c4ba64024bccad95ecc909d03ae0 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:54.915457 18985 raft_consensus.cc:374] T a0d3c4ba64024bccad95ecc909d03ae0 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:54.915560 18985 raft_consensus.cc:729] T a0d3c4ba64024bccad95ecc909d03ae0 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:54.915838 18985 consensus_queue.cc:260] T a0d3c4ba64024bccad95ecc909d03ae0 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:54.916411 18985 ts_tablet_manager.cc:1398] T a0d3c4ba64024bccad95ecc909d03ae0 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240430 20:11:54.916867 18985 tablet_bootstrap.cc:492] T f8bfbf673ea346529d1b2b376a3da496 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:54.834880 18620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "330453ddeea6456e8ef93fa92608b0d3" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:54.918083 18620 raft_consensus.cc:2379] T 330453ddeea6456e8ef93fa92608b0d3 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 2efb542aa55e439b99ccb98d1c5c884a in current term 1: Already voted for candidate 3c51c33d5616482cafb68e26688820f4 in this term.
I20240430 20:11:54.837657 21481 tablet.cc:1722] T 724e51ff05ec4baebc2bb55e05c8eddc P 5e1c6397f09747f091534faf93542710: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240430 20:11:54.918365 18985 tablet_bootstrap.cc:654] T f8bfbf673ea346529d1b2b376a3da496 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:54.838879 21489 tablet.cc:1722] T ca64de16d40f4a44b8b411f5526ba11a P 5e1c6397f09747f091534faf93542710: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240430 20:11:54.855706 18610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a663030d37614053a7c0369841403422" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:54.919277 18610 raft_consensus.cc:2454] T a663030d37614053a7c0369841403422 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 0.
I20240430 20:11:54.858906 18606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a663030d37614053a7c0369841403422" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:54.919595 18606 raft_consensus.cc:3037] T a663030d37614053a7c0369841403422 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:54.926218 21397 ts_tablet_manager.cc:1875] T dad16ae02b8b4e29a724d8eec67bb4b8 P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:54.926705 19946 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 266fd23786bc435db5083370f3330ed0 (table default.Table_cf60b652e9dc49e7a948ce66fd1cbffe [id=9734bb8dd05a400ca4244b3829f01dee]). Expected version 1 got 0
I20240430 20:11:54.927793 19946 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet dd1de890e4394db287cf38bb411f5586 (table default.Table_84084a27f7504dd992fce07448c10df3 [id=6257559b0e9545c58848a76af0c5a5f7]). Expected version 2 got 0
I20240430 20:11:54.929988 19946 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 89b774967624498fa2be499878b71124 (table default.Table_2ec9e316bc3347b189a13b7a8bbcf829 [id=9fa22833979f4c2f978de651145acef9]). Expected version 1 got 0
I20240430 20:11:54.930522 19946 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 130007acc8ba4e709c3eb85fabe75903 (table default.Table_78987875650940c293247bc43db345ab [id=36f478a20a1f4a53a52fc29ade62b7f6]). Expected version 1 got 0
I20240430 20:11:54.930845 21489 tablet.cc:1722] T 89b774967624498fa2be499878b71124 P 5e1c6397f09747f091534faf93542710: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240430 20:11:54.931124 19946 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet dac141d1b70c48d785caddd7182811ee (table default.Table_fc0a4540f98b4eea95099b8ab46ea235 [id=f3c53ccffe64410495cc8c379a05cfae]). Expected version 1 got 0
I20240430 20:11:54.931850 21481 tablet.cc:1722] T 2e79df8744e74b5991a79ce978ff0202 P 5e1c6397f09747f091534faf93542710: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240430 20:11:54.935554 21489 tablet.cc:1722] T 99203cc522de424184c6f97453f166fd P 5e1c6397f09747f091534faf93542710: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240430 20:11:54.936594 18606 raft_consensus.cc:2454] T a663030d37614053a7c0369841403422 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:54.938123 21481 tablet.cc:1722] T 8e9711d0a382424f9f758bfa883866a0 P 5e1c6397f09747f091534faf93542710: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240430 20:11:54.937103 18696 leader_election.cc:302] T a663030d37614053a7c0369841403422 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:54.938769 21487 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:45934 (request call id 85) took 1103 ms. Trace:
I20240430 20:11:54.939067 21306 raft_consensus.cc:2781] T a663030d37614053a7c0369841403422 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:54.939198 21306 raft_consensus.cc:686] T a663030d37614053a7c0369841403422 P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Becoming Leader. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Running, Role: LEADER
I20240430 20:11:54.938874 21487 rpcz_store.cc:276] 0430 20:11:53.835240 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:53.835471 (+ 231us) service_pool.cc:225] Handling call
0430 20:11:54.938743 (+1103272us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:53.919806 (+ 0us) alter_schema_op.cc:96] PREPARE ALTER-SCHEMA: Starting
0430 20:11:53.920014 (+ 208us) alter_schema_op.cc:57] Acquiring schema lock in exclusive mode
0430 20:11:53.920024 (+ 10us) alter_schema_op.cc:59] Acquired schema lock
0430 20:11:53.920030 (+ 6us) alter_schema_op.cc:111] PREPARE ALTER-SCHEMA: finished
0430 20:11:53.920122 (+ 92us) alter_schema_op.cc:119] START. Timestamp: 1714507913920082
0430 20:11:53.920125 (+ 3us) op_driver.cc:331] REPLICATION: starting
0430 20:11:53.920381 (+ 256us) log.cc:842] Serialized 173 byte log entry
0430 20:11:54.799975 (+879594us) op_driver.cc:444] REPLICATION: finished
0430 20:11:54.800660 (+ 685us) alter_schema_op.cc:124] APPLY ALTER-SCHEMA: Starting
0430 20:11:54.801100 (+ 440us) log.cc:842] Serialized 5 byte log entry
0430 20:11:54.861781 (+ 60681us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:54.862140 (+ 359us) log.cc:842] Serialized 14 byte log entry
0430 20:11:54.938704 (+ 76564us) alter_schema_op.cc:65] Released schema lock
0430 20:11:54.938711 (+ 7us) alter_schema_op.cc:167] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":624,"prepare.queue_time_us":83870,"prepare.run_cpu_time_us":1987,"prepare.run_wall_time_us":52380,"replication_time_us":879824,"thread_start_us":143328,"threads_started":2,"wal-append.queue_time_us":50387}]]}
I20240430 20:11:54.939437 21306 consensus_queue.cc:237] T a663030d37614053a7c0369841403422 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:54.939690 21487 tablet.cc:1722] T 78ba48dadef548708153878810cc46f8 P 5e1c6397f09747f091534faf93542710: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240430 20:11:54.939822 18748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13c2e69ae91f4957887fd64b90f6e435" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:54.939985 18748 raft_consensus.cc:3037] T 13c2e69ae91f4957887fd64b90f6e435 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:54.869529 21567 tablet.cc:1722] T 66dd767bd3f8494f94061fb89e5e5680 P 5e1c6397f09747f091534faf93542710: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240430 20:11:54.876482 18557 leader_election.cc:302] T 086b3dfa84c045a18c8c4f177dce606a P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:54.871863 18617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69f88532952a4591bf1577541c3430e6" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:54.941666 20489 raft_consensus.cc:2781] T 086b3dfa84c045a18c8c4f177dce606a P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election won for term 1
W20240430 20:11:54.941628 18557 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.177.3:44867, user_credentials={real_user=slave}} blocked reactor thread for 65182.1us
I20240430 20:11:54.941798 20489 raft_consensus.cc:686] T 086b3dfa84c045a18c8c4f177dce606a P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Becoming Leader. State: Replica: 5e1c6397f09747f091534faf93542710, State: Running, Role: LEADER
I20240430 20:11:54.942016 20489 consensus_queue.cc:237] T 086b3dfa84c045a18c8c4f177dce606a P 5e1c6397f09747f091534faf93542710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:54.942734 18988 tablet_bootstrap.cc:654] T f8bfbf673ea346529d1b2b376a3da496 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:54.943547 21486 tablet.cc:1722] T 724e51ff05ec4baebc2bb55e05c8eddc P 3c51c33d5616482cafb68e26688820f4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240430 20:11:54.946159 21395 ts_tablet_manager.cc:1888] T 6db55bb510b24b0cae897936211e15f0 P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:54.946274 21395 log.cc:1196] T 6db55bb510b24b0cae897936211e15f0 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/6db55bb510b24b0cae897936211e15f0
I20240430 20:11:54.947654 21496 tablet.cc:1722] T ca64de16d40f4a44b8b411f5526ba11a P 3c51c33d5616482cafb68e26688820f4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240430 20:11:54.948617 19946 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet ed0c37a71ede4bfc9eb5ad1a4365f25e (table default.Table_a2e85efc6b454524b0b122102ec07002 [id=86d795a82e674d859104741ebd04f4e8]). Expected version 1 got 0
I20240430 20:11:54.949134 19946 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 12795670f21447a8964fe1efc6e92398 (table default.Table_896bbae1f6944005b0272ca8ebc97218 [id=d5cea90cbc2c48df9d8b80da1eccc949]). Expected version 3 got 0
I20240430 20:11:54.950325 19946 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet a482f4622f774d60b4efd6827b8be7d7 (table default.Table_e686cdfcbcc046749fa5b51578a64794 [id=c3dc9f05316d46d6a3ef4d03f40877ec]). Expected version 1 got 0
I20240430 20:11:54.951246 18763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c73d0a6bc414d3b9900570848d19a51" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:54.951440 18763 raft_consensus.cc:3037] T 9c73d0a6bc414d3b9900570848d19a51 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Advancing to term 2
I20240430 20:11:54.951607 19946 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 677481aee5fa4eb4b189322d7a3f8f2e (table default.Table_1ce204e15fce4c469816cf4e7f80cac7 [id=d268c31d82df4871bc1a78aea5871b33]). Expected version 1 got 0
I20240430 20:11:54.952085 18617 raft_consensus.cc:2379] T 69f88532952a4591bf1577541c3430e6 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 2efb542aa55e439b99ccb98d1c5c884a in current term 1: Already voted for candidate 3c51c33d5616482cafb68e26688820f4 in this term.
I20240430 20:11:54.952059 19946 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 5b26319488644d6a8b6be60460a5e7af (table default.Table_69892a8264a4477db48ee6248758e046 [id=5a1c4a7be2fc465fac89a50854eb3322]). Expected version 3 got 0
I20240430 20:11:54.952376 19946 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 0368d37dd9e2452982fb81f539c6805b (table default.Table_404b186c544e4b30bb36cb77d1a14120 [id=4330626e342a48699ea939add5c91b11]). Expected version 1 got 0
I20240430 20:11:54.958413 21486 tablet.cc:1722] T 2e79df8744e74b5991a79ce978ff0202 P 3c51c33d5616482cafb68e26688820f4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240430 20:11:54.964229 21487 tablet.cc:1722] T 47bf64ecc41b445da4ff697e141243dd P 5e1c6397f09747f091534faf93542710: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240430 20:11:54.966257 21489 tablet.cc:1722] T 677481aee5fa4eb4b189322d7a3f8f2e P 5e1c6397f09747f091534faf93542710: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240430 20:11:54.966778 21395 ts_tablet_manager.cc:1909] T 6db55bb510b24b0cae897936211e15f0 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:54.967090 21476 tablet.cc:1722] T 47bf64ecc41b445da4ff697e141243dd P 3c51c33d5616482cafb68e26688820f4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240430 20:11:54.967248 21395 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41450 (request call id 59) took 1164 ms. Trace:
I20240430 20:11:54.967344 21395 rpcz_store.cc:276] 0430 20:11:53.802356 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:53.802423 (+ 67us) service_pool.cc:225] Handling call
0430 20:11:54.790035 (+987612us) ts_tablet_manager.cc:1120] Deleting tablet 6db55bb510b24b0cae897936211e15f0
0430 20:11:54.790049 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:54.910113 (+120064us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:54.946043 (+ 35930us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:54.967222 (+ 21179us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":987288}
I20240430 20:11:54.968425 21395 tablet_replica.cc:330] T 47e98935eb21449abd3d4d24327778d5 P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
W20240430 20:11:54.970856 18415 consensus_peers.cc:485] T 9f0b341a31224bf1be525b84b9500304 P 2efb542aa55e439b99ccb98d1c5c884a -> Peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Couldn't send request to peer 5e1c6397f09747f091534faf93542710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9f0b341a31224bf1be525b84b9500304. This is attempt 1: this message will repeat every 5th retry.
W20240430 20:11:54.972246 18415 leader_election.cc:341] T 95383d9d5a234a4d9d841a8b114428b0 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Not found: Tablet not found: 95383d9d5a234a4d9d841a8b114428b0
I20240430 20:11:54.978199 18415 leader_election.cc:302] T 95383d9d5a234a4d9d841a8b114428b0 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a; no voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710
I20240430 20:11:54.978674 21544 raft_consensus.cc:2726] T 95383d9d5a234a4d9d841a8b114428b0 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:54.981106 21395 raft_consensus.cc:2229] T 47e98935eb21449abd3d4d24327778d5 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:54.981310 21395 raft_consensus.cc:2258] T 47e98935eb21449abd3d4d24327778d5 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:54.981748 19946 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 99203cc522de424184c6f97453f166fd (table default.Table_cb2a4b9ceaf4431facfff2c9bcf94c5d [id=3f8690a437ae49a3beb009fa7899d5d0]). Expected version 1 got 0
I20240430 20:11:54.982132 19946 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 2da900b6bbfa4fa99fe8494fd2e68da8 (table default.Table_19a91a7f17ff456ebb145e8dd89c0940 [id=67283dd3995a44c6a7a7bc8722e4f955]). Expected version 1 got 0
I20240430 20:11:54.982340 19946 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 17ddcaf971154a52b606bbf29a539bf0 (table default.Table_f2f38f5d69e34eaa9e3dbf24cfd1cdd6 [id=872c2faa4a464edd98a51df2e1b3f817]). Expected version 1 got 0
I20240430 20:11:54.982779 18763 raft_consensus.cc:2454] T 9c73d0a6bc414d3b9900570848d19a51 P 3c51c33d5616482cafb68e26688820f4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 2.
I20240430 20:11:54.988806 18748 raft_consensus.cc:2454] T 13c2e69ae91f4957887fd64b90f6e435 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 1.
I20240430 20:11:54.989748 21397 ts_tablet_manager.cc:1888] T dad16ae02b8b4e29a724d8eec67bb4b8 P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:54.989923 21397 log.cc:1196] T dad16ae02b8b4e29a724d8eec67bb4b8 P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/dad16ae02b8b4e29a724d8eec67bb4b8
I20240430 20:11:54.990319 21397 ts_tablet_manager.cc:1909] T dad16ae02b8b4e29a724d8eec67bb4b8 P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:54.992959 19946 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 330453ddeea6456e8ef93fa92608b0d3 (table default.Table_c1f36f7df5ae4e7c8eb8b4e99b89748b [id=a50abf28790d4284a76ac6ee22c02606]). Expected version 1 got 0
I20240430 20:11:54.993480 19946 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 22fef8bb6b034db098c3372ae992f7a7 (table default.Table_4440df38610340758047115845b42b95 [id=7d186ca20c094cc19aef792fe1c094e9]). Expected version 1 got 0
I20240430 20:11:54.993947 19946 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 086b3dfa84c045a18c8c4f177dce606a (table default.Table_df19cb8a6f4d46699350ec5b69c14e5a [id=197923633fc54a4582207942b51530e2]). Expected version 2 got 0
I20240430 20:11:54.994482 19946 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 1050ab528c7d41559831652a1437d358 (table default.Table_acddc4cbf6334546b5fb3e4752fef2a5 [id=274cb15fd751410e83327ac4e8dc4629]). Expected version 2 got 0
I20240430 20:11:55.003834 21486 tablet.cc:1722] T ae20f44df50343289b0a512a4ea53692 P 3c51c33d5616482cafb68e26688820f4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240430 20:11:55.004140 21496 tablet.cc:1722] T 331c031d44854d679a27d10d9151d0ea P 3c51c33d5616482cafb68e26688820f4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240430 20:11:55.005209 21476 tablet.cc:1722] T a59ef6c545dd41a28265a4c1b8ce10e0 P 3c51c33d5616482cafb68e26688820f4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240430 20:11:55.005714 21475 tablet.cc:1722] T f5eaf2413e1d4e69b56fae61f768f0ce P 3c51c33d5616482cafb68e26688820f4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20240430 20:11:55.007819 18415 leader_election.cc:341] T 9c73d0a6bc414d3b9900570848d19a51 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Not found: Tablet not found: 9c73d0a6bc414d3b9900570848d19a51
I20240430 20:11:55.014757 21467 consensus_queue.cc:1035] T 436579f473844c6ca1038b496201a550 P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:55.019934 21476 tablet.cc:1722] T 3a03b6997921408cb5a65f4cae223e9c P 3c51c33d5616482cafb68e26688820f4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240430 20:11:55.021634 21397 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56540 (request call id 57) took 1286 ms. Trace:
I20240430 20:11:55.021631 21548 tablet.cc:1722] T fbdeb4714b8c4b93ab72822c43329aae P 2efb542aa55e439b99ccb98d1c5c884a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240430 20:11:55.021751 21397 rpcz_store.cc:276] 0430 20:11:53.735609 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:53.735878 (+ 269us) service_pool.cc:225] Handling call
0430 20:11:54.911853 (+1175975us) ts_tablet_manager.cc:1120] Deleting tablet dad16ae02b8b4e29a724d8eec67bb4b8
0430 20:11:54.911864 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:54.957184 (+ 45320us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:54.989639 (+ 32455us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:55.021617 (+ 31978us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1175865}
I20240430 20:11:55.022243 19900 catalog_manager.cc:4830] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): tablet dad16ae02b8b4e29a724d8eec67bb4b8 (table default.Table_8a6b5152d20b4890918aedb454764ee9 [id=121444fdfd2642a38cdbd3a6c411c4fd]) successfully deleted
I20240430 20:11:55.022533 21397 tablet_replica.cc:330] T 6db55bb510b24b0cae897936211e15f0 P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:55.022480 21535 tablet.cc:1722] T 2da900b6bbfa4fa99fe8494fd2e68da8 P 2efb542aa55e439b99ccb98d1c5c884a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240430 20:11:55.022764 21397 raft_consensus.cc:2229] T 6db55bb510b24b0cae897936211e15f0 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:55.022944 21397 raft_consensus.cc:2258] T 6db55bb510b24b0cae897936211e15f0 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240430 20:11:55.024042 18417 consensus_peers.cc:485] T 9f0b341a31224bf1be525b84b9500304 P 2efb542aa55e439b99ccb98d1c5c884a -> Peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Couldn't send request to peer 3c51c33d5616482cafb68e26688820f4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9f0b341a31224bf1be525b84b9500304. This is attempt 1: this message will repeat every 5th retry.
I20240430 20:11:55.024331 18417 leader_election.cc:302] T 9c73d0a6bc414d3b9900570848d19a51 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters:
I20240430 20:11:55.024670 18417 leader_election.cc:302] T 13c2e69ae91f4957887fd64b90f6e435 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters:
I20240430 20:11:55.024900 21437 raft_consensus.cc:2781] T 9c73d0a6bc414d3b9900570848d19a51 P 2efb542aa55e439b99ccb98d1c5c884a [term 2 FOLLOWER]: Leader election won for term 2
I20240430 20:11:55.026400 21439 raft_consensus.cc:2781] T 13c2e69ae91f4957887fd64b90f6e435 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:55.028470 21397 ts_tablet_manager.cc:1875] T 6db55bb510b24b0cae897936211e15f0 P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:55.034993 21486 tablet.cc:1722] T a0baee91f5b44945b595d0e234af018e P 3c51c33d5616482cafb68e26688820f4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240430 20:11:55.034556 21396 ts_tablet_manager.cc:1888] T c1017a50d1d646c486c0b9a75cb88ce4 P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:55.039093 21396 log.cc:1196] T c1017a50d1d646c486c0b9a75cb88ce4 P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/c1017a50d1d646c486c0b9a75cb88ce4
I20240430 20:11:55.039500 21396 ts_tablet_manager.cc:1909] T c1017a50d1d646c486c0b9a75cb88ce4 P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:55.053797 21450 raft_consensus.cc:482] T 115c4a12440647a4bb46eb6824f7ea98 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5e1c6397f09747f091534faf93542710)
I20240430 20:11:55.059909 21396 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:45934 (request call id 66) took 1297 ms. Trace:
I20240430 20:11:55.060164 21396 rpcz_store.cc:276] 0430 20:11:53.762709 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:53.762755 (+ 46us) service_pool.cc:225] Handling call
0430 20:11:54.719727 (+956972us) ts_tablet_manager.cc:1120] Deleting tablet c1017a50d1d646c486c0b9a75cb88ce4
0430 20:11:54.719741 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:54.860531 (+140790us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:55.034451 (+173920us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:55.059879 (+ 25428us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":956837}
I20240430 20:11:55.060741 21396 tablet_replica.cc:330] T f940d17b02b541289121463e05b0bb5f P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:55.061015 21396 raft_consensus.cc:2229] T f940d17b02b541289121463e05b0bb5f P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Raft consensus shutting down.
I20240430 20:11:55.060956 19895 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet c1017a50d1d646c486c0b9a75cb88ce4 (table default.Table_d95c70be474e415cba6a345438fad1ba [id=095b8b568ebc475aa3c21e0c03ef5d79]) successfully deleted
I20240430 20:11:55.061611 21396 raft_consensus.cc:2258] T f940d17b02b541289121463e05b0bb5f P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:55.054175 21450 raft_consensus.cc:504] T 115c4a12440647a4bb46eb6824f7ea98 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:55.077302 21396 ts_tablet_manager.cc:1875] T f940d17b02b541289121463e05b0bb5f P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240430 20:11:55.079389 18696 leader_election.cc:341] T 115c4a12440647a4bb46eb6824f7ea98 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Not found: Tablet not found: 115c4a12440647a4bb46eb6824f7ea98
W20240430 20:11:55.081212 18415 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.177.2:46027, user_credentials={real_user=slave}} blocked reactor thread for 73045.7us
I20240430 20:11:55.082837 21542 tablet.cc:1722] T 488d0e609b224674932f890939664df2 P 2efb542aa55e439b99ccb98d1c5c884a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20240430 20:11:55.085671 18415 leader_election.cc:341] T 13c2e69ae91f4957887fd64b90f6e435 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Not found: Tablet not found: 13c2e69ae91f4957887fd64b90f6e435
I20240430 20:11:55.087306 19946 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet e041756c962b4a9f97c2cb314e7d8c05 (table default.Table_de88cafd92b149f9b545369bff71033c [id=c4bf4806f3d84579b16a3038d8d982c3]). Expected version 1 got 0
I20240430 20:11:55.087780 19946 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 8e9711d0a382424f9f758bfa883866a0 (table default.Table_f1dc481c6d8c43f589fe2b1892a4cb10 [id=db696ce66005401fa73746e55c071f44]). Expected version 1 got 0
I20240430 20:11:55.088166 19946 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 47bf64ecc41b445da4ff697e141243dd (table default.Table_f1ad29d5986d4b799cfc67afb2b0dfdf [id=838fc8b61a864dea8877a6f144e4cb4f]). Expected version 2 got 0
I20240430 20:11:55.078552 21450 leader_election.cc:288] T 115c4a12440647a4bb46eb6824f7ea98 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:55.093137 21366 raft_consensus.cc:482] T a0d3c4ba64024bccad95ecc909d03ae0 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240430 20:11:55.094661 18415 leader_election.cc:341] T 9c73d0a6bc414d3b9900570848d19a51 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Not found: Tablet not found: 9c73d0a6bc414d3b9900570848d19a51
I20240430 20:11:55.095016 18988 tablet_bootstrap.cc:492] T f8bfbf673ea346529d1b2b376a3da496 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:55.095229 18988 ts_tablet_manager.cc:1367] T f8bfbf673ea346529d1b2b376a3da496 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.222s user 0.006s sys 0.013s
I20240430 20:11:55.095994 18988 raft_consensus.cc:348] T f8bfbf673ea346529d1b2b376a3da496 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:55.096326 18988 raft_consensus.cc:374] T f8bfbf673ea346529d1b2b376a3da496 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:55.096426 18988 raft_consensus.cc:729] T f8bfbf673ea346529d1b2b376a3da496 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:55.096748 18988 consensus_queue.cc:260] T f8bfbf673ea346529d1b2b376a3da496 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:55.097548 18988 ts_tablet_manager.cc:1398] T f8bfbf673ea346529d1b2b376a3da496 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240430 20:11:55.097573 19946 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet f5eaf2413e1d4e69b56fae61f768f0ce (table default.Table_1280cfe32d5b41abb4d3eb1bd70641bf [id=8489b872074a42b6abb713e420968141]). Expected version 1 got 0
I20240430 20:11:55.093317 21366 raft_consensus.cc:504] T a0d3c4ba64024bccad95ecc909d03ae0 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:55.099529 21366 leader_election.cc:288] T a0d3c4ba64024bccad95ecc909d03ae0 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:55.099959 18983 tablet_bootstrap.cc:492] T 4391b5227fb64ef481542a962a5e21a5 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:55.100078 18983 ts_tablet_manager.cc:1367] T 4391b5227fb64ef481542a962a5e21a5 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.297s user 0.002s sys 0.003s
I20240430 20:11:55.100764 18983 raft_consensus.cc:348] T 4391b5227fb64ef481542a962a5e21a5 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:55.100936 18983 raft_consensus.cc:374] T 4391b5227fb64ef481542a962a5e21a5 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:55.100919 18754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0d3c4ba64024bccad95ecc909d03ae0" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:55.100996 18983 raft_consensus.cc:729] T 4391b5227fb64ef481542a962a5e21a5 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:55.101109 18754 raft_consensus.cc:2454] T a0d3c4ba64024bccad95ecc909d03ae0 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 0.
I20240430 20:11:55.101202 18983 consensus_queue.cc:260] T 4391b5227fb64ef481542a962a5e21a5 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:55.101505 18557 leader_election.cc:302] T a0d3c4ba64024bccad95ecc909d03ae0 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:55.101742 18983 ts_tablet_manager.cc:1398] T 4391b5227fb64ef481542a962a5e21a5 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240430 20:11:55.101828 20489 raft_consensus.cc:2781] T a0d3c4ba64024bccad95ecc909d03ae0 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:55.101893 20489 raft_consensus.cc:482] T a0d3c4ba64024bccad95ecc909d03ae0 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:55.101958 20489 raft_consensus.cc:3037] T a0d3c4ba64024bccad95ecc909d03ae0 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:55.104619 20489 raft_consensus.cc:504] T a0d3c4ba64024bccad95ecc909d03ae0 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:55.105369 20489 leader_election.cc:288] T a0d3c4ba64024bccad95ecc909d03ae0 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:55.105829 18754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0d3c4ba64024bccad95ecc909d03ae0" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:55.106068 18754 raft_consensus.cc:3037] T a0d3c4ba64024bccad95ecc909d03ae0 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:55.108856 18754 raft_consensus.cc:2454] T a0d3c4ba64024bccad95ecc909d03ae0 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
I20240430 20:11:55.109517 18557 leader_election.cc:302] T a0d3c4ba64024bccad95ecc909d03ae0 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:55.110169 21366 raft_consensus.cc:2781] T a0d3c4ba64024bccad95ecc909d03ae0 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:55.110289 21366 raft_consensus.cc:686] T a0d3c4ba64024bccad95ecc909d03ae0 P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Becoming Leader. State: Replica: 5e1c6397f09747f091534faf93542710, State: Running, Role: LEADER
I20240430 20:11:55.110538 21366 consensus_queue.cc:237] T a0d3c4ba64024bccad95ecc909d03ae0 P 5e1c6397f09747f091534faf93542710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:55.115063 18983 tablet_bootstrap.cc:492] T fd1b8407071e4e4287fc172009777414 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:55.117535 18983 tablet_bootstrap.cc:654] T fd1b8407071e4e4287fc172009777414 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:55.118494 18988 tablet_bootstrap.cc:492] T fd1b8407071e4e4287fc172009777414 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:55.122107 21397 ts_tablet_manager.cc:1888] T 6db55bb510b24b0cae897936211e15f0 P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:55.123845 21397 log.cc:1196] T 6db55bb510b24b0cae897936211e15f0 P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/6db55bb510b24b0cae897936211e15f0
I20240430 20:11:55.125044 18988 tablet_bootstrap.cc:654] T fd1b8407071e4e4287fc172009777414 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:55.125387 18668 heartbeater.cc:503] Master 127.17.177.62:36441 was elected leader, sending a full tablet report...
I20240430 20:11:55.128456 21397 ts_tablet_manager.cc:1909] T 6db55bb510b24b0cae897936211e15f0 P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:55.129853 21397 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56540 (request call id 59) took 1394 ms. Trace:
I20240430 20:11:55.138650 21397 rpcz_store.cc:276] 0430 20:11:53.735862 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:53.736005 (+ 143us) service_pool.cc:225] Handling call
0430 20:11:55.022036 (+1286031us) ts_tablet_manager.cc:1120] Deleting tablet 6db55bb510b24b0cae897936211e15f0
0430 20:11:55.022475 (+ 439us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:55.088341 (+ 65866us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:55.122006 (+ 33665us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:55.129828 (+ 7822us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1285910}
I20240430 20:11:55.139307 21397 tablet_replica.cc:330] T 90a694bd5db649afa50f3c4e752939c1 P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:55.139245 19900 catalog_manager.cc:4830] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): tablet 6db55bb510b24b0cae897936211e15f0 (table default.Table_3471b34c1c0d4d829bb27a8520eff502 [id=f3498e7e44974f65bbd2e92c6c8c119f]) successfully deleted
I20240430 20:11:55.139540 21397 raft_consensus.cc:2229] T 90a694bd5db649afa50f3c4e752939c1 P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Raft consensus shutting down.
I20240430 20:11:55.139991 21397 raft_consensus.cc:2258] T 90a694bd5db649afa50f3c4e752939c1 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:55.140772 21397 ts_tablet_manager.cc:1875] T 90a694bd5db649afa50f3c4e752939c1 P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:55.153533 21396 ts_tablet_manager.cc:1888] T f940d17b02b541289121463e05b0bb5f P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:55.154943 18988 tablet_bootstrap.cc:492] T fd1b8407071e4e4287fc172009777414 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:55.155200 18988 ts_tablet_manager.cc:1367] T fd1b8407071e4e4287fc172009777414 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.057s user 0.009s sys 0.007s
I20240430 20:11:55.155355 21396 log.cc:1196] T f940d17b02b541289121463e05b0bb5f P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/f940d17b02b541289121463e05b0bb5f
I20240430 20:11:55.156028 18988 raft_consensus.cc:348] T fd1b8407071e4e4287fc172009777414 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:55.156260 21396 ts_tablet_manager.cc:1909] T f940d17b02b541289121463e05b0bb5f P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:55.156358 18988 raft_consensus.cc:374] T fd1b8407071e4e4287fc172009777414 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:55.156474 18988 raft_consensus.cc:729] T fd1b8407071e4e4287fc172009777414 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:55.156594 21396 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:45934 (request call id 67) took 1393 ms. Trace:
I20240430 20:11:55.156693 21396 rpcz_store.cc:276] 0430 20:11:53.762974 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:53.763003 (+ 29us) service_pool.cc:225] Handling call
0430 20:11:55.060690 (+1297687us) ts_tablet_manager.cc:1120] Deleting tablet f940d17b02b541289121463e05b0bb5f
0430 20:11:55.060701 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:55.104714 (+ 44013us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:55.153404 (+ 48690us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:55.156571 (+ 3167us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1297583}
I20240430 20:11:55.156827 18988 consensus_queue.cc:260] T fd1b8407071e4e4287fc172009777414 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:55.157187 19895 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet f940d17b02b541289121463e05b0bb5f (table default.Table_07993f6164104b888058e1b35c9f4ed7 [id=f2e6cac20f6a4261bb60f2e40f210491]) successfully deleted
I20240430 20:11:55.157754 18988 ts_tablet_manager.cc:1398] T fd1b8407071e4e4287fc172009777414 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240430 20:11:55.158116 18988 tablet_bootstrap.cc:492] T 8668f7a285de43f48d83abecb4b2e96c P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:55.159906 18988 tablet_bootstrap.cc:654] T 8668f7a285de43f48d83abecb4b2e96c P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:55.162119 21396 tablet_replica.cc:330] T 04b3dff86d5c48a09d03da58bd48f360 P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:55.162559 21396 raft_consensus.cc:2229] T 04b3dff86d5c48a09d03da58bd48f360 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:55.163194 21396 raft_consensus.cc:2258] T 04b3dff86d5c48a09d03da58bd48f360 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:55.164400 21396 ts_tablet_manager.cc:1875] T 04b3dff86d5c48a09d03da58bd48f360 P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:55.168897 19902 catalog_manager.cc:4830] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): tablet a692b253db914531b8fc6ffaf3e350b5 (table default.Table_96aea8c4ff80428da8a0394f1bf7b493 [id=03ccb721140c4e0193e2e48d509430f9]) successfully deleted
I20240430 20:11:55.096750 21395 ts_tablet_manager.cc:1875] T 47e98935eb21449abd3d4d24327778d5 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:55.098860 18985 tablet_bootstrap.cc:492] T f8bfbf673ea346529d1b2b376a3da496 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:55.171806 18985 ts_tablet_manager.cc:1367] T f8bfbf673ea346529d1b2b376a3da496 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.255s user 0.023s sys 0.010s
I20240430 20:11:55.172518 18985 raft_consensus.cc:348] T f8bfbf673ea346529d1b2b376a3da496 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:55.172708 18985 raft_consensus.cc:374] T f8bfbf673ea346529d1b2b376a3da496 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:55.172752 18985 raft_consensus.cc:729] T f8bfbf673ea346529d1b2b376a3da496 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:55.172942 18985 consensus_queue.cc:260] T f8bfbf673ea346529d1b2b376a3da496 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:55.173442 18985 ts_tablet_manager.cc:1398] T f8bfbf673ea346529d1b2b376a3da496 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240430 20:11:55.173751 18985 tablet_bootstrap.cc:492] T 8668f7a285de43f48d83abecb4b2e96c P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:55.174683 21439 raft_consensus.cc:686] T 13c2e69ae91f4957887fd64b90f6e435 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Becoming Leader. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Running, Role: LEADER
I20240430 20:11:55.174983 21439 consensus_queue.cc:237] T 13c2e69ae91f4957887fd64b90f6e435 P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
W20240430 20:11:55.175984 18415 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.177.2:46027, user_credentials={real_user=slave}} blocked reactor thread for 81332.8us
I20240430 20:11:55.176664 21580 consensus_queue.cc:1035] T 1fe7c6f1ab54499facd4d01573d3e7bc P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:55.177819 18415 leader_election.cc:302] T 330453ddeea6456e8ef93fa92608b0d3 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a; no voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710
I20240430 20:11:55.178112 21580 raft_consensus.cc:2726] T 330453ddeea6456e8ef93fa92608b0d3 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240430 20:11:55.098070 19946 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 5df364628561475391c9a5e10ff5a05f (table default.Table_3c8f9f7b22f64fb1bdc3f1550987233a [id=7fd6f8576f4c4cbc8ffcdb4ab95e15dc]). Expected version 1 got 0
I20240430 20:11:55.178931 19946 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 821c55edb3414d72b1489eea97d3f73a (table default.Table_4d68c38d9ba7452a829f8318b2b267b8 [id=d06e3e9045b1429aaeebd1fd67d841de]). Expected version 1 got 0
I20240430 20:11:55.104223 21535 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:41450 (request call id 88) took 1041 ms. Trace:
I20240430 20:11:55.180075 21535 rpcz_store.cc:276] 0430 20:11:54.062534 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:54.062576 (+ 42us) service_pool.cc:225] Handling call
0430 20:11:55.104193 (+1041617us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:54.131094 (+ 0us) alter_schema_op.cc:96] PREPARE ALTER-SCHEMA: Starting
0430 20:11:54.208782 (+ 77688us) alter_schema_op.cc:57] Acquiring schema lock in exclusive mode
0430 20:11:54.208796 (+ 14us) alter_schema_op.cc:59] Acquired schema lock
0430 20:11:54.208808 (+ 12us) alter_schema_op.cc:111] PREPARE ALTER-SCHEMA: finished
0430 20:11:54.212135 (+ 3327us) alter_schema_op.cc:119] START. Timestamp: 1714507914212086
0430 20:11:54.212139 (+ 4us) op_driver.cc:331] REPLICATION: starting
0430 20:11:54.212367 (+ 228us) log.cc:842] Serialized 173 byte log entry
0430 20:11:55.017931 (+805564us) op_driver.cc:444] REPLICATION: finished
0430 20:11:55.022458 (+ 4527us) alter_schema_op.cc:124] APPLY ALTER-SCHEMA: Starting
0430 20:11:55.023039 (+ 581us) log.cc:842] Serialized 5 byte log entry
0430 20:11:55.103249 (+ 80210us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:55.104114 (+ 865us) log.cc:842] Serialized 14 byte log entry
0430 20:11:55.104155 (+ 41us) alter_schema_op.cc:65] Released schema lock
0430 20:11:55.104159 (+ 4us) alter_schema_op.cc:167] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":4472,"mutex_wait_us":3489,"prepare.queue_time_us":52143,"prepare.run_cpu_time_us":10214,"prepare.run_wall_time_us":339389,"raft.queue_time_us":588,"raft.run_cpu_time_us":578,"raft.run_wall_time_us":864,"replication_time_us":805777,"thread_start_us":464527,"threads_started":2,"wal-append.queue_time_us":252243}]]}
I20240430 20:11:55.180735 19946 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 37b2ea4be7494f0985808ade788dbe2b (table default.Table_5f6e0fce999b46de927f9271e5de5071 [id=0baad83a8c0f4aeebc7ddad068625a69]). Expected version 1 got 0
I20240430 20:11:55.181162 19946 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet aaef85eea87f4443943fd3692bb2a38f (table default.Table_3da5b4c8668c470483ae2a0c7773eaaf [id=e292d818ee3c45df90522f08eb2d6227]). Expected version 1 got 0
I20240430 20:11:55.181344 19946 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet a59ef6c545dd41a28265a4c1b8ce10e0 (table default.Table_07e40af8f1c546048975c8edf88170d7 [id=86df5d161d8641c89070e7eed9a39360]). Expected version 2 got 0
I20240430 20:11:55.112214 21548 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:41450 (request call id 85) took 1050 ms. Trace:
I20240430 20:11:55.181488 21548 rpcz_store.cc:276] 0430 20:11:54.062101 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:54.065860 (+ 3759us) service_pool.cc:225] Handling call
0430 20:11:55.112188 (+1046328us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:54.127916 (+ 0us) alter_schema_op.cc:96] PREPARE ALTER-SCHEMA: Starting
0430 20:11:54.128043 (+ 127us) alter_schema_op.cc:57] Acquiring schema lock in exclusive mode
0430 20:11:54.128052 (+ 9us) alter_schema_op.cc:59] Acquired schema lock
0430 20:11:54.128057 (+ 5us) alter_schema_op.cc:111] PREPARE ALTER-SCHEMA: finished
0430 20:11:54.128088 (+ 31us) alter_schema_op.cc:119] START. Timestamp: 1714507914128026
0430 20:11:54.128091 (+ 3us) op_driver.cc:331] REPLICATION: starting
0430 20:11:54.145807 (+ 17716us) log.cc:842] Serialized 173 byte log entry
0430 20:11:55.017912 (+872105us) op_driver.cc:444] REPLICATION: finished
0430 20:11:55.021599 (+ 3687us) alter_schema_op.cc:124] APPLY ALTER-SCHEMA: Starting
0430 20:11:55.022074 (+ 475us) log.cc:842] Serialized 5 byte log entry
0430 20:11:55.111794 (+ 89720us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:55.112109 (+ 315us) log.cc:842] Serialized 14 byte log entry
0430 20:11:55.112153 (+ 44us) alter_schema_op.cc:65] Released schema lock
0430 20:11:55.112156 (+ 3us) alter_schema_op.cc:167] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":3617,"prepare.queue_time_us":50614,"prepare.run_cpu_time_us":5269,"prepare.run_wall_time_us":154871,"raft.queue_time_us":266,"raft.run_cpu_time_us":372,"raft.run_wall_time_us":549,"replication_time_us":889787,"thread_start_us":125889,"threads_started":1,"wal-append.queue_time_us":133319}]]}
I20240430 20:11:55.182060 19946 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 69f88532952a4591bf1577541c3430e6 (table default.Table_1177828cad1f4a24bd397619dad96170 [id=a95fed1bc3fa4865bf9a35dfc51b659c]). Expected version 2 got 0
I20240430 20:11:55.182317 19946 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet eb6c96047b114ebba73db7a371f2da8e (table default.Table_7d5f6f18cbd64c869739872a7dff9be2 [id=e67d619e36c0485f8f22b4a81cb4cb6c]). Expected version 1 got 0
I20240430 20:11:55.183210 18985 tablet_bootstrap.cc:654] T 8668f7a285de43f48d83abecb4b2e96c P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:55.112576 21542 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:41450 (request call id 87) took 1050 ms. Trace:
I20240430 20:11:55.119298 21437 raft_consensus.cc:686] T 9c73d0a6bc414d3b9900570848d19a51 P 2efb542aa55e439b99ccb98d1c5c884a [term 2 LEADER]: Becoming Leader. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Running, Role: LEADER
W20240430 20:11:55.129474 18417 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.177.3:44867, user_credentials={real_user=slave}} blocked reactor thread for 103479us
I20240430 20:11:55.184430 21437 consensus_queue.cc:237] T 9c73d0a6bc414d3b9900570848d19a51 P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:55.184772 21580 consensus_queue.cc:1035] T 1fe7c6f1ab54499facd4d01573d3e7bc P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:55.186501 21542 rpcz_store.cc:276] 0430 20:11:54.062344 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:54.062397 (+ 53us) service_pool.cc:225] Handling call
0430 20:11:55.112558 (+1050161us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:54.086470 (+ 0us) alter_schema_op.cc:96] PREPARE ALTER-SCHEMA: Starting
0430 20:11:54.096098 (+ 9628us) alter_schema_op.cc:57] Acquiring schema lock in exclusive mode
0430 20:11:54.096109 (+ 11us) alter_schema_op.cc:59] Acquired schema lock
0430 20:11:54.096117 (+ 8us) alter_schema_op.cc:111] PREPARE ALTER-SCHEMA: finished
0430 20:11:54.096155 (+ 38us) alter_schema_op.cc:119] START. Timestamp: 1714507914096092
0430 20:11:54.096159 (+ 4us) op_driver.cc:331] REPLICATION: starting
0430 20:11:54.096340 (+ 181us) log.cc:842] Serialized 173 byte log entry
0430 20:11:55.082702 (+986362us) op_driver.cc:444] REPLICATION: finished
0430 20:11:55.082810 (+ 108us) alter_schema_op.cc:124] APPLY ALTER-SCHEMA: Starting
0430 20:11:55.083300 (+ 490us) log.cc:842] Serialized 5 byte log entry
0430 20:11:55.108979 (+ 25679us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:55.109432 (+ 453us) log.cc:842] Serialized 14 byte log entry
0430 20:11:55.112529 (+ 3097us) alter_schema_op.cc:65] Released schema lock
0430 20:11:55.112533 (+ 4us) alter_schema_op.cc:167] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"spinlock_wait_cycles":8576,"child_traces":[["op",{"apply.queue_time_us":59,"prepare.queue_time_us":9514,"prepare.run_cpu_time_us":9884,"prepare.run_wall_time_us":341019,"raft.queue_time_us":69,"raft.run_cpu_time_us":587,"raft.run_wall_time_us":586,"replication_time_us":986508,"thread_start_us":330253,"threads_started":1,"wal-append.queue_time_us":325702}]]}
I20240430 20:11:55.194875 18810 heartbeater.cc:503] Master 127.17.177.60:37225 was elected leader, sending a full tablet report...
I20240430 20:11:55.196879 21486 tablet.cc:1722] T 2da900b6bbfa4fa99fe8494fd2e68da8 P 3c51c33d5616482cafb68e26688820f4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240430 20:11:55.197515 21496 tablet.cc:1722] T 8e9711d0a382424f9f758bfa883866a0 P 3c51c33d5616482cafb68e26688820f4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240430 20:11:55.197747 21475 tablet.cc:1722] T 66dd767bd3f8494f94061fb89e5e5680 P 3c51c33d5616482cafb68e26688820f4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240430 20:11:55.198256 21476 tablet.cc:1722] T 677481aee5fa4eb4b189322d7a3f8f2e P 3c51c33d5616482cafb68e26688820f4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240430 20:11:55.207093 21476 tablet.cc:1722] T 99203cc522de424184c6f97453f166fd P 3c51c33d5616482cafb68e26688820f4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240430 20:11:55.209931 21475 tablet.cc:1722] T 78ba48dadef548708153878810cc46f8 P 3c51c33d5616482cafb68e26688820f4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240430 20:11:55.210096 21486 tablet.cc:1722] T 488d0e609b224674932f890939664df2 P 3c51c33d5616482cafb68e26688820f4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240430 20:11:55.210700 21476 tablet.cc:1722] T 89b774967624498fa2be499878b71124 P 3c51c33d5616482cafb68e26688820f4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240430 20:11:55.214188 21475 tablet.cc:1722] T fbdeb4714b8c4b93ab72822c43329aae P 3c51c33d5616482cafb68e26688820f4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240430 20:11:55.199071 18988 tablet_bootstrap.cc:492] T 8668f7a285de43f48d83abecb4b2e96c P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:55.199837 18699 leader_election.cc:302] T 13c2e69ae91f4957887fd64b90f6e435 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters: 5e1c6397f09747f091534faf93542710
I20240430 20:11:55.215404 21493 raft_consensus.cc:2741] T 13c2e69ae91f4957887fd64b90f6e435 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
W20240430 20:11:55.215448 18699 leader_election.cc:341] T a0d3c4ba64024bccad95ecc909d03ae0 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:55.215562 18699 leader_election.cc:302] T a0d3c4ba64024bccad95ecc909d03ae0 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710
I20240430 20:11:55.215792 21436 raft_consensus.cc:2726] T a0d3c4ba64024bccad95ecc909d03ae0 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:55.217483 18988 ts_tablet_manager.cc:1367] T 8668f7a285de43f48d83abecb4b2e96c P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.059s user 0.003s sys 0.000s
I20240430 20:11:55.218277 18988 raft_consensus.cc:348] T 8668f7a285de43f48d83abecb4b2e96c P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:55.219668 18415 leader_election.cc:302] T a663030d37614053a7c0369841403422 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters: 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:55.225459 21396 ts_tablet_manager.cc:1888] T 04b3dff86d5c48a09d03da58bd48f360 P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:55.225644 21396 log.cc:1196] T 04b3dff86d5c48a09d03da58bd48f360 P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/04b3dff86d5c48a09d03da58bd48f360
I20240430 20:11:55.226172 21396 ts_tablet_manager.cc:1909] T 04b3dff86d5c48a09d03da58bd48f360 P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:55.226671 21396 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:45934 (request call id 68) took 1463 ms. Trace:
I20240430 20:11:55.226823 21396 rpcz_store.cc:276] 0430 20:11:53.763170 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:53.763196 (+ 26us) service_pool.cc:225] Handling call
0430 20:11:55.161835 (+1398639us) ts_tablet_manager.cc:1120] Deleting tablet 04b3dff86d5c48a09d03da58bd48f360
0430 20:11:55.162060 (+ 225us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:55.185172 (+ 23112us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:55.225306 (+ 40134us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:55.226647 (+ 1341us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1398546}
I20240430 20:11:55.227068 18988 raft_consensus.cc:374] T 8668f7a285de43f48d83abecb4b2e96c P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:55.227182 18988 raft_consensus.cc:729] T 8668f7a285de43f48d83abecb4b2e96c P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:55.227492 19895 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet 04b3dff86d5c48a09d03da58bd48f360 (table default.Table_2ec9e316bc3347b189a13b7a8bbcf829 [id=9fa22833979f4c2f978de651145acef9]) successfully deleted
I20240430 20:11:55.227685 21396 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:45934 (request call id 61) took 1465 ms. Trace:
I20240430 20:11:55.227766 21396 rpcz_store.cc:276] 0430 20:11:53.762148 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:53.763360 (+ 1212us) service_pool.cc:225] Handling call
0430 20:11:55.227545 (+1464185us) ts_tablet_manager.cc:1120] Deleting tablet 3b7fdc6ec7e046c5af894f9a6eda17b0
0430 20:11:55.227556 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:55.227669 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1464009}
I20240430 20:11:55.227834 18988 consensus_queue.cc:260] T 8668f7a285de43f48d83abecb4b2e96c P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:55.228283 21396 tablet_replica.cc:330] T d7c1fc7e81364fb890bbc13d7fb6848b P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:55.228545 21396 raft_consensus.cc:2229] T d7c1fc7e81364fb890bbc13d7fb6848b P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus shutting down.
W20240430 20:11:55.228535 19895 catalog_manager.cc:4810] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): delete failed for tablet 3b7fdc6ec7e046c5af894f9a6eda17b0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3b7fdc6ec7e046c5af894f9a6eda17b0 already in progress: creating tablet
I20240430 20:11:55.228770 21396 raft_consensus.cc:2258] T d7c1fc7e81364fb890bbc13d7fb6848b P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:55.229460 21396 ts_tablet_manager.cc:1875] T d7c1fc7e81364fb890bbc13d7fb6848b P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:55.229931 18983 tablet_bootstrap.cc:492] T fd1b8407071e4e4287fc172009777414 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:55.230132 18983 ts_tablet_manager.cc:1367] T fd1b8407071e4e4287fc172009777414 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.115s user 0.003s sys 0.010s
I20240430 20:11:55.230351 21580 raft_consensus.cc:482] T c6378f1495144c458af49b3c19a2b67c P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:55.230759 21580 raft_consensus.cc:504] T c6378f1495144c458af49b3c19a2b67c P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:55.232036 18983 raft_consensus.cc:348] T fd1b8407071e4e4287fc172009777414 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:55.232245 21580 leader_election.cc:288] T c6378f1495144c458af49b3c19a2b67c P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:55.232280 18983 raft_consensus.cc:374] T fd1b8407071e4e4287fc172009777414 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:55.232435 18983 raft_consensus.cc:729] T fd1b8407071e4e4287fc172009777414 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:55.232692 18983 consensus_queue.cc:260] T fd1b8407071e4e4287fc172009777414 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:55.233412 18983 ts_tablet_manager.cc:1398] T fd1b8407071e4e4287fc172009777414 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240430 20:11:55.233847 18983 tablet_bootstrap.cc:492] T 00be6ddd91e84865a48743fb4d077eec P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:55.235270 18988 ts_tablet_manager.cc:1398] T 8668f7a285de43f48d83abecb4b2e96c P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.018s user 0.002s sys 0.000s
I20240430 20:11:55.235677 18988 tablet_bootstrap.cc:492] T dbb427aefa6b47b0805d1751fea4efcb P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:55.237701 18988 tablet_bootstrap.cc:654] T dbb427aefa6b47b0805d1751fea4efcb P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:55.239854 18473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5b85a066cdc46d48ccf8201e5423c29" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:55.241695 18415 leader_election.cc:302] T 69f88532952a4591bf1577541c3430e6 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a; no voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710
I20240430 20:11:55.242053 21579 raft_consensus.cc:2781] T a663030d37614053a7c0369841403422 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:55.242182 21579 raft_consensus.cc:482] T a663030d37614053a7c0369841403422 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:55.242316 21579 raft_consensus.cc:3037] T a663030d37614053a7c0369841403422 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
W20240430 20:11:55.245303 18417 leader_election.cc:341] T c6378f1495144c458af49b3c19a2b67c P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Not found: Tablet not found: c6378f1495144c458af49b3c19a2b67c
I20240430 20:11:55.246799 18983 tablet_bootstrap.cc:654] T 00be6ddd91e84865a48743fb4d077eec P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:55.246039 19946 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 7a0176f8a74c47efbbe3d3343b3dc896 (table default.Table_144e5402c1864d818436015b26648024 [id=73e5c9f0470d47849109819c063e1ce4]). Expected version 2 got 0
I20240430 20:11:55.255038 19946 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 94fbd4a57a1e438891fcb456f6b40be3 (table default.Table_25ddc3adef6b4ccf8b9c00cc6772031c [id=4fa623ffdcf347639150a355806038af]). Expected version 1 got 0
I20240430 20:11:55.255234 21396 ts_tablet_manager.cc:1888] T d7c1fc7e81364fb890bbc13d7fb6848b P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:55.252578 18483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086b3dfa84c045a18c8c4f177dce606a" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:55.255599 18483 raft_consensus.cc:3037] T 086b3dfa84c045a18c8c4f177dce606a P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:55.256008 21437 raft_consensus.cc:2726] T 69f88532952a4591bf1577541c3430e6 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240430 20:11:55.256129 21396 log.cc:1196] T d7c1fc7e81364fb890bbc13d7fb6848b P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/d7c1fc7e81364fb890bbc13d7fb6848b
I20240430 20:11:55.256522 21396 ts_tablet_manager.cc:1909] T d7c1fc7e81364fb890bbc13d7fb6848b P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:55.258003 21487 tablet.cc:1722] T 2da900b6bbfa4fa99fe8494fd2e68da8 P 5e1c6397f09747f091534faf93542710: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240430 20:11:55.258347 21489 tablet.cc:1722] T fbdeb4714b8c4b93ab72822c43329aae P 5e1c6397f09747f091534faf93542710: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240430 20:11:55.258527 18983 tablet_bootstrap.cc:492] T 00be6ddd91e84865a48743fb4d077eec P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:55.259647 18983 ts_tablet_manager.cc:1367] T 00be6ddd91e84865a48743fb4d077eec P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.026s user 0.002s sys 0.002s
I20240430 20:11:55.259716 21395 ts_tablet_manager.cc:1888] T 47e98935eb21449abd3d4d24327778d5 P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:55.258903 21567 tablet.cc:1722] T 488d0e609b224674932f890939664df2 P 5e1c6397f09747f091534faf93542710: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240430 20:11:55.259821 21395 log.cc:1196] T 47e98935eb21449abd3d4d24327778d5 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/47e98935eb21449abd3d4d24327778d5
I20240430 20:11:55.260501 18983 raft_consensus.cc:348] T 00be6ddd91e84865a48743fb4d077eec P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:55.260707 18983 raft_consensus.cc:374] T 00be6ddd91e84865a48743fb4d077eec P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:55.260762 18983 raft_consensus.cc:729] T 00be6ddd91e84865a48743fb4d077eec P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:55.260963 18983 consensus_queue.cc:260] T 00be6ddd91e84865a48743fb4d077eec P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:55.261638 18983 ts_tablet_manager.cc:1398] T 00be6ddd91e84865a48743fb4d077eec P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240430 20:11:55.261905 18983 tablet_bootstrap.cc:492] T dbb427aefa6b47b0805d1751fea4efcb P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:55.262506 21395 ts_tablet_manager.cc:1909] T 47e98935eb21449abd3d4d24327778d5 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:55.263370 21395 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41450 (request call id 60) took 1460 ms. Trace:
I20240430 20:11:55.263553 18983 tablet_bootstrap.cc:654] T dbb427aefa6b47b0805d1751fea4efcb P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:55.277717 21396 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:45934 (request call id 69) took 1514 ms. Trace:
I20240430 20:11:55.281419 18988 tablet_bootstrap.cc:492] T dbb427aefa6b47b0805d1751fea4efcb P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:55.281355 21396 rpcz_store.cc:276] 0430 20:11:53.763423 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:53.763466 (+ 43us) service_pool.cc:225] Handling call
0430 20:11:55.228238 (+1464772us) ts_tablet_manager.cc:1120] Deleting tablet d7c1fc7e81364fb890bbc13d7fb6848b
0430 20:11:55.228247 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:55.234820 (+ 6573us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:55.255100 (+ 20280us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:55.277678 (+ 22578us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1464610}
I20240430 20:11:55.286784 18988 ts_tablet_manager.cc:1367] T dbb427aefa6b47b0805d1751fea4efcb P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.051s user 0.004s sys 0.000s
I20240430 20:11:55.289351 18988 raft_consensus.cc:348] T dbb427aefa6b47b0805d1751fea4efcb P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:55.294004 18988 raft_consensus.cc:374] T dbb427aefa6b47b0805d1751fea4efcb P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:55.295358 21396 tablet_replica.cc:330] T 489dc838744a49eba542009c86688aa5 P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:55.295672 21396 raft_consensus.cc:2229] T 489dc838744a49eba542009c86688aa5 P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Raft consensus shutting down.
I20240430 20:11:55.296108 21396 raft_consensus.cc:2258] T 489dc838744a49eba542009c86688aa5 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:55.297358 21396 ts_tablet_manager.cc:1875] T 489dc838744a49eba542009c86688aa5 P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:55.298745 18988 raft_consensus.cc:729] T dbb427aefa6b47b0805d1751fea4efcb P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:55.300357 18988 consensus_queue.cc:260] T dbb427aefa6b47b0805d1751fea4efcb P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:55.257498 18480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "347a8224114b49e5b7dcb75d9b726e0a" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:55.281392 21579 raft_consensus.cc:504] T a663030d37614053a7c0369841403422 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:55.286865 18483 raft_consensus.cc:2454] T 086b3dfa84c045a18c8c4f177dce606a P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
I20240430 20:11:55.301786 18988 ts_tablet_manager.cc:1398] T dbb427aefa6b47b0805d1751fea4efcb P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.014s user 0.001s sys 0.001s
I20240430 20:11:55.302491 21579 leader_election.cc:288] T a663030d37614053a7c0369841403422 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:55.302810 18756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a663030d37614053a7c0369841403422" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:55.308260 18985 tablet_bootstrap.cc:492] T 8668f7a285de43f48d83abecb4b2e96c P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:55.308504 18985 ts_tablet_manager.cc:1367] T 8668f7a285de43f48d83abecb4b2e96c P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.135s user 0.012s sys 0.005s
I20240430 20:11:55.310076 21396 ts_tablet_manager.cc:1888] T 489dc838744a49eba542009c86688aa5 P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:55.308645 21439 raft_consensus.cc:482] T a0d3c4ba64024bccad95ecc909d03ae0 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:55.311110 21439 raft_consensus.cc:504] T a0d3c4ba64024bccad95ecc909d03ae0 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:55.317561 21439 leader_election.cc:288] T a0d3c4ba64024bccad95ecc909d03ae0 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:55.317507 18756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0d3c4ba64024bccad95ecc909d03ae0" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:55.315508 18983 tablet_bootstrap.cc:492] T dbb427aefa6b47b0805d1751fea4efcb P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:55.317800 18756 raft_consensus.cc:2379] T a0d3c4ba64024bccad95ecc909d03ae0 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2efb542aa55e439b99ccb98d1c5c884a in current term 1: Already voted for candidate 5e1c6397f09747f091534faf93542710 in this term.
I20240430 20:11:55.317971 18983 ts_tablet_manager.cc:1367] T dbb427aefa6b47b0805d1751fea4efcb P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.056s user 0.001s sys 0.005s
I20240430 20:11:55.318994 18983 raft_consensus.cc:348] T dbb427aefa6b47b0805d1751fea4efcb P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:55.319310 18983 raft_consensus.cc:374] T dbb427aefa6b47b0805d1751fea4efcb P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:55.319422 18983 raft_consensus.cc:729] T dbb427aefa6b47b0805d1751fea4efcb P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:55.319698 18983 consensus_queue.cc:260] T dbb427aefa6b47b0805d1751fea4efcb P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:55.320343 21542 tablet.cc:1722] T acd23f0a63d5448aaa7ed78422e67bad P 2efb542aa55e439b99ccb98d1c5c884a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240430 20:11:55.320469 18983 ts_tablet_manager.cc:1398] T dbb427aefa6b47b0805d1751fea4efcb P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240430 20:11:55.320598 21396 log.cc:1196] T 489dc838744a49eba542009c86688aa5 P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/489dc838744a49eba542009c86688aa5
I20240430 20:11:55.320976 21396 ts_tablet_manager.cc:1909] T 489dc838744a49eba542009c86688aa5 P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:55.321187 18810 heartbeater.cc:503] Master 127.17.177.60:37225 was elected leader, sending a full tablet report...
I20240430 20:11:55.325764 18983 tablet_bootstrap.cc:492] T b7d8da6e2fe8499d9d53af84a25f2faf P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:55.326404 18985 raft_consensus.cc:348] T 8668f7a285de43f48d83abecb4b2e96c P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:55.326676 18985 raft_consensus.cc:374] T 8668f7a285de43f48d83abecb4b2e96c P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:55.326740 18985 raft_consensus.cc:729] T 8668f7a285de43f48d83abecb4b2e96c P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:55.326936 18985 consensus_queue.cc:260] T 8668f7a285de43f48d83abecb4b2e96c P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:55.327502 18985 ts_tablet_manager.cc:1398] T 8668f7a285de43f48d83abecb4b2e96c P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.019s user 0.001s sys 0.000s
I20240430 20:11:55.327625 18983 tablet_bootstrap.cc:654] T b7d8da6e2fe8499d9d53af84a25f2faf P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:55.327771 18985 tablet_bootstrap.cc:492] T 347a8224114b49e5b7dcb75d9b726e0a P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:55.329138 21396 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:45934 (request call id 62) took 1566 ms. Trace:
I20240430 20:11:55.329329 18985 tablet_bootstrap.cc:654] T 347a8224114b49e5b7dcb75d9b726e0a P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:55.329241 21396 rpcz_store.cc:276] 0430 20:11:53.762233 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:53.763572 (+ 1339us) service_pool.cc:225] Handling call
0430 20:11:55.282899 (+1519327us) ts_tablet_manager.cc:1120] Deleting tablet 489dc838744a49eba542009c86688aa5
0430 20:11:55.295309 (+ 12410us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:55.302886 (+ 7577us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:55.309947 (+ 7061us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:55.329115 (+ 19168us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1519212}
I20240430 20:11:55.329563 21396 tablet_replica.cc:330] T 47e98935eb21449abd3d4d24327778d5 P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:55.329747 21396 raft_consensus.cc:2229] T 47e98935eb21449abd3d4d24327778d5 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:55.329897 21396 raft_consensus.cc:2258] T 47e98935eb21449abd3d4d24327778d5 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:55.330405 21396 ts_tablet_manager.cc:1875] T 47e98935eb21449abd3d4d24327778d5 P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:55.339637 21535 tablet.cc:1722] T eb6c96047b114ebba73db7a371f2da8e P 2efb542aa55e439b99ccb98d1c5c884a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240430 20:11:55.339821 21548 tablet.cc:1722] T a59ef6c545dd41a28265a4c1b8ce10e0 P 2efb542aa55e439b99ccb98d1c5c884a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240430 20:11:55.341272 21395 rpcz_store.cc:276] 0430 20:11:53.802976 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:53.803028 (+ 52us) service_pool.cc:225] Handling call
0430 20:11:54.968129 (+1165101us) ts_tablet_manager.cc:1120] Deleting tablet 47e98935eb21449abd3d4d24327778d5
0430 20:11:54.968143 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:55.096720 (+128577us) spinlock_profiling.cc:243] Waited 4.79 ms on lock 0x616000144754. stack: 00007f9c64594756 00007f9c64594510 00007f9c635c748d 0000000000955804 00000000009557cd 000000000095519a 00007f9c6fe0a1aa 00007f9c73b6f58e 00007f9c73ba1814 00007f9c73b82b00 00007f9c73b8288d 0000000000a60bcc 00007f9c645dab74 00007f9c645de19b 00007f9c645de018 0000000000a60bcc
0430 20:11:55.196711 (+ 99991us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:55.259603 (+ 62892us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:55.263348 (+ 3745us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":11027328,"tablet-delete.queue_time_us":1164757}
I20240430 20:11:55.341806 21395 tablet_replica.cc:330] T f3711610610542128c4a84e17bce4a3e P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:55.342067 21395 raft_consensus.cc:2229] T f3711610610542128c4a84e17bce4a3e P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:55.348304 21396 ts_tablet_manager.cc:1888] T 47e98935eb21449abd3d4d24327778d5 P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:55.349515 21396 log.cc:1196] T 47e98935eb21449abd3d4d24327778d5 P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/47e98935eb21449abd3d4d24327778d5
I20240430 20:11:55.282953 19895 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet d7c1fc7e81364fb890bbc13d7fb6848b (table default.Table_fa64149f685c4784bda92ae4f578dd92 [id=3d6e1fcb9c0f403394d8b436d100b9d2]) successfully deleted
W20240430 20:11:55.351281 19895 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.17.177.2:46027, user_credentials={real_user=slave}} blocked reactor thread for 68364.1us
I20240430 20:11:55.349586 18669 heartbeater.cc:347] Connected to a master server at 127.17.177.60:37225
I20240430 20:11:55.351735 19895 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet 489dc838744a49eba542009c86688aa5 (table default.Table_4493a13725004bf5a2996bc9369a0892 [id=6f4ee22d8d924af08a838970664ecda0]) successfully deleted
I20240430 20:11:55.352303 19946 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 2e79df8744e74b5991a79ce978ff0202 (table default.Table_276bbdc7cfe542d1a2936485a2aad1ec [id=f146a15e6e8a4acbb4c734c1c863da57]). Expected version 1 got 0
I20240430 20:11:55.353623 21397 ts_tablet_manager.cc:1888] T 90a694bd5db649afa50f3c4e752939c1 P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:55.354578 21548 tablet.cc:1722] T a0baee91f5b44945b595d0e234af018e P 2efb542aa55e439b99ccb98d1c5c884a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240430 20:11:55.355461 21396 ts_tablet_manager.cc:1909] T 47e98935eb21449abd3d4d24327778d5 P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:55.359237 21396 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:45934 (request call id 64) took 1596 ms. Trace:
I20240430 20:11:55.361312 21548 tablet.cc:1722] T 331c031d44854d679a27d10d9151d0ea P 2efb542aa55e439b99ccb98d1c5c884a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240430 20:11:55.362780 21396 rpcz_store.cc:276] 0430 20:11:53.762427 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:53.763708 (+ 1281us) service_pool.cc:225] Handling call
0430 20:11:55.329515 (+1565807us) ts_tablet_manager.cc:1120] Deleting tablet 47e98935eb21449abd3d4d24327778d5
0430 20:11:55.329526 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:55.333523 (+ 3997us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:55.348180 (+ 14657us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:55.359103 (+ 10923us) spinlock_profiling.cc:243] Waited 2.78 ms on lock 0x616000274c54. stack: 00007ffa8b97e756 00007ffa8b97e510 00007ffa8a9b148d 0000000000955804 00000000009557cd 000000000095519a 00007ffa971ec88a 00007ffa971ac1c4 00007ffa984c86a6 00007ffa9af598a8 00007ffa9af8b814 00007ffa9af6cb00 00007ffa9af6c88d 0000000000a60bcc 00007ffa8b9c4b74 00007ffa8b9c819b
0430 20:11:55.359207 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6395136,"tablet-delete.queue_time_us":1565680}
I20240430 20:11:55.363819 19895 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet 47e98935eb21449abd3d4d24327778d5 (table default.Table_67d51131549440cc99629dda924693f7 [id=842d2b7e31cb4e7793d88103997a88d8]) successfully deleted
I20240430 20:11:55.364209 21396 tablet_replica.cc:330] T a663030d37614053a7c0369841403422 P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:55.364444 19946 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet acd23f0a63d5448aaa7ed78422e67bad (table default.Table_972f13be4dca4cd19b9a04534dc59f51 [id=69f37db4de634a3185613e40013e0ed7]). Expected version 1 got 0
I20240430 20:11:55.364553 21396 raft_consensus.cc:2229] T a663030d37614053a7c0369841403422 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:55.365010 21396 raft_consensus.cc:2258] T a663030d37614053a7c0369841403422 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:55.366005 21396 ts_tablet_manager.cc:1875] T a663030d37614053a7c0369841403422 P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:55.366672 21535 tablet.cc:1722] T 3a03b6997921408cb5a65f4cae223e9c P 2efb542aa55e439b99ccb98d1c5c884a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240430 20:11:55.377063 18983 tablet_bootstrap.cc:492] T b7d8da6e2fe8499d9d53af84a25f2faf P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:55.377322 18983 ts_tablet_manager.cc:1367] T b7d8da6e2fe8499d9d53af84a25f2faf P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.052s user 0.008s sys 0.002s
I20240430 20:11:55.378746 18983 raft_consensus.cc:348] T b7d8da6e2fe8499d9d53af84a25f2faf P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:55.379045 18983 raft_consensus.cc:374] T b7d8da6e2fe8499d9d53af84a25f2faf P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:55.379138 18983 raft_consensus.cc:729] T b7d8da6e2fe8499d9d53af84a25f2faf P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:55.379482 18983 consensus_queue.cc:260] T b7d8da6e2fe8499d9d53af84a25f2faf P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:55.384950 21395 raft_consensus.cc:2258] T f3711610610542128c4a84e17bce4a3e P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:55.385617 21395 ts_tablet_manager.cc:1875] T f3711610610542128c4a84e17bce4a3e P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:55.387039 20489 raft_consensus.cc:482] T f8bfbf673ea346529d1b2b376a3da496 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:55.387210 20489 raft_consensus.cc:504] T f8bfbf673ea346529d1b2b376a3da496 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:55.387877 20489 leader_election.cc:288] T f8bfbf673ea346529d1b2b376a3da496 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:55.388893 20489 raft_consensus.cc:482] T 4391b5227fb64ef481542a962a5e21a5 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:55.389001 20489 raft_consensus.cc:504] T 4391b5227fb64ef481542a962a5e21a5 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:55.389601 20489 leader_election.cc:288] T 4391b5227fb64ef481542a962a5e21a5 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:55.390583 19946 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 724e51ff05ec4baebc2bb55e05c8eddc (table default.Table_19df33614a6c4f94b6419a5b441e3ef8 [id=4dce8a767f0541e1beb5464b2ed95d06]). Expected version 2 got 1
I20240430 20:11:55.365618 18988 tablet_bootstrap.cc:492] T 00be6ddd91e84865a48743fb4d077eec P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:55.391410 18983 ts_tablet_manager.cc:1398] T b7d8da6e2fe8499d9d53af84a25f2faf P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20240430 20:11:55.391708 18983 tablet_bootstrap.cc:492] T 3b7fdc6ec7e046c5af894f9a6eda17b0 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:55.392604 18988 tablet_bootstrap.cc:654] T 00be6ddd91e84865a48743fb4d077eec P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:55.393229 18983 tablet_bootstrap.cc:654] T 3b7fdc6ec7e046c5af894f9a6eda17b0 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:55.376215 18810 heartbeater.cc:503] Master 127.17.177.60:37225 was elected leader, sending a full tablet report...
I20240430 20:11:55.384249 21463 consensus_queue.cc:1035] T 1050ab528c7d41559831652a1437d358 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:55.388760 18756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8bfbf673ea346529d1b2b376a3da496" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:55.396358 18756 raft_consensus.cc:2454] T f8bfbf673ea346529d1b2b376a3da496 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 0.
I20240430 20:11:55.390237 18749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4391b5227fb64ef481542a962a5e21a5" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:55.396842 18557 leader_election.cc:302] T f8bfbf673ea346529d1b2b376a3da496 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:55.397105 20489 raft_consensus.cc:2781] T f8bfbf673ea346529d1b2b376a3da496 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:55.397181 20489 raft_consensus.cc:482] T f8bfbf673ea346529d1b2b376a3da496 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:55.397243 20489 raft_consensus.cc:3037] T f8bfbf673ea346529d1b2b376a3da496 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:55.401585 20489 raft_consensus.cc:504] T f8bfbf673ea346529d1b2b376a3da496 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:55.403530 18749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8bfbf673ea346529d1b2b376a3da496" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:55.403719 18749 raft_consensus.cc:3037] T f8bfbf673ea346529d1b2b376a3da496 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
W20240430 20:11:55.403724 18557 leader_election.cc:341] T 4391b5227fb64ef481542a962a5e21a5 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:55.405086 20489 leader_election.cc:288] T f8bfbf673ea346529d1b2b376a3da496 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:55.406437 18749 raft_consensus.cc:2454] T f8bfbf673ea346529d1b2b376a3da496 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
I20240430 20:11:55.406889 18557 leader_election.cc:302] T f8bfbf673ea346529d1b2b376a3da496 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:55.407157 20489 raft_consensus.cc:2781] T f8bfbf673ea346529d1b2b376a3da496 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:55.407397 20489 raft_consensus.cc:686] T f8bfbf673ea346529d1b2b376a3da496 P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Becoming Leader. State: Replica: 5e1c6397f09747f091534faf93542710, State: Running, Role: LEADER
I20240430 20:11:55.407711 20489 consensus_queue.cc:237] T f8bfbf673ea346529d1b2b376a3da496 P 5e1c6397f09747f091534faf93542710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:55.416415 21396 ts_tablet_manager.cc:1888] T a663030d37614053a7c0369841403422 P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:55.416618 21396 log.cc:1196] T a663030d37614053a7c0369841403422 P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/a663030d37614053a7c0369841403422
I20240430 20:11:55.417037 21396 ts_tablet_manager.cc:1909] T a663030d37614053a7c0369841403422 P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:55.418911 21396 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:45934 (request call id 71) took 1640 ms. Trace:
I20240430 20:11:55.419029 18983 tablet_bootstrap.cc:492] T 3b7fdc6ec7e046c5af894f9a6eda17b0 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:55.419220 18983 ts_tablet_manager.cc:1367] T 3b7fdc6ec7e046c5af894f9a6eda17b0 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.028s user 0.003s sys 0.001s
I20240430 20:11:55.419337 21397 log.cc:1196] T 90a694bd5db649afa50f3c4e752939c1 P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/90a694bd5db649afa50f3c4e752939c1
I20240430 20:11:55.419987 21397 ts_tablet_manager.cc:1909] T 90a694bd5db649afa50f3c4e752939c1 P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:55.420063 18983 raft_consensus.cc:348] T 3b7fdc6ec7e046c5af894f9a6eda17b0 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:55.420331 18983 raft_consensus.cc:374] T 3b7fdc6ec7e046c5af894f9a6eda17b0 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:55.419119 21396 rpcz_store.cc:276] 0430 20:11:53.778397 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:53.778458 (+ 61us) service_pool.cc:225] Handling call
0430 20:11:55.363884 (+1585426us) ts_tablet_manager.cc:1120] Deleting tablet a663030d37614053a7c0369841403422
0430 20:11:55.363899 (+ 15us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:55.370369 (+ 6470us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:55.416281 (+ 45912us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:55.418893 (+ 2612us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1585170}
I20240430 20:11:55.421126 21396 tablet_replica.cc:330] T 436579f473844c6ca1038b496201a550 P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:55.421351 21396 raft_consensus.cc:2229] T 436579f473844c6ca1038b496201a550 P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Raft consensus shutting down.
I20240430 20:11:55.421666 21396 raft_consensus.cc:2258] T 436579f473844c6ca1038b496201a550 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:55.422345 21396 ts_tablet_manager.cc:1875] T 436579f473844c6ca1038b496201a550 P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:55.423312 19946 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet ca64de16d40f4a44b8b411f5526ba11a (table default.Table_1cf914acf7fe4e98b8dbc297b6372f21 [id=f4ba9536e9c141deb68f4f62b7e8fa70]). Expected version 1 got 0
I20240430 20:11:55.388119 21535 tablet.cc:1722] T ed0c37a71ede4bfc9eb5ad1a4365f25e P 2efb542aa55e439b99ccb98d1c5c884a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20240430 20:11:55.394790 18415 leader_election.cc:341] T c6378f1495144c458af49b3c19a2b67c P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Not found: Tablet not found: c6378f1495144c458af49b3c19a2b67c
I20240430 20:11:55.423938 18415 leader_election.cc:302] T c6378f1495144c458af49b3c19a2b67c P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a; no voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710
W20240430 20:11:55.424611 18415 consensus_peers.cc:485] T 04b3dff86d5c48a09d03da58bd48f360 P 2efb542aa55e439b99ccb98d1c5c884a -> Peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Couldn't send request to peer 5e1c6397f09747f091534faf93542710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 04b3dff86d5c48a09d03da58bd48f360. This is attempt 1: this message will repeat every 5th retry.
I20240430 20:11:55.425436 21437 raft_consensus.cc:2726] T c6378f1495144c458af49b3c19a2b67c P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240430 20:11:55.428436 18606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0d3c4ba64024bccad95ecc909d03ae0" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:55.435272 18472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0d3c4ba64024bccad95ecc909d03ae0" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:55.435482 18472 raft_consensus.cc:2454] T a0d3c4ba64024bccad95ecc909d03ae0 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 0.
I20240430 20:11:55.435972 18472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0d3c4ba64024bccad95ecc909d03ae0" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:55.436118 18472 raft_consensus.cc:3037] T a0d3c4ba64024bccad95ecc909d03ae0 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:55.436982 18415 leader_election.cc:302] T a0d3c4ba64024bccad95ecc909d03ae0 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a; no voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710
W20240430 20:11:55.437633 18415 leader_election.cc:341] T a663030d37614053a7c0369841403422 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Not found: Tablet not found: a663030d37614053a7c0369841403422
I20240430 20:11:55.437734 18415 leader_election.cc:302] T a663030d37614053a7c0369841403422 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a; no voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710
I20240430 20:11:55.438637 21581 raft_consensus.cc:2726] T a663030d37614053a7c0369841403422 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240430 20:11:55.439041 18472 raft_consensus.cc:2454] T a0d3c4ba64024bccad95ecc909d03ae0 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
I20240430 20:11:55.439255 20630 raft_consensus.cc:2726] T a0d3c4ba64024bccad95ecc909d03ae0 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:55.442543 18983 raft_consensus.cc:729] T 3b7fdc6ec7e046c5af894f9a6eda17b0 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:55.442727 21366 raft_consensus.cc:482] T b7d8da6e2fe8499d9d53af84a25f2faf P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:55.442873 21366 raft_consensus.cc:504] T b7d8da6e2fe8499d9d53af84a25f2faf P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:55.443102 18983 consensus_queue.cc:260] T 3b7fdc6ec7e046c5af894f9a6eda17b0 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:55.443476 21366 leader_election.cc:288] T b7d8da6e2fe8499d9d53af84a25f2faf P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:55.444748 18983 ts_tablet_manager.cc:1398] T 3b7fdc6ec7e046c5af894f9a6eda17b0 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.025s user 0.013s sys 0.000s
I20240430 20:11:55.445122 18983 tablet_bootstrap.cc:492] T 9e741b00c74d43b5b6243ba7003f8804 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:55.446991 18983 tablet_bootstrap.cc:654] T 9e741b00c74d43b5b6243ba7003f8804 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:55.447849 21397 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56540 (request call id 60) took 1711 ms. Trace:
I20240430 20:11:55.431994 18988 tablet_bootstrap.cc:492] T 00be6ddd91e84865a48743fb4d077eec P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:55.444020 18756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7d8da6e2fe8499d9d53af84a25f2faf" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:55.448101 21454 raft_consensus.cc:482] T 3b7fdc6ec7e046c5af894f9a6eda17b0 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:55.448197 18988 ts_tablet_manager.cc:1367] T 00be6ddd91e84865a48743fb4d077eec P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.083s user 0.002s sys 0.006s
I20240430 20:11:55.448207 21454 raft_consensus.cc:504] T 3b7fdc6ec7e046c5af894f9a6eda17b0 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
W20240430 20:11:55.448648 18557 leader_election.cc:341] T b7d8da6e2fe8499d9d53af84a25f2faf P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:55.445027 21493 raft_consensus.cc:482] T fd1b8407071e4e4287fc172009777414 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:55.448844 21493 raft_consensus.cc:504] T fd1b8407071e4e4287fc172009777414 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:55.448902 18988 raft_consensus.cc:348] T 00be6ddd91e84865a48743fb4d077eec P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:55.449086 18988 raft_consensus.cc:374] T 00be6ddd91e84865a48743fb4d077eec P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:55.449151 18988 raft_consensus.cc:729] T 00be6ddd91e84865a48743fb4d077eec P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:55.449333 18988 consensus_queue.cc:260] T 00be6ddd91e84865a48743fb4d077eec P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:55.449470 21493 leader_election.cc:288] T fd1b8407071e4e4287fc172009777414 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:55.449545 18617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b7fdc6ec7e046c5af894f9a6eda17b0" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:55.449726 18617 raft_consensus.cc:2454] T 3b7fdc6ec7e046c5af894f9a6eda17b0 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:55.449968 18988 ts_tablet_manager.cc:1398] T 00be6ddd91e84865a48743fb4d077eec P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240430 20:11:55.448920 21454 leader_election.cc:288] T 3b7fdc6ec7e046c5af894f9a6eda17b0 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:55.450237 18988 tablet_bootstrap.cc:492] T 347a8224114b49e5b7dcb75d9b726e0a P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:55.450271 18696 leader_election.cc:302] T 3b7fdc6ec7e046c5af894f9a6eda17b0 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:55.450675 21454 raft_consensus.cc:2781] T 3b7fdc6ec7e046c5af894f9a6eda17b0 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:55.450744 21454 raft_consensus.cc:482] T 3b7fdc6ec7e046c5af894f9a6eda17b0 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:55.450798 21454 raft_consensus.cc:3037] T 3b7fdc6ec7e046c5af894f9a6eda17b0 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:55.451807 18988 tablet_bootstrap.cc:654] T 347a8224114b49e5b7dcb75d9b726e0a P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:55.453634 21454 raft_consensus.cc:504] T 3b7fdc6ec7e046c5af894f9a6eda17b0 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:55.421152 19895 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet a663030d37614053a7c0369841403422 (table default.Table_e34613952dcb46618c3bc7cecb31fdaf [id=8f1ad1f8d0224b3daa5d84dc6916227b]) successfully deleted
I20240430 20:11:55.455054 19946 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 81787eccca6c42c1b421377a3bf9ebe6 (table default.Table_0b4ca4cd0af148399dfa21f2d32114e3 [id=7f602584cef9486eb7d9b4389be3f43e]). Expected version 3 got 2
I20240430 20:11:55.455233 18607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b7fdc6ec7e046c5af894f9a6eda17b0" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:55.455428 18607 raft_consensus.cc:3037] T 3b7fdc6ec7e046c5af894f9a6eda17b0 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:55.455595 19946 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 66dd767bd3f8494f94061fb89e5e5680 (table default.Table_b064d5659f7243f5b0ec167fb6c3f404 [id=5227206f44d04fa89b4842b899f33afb]). Expected version 2 got 0
I20240430 20:11:55.455782 19946 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet f1073fc792954c5795a05aa7d0ec24df (table default.Table_ba8b218e936b4bfcb073f8182f474cca [id=aedf11fda8914a3b843a57f8a7b69a53]). Expected version 1 got 0
I20240430 20:11:55.456434 19946 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet fbdeb4714b8c4b93ab72822c43329aae (table default.Table_c12ca6938d424fbfa2e1df5cdb00fdbc [id=0fb93f36bc17408ea56bc4cafcfd8580]). Expected version 1 got 0
I20240430 20:11:55.457057 19946 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 488d0e609b224674932f890939664df2 (table default.Table_58f35a2120874472978ca0e5f816e86c [id=ff18001c6d814afc8c1768d0c84a6bac]). Expected version 1 got 0
I20240430 20:11:55.457738 18617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd1b8407071e4e4287fc172009777414" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:55.457978 18617 raft_consensus.cc:2454] T fd1b8407071e4e4287fc172009777414 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:55.458160 19946 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 78ba48dadef548708153878810cc46f8 (table default.Table_b9983e101a6547649eb44ee04b5e1147 [id=29769be2b9ec4ecba9de5c0a65692f85]). Expected version 1 got 0
I20240430 20:11:55.458323 18607 raft_consensus.cc:2454] T 3b7fdc6ec7e046c5af894f9a6eda17b0 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:55.458401 19946 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 2cd41b407d6b444d844c4c7c0ebf82e5 (table default.Table_6f8e0eff8f5a484dbd6a9c3fdb5d03f6 [id=3813b5e9778c4bd683a06598cef0d86f]). Expected version 1 got 0
I20240430 20:11:55.459620 18985 tablet_bootstrap.cc:492] T 347a8224114b49e5b7dcb75d9b726e0a P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
W20240430 20:11:55.459734 18696 consensus_peers.cc:485] T 47e98935eb21449abd3d4d24327778d5 P 3c51c33d5616482cafb68e26688820f4 -> Peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Couldn't send request to peer 5e1c6397f09747f091534faf93542710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 47e98935eb21449abd3d4d24327778d5. This is attempt 1: this message will repeat every 5th retry.
I20240430 20:11:55.459767 18985 ts_tablet_manager.cc:1367] T 347a8224114b49e5b7dcb75d9b726e0a P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.132s user 0.000s sys 0.002s
I20240430 20:11:55.459899 18696 leader_election.cc:302] T fd1b8407071e4e4287fc172009777414 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:55.460143 18696 leader_election.cc:302] T 3b7fdc6ec7e046c5af894f9a6eda17b0 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:55.460361 21493 raft_consensus.cc:2781] T fd1b8407071e4e4287fc172009777414 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:55.460433 21493 raft_consensus.cc:482] T fd1b8407071e4e4287fc172009777414 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:55.460489 21493 raft_consensus.cc:3037] T fd1b8407071e4e4287fc172009777414 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:55.460610 21436 raft_consensus.cc:2781] T 3b7fdc6ec7e046c5af894f9a6eda17b0 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:55.460584 18985 raft_consensus.cc:348] T 347a8224114b49e5b7dcb75d9b726e0a P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:55.460718 21436 raft_consensus.cc:686] T 3b7fdc6ec7e046c5af894f9a6eda17b0 P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Becoming Leader. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Running, Role: LEADER
I20240430 20:11:55.460784 18985 raft_consensus.cc:374] T 347a8224114b49e5b7dcb75d9b726e0a P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:55.460844 18985 raft_consensus.cc:729] T 347a8224114b49e5b7dcb75d9b726e0a P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:55.460966 21436 consensus_queue.cc:237] T 3b7fdc6ec7e046c5af894f9a6eda17b0 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:55.461062 18985 consensus_queue.cc:260] T 347a8224114b49e5b7dcb75d9b726e0a P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:55.461877 21396 ts_tablet_manager.cc:1888] T 436579f473844c6ca1038b496201a550 P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:55.461977 21396 log.cc:1196] T 436579f473844c6ca1038b496201a550 P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/436579f473844c6ca1038b496201a550
I20240430 20:11:55.462090 18985 ts_tablet_manager.cc:1398] T 347a8224114b49e5b7dcb75d9b726e0a P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240430 20:11:55.462304 21396 ts_tablet_manager.cc:1909] T 436579f473844c6ca1038b496201a550 P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:55.464030 21493 raft_consensus.cc:504] T fd1b8407071e4e4287fc172009777414 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:55.464937 21493 leader_election.cc:288] T fd1b8407071e4e4287fc172009777414 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Requested vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:55.465366 21450 raft_consensus.cc:482] T 8668f7a285de43f48d83abecb4b2e96c P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:55.465592 21450 raft_consensus.cc:504] T 8668f7a285de43f48d83abecb4b2e96c P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:55.465875 18810 heartbeater.cc:503] Master 127.17.177.60:37225 was elected leader, sending a full tablet report...
I20240430 20:11:55.466127 21450 leader_election.cc:288] T 8668f7a285de43f48d83abecb4b2e96c P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:55.467947 21535 tablet.cc:1722] T 62732af95c6a4c749c5b726f0460dc72 P 2efb542aa55e439b99ccb98d1c5c884a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240430 20:11:55.468736 18607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd1b8407071e4e4287fc172009777414" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:55.468919 18607 raft_consensus.cc:3037] T fd1b8407071e4e4287fc172009777414 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:55.469014 21396 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:45934 (request call id 72) took 1690 ms. Trace:
I20240430 20:11:55.469082 21396 rpcz_store.cc:276] 0430 20:11:53.778494 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:53.778939 (+ 445us) service_pool.cc:225] Handling call
0430 20:11:55.421074 (+1642135us) ts_tablet_manager.cc:1120] Deleting tablet 436579f473844c6ca1038b496201a550
0430 20:11:55.421084 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:55.458880 (+ 37796us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:55.461778 (+ 2898us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:55.468998 (+ 7220us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1641941}
I20240430 20:11:55.469235 18472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8bfbf673ea346529d1b2b376a3da496" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:55.469331 18619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8668f7a285de43f48d83abecb4b2e96c" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:55.469412 18472 raft_consensus.cc:2454] T f8bfbf673ea346529d1b2b376a3da496 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 0.
I20240430 20:11:55.469442 21396 tablet_replica.cc:330] T dad16ae02b8b4e29a724d8eec67bb4b8 P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:55.469480 18619 raft_consensus.cc:2454] T 8668f7a285de43f48d83abecb4b2e96c P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:55.469643 21396 raft_consensus.cc:2229] T dad16ae02b8b4e29a724d8eec67bb4b8 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:55.469753 18472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4391b5227fb64ef481542a962a5e21a5" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:55.469779 19895 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet 436579f473844c6ca1038b496201a550 (table default.Table_273caa9a42a9474e8787c37032368654 [id=d89426aa652c42e49f956ea85efb0d43]) successfully deleted
I20240430 20:11:55.470185 18696 leader_election.cc:302] T 8668f7a285de43f48d83abecb4b2e96c P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:55.470185 18472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8bfbf673ea346529d1b2b376a3da496" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:55.470319 18472 raft_consensus.cc:3037] T f8bfbf673ea346529d1b2b376a3da496 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:55.470451 21450 raft_consensus.cc:2781] T 8668f7a285de43f48d83abecb4b2e96c P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:55.470537 21450 raft_consensus.cc:482] T 8668f7a285de43f48d83abecb4b2e96c P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:55.470639 21450 raft_consensus.cc:3037] T 8668f7a285de43f48d83abecb4b2e96c P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:55.470491 21396 raft_consensus.cc:2258] T dad16ae02b8b4e29a724d8eec67bb4b8 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:55.470721 18480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7d8da6e2fe8499d9d53af84a25f2faf" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:55.471341 21396 ts_tablet_manager.cc:1875] T dad16ae02b8b4e29a724d8eec67bb4b8 P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:55.471478 18480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69f88532952a4591bf1577541c3430e6" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:55.471668 18480 raft_consensus.cc:2379] T 69f88532952a4591bf1577541c3430e6 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c51c33d5616482cafb68e26688820f4 in current term 1: Already voted for candidate 2efb542aa55e439b99ccb98d1c5c884a in this term.
I20240430 20:11:55.473248 21450 raft_consensus.cc:504] T 8668f7a285de43f48d83abecb4b2e96c P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:55.474115 21450 leader_election.cc:288] T 8668f7a285de43f48d83abecb4b2e96c P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Requested vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:55.471864 18607 raft_consensus.cc:2454] T fd1b8407071e4e4287fc172009777414 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:55.473500 18983 tablet_bootstrap.cc:492] T 9e741b00c74d43b5b6243ba7003f8804 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:55.475816 21454 leader_election.cc:288] T 3b7fdc6ec7e046c5af894f9a6eda17b0 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Requested vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:55.475840 18983 ts_tablet_manager.cc:1367] T 9e741b00c74d43b5b6243ba7003f8804 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.031s user 0.006s sys 0.000s
I20240430 20:11:55.476190 18696 leader_election.cc:302] T fd1b8407071e4e4287fc172009777414 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:55.476496 21454 raft_consensus.cc:2781] T fd1b8407071e4e4287fc172009777414 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:55.476460 18983 raft_consensus.cc:348] T 9e741b00c74d43b5b6243ba7003f8804 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:55.476648 18983 raft_consensus.cc:374] T 9e741b00c74d43b5b6243ba7003f8804 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:55.476716 18983 raft_consensus.cc:729] T 9e741b00c74d43b5b6243ba7003f8804 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:55.476698 21454 raft_consensus.cc:686] T fd1b8407071e4e4287fc172009777414 P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Becoming Leader. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Running, Role: LEADER
I20240430 20:11:55.474656 18619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8668f7a285de43f48d83abecb4b2e96c" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:55.476919 18619 raft_consensus.cc:3037] T 8668f7a285de43f48d83abecb4b2e96c P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:55.447949 21397 rpcz_store.cc:276] 0430 20:11:53.735964 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:53.736005 (+ 41us) service_pool.cc:225] Handling call
0430 20:11:55.139240 (+1403235us) ts_tablet_manager.cc:1120] Deleting tablet 90a694bd5db649afa50f3c4e752939c1
0430 20:11:55.139254 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:55.235589 (+ 96335us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:55.353469 (+117880us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:55.447817 (+ 94348us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1403088}
I20240430 20:11:55.477142 21454 consensus_queue.cc:237] T fd1b8407071e4e4287fc172009777414 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:55.477414 19900 catalog_manager.cc:4830] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): tablet 90a694bd5db649afa50f3c4e752939c1 (table default.Table_97a0991bf9284215b482c532e9dfc113 [id=7e3ac0868c0a4cc581427f121f305323]) successfully deleted
I20240430 20:11:55.477641 21548 tablet.cc:1722] T 12795670f21447a8964fe1efc6e92398 P 2efb542aa55e439b99ccb98d1c5c884a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240430 20:11:55.477749 21397 tablet_replica.cc:330] T c1017a50d1d646c486c0b9a75cb88ce4 P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:55.477918 21397 raft_consensus.cc:2229] T c1017a50d1d646c486c0b9a75cb88ce4 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:55.478088 21397 raft_consensus.cc:2258] T c1017a50d1d646c486c0b9a75cb88ce4 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:55.479588 18983 consensus_queue.cc:260] T 9e741b00c74d43b5b6243ba7003f8804 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:55.481323 18983 ts_tablet_manager.cc:1398] T 9e741b00c74d43b5b6243ba7003f8804 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240430 20:11:55.482053 18619 raft_consensus.cc:2454] T 8668f7a285de43f48d83abecb4b2e96c P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:55.482334 18983 tablet_bootstrap.cc:492] T fcc16db4d6be42edb7bd1ce18fe0190d P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:55.482806 18696 leader_election.cc:302] T 8668f7a285de43f48d83abecb4b2e96c P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:55.483109 21450 raft_consensus.cc:2781] T 8668f7a285de43f48d83abecb4b2e96c P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:55.483350 21450 raft_consensus.cc:686] T 8668f7a285de43f48d83abecb4b2e96c P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Becoming Leader. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Running, Role: LEADER
I20240430 20:11:55.483604 21450 consensus_queue.cc:237] T 8668f7a285de43f48d83abecb4b2e96c P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:55.483788 18983 tablet_bootstrap.cc:654] T fcc16db4d6be42edb7bd1ce18fe0190d P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:55.486901 21397 ts_tablet_manager.cc:1875] T c1017a50d1d646c486c0b9a75cb88ce4 P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:55.489758 18471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a663030d37614053a7c0369841403422" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:55.490044 18471 raft_consensus.cc:2379] T a663030d37614053a7c0369841403422 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c51c33d5616482cafb68e26688820f4 in current term 1: Already voted for candidate 2efb542aa55e439b99ccb98d1c5c884a in this term.
I20240430 20:11:55.490924 18474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a663030d37614053a7c0369841403422" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:55.491164 18474 raft_consensus.cc:2379] T a663030d37614053a7c0369841403422 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 3c51c33d5616482cafb68e26688820f4 in current term 1: Already voted for candidate 2efb542aa55e439b99ccb98d1c5c884a in this term.
I20240430 20:11:55.491978 21605 tablet.cc:1722] T ae20f44df50343289b0a512a4ea53692 P 2efb542aa55e439b99ccb98d1c5c884a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240430 20:11:55.492408 20630 raft_consensus.cc:482] T 3ec3993042f5498b818bdf32d5087928 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:55.492523 20630 raft_consensus.cc:504] T 3ec3993042f5498b818bdf32d5087928 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:55.494655 18475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c73d0a6bc414d3b9900570848d19a51" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:55.496204 21542 tablet.cc:1722] T 130007acc8ba4e709c3eb85fabe75903 P 2efb542aa55e439b99ccb98d1c5c884a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240430 20:11:55.496727 21396 ts_tablet_manager.cc:1888] T dad16ae02b8b4e29a724d8eec67bb4b8 P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:55.496941 21396 log.cc:1196] T dad16ae02b8b4e29a724d8eec67bb4b8 P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/dad16ae02b8b4e29a724d8eec67bb4b8
I20240430 20:11:55.497489 21396 ts_tablet_manager.cc:1909] T dad16ae02b8b4e29a724d8eec67bb4b8 P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:55.498548 21396 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:45934 (request call id 73) took 1719 ms. Trace:
I20240430 20:11:55.502854 19946 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 772eba5f24c549c791cad75565858285 (table default.Table_5fc77d5f60ae44f4bb4ca9f7f183672c [id=6ca223ca6354449c959d85ed2ac6a9c8]). Expected version 1 got 0
I20240430 20:11:55.507647 21396 rpcz_store.cc:276] 0430 20:11:53.779308 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:53.779650 (+ 342us) service_pool.cc:225] Handling call
0430 20:11:55.469392 (+1689742us) ts_tablet_manager.cc:1120] Deleting tablet dad16ae02b8b4e29a724d8eec67bb4b8
0430 20:11:55.469402 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:55.482215 (+ 12813us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:55.496585 (+ 14370us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:55.498523 (+ 1938us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1689572}
I20240430 20:11:55.508427 19895 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet dad16ae02b8b4e29a724d8eec67bb4b8 (table default.Table_8a6b5152d20b4890918aedb454764ee9 [id=121444fdfd2642a38cdbd3a6c411c4fd]) successfully deleted
I20240430 20:11:55.508512 21396 tablet_replica.cc:330] T fd1b8407071e4e4287fc172009777414 P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:55.508914 21396 raft_consensus.cc:2229] T fd1b8407071e4e4287fc172009777414 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:55.494498 18479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "330453ddeea6456e8ef93fa92608b0d3" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:55.509327 18479 raft_consensus.cc:2379] T 330453ddeea6456e8ef93fa92608b0d3 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 3c51c33d5616482cafb68e26688820f4 in current term 1: Already voted for candidate 2efb542aa55e439b99ccb98d1c5c884a in this term.
W20240430 20:11:55.510318 18417 leader_election.cc:341] T 3ec3993042f5498b818bdf32d5087928 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Not found: Tablet not found: 3ec3993042f5498b818bdf32d5087928
W20240430 20:11:55.511106 18415 consensus_peers.cc:485] T d7c1fc7e81364fb890bbc13d7fb6848b P 2efb542aa55e439b99ccb98d1c5c884a -> Peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Couldn't send request to peer 5e1c6397f09747f091534faf93542710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d7c1fc7e81364fb890bbc13d7fb6848b. This is attempt 1: this message will repeat every 5th retry.
I20240430 20:11:55.511230 18988 tablet_bootstrap.cc:492] T 347a8224114b49e5b7dcb75d9b726e0a P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:55.511416 18988 ts_tablet_manager.cc:1367] T 347a8224114b49e5b7dcb75d9b726e0a P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.061s user 0.002s sys 0.001s
I20240430 20:11:55.511633 21396 raft_consensus.cc:2258] T fd1b8407071e4e4287fc172009777414 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:55.512444 21396 ts_tablet_manager.cc:1875] T fd1b8407071e4e4287fc172009777414 P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:55.512406 18988 raft_consensus.cc:348] T 347a8224114b49e5b7dcb75d9b726e0a P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:55.512681 18988 raft_consensus.cc:374] T 347a8224114b49e5b7dcb75d9b726e0a P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:55.512754 18988 raft_consensus.cc:729] T 347a8224114b49e5b7dcb75d9b726e0a P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:55.513056 18988 consensus_queue.cc:260] T 347a8224114b49e5b7dcb75d9b726e0a P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:55.514318 18988 ts_tablet_manager.cc:1398] T 347a8224114b49e5b7dcb75d9b726e0a P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240430 20:11:55.515156 18988 tablet_bootstrap.cc:492] T e5b85a066cdc46d48ccf8201e5423c29 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:55.519448 18983 tablet_bootstrap.cc:492] T fcc16db4d6be42edb7bd1ce18fe0190d P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:55.519595 18983 ts_tablet_manager.cc:1367] T fcc16db4d6be42edb7bd1ce18fe0190d P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.037s user 0.011s sys 0.002s
I20240430 20:11:55.520663 18983 raft_consensus.cc:348] T fcc16db4d6be42edb7bd1ce18fe0190d P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:55.520879 18983 raft_consensus.cc:374] T fcc16db4d6be42edb7bd1ce18fe0190d P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:55.520956 18983 raft_consensus.cc:729] T fcc16db4d6be42edb7bd1ce18fe0190d P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:55.521190 18983 consensus_queue.cc:260] T fcc16db4d6be42edb7bd1ce18fe0190d P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:55.521800 18983 ts_tablet_manager.cc:1398] T fcc16db4d6be42edb7bd1ce18fe0190d P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240430 20:11:55.526433 20630 leader_election.cc:288] T 3ec3993042f5498b818bdf32d5087928 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:55.526705 18481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69f88532952a4591bf1577541c3430e6" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:55.526960 18481 raft_consensus.cc:2379] T 69f88532952a4591bf1577541c3430e6 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 3c51c33d5616482cafb68e26688820f4 in current term 1: Already voted for candidate 2efb542aa55e439b99ccb98d1c5c884a in this term.
W20240430 20:11:55.528548 18415 leader_election.cc:341] T 3ec3993042f5498b818bdf32d5087928 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Not found: Tablet not found: 3ec3993042f5498b818bdf32d5087928
I20240430 20:11:55.528678 18415 leader_election.cc:302] T 3ec3993042f5498b818bdf32d5087928 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a; no voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710
I20240430 20:11:55.529126 20630 raft_consensus.cc:2726] T 3ec3993042f5498b818bdf32d5087928 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240430 20:11:55.546958 18983 tablet_bootstrap.cc:492] T f9bf604e6c234349a9f72f45a4c4aef6 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:55.548444 18983 tablet_bootstrap.cc:654] T f9bf604e6c234349a9f72f45a4c4aef6 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:55.549192 21535 tablet.cc:1722] T a482f4622f774d60b4efd6827b8be7d7 P 2efb542aa55e439b99ccb98d1c5c884a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240430 20:11:55.549657 21396 ts_tablet_manager.cc:1888] T fd1b8407071e4e4287fc172009777414 P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:55.554723 21396 log.cc:1196] T fd1b8407071e4e4287fc172009777414 P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/fd1b8407071e4e4287fc172009777414
I20240430 20:11:55.555661 18472 raft_consensus.cc:2454] T f8bfbf673ea346529d1b2b376a3da496 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
I20240430 20:11:55.558975 21396 ts_tablet_manager.cc:1909] T fd1b8407071e4e4287fc172009777414 P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:55.560310 21396 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:45934 (request call id 74) took 1780 ms. Trace:
I20240430 20:11:55.560482 21396 rpcz_store.cc:276] 0430 20:11:53.780235 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:53.780464 (+ 229us) service_pool.cc:225] Handling call
0430 20:11:55.508437 (+1727973us) ts_tablet_manager.cc:1120] Deleting tablet fd1b8407071e4e4287fc172009777414
0430 20:11:55.508453 (+ 16us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:55.515415 (+ 6962us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:55.549566 (+ 34151us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:55.560290 (+ 10724us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1727805}
I20240430 20:11:55.560763 21581 consensus_queue.cc:1035] T 4dac8b3838954d839094021f1c0503da P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:55.561215 21396 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:45934 (request call id 76) took 1750 ms. Trace:
I20240430 20:11:55.561290 21396 rpcz_store.cc:276] 0430 20:11:53.811093 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:53.811137 (+ 44us) service_pool.cc:225] Handling call
0430 20:11:55.561084 (+1749947us) ts_tablet_manager.cc:1120] Deleting tablet a37d917bcc45468f9e3ee805640fba98
0430 20:11:55.561094 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:55.561197 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1749706}
I20240430 20:11:55.561430 19895 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet fd1b8407071e4e4287fc172009777414 (table default.Table_5572f0acadaa432e8570b159fd9b76bc [id=d0f3b1709ac94aed8ec568c0327eab34]) successfully deleted
I20240430 20:11:55.561592 21396 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:45934 (request call id 80) took 1747 ms. Trace:
I20240430 20:11:55.561666 21396 rpcz_store.cc:276] 0430 20:11:53.814647 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:53.814697 (+ 50us) service_pool.cc:225] Handling call
0430 20:11:55.561508 (+1746811us) ts_tablet_manager.cc:1120] Deleting tablet 7bde1d6934784c9aa5911181e8df9eeb
0430 20:11:55.561518 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:55.561579 (+ 61us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1746577}
W20240430 20:11:55.561775 19895 catalog_manager.cc:4810] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): delete failed for tablet a37d917bcc45468f9e3ee805640fba98 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a37d917bcc45468f9e3ee805640fba98 already in progress: creating tablet
W20240430 20:11:55.562044 19895 catalog_manager.cc:4797] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): delete failed for tablet 7bde1d6934784c9aa5911181e8df9eeb because the tablet was not found. No further retry: Not found: Tablet not found: 7bde1d6934784c9aa5911181e8df9eeb
I20240430 20:11:55.563339 18983 tablet_bootstrap.cc:492] T f9bf604e6c234349a9f72f45a4c4aef6 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:55.563464 18983 ts_tablet_manager.cc:1367] T f9bf604e6c234349a9f72f45a4c4aef6 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.001s
I20240430 20:11:55.564105 18983 raft_consensus.cc:348] T f9bf604e6c234349a9f72f45a4c4aef6 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:55.564298 18983 raft_consensus.cc:374] T f9bf604e6c234349a9f72f45a4c4aef6 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:55.564357 18983 raft_consensus.cc:729] T f9bf604e6c234349a9f72f45a4c4aef6 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:55.564543 18983 consensus_queue.cc:260] T f9bf604e6c234349a9f72f45a4c4aef6 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:55.565088 18983 ts_tablet_manager.cc:1398] T f9bf604e6c234349a9f72f45a4c4aef6 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240430 20:11:55.565913 21436 consensus_queue.cc:1035] T 3191aa54a9644783a2403991c3be27c8 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:55.568771 21535 tablet.cc:1722] T 0368d37dd9e2452982fb81f539c6805b P 2efb542aa55e439b99ccb98d1c5c884a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240430 20:11:55.569775 18983 tablet_bootstrap.cc:492] T 8e7f08e9ac274bfa86763eec895bbfda P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:55.571553 18983 tablet_bootstrap.cc:654] T 8e7f08e9ac274bfa86763eec895bbfda P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:55.572542 21396 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:45934 (request call id 79) took 1758 ms. Trace:
I20240430 20:11:55.572664 21396 rpcz_store.cc:276] 0430 20:11:53.814463 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:53.815583 (+ 1120us) service_pool.cc:225] Handling call
0430 20:11:55.565219 (+1749636us) ts_tablet_manager.cc:1120] Deleting tablet 024069ab786749028c020abd1c836cf5
0430 20:11:55.572367 (+ 7148us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:55.572517 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1746094}
W20240430 20:11:55.573267 19895 catalog_manager.cc:4810] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): delete failed for tablet 024069ab786749028c020abd1c836cf5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 024069ab786749028c020abd1c836cf5 already in progress: creating tablet
I20240430 20:11:55.573323 21396 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:45934 (request call id 78) took 1759 ms. Trace:
I20240430 20:11:55.573411 21396 rpcz_store.cc:276] 0430 20:11:53.814337 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:53.815807 (+ 1470us) service_pool.cc:225] Handling call
0430 20:11:55.573187 (+1757380us) ts_tablet_manager.cc:1120] Deleting tablet 2fc6908ecf954020a676b9d10a2b00e7
0430 20:11:55.573201 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:55.573303 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1757218}
W20240430 20:11:55.573868 19895 catalog_manager.cc:4797] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): delete failed for tablet 2fc6908ecf954020a676b9d10a2b00e7 because the tablet was not found. No further retry: Not found: Tablet not found: 2fc6908ecf954020a676b9d10a2b00e7
I20240430 20:11:55.573913 21396 tablet_replica.cc:330] T 2797eed593fb4b06a3c4d6a28e8075a2 P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:55.574154 21396 raft_consensus.cc:2229] T 2797eed593fb4b06a3c4d6a28e8075a2 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:55.574308 21396 raft_consensus.cc:2258] T 2797eed593fb4b06a3c4d6a28e8075a2 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:55.585669 21548 tablet.cc:1722] T 81787eccca6c42c1b421377a3bf9ebe6 P 2efb542aa55e439b99ccb98d1c5c884a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240430 20:11:55.587678 18983 tablet_bootstrap.cc:492] T 8e7f08e9ac274bfa86763eec895bbfda P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:55.587833 18983 ts_tablet_manager.cc:1367] T 8e7f08e9ac274bfa86763eec895bbfda P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.000s
I20240430 20:11:55.588562 18983 raft_consensus.cc:348] T 8e7f08e9ac274bfa86763eec895bbfda P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:55.588778 18983 raft_consensus.cc:374] T 8e7f08e9ac274bfa86763eec895bbfda P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:55.588833 18983 raft_consensus.cc:729] T 8e7f08e9ac274bfa86763eec895bbfda P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:55.589092 18983 consensus_queue.cc:260] T 8e7f08e9ac274bfa86763eec895bbfda P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:55.589619 18983 ts_tablet_manager.cc:1398] T 8e7f08e9ac274bfa86763eec895bbfda P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240430 20:11:55.589869 18983 tablet_bootstrap.cc:492] T 17ddcaf971154a52b606bbf29a539bf0 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:55.591539 18983 tablet_bootstrap.cc:654] T 17ddcaf971154a52b606bbf29a539bf0 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:55.592640 18985 tablet_bootstrap.cc:492] T e5b85a066cdc46d48ccf8201e5423c29 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:55.594122 18985 tablet_bootstrap.cc:654] T e5b85a066cdc46d48ccf8201e5423c29 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:55.603538 21395 ts_tablet_manager.cc:1888] T f3711610610542128c4a84e17bce4a3e P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:55.603775 18983 tablet_bootstrap.cc:492] T 17ddcaf971154a52b606bbf29a539bf0 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:55.603802 21395 log.cc:1196] T f3711610610542128c4a84e17bce4a3e P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/f3711610610542128c4a84e17bce4a3e
I20240430 20:11:55.603938 18983 ts_tablet_manager.cc:1367] T 17ddcaf971154a52b606bbf29a539bf0 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.003s
I20240430 20:11:55.604619 18983 raft_consensus.cc:348] T 17ddcaf971154a52b606bbf29a539bf0 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:55.604892 18983 raft_consensus.cc:374] T 17ddcaf971154a52b606bbf29a539bf0 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:55.604975 18983 raft_consensus.cc:729] T 17ddcaf971154a52b606bbf29a539bf0 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:55.605867 18983 consensus_queue.cc:260] T 17ddcaf971154a52b606bbf29a539bf0 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:55.606786 18983 ts_tablet_manager.cc:1398] T 17ddcaf971154a52b606bbf29a539bf0 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240430 20:11:55.607120 18983 tablet_bootstrap.cc:492] T 94fbd4a57a1e438891fcb456f6b40be3 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:55.609019 20489 raft_consensus.cc:482] T 00be6ddd91e84865a48743fb4d077eec P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:55.609148 20489 raft_consensus.cc:504] T 00be6ddd91e84865a48743fb4d077eec P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:55.609313 18983 tablet_bootstrap.cc:654] T 94fbd4a57a1e438891fcb456f6b40be3 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:55.609824 20489 leader_election.cc:288] T 00be6ddd91e84865a48743fb4d077eec P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:55.610707 18756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00be6ddd91e84865a48743fb4d077eec" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:55.610994 18756 raft_consensus.cc:2454] T 00be6ddd91e84865a48743fb4d077eec P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 0.
I20240430 20:11:55.612032 18557 leader_election.cc:302] T 00be6ddd91e84865a48743fb4d077eec P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:55.612743 21468 raft_consensus.cc:2781] T 00be6ddd91e84865a48743fb4d077eec P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:55.612828 21468 raft_consensus.cc:482] T 00be6ddd91e84865a48743fb4d077eec P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:55.612898 21468 raft_consensus.cc:3037] T 00be6ddd91e84865a48743fb4d077eec P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:55.615535 21468 raft_consensus.cc:504] T 00be6ddd91e84865a48743fb4d077eec P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:55.616223 21468 leader_election.cc:288] T 00be6ddd91e84865a48743fb4d077eec P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:55.503512 19946 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 1faa2f2c84184b129ba6e4d2857822f0 (table default.Table_0dd55b391be94213b4af60ea6e5a979f [id=1a382ad8009b4522a50d55c307db8f96]). Expected version 1 got 0
I20240430 20:11:55.616802 18756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00be6ddd91e84865a48743fb4d077eec" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:55.616998 19946 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 3a03b6997921408cb5a65f4cae223e9c (table default.Table_744b5c96ea014bdb8feae740356a790e [id=83b1ac2e9c3d4a10b02f96ebb17faf44]). Expected version 1 got 0
I20240430 20:11:55.617062 18756 raft_consensus.cc:3037] T 00be6ddd91e84865a48743fb4d077eec P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:55.617170 19946 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet ae20f44df50343289b0a512a4ea53692 (table default.Table_1844f6b1bda84812825384e14a2fd94e [id=71da9acc3cff4d95bbac4a686b708fd5]). Expected version 1 got 0
I20240430 20:11:55.617457 19946 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 331c031d44854d679a27d10d9151d0ea (table default.Table_763f0800e8bb4fa69976886416250f42 [id=61c8a5b99ef34a0baeeb343fef7cfd05]). Expected version 1 got 0
I20240430 20:11:55.617606 19946 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 7ce09fcbe1ac457a9c05efaf000c3263 (table default.Table_edc4c27ed846454288f73f6a0d88cb69 [id=78fd3f516083489ab3b42d90dcef7e4d]). Expected version 1 got 0
I20240430 20:11:55.620126 18756 raft_consensus.cc:2454] T 00be6ddd91e84865a48743fb4d077eec P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e1c6397f09747f091534faf93542710 in term 1.
I20240430 20:11:55.620602 18557 leader_election.cc:302] T 00be6ddd91e84865a48743fb4d077eec P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:55.620934 21366 raft_consensus.cc:2781] T 00be6ddd91e84865a48743fb4d077eec P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:55.621091 21366 raft_consensus.cc:686] T 00be6ddd91e84865a48743fb4d077eec P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Becoming Leader. State: Replica: 5e1c6397f09747f091534faf93542710, State: Running, Role: LEADER
I20240430 20:11:55.621371 21366 consensus_queue.cc:237] T 00be6ddd91e84865a48743fb4d077eec P 5e1c6397f09747f091534faf93542710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:55.635288 21396 ts_tablet_manager.cc:1875] T 2797eed593fb4b06a3c4d6a28e8075a2 P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:55.638513 18983 tablet_bootstrap.cc:492] T 94fbd4a57a1e438891fcb456f6b40be3 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:55.638761 18983 ts_tablet_manager.cc:1367] T 94fbd4a57a1e438891fcb456f6b40be3 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.032s user 0.005s sys 0.002s
I20240430 20:11:55.639468 18983 raft_consensus.cc:348] T 94fbd4a57a1e438891fcb456f6b40be3 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:55.639741 18983 raft_consensus.cc:374] T 94fbd4a57a1e438891fcb456f6b40be3 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:55.639835 18983 raft_consensus.cc:729] T 94fbd4a57a1e438891fcb456f6b40be3 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:55.640156 18983 consensus_queue.cc:260] T 94fbd4a57a1e438891fcb456f6b40be3 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:55.640640 18983 ts_tablet_manager.cc:1398] T 94fbd4a57a1e438891fcb456f6b40be3 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240430 20:11:55.643707 18983 tablet_bootstrap.cc:492] T e041756c962b4a9f97c2cb314e7d8c05 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:55.645352 18983 tablet_bootstrap.cc:654] T e041756c962b4a9f97c2cb314e7d8c05 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:55.669572 18983 tablet_bootstrap.cc:492] T e041756c962b4a9f97c2cb314e7d8c05 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:55.669835 18983 ts_tablet_manager.cc:1367] T e041756c962b4a9f97c2cb314e7d8c05 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.026s user 0.002s sys 0.001s
I20240430 20:11:55.670522 18983 raft_consensus.cc:348] T e041756c962b4a9f97c2cb314e7d8c05 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:55.670817 18983 raft_consensus.cc:374] T e041756c962b4a9f97c2cb314e7d8c05 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:55.670920 18983 raft_consensus.cc:729] T e041756c962b4a9f97c2cb314e7d8c05 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:55.671197 18983 consensus_queue.cc:260] T e041756c962b4a9f97c2cb314e7d8c05 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:55.671736 18983 ts_tablet_manager.cc:1398] T e041756c962b4a9f97c2cb314e7d8c05 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240430 20:11:55.672044 18983 tablet_bootstrap.cc:492] T cdeef95b54ca42d38d25b57738d84ef1 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:55.673362 18983 tablet_bootstrap.cc:654] T cdeef95b54ca42d38d25b57738d84ef1 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:55.675436 19902 catalog_manager.cc:4830] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): tablet 3e5225f1f5f544ecbf8dfa470af3daca (table default.Table_c7c87a8e019944cab3ec99e9563ac01a [id=abf3765074eb4d7d9d2bcc4ab721e6bd]) successfully deleted
I20240430 20:11:55.675990 19902 catalog_manager.cc:4830] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): tablet 6db55bb510b24b0cae897936211e15f0 (table default.Table_3471b34c1c0d4d829bb27a8520eff502 [id=f3498e7e44974f65bbd2e92c6c8c119f]) successfully deleted
W20240430 20:11:55.676002 18558 leader_election.cc:341] T e5b85a066cdc46d48ccf8201e5423c29 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:55.676118 18558 leader_election.cc:302] T e5b85a066cdc46d48ccf8201e5423c29 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:55.676903 20489 raft_consensus.cc:2726] T e5b85a066cdc46d48ccf8201e5423c29 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:55.679344 21548 tablet.cc:1722] T c6e3d09402574f1e8fe9e4160c70a3ee P 2efb542aa55e439b99ccb98d1c5c884a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240430 20:11:55.616084 21542 tablet.cc:1722] T 22fef8bb6b034db098c3372ae992f7a7 P 2efb542aa55e439b99ccb98d1c5c884a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240430 20:11:55.630352 21580 consensus_queue.cc:1035] T 4dac8b3838954d839094021f1c0503da P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:55.631709 18467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b7fdc6ec7e046c5af894f9a6eda17b0" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:55.632123 18472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd1b8407071e4e4287fc172009777414" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:55.680745 18467 raft_consensus.cc:2454] T 3b7fdc6ec7e046c5af894f9a6eda17b0 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:55.632196 18481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd1b8407071e4e4287fc172009777414" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:55.632504 18470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8668f7a285de43f48d83abecb4b2e96c" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:55.681080 18470 raft_consensus.cc:2454] T 8668f7a285de43f48d83abecb4b2e96c P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:55.632831 18469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8668f7a285de43f48d83abecb4b2e96c" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:55.681236 18469 raft_consensus.cc:3037] T 8668f7a285de43f48d83abecb4b2e96c P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:55.633185 18483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b7fdc6ec7e046c5af894f9a6eda17b0" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:55.682440 18483 raft_consensus.cc:3037] T 3b7fdc6ec7e046c5af894f9a6eda17b0 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:55.686071 18983 tablet_bootstrap.cc:492] T cdeef95b54ca42d38d25b57738d84ef1 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:55.650571 21535 tablet.cc:1722] T 7ce09fcbe1ac457a9c05efaf000c3263 P 2efb542aa55e439b99ccb98d1c5c884a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240430 20:11:55.686226 18983 ts_tablet_manager.cc:1367] T cdeef95b54ca42d38d25b57738d84ef1 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.001s
I20240430 20:11:55.653409 18985 tablet_bootstrap.cc:492] T e5b85a066cdc46d48ccf8201e5423c29 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:55.686493 18985 ts_tablet_manager.cc:1367] T e5b85a066cdc46d48ccf8201e5423c29 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.094s user 0.003s sys 0.000s
I20240430 20:11:55.686967 18983 raft_consensus.cc:348] T cdeef95b54ca42d38d25b57738d84ef1 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:55.687168 18983 raft_consensus.cc:374] T cdeef95b54ca42d38d25b57738d84ef1 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:55.687245 18983 raft_consensus.cc:729] T cdeef95b54ca42d38d25b57738d84ef1 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:55.687470 18983 consensus_queue.cc:260] T cdeef95b54ca42d38d25b57738d84ef1 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:55.688091 18983 ts_tablet_manager.cc:1398] T cdeef95b54ca42d38d25b57738d84ef1 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240430 20:11:55.690451 18983 tablet_bootstrap.cc:492] T d34a450a91d542c98261456d1f635e9c P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:55.697398 18983 tablet_bootstrap.cc:654] T d34a450a91d542c98261456d1f635e9c P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:55.700747 19902 catalog_manager.cc:4830] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): tablet 47e98935eb21449abd3d4d24327778d5 (table default.Table_67d51131549440cc99629dda924693f7 [id=842d2b7e31cb4e7793d88103997a88d8]) successfully deleted
I20240430 20:11:55.654520 21397 ts_tablet_manager.cc:1888] T c1017a50d1d646c486c0b9a75cb88ce4 P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:55.664548 21454 consensus_queue.cc:1035] T 69f88532952a4591bf1577541c3430e6 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:55.712159 21397 log.cc:1196] T c1017a50d1d646c486c0b9a75cb88ce4 P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/c1017a50d1d646c486c0b9a75cb88ce4
W20240430 20:11:55.712097 18558 leader_election.cc:341] T 347a8224114b49e5b7dcb75d9b726e0a P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:55.712270 18558 leader_election.cc:302] T 347a8224114b49e5b7dcb75d9b726e0a P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:55.712769 21397 ts_tablet_manager.cc:1909] T c1017a50d1d646c486c0b9a75cb88ce4 P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:55.713168 21397 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56540 (request call id 61) took 1977 ms. Trace:
I20240430 20:11:55.678122 18988 tablet_bootstrap.cc:654] T e5b85a066cdc46d48ccf8201e5423c29 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
W20240430 20:11:55.702941 18699 leader_election.cc:341] T 2cdccf4be7094ca5b863dd5c3432d55d P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Not found: Tablet not found: 2cdccf4be7094ca5b863dd5c3432d55d
I20240430 20:11:55.713843 18699 leader_election.cc:302] T 2cdccf4be7094ca5b863dd5c3432d55d P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710
I20240430 20:11:55.714483 21463 raft_consensus.cc:2726] T 2cdccf4be7094ca5b863dd5c3432d55d P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:55.714705 21638 raft_consensus.cc:2726] T 347a8224114b49e5b7dcb75d9b726e0a P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:55.716507 18983 tablet_bootstrap.cc:492] T d34a450a91d542c98261456d1f635e9c P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:55.716637 18983 ts_tablet_manager.cc:1367] T d34a450a91d542c98261456d1f635e9c P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.026s user 0.005s sys 0.000s
I20240430 20:11:55.717337 18983 raft_consensus.cc:348] T d34a450a91d542c98261456d1f635e9c P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:55.717746 18983 raft_consensus.cc:374] T d34a450a91d542c98261456d1f635e9c P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:55.717818 18983 raft_consensus.cc:729] T d34a450a91d542c98261456d1f635e9c P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:55.718019 18983 consensus_queue.cc:260] T d34a450a91d542c98261456d1f635e9c P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:55.719410 18983 ts_tablet_manager.cc:1398] T d34a450a91d542c98261456d1f635e9c P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240430 20:11:55.719723 18983 tablet_bootstrap.cc:492] T 9a76db2c7ff146758c54908e428a0e02 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:55.721168 18983 tablet_bootstrap.cc:654] T 9a76db2c7ff146758c54908e428a0e02 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:55.725677 21454 consensus_queue.cc:1035] T 2e1cac64cefe4336bd3207b5ca58b631 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:55.727231 18699 leader_election.cc:302] T 9c73d0a6bc414d3b9900570848d19a51 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710
I20240430 20:11:55.727744 21454 raft_consensus.cc:2726] T 9c73d0a6bc414d3b9900570848d19a51 P 3c51c33d5616482cafb68e26688820f4 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240430 20:11:55.728440 18558 consensus_peers.cc:485] T f3711610610542128c4a84e17bce4a3e P 5e1c6397f09747f091534faf93542710 -> Peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Couldn't send request to peer 2efb542aa55e439b99ccb98d1c5c884a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: STOPPING. This is attempt 1: this message will repeat every 5th retry.
W20240430 20:11:55.729552 18699 leader_election.cc:341] T 115c4a12440647a4bb46eb6824f7ea98 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Not found: Tablet not found: 115c4a12440647a4bb46eb6824f7ea98
I20240430 20:11:55.729678 18699 leader_election.cc:302] T 115c4a12440647a4bb46eb6824f7ea98 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710
I20240430 20:11:55.729882 21454 raft_consensus.cc:2726] T 115c4a12440647a4bb46eb6824f7ea98 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240430 20:11:55.731036 21454 consensus_queue.cc:1035] T 2797eed593fb4b06a3c4d6a28e8075a2 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240430 20:11:55.732375 18558 leader_election.cc:341] T 4391b5227fb64ef481542a962a5e21a5 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:55.732587 18558 leader_election.cc:302] T 4391b5227fb64ef481542a962a5e21a5 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:55.732877 21468 raft_consensus.cc:2726] T 4391b5227fb64ef481542a962a5e21a5 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240430 20:11:55.733044 18558 leader_election.cc:341] T b7d8da6e2fe8499d9d53af84a25f2faf P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:55.733162 18558 leader_election.cc:302] T b7d8da6e2fe8499d9d53af84a25f2faf P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:55.733394 21468 raft_consensus.cc:2726] T b7d8da6e2fe8499d9d53af84a25f2faf P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:55.748508 18725 tablet_service.cc:1467] Processing CreateTablet for tablet 1a65954286604fe88f5d00f2573a0d99 (DEFAULT_TABLE table=default.Table_2c97168cc0e24d8fbece3e129cb15067 [id=f6db507a92d246d8927fa3b570301a2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:55.748750 18726 tablet_service.cc:1467] Processing CreateTablet for tablet 53bd608ff24a47bab1ae07c16748c0ed (DEFAULT_TABLE table=default.Table_04cfbe148d8b4fc8aca451328df01be6 [id=484b2118174e454092e40e47ad2fd32c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:55.749024 18725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a65954286604fe88f5d00f2573a0d99. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:55.749245 18726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53bd608ff24a47bab1ae07c16748c0ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:55.750121 18470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00be6ddd91e84865a48743fb4d077eec" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:55.750231 18450 tablet_service.cc:1467] Processing CreateTablet for tablet 1a65954286604fe88f5d00f2573a0d99 (DEFAULT_TABLE table=default.Table_2c97168cc0e24d8fbece3e129cb15067 [id=f6db507a92d246d8927fa3b570301a2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:55.750854 21395 ts_tablet_manager.cc:1909] T f3711610610542128c4a84e17bce4a3e P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:55.751152 21395 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41450 (request call id 61) took 1947 ms. Trace:
I20240430 20:11:55.751230 21395 rpcz_store.cc:276] 0430 20:11:53.803409 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:53.803462 (+ 53us) service_pool.cc:225] Handling call
0430 20:11:55.341734 (+1538272us) ts_tablet_manager.cc:1120] Deleting tablet f3711610610542128c4a84e17bce4a3e
0430 20:11:55.341751 (+ 17us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:55.484110 (+142359us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:55.603418 (+119308us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:55.751135 (+147717us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1538063}
I20240430 20:11:55.751368 18603 tablet_service.cc:1467] Processing CreateTablet for tablet 1a65954286604fe88f5d00f2573a0d99 (DEFAULT_TABLE table=default.Table_2c97168cc0e24d8fbece3e129cb15067 [id=f6db507a92d246d8927fa3b570301a2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:55.751524 21395 tablet_replica.cc:330] T 1f4e860484784aa9a2d28af2e9fb611c P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:55.751726 21395 raft_consensus.cc:2229] T 1f4e860484784aa9a2d28af2e9fb611c P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Raft consensus shutting down.
I20240430 20:11:55.751892 18601 tablet_service.cc:1467] Processing CreateTablet for tablet 53bd608ff24a47bab1ae07c16748c0ed (DEFAULT_TABLE table=default.Table_04cfbe148d8b4fc8aca451328df01be6 [id=484b2118174e454092e40e47ad2fd32c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:55.752071 21395 raft_consensus.cc:2258] T 1f4e860484784aa9a2d28af2e9fb611c P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:55.752288 18592 tablet_service.cc:1467] Processing CreateTablet for tablet 3ba47468dde4470ea1a5597b6e4679f7 (DEFAULT_TABLE table=default.Table_10a24c59b9bc473da9f773530f720f1d [id=46cfd99fc9274a87a0f86485e51da5dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:55.752641 21395 ts_tablet_manager.cc:1875] T 1f4e860484784aa9a2d28af2e9fb611c P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:55.754917 18733 tablet_service.cc:1467] Processing CreateTablet for tablet 3ba47468dde4470ea1a5597b6e4679f7 (DEFAULT_TABLE table=default.Table_10a24c59b9bc473da9f773530f720f1d [id=46cfd99fc9274a87a0f86485e51da5dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:55.755384 18733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ba47468dde4470ea1a5597b6e4679f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:55.750530 18481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00be6ddd91e84865a48743fb4d077eec" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:55.759210 18724 tablet_service.cc:1467] Processing CreateTablet for tablet 9e1fed3070794d8da6e31692adfcb8cc (DEFAULT_TABLE table=default.Table_47217f7ee5ef4a50a2e9f146412b198a [id=3b5dcd10bb0d4f2493dcfaae7a2b9d19]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:55.759688 18724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e1fed3070794d8da6e31692adfcb8cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:55.760608 18595 tablet_service.cc:1467] Processing CreateTablet for tablet 9e1fed3070794d8da6e31692adfcb8cc (DEFAULT_TABLE table=default.Table_47217f7ee5ef4a50a2e9f146412b198a [id=3b5dcd10bb0d4f2493dcfaae7a2b9d19]), partition=RANGE (key) PARTITION UNBOUNDED
W20240430 20:11:55.761463 18699 consensus_peers.cc:485] T 47e98935eb21449abd3d4d24327778d5 P 3c51c33d5616482cafb68e26688820f4 -> Peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Couldn't send request to peer 2efb542aa55e439b99ccb98d1c5c884a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 47e98935eb21449abd3d4d24327778d5. This is attempt 1: this message will repeat every 5th retry.
I20240430 20:11:55.761709 21454 consensus_queue.cc:1035] T 1050ab528c7d41559831652a1437d358 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:55.762224 18590 tablet_service.cc:1467] Processing CreateTablet for tablet fe84ec08c70b4e23acb35884066d6129 (DEFAULT_TABLE table=default.Table_dac5789e49e24b798431be43e39edb7c [id=d84dff22dabd4fc2b07c8d77555ca2db]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:55.762346 18729 tablet_service.cc:1467] Processing CreateTablet for tablet fe84ec08c70b4e23acb35884066d6129 (DEFAULT_TABLE table=default.Table_dac5789e49e24b798431be43e39edb7c [id=d84dff22dabd4fc2b07c8d77555ca2db]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:55.762709 18729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe84ec08c70b4e23acb35884066d6129. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:55.765916 21454 consensus_queue.cc:1035] T 3191aa54a9644783a2403991c3be27c8 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240430 20:11:55.768203 18699 leader_election.cc:341] T fd1b8407071e4e4287fc172009777414 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
W20240430 20:11:55.768579 18699 leader_election.cc:341] T fd1b8407071e4e4287fc172009777414 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:55.771266 18456 tablet_service.cc:1467] Processing CreateTablet for tablet 53bd608ff24a47bab1ae07c16748c0ed (DEFAULT_TABLE table=default.Table_04cfbe148d8b4fc8aca451328df01be6 [id=484b2118174e454092e40e47ad2fd32c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:55.771664 18456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53bd608ff24a47bab1ae07c16748c0ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:55.772800 18447 tablet_service.cc:1467] Processing CreateTablet for tablet 9e1fed3070794d8da6e31692adfcb8cc (DEFAULT_TABLE table=default.Table_47217f7ee5ef4a50a2e9f146412b198a [id=3b5dcd10bb0d4f2493dcfaae7a2b9d19]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:55.773259 18447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e1fed3070794d8da6e31692adfcb8cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:55.773736 18595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e1fed3070794d8da6e31692adfcb8cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:55.774241 18590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe84ec08c70b4e23acb35884066d6129. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:55.774745 18592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ba47468dde4470ea1a5597b6e4679f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:55.775148 18601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53bd608ff24a47bab1ae07c16748c0ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:55.775614 18603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a65954286604fe88f5d00f2573a0d99. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:55.779551 21396 ts_tablet_manager.cc:1888] T 2797eed593fb4b06a3c4d6a28e8075a2 P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:55.780470 21396 log.cc:1196] T 2797eed593fb4b06a3c4d6a28e8075a2 P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/2797eed593fb4b06a3c4d6a28e8075a2
I20240430 20:11:55.780586 21445 raft_consensus.cc:482] T 2fc6908ecf954020a676b9d10a2b00e7 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:55.780711 21445 raft_consensus.cc:504] T 2fc6908ecf954020a676b9d10a2b00e7 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:55.784137 18453 tablet_service.cc:1467] Processing CreateTablet for tablet 3ba47468dde4470ea1a5597b6e4679f7 (DEFAULT_TABLE table=default.Table_10a24c59b9bc473da9f773530f720f1d [id=46cfd99fc9274a87a0f86485e51da5dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:55.785619 19902 catalog_manager.cc:4830] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): tablet f3711610610542128c4a84e17bce4a3e (table default.Table_64d14df4e12146ecb85239e7ca03cf6e [id=c376119401934812aef62a3a18eeb368]) successfully deleted
I20240430 20:11:55.786243 18453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ba47468dde4470ea1a5597b6e4679f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:55.786433 18450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a65954286604fe88f5d00f2573a0d99. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:55.713333 21397 rpcz_store.cc:276] 0430 20:11:53.736072 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:53.736155 (+ 83us) service_pool.cc:225] Handling call
0430 20:11:55.477700 (+1741545us) ts_tablet_manager.cc:1120] Deleting tablet c1017a50d1d646c486c0b9a75cb88ce4
0430 20:11:55.477711 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:55.651274 (+173563us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:55.654419 (+ 3145us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:55.713150 (+ 58731us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1741435}
I20240430 20:11:55.787585 19900 catalog_manager.cc:4830] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): tablet c1017a50d1d646c486c0b9a75cb88ce4 (table default.Table_d95c70be474e415cba6a345438fad1ba [id=095b8b568ebc475aa3c21e0c03ef5d79]) successfully deleted
I20240430 20:11:55.787755 21397 tablet_replica.cc:330] T 898b2c91adf34545808df9c786e218c9 P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:55.788203 21397 raft_consensus.cc:2229] T 898b2c91adf34545808df9c786e218c9 P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Raft consensus shutting down.
W20240430 20:11:55.785259 18558 leader_election.cc:341] T 00be6ddd91e84865a48743fb4d077eec P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
W20240430 20:11:55.790253 18558 leader_election.cc:341] T 00be6ddd91e84865a48743fb4d077eec P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:55.790238 21397 raft_consensus.cc:2258] T 898b2c91adf34545808df9c786e218c9 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:55.795280 21397 ts_tablet_manager.cc:1875] T 898b2c91adf34545808df9c786e218c9 P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:55.799163 18458 tablet_service.cc:1467] Processing CreateTablet for tablet fe84ec08c70b4e23acb35884066d6129 (DEFAULT_TABLE table=default.Table_dac5789e49e24b798431be43e39edb7c [id=d84dff22dabd4fc2b07c8d77555ca2db]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:55.799602 18458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe84ec08c70b4e23acb35884066d6129. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:55.801962 18988 tablet_bootstrap.cc:492] T e5b85a066cdc46d48ccf8201e5423c29 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:55.802109 18988 ts_tablet_manager.cc:1367] T e5b85a066cdc46d48ccf8201e5423c29 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.287s user 0.004s sys 0.003s
I20240430 20:11:55.803331 21535 tablet.cc:1722] T f5eaf2413e1d4e69b56fae61f768f0ce P 2efb542aa55e439b99ccb98d1c5c884a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240430 20:11:55.809238 18988 raft_consensus.cc:348] T e5b85a066cdc46d48ccf8201e5423c29 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:55.809469 18988 raft_consensus.cc:374] T e5b85a066cdc46d48ccf8201e5423c29 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:55.809527 18988 raft_consensus.cc:729] T e5b85a066cdc46d48ccf8201e5423c29 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:55.811075 18483 raft_consensus.cc:2454] T 3b7fdc6ec7e046c5af894f9a6eda17b0 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:55.811924 18988 consensus_queue.cc:260] T e5b85a066cdc46d48ccf8201e5423c29 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:55.812868 21445 leader_election.cc:288] T 2fc6908ecf954020a676b9d10a2b00e7 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:55.815032 18985 raft_consensus.cc:348] T e5b85a066cdc46d48ccf8201e5423c29 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:55.815227 18985 raft_consensus.cc:374] T e5b85a066cdc46d48ccf8201e5423c29 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:55.815279 18985 raft_consensus.cc:729] T e5b85a066cdc46d48ccf8201e5423c29 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
W20240430 20:11:55.815811 18415 leader_election.cc:341] T 2fc6908ecf954020a676b9d10a2b00e7 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Not found: Tablet not found: 2fc6908ecf954020a676b9d10a2b00e7
I20240430 20:11:55.816602 21396 ts_tablet_manager.cc:1909] T 2797eed593fb4b06a3c4d6a28e8075a2 P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:55.819942 21396 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:45934 (request call id 81) took 2003 ms. Trace:
I20240430 20:11:55.822160 18985 consensus_queue.cc:260] T e5b85a066cdc46d48ccf8201e5423c29 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:55.826393 18985 ts_tablet_manager.cc:1398] T e5b85a066cdc46d48ccf8201e5423c29 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.140s user 0.003s sys 0.000s
I20240430 20:11:55.827858 18985 tablet_bootstrap.cc:492] T fd1b8407071e4e4287fc172009777414 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:55.835367 21395 ts_tablet_manager.cc:1888] T 1f4e860484784aa9a2d28af2e9fb611c P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:55.835520 21395 log.cc:1196] T 1f4e860484784aa9a2d28af2e9fb611c P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/1f4e860484784aa9a2d28af2e9fb611c
I20240430 20:11:55.835903 21395 ts_tablet_manager.cc:1909] T 1f4e860484784aa9a2d28af2e9fb611c P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:55.836220 21395 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41450 (request call id 62) took 2032 ms. Trace:
I20240430 20:11:55.836405 21395 rpcz_store.cc:276] 0430 20:11:53.803841 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:53.803909 (+ 68us) service_pool.cc:225] Handling call
0430 20:11:55.751486 (+1947577us) ts_tablet_manager.cc:1120] Deleting tablet 1f4e860484784aa9a2d28af2e9fb611c
0430 20:11:55.751494 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:55.769160 (+ 17666us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:55.834576 (+ 65416us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:55.836202 (+ 1626us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1947409}
I20240430 20:11:55.836740 21395 tablet_replica.cc:330] T f7d3dea34cf54fe2b949d1b7ca68577a P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:55.836943 21395 raft_consensus.cc:2229] T f7d3dea34cf54fe2b949d1b7ca68577a P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:55.837100 21395 raft_consensus.cc:2258] T f7d3dea34cf54fe2b949d1b7ca68577a P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:55.820083 21396 rpcz_store.cc:276] 0430 20:11:53.816462 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:53.816527 (+ 65us) service_pool.cc:225] Handling call
0430 20:11:55.573841 (+1757314us) ts_tablet_manager.cc:1120] Deleting tablet 2797eed593fb4b06a3c4d6a28e8075a2
0430 20:11:55.573853 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:55.696573 (+122720us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:55.779429 (+ 82856us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:55.819911 (+ 40482us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1757129}
I20240430 20:11:55.837637 21395 ts_tablet_manager.cc:1875] T f7d3dea34cf54fe2b949d1b7ca68577a P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:55.837788 19895 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet 2797eed593fb4b06a3c4d6a28e8075a2 (table default.Table_88b0eaf3d0a1461ab5e2130cb69f0dee [id=6ddeb7090daf419db901940d7a86acee]) successfully deleted
W20240430 20:11:55.839002 21605 tablet_replica.cc:1402] Time spent flushing the Log queue took a long time: real 0.347s user 0.000s sys 0.014s
I20240430 20:11:55.840620 21396 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:45934 (request call id 82) took 2006 ms. Trace:
I20240430 20:11:55.840785 21396 rpcz_store.cc:276] 0430 20:11:53.834001 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:53.834059 (+ 58us) service_pool.cc:225] Handling call
0430 20:11:55.838187 (+2004128us) ts_tablet_manager.cc:1120] Deleting tablet b7141e215d6d44bc8406c80508190874
0430 20:11:55.840494 (+ 2307us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:55.840603 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2003846}
W20240430 20:11:55.841260 19895 catalog_manager.cc:4810] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): delete failed for tablet b7141e215d6d44bc8406c80508190874 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b7141e215d6d44bc8406c80508190874 already in progress: creating tablet
I20240430 20:11:55.844604 21396 tablet_replica.cc:330] T dbb427aefa6b47b0805d1751fea4efcb P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:55.844954 21396 raft_consensus.cc:2229] T dbb427aefa6b47b0805d1751fea4efcb P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:55.845273 21396 raft_consensus.cc:2258] T dbb427aefa6b47b0805d1751fea4efcb P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:55.846028 21396 ts_tablet_manager.cc:1875] T dbb427aefa6b47b0805d1751fea4efcb P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:55.849560 18983 tablet_bootstrap.cc:492] T 9a76db2c7ff146758c54908e428a0e02 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:55.849702 18983 ts_tablet_manager.cc:1367] T 9a76db2c7ff146758c54908e428a0e02 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.130s user 0.008s sys 0.031s
W20240430 20:11:55.850246 19902 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 1faa2f2c84184b129ba6e4d2857822f0 (table default.Table_0dd55b391be94213b4af60ea6e5a979f [id=1a382ad8009b4522a50d55c307db8f96], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 1faa2f2c84184b129ba6e4d2857822f0 consensus state has no leader
I20240430 20:11:55.850550 19942 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 3a03b6997921408cb5a65f4cae223e9c (table default.Table_744b5c96ea014bdb8feae740356a790e [id=83b1ac2e9c3d4a10b02f96ebb17faf44]). Expected version 1 got 0
I20240430 20:11:55.851171 19942 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet ae20f44df50343289b0a512a4ea53692 (table default.Table_1844f6b1bda84812825384e14a2fd94e [id=71da9acc3cff4d95bbac4a686b708fd5]). Expected version 1 got 0
I20240430 20:11:55.852308 19942 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 1faa2f2c84184b129ba6e4d2857822f0 (table default.Table_0dd55b391be94213b4af60ea6e5a979f [id=1a382ad8009b4522a50d55c307db8f96]). Expected version 1 got 0
I20240430 20:11:55.852788 19942 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet eb6c96047b114ebba73db7a371f2da8e (table default.Table_7d5f6f18cbd64c869739872a7dff9be2 [id=e67d619e36c0485f8f22b4a81cb4cb6c]). Expected version 1 got 0
I20240430 20:11:55.853062 18983 raft_consensus.cc:348] T 9a76db2c7ff146758c54908e428a0e02 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:55.853516 19942 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 6b51cbc731794083a88fe4bed1ddeebf (table default.Table_c6c962da746e4fd8a4d7200d09f08e9e [id=8e0f5d8c941d423ea7b30a89513d5170]). Expected version 1 got 0
I20240430 20:11:55.853655 18983 raft_consensus.cc:374] T 9a76db2c7ff146758c54908e428a0e02 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:55.854122 18983 raft_consensus.cc:729] T 9a76db2c7ff146758c54908e428a0e02 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:55.854383 18983 consensus_queue.cc:260] T 9a76db2c7ff146758c54908e428a0e02 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:55.854812 19942 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 62732af95c6a4c749c5b726f0460dc72 (table default.Table_e3160f5ced1b4718b479401ca4df675c [id=2d44ec94131148048021b2ee6366ba30]). Expected version 3 got 0
I20240430 20:11:55.855185 19942 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet a0baee91f5b44945b595d0e234af018e (table default.Table_5ed3ea587cf843db9a1ff1ffab40b678 [id=b3d1c2f97c4648518dbea83a46cadd85]). Expected version 2 got 0
I20240430 20:11:55.855716 18983 ts_tablet_manager.cc:1398] T 9a76db2c7ff146758c54908e428a0e02 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240430 20:11:55.855973 19942 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 266fd23786bc435db5083370f3330ed0 (table default.Table_cf60b652e9dc49e7a948ce66fd1cbffe [id=9734bb8dd05a400ca4244b3829f01dee]). Expected version 1 got 0
I20240430 20:11:55.856650 19942 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 89b774967624498fa2be499878b71124 (table default.Table_2ec9e316bc3347b189a13b7a8bbcf829 [id=9fa22833979f4c2f978de651145acef9]). Expected version 1 got 0
I20240430 20:11:55.857097 19942 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet ed0c37a71ede4bfc9eb5ad1a4365f25e (table default.Table_a2e85efc6b454524b0b122102ec07002 [id=86d795a82e674d859104741ebd04f4e8]). Expected version 1 got 0
I20240430 20:11:55.857594 19942 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 12795670f21447a8964fe1efc6e92398 (table default.Table_896bbae1f6944005b0272ca8ebc97218 [id=d5cea90cbc2c48df9d8b80da1eccc949]). Expected version 3 got 0
I20240430 20:11:55.858331 19942 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet dd1de890e4394db287cf38bb411f5586 (table default.Table_84084a27f7504dd992fce07448c10df3 [id=6257559b0e9545c58848a76af0c5a5f7]). Expected version 2 got 0
W20240430 20:11:55.858377 18696 consensus_peers.cc:485] T a663030d37614053a7c0369841403422 P 3c51c33d5616482cafb68e26688820f4 -> Peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Couldn't send request to peer 5e1c6397f09747f091534faf93542710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a663030d37614053a7c0369841403422. This is attempt 1: this message will repeat every 5th retry.
I20240430 20:11:55.858852 19942 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 130007acc8ba4e709c3eb85fabe75903 (table default.Table_78987875650940c293247bc43db345ab [id=36f478a20a1f4a53a52fc29ade62b7f6]). Expected version 1 got 0
I20240430 20:11:55.859578 18988 ts_tablet_manager.cc:1398] T e5b85a066cdc46d48ccf8201e5423c29 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.057s user 0.001s sys 0.001s
I20240430 20:11:55.860014 19902 catalog_manager.cc:4830] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): tablet 1f4e860484784aa9a2d28af2e9fb611c (table default.Table_45d50fc81325443097375ae6e65a1633 [id=9c861883de1d49c78146d65e9d4dd39f]) successfully deleted
I20240430 20:11:55.859840 19942 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet dac141d1b70c48d785caddd7182811ee (table default.Table_fc0a4540f98b4eea95099b8ab46ea235 [id=f3c53ccffe64410495cc8c379a05cfae]). Expected version 1 got 0
I20240430 20:11:55.862006 19942 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet a482f4622f774d60b4efd6827b8be7d7 (table default.Table_e686cdfcbcc046749fa5b51578a64794 [id=c3dc9f05316d46d6a3ef4d03f40877ec]). Expected version 1 got 0
I20240430 20:11:55.862748 19942 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 330453ddeea6456e8ef93fa92608b0d3 (table default.Table_c1f36f7df5ae4e7c8eb8b4e99b89748b [id=a50abf28790d4284a76ac6ee22c02606]). Expected version 1 got 0
I20240430 20:11:55.867322 19942 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 22fef8bb6b034db098c3372ae992f7a7 (table default.Table_4440df38610340758047115845b42b95 [id=7d186ca20c094cc19aef792fe1c094e9]). Expected version 1 got 0
I20240430 20:11:55.867787 19942 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 1050ab528c7d41559831652a1437d358 (table default.Table_acddc4cbf6334546b5fb3e4752fef2a5 [id=274cb15fd751410e83327ac4e8dc4629]). Expected version 2 got 0
I20240430 20:11:55.868158 19942 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 086b3dfa84c045a18c8c4f177dce606a (table default.Table_df19cb8a6f4d46699350ec5b69c14e5a [id=197923633fc54a4582207942b51530e2]). Expected version 2 got 0
I20240430 20:11:55.868716 19942 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet e041756c962b4a9f97c2cb314e7d8c05 (table default.Table_de88cafd92b149f9b545369bff71033c [id=c4bf4806f3d84579b16a3038d8d982c3]). Expected version 1 got 0
I20240430 20:11:55.869031 19942 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 8e9711d0a382424f9f758bfa883866a0 (table default.Table_f1dc481c6d8c43f589fe2b1892a4cb10 [id=db696ce66005401fa73746e55c071f44]). Expected version 1 got 0
I20240430 20:11:55.869395 19942 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet f5eaf2413e1d4e69b56fae61f768f0ce (table default.Table_1280cfe32d5b41abb4d3eb1bd70641bf [id=8489b872074a42b6abb713e420968141]). Expected version 1 got 0
I20240430 20:11:55.869704 19942 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 5df364628561475391c9a5e10ff5a05f (table default.Table_3c8f9f7b22f64fb1bdc3f1550987233a [id=7fd6f8576f4c4cbc8ffcdb4ab95e15dc]). Expected version 1 got 0
I20240430 20:11:55.869961 19942 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 821c55edb3414d72b1489eea97d3f73a (table default.Table_4d68c38d9ba7452a829f8318b2b267b8 [id=d06e3e9045b1429aaeebd1fd67d841de]). Expected version 1 got 0
I20240430 20:11:55.870206 19942 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 47bf64ecc41b445da4ff697e141243dd (table default.Table_f1ad29d5986d4b799cfc67afb2b0dfdf [id=838fc8b61a864dea8877a6f144e4cb4f]). Expected version 2 got 0
I20240430 20:11:55.870906 19942 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 99203cc522de424184c6f97453f166fd (table default.Table_cb2a4b9ceaf4431facfff2c9bcf94c5d [id=3f8690a437ae49a3beb009fa7899d5d0]). Expected version 1 got 0
I20240430 20:11:55.871419 18983 tablet_bootstrap.cc:492] T 9cc9251abd4e40d59a2bd2d0fada6165 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:55.872610 18469 raft_consensus.cc:2454] T 8668f7a285de43f48d83abecb4b2e96c P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:55.872679 19942 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 9e8ab75e7d9c4b64a7b7dd2678569490 (table default.Table_b2d07520b9944a89ad62fca70396174e [id=1c8e0a3a44ec4a798770c0165ed3dffa]). Expected version 1 got 0
I20240430 20:11:55.873034 19942 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 37b2ea4be7494f0985808ade788dbe2b (table default.Table_5f6e0fce999b46de927f9271e5de5071 [id=0baad83a8c0f4aeebc7ddad068625a69]). Expected version 1 got 0
I20240430 20:11:55.873167 18983 tablet_bootstrap.cc:654] T 9cc9251abd4e40d59a2bd2d0fada6165 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
W20240430 20:11:55.873185 18417 consensus_peers.cc:485] T e87fb2f0361d4aa3ada27bbdfa88e94c P 2efb542aa55e439b99ccb98d1c5c884a -> Peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Couldn't send request to peer 3c51c33d5616482cafb68e26688820f4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e87fb2f0361d4aa3ada27bbdfa88e94c. This is attempt 6: this message will repeat every 5th retry.
W20240430 20:11:55.873607 18417 consensus_peers.cc:485] T dad16ae02b8b4e29a724d8eec67bb4b8 P 2efb542aa55e439b99ccb98d1c5c884a -> Peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Couldn't send request to peer 3c51c33d5616482cafb68e26688820f4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dad16ae02b8b4e29a724d8eec67bb4b8. This is attempt 1: this message will repeat every 5th retry.
W20240430 20:11:55.874541 18417 leader_election.cc:341] T 2fc6908ecf954020a676b9d10a2b00e7 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Not found: Tablet not found: 2fc6908ecf954020a676b9d10a2b00e7
I20240430 20:11:55.874728 18417 leader_election.cc:302] T 2fc6908ecf954020a676b9d10a2b00e7 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a; no voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710
I20240430 20:11:55.875308 21468 raft_consensus.cc:482] T cdeef95b54ca42d38d25b57738d84ef1 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:55.875456 21468 raft_consensus.cc:504] T cdeef95b54ca42d38d25b57738d84ef1 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:55.876120 21468 leader_election.cc:288] T cdeef95b54ca42d38d25b57738d84ef1 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:55.876708 21445 raft_consensus.cc:2726] T 2fc6908ecf954020a676b9d10a2b00e7 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:55.878610 21454 consensus_queue.cc:1035] T 69f88532952a4591bf1577541c3430e6 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:55.882189 18983 tablet_bootstrap.cc:492] T 9cc9251abd4e40d59a2bd2d0fada6165 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:55.882313 18983 ts_tablet_manager.cc:1367] T 9cc9251abd4e40d59a2bd2d0fada6165 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.001s
I20240430 20:11:55.882997 18983 raft_consensus.cc:348] T 9cc9251abd4e40d59a2bd2d0fada6165 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:55.883173 18983 raft_consensus.cc:374] T 9cc9251abd4e40d59a2bd2d0fada6165 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:55.883234 18983 raft_consensus.cc:729] T 9cc9251abd4e40d59a2bd2d0fada6165 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:55.883301 19942 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet aaef85eea87f4443943fd3692bb2a38f (table default.Table_3da5b4c8668c470483ae2a0c7773eaaf [id=e292d818ee3c45df90522f08eb2d6227]). Expected version 1 got 0
I20240430 20:11:55.883476 18983 consensus_queue.cc:260] T 9cc9251abd4e40d59a2bd2d0fada6165 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:55.883847 19942 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet a59ef6c545dd41a28265a4c1b8ce10e0 (table default.Table_07e40af8f1c546048975c8edf88170d7 [id=86df5d161d8641c89070e7eed9a39360]). Expected version 2 got 0
I20240430 20:11:55.884004 18983 ts_tablet_manager.cc:1398] T 9cc9251abd4e40d59a2bd2d0fada6165 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240430 20:11:55.884253 18983 tablet_bootstrap.cc:492] T a37d917bcc45468f9e3ee805640fba98 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:55.884512 19942 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 69f88532952a4591bf1577541c3430e6 (table default.Table_1177828cad1f4a24bd397619dad96170 [id=a95fed1bc3fa4865bf9a35dfc51b659c]). Expected version 2 got 0
I20240430 20:11:55.884878 19942 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet acd23f0a63d5448aaa7ed78422e67bad (table default.Table_972f13be4dca4cd19b9a04534dc59f51 [id=69f37db4de634a3185613e40013e0ed7]). Expected version 1 got 0
I20240430 20:11:55.886032 18983 tablet_bootstrap.cc:654] T a37d917bcc45468f9e3ee805640fba98 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:55.886273 19942 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 2e79df8744e74b5991a79ce978ff0202 (table default.Table_276bbdc7cfe542d1a2936485a2aad1ec [id=f146a15e6e8a4acbb4c734c1c863da57]). Expected version 1 got 0
I20240430 20:11:55.886528 19942 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 7a0176f8a74c47efbbe3d3343b3dc896 (table default.Table_144e5402c1864d818436015b26648024 [id=73e5c9f0470d47849109819c063e1ce4]). Expected version 2 got 0
I20240430 20:11:55.886811 18985 tablet_bootstrap.cc:654] T fd1b8407071e4e4287fc172009777414 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:55.887029 19942 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 94fbd4a57a1e438891fcb456f6b40be3 (table default.Table_25ddc3adef6b4ccf8b9c00cc6772031c [id=4fa623ffdcf347639150a355806038af]). Expected version 1 got 0
I20240430 20:11:55.887221 19942 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 724e51ff05ec4baebc2bb55e05c8eddc (table default.Table_19df33614a6c4f94b6419a5b441e3ef8 [id=4dce8a767f0541e1beb5464b2ed95d06]). Expected version 2 got 1
I20240430 20:11:55.887414 19942 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet ca64de16d40f4a44b8b411f5526ba11a (table default.Table_1cf914acf7fe4e98b8dbc297b6372f21 [id=f4ba9536e9c141deb68f4f62b7e8fa70]). Expected version 1 got 0
I20240430 20:11:55.887570 18469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdeef95b54ca42d38d25b57738d84ef1" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:55.888763 21396 ts_tablet_manager.cc:1888] T dbb427aefa6b47b0805d1751fea4efcb P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:55.888928 21396 log.cc:1196] T dbb427aefa6b47b0805d1751fea4efcb P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/dbb427aefa6b47b0805d1751fea4efcb
I20240430 20:11:55.889397 18760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdeef95b54ca42d38d25b57738d84ef1" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:55.890606 19942 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet fbdeb4714b8c4b93ab72822c43329aae (table default.Table_c12ca6938d424fbfa2e1df5cdb00fdbc [id=0fb93f36bc17408ea56bc4cafcfd8580]). Expected version 1 got 0
I20240430 20:11:55.891393 19942 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 772eba5f24c549c791cad75565858285 (table default.Table_5fc77d5f60ae44f4bb4ca9f7f183672c [id=6ca223ca6354449c959d85ed2ac6a9c8]). Expected version 1 got 0
I20240430 20:11:55.891731 19942 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 78ba48dadef548708153878810cc46f8 (table default.Table_b9983e101a6547649eb44ee04b5e1147 [id=29769be2b9ec4ecba9de5c0a65692f85]). Expected version 1 got 0
I20240430 20:11:55.892020 19942 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 2cd41b407d6b444d844c4c7c0ebf82e5 (table default.Table_6f8e0eff8f5a484dbd6a9c3fdb5d03f6 [id=3813b5e9778c4bd683a06598cef0d86f]). Expected version 1 got 0
I20240430 20:11:55.892441 19942 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 488d0e609b224674932f890939664df2 (table default.Table_58f35a2120874472978ca0e5f816e86c [id=ff18001c6d814afc8c1768d0c84a6bac]). Expected version 1 got 0
I20240430 20:11:55.892807 19942 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 17ddcaf971154a52b606bbf29a539bf0 (table default.Table_f2f38f5d69e34eaa9e3dbf24cfd1cdd6 [id=872c2faa4a464edd98a51df2e1b3f817]). Expected version 1 got 0
I20240430 20:11:55.893115 19942 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 2da900b6bbfa4fa99fe8494fd2e68da8 (table default.Table_19a91a7f17ff456ebb145e8dd89c0940 [id=67283dd3995a44c6a7a7bc8722e4f955]). Expected version 1 got 0
I20240430 20:11:55.893383 19942 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 0368d37dd9e2452982fb81f539c6805b (table default.Table_404b186c544e4b30bb36cb77d1a14120 [id=4330626e342a48699ea939add5c91b11]). Expected version 1 got 0
I20240430 20:11:55.893666 19942 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 331c031d44854d679a27d10d9151d0ea (table default.Table_763f0800e8bb4fa69976886416250f42 [id=61c8a5b99ef34a0baeeb343fef7cfd05]). Expected version 1 got 0
I20240430 20:11:55.893889 19942 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 7ce09fcbe1ac457a9c05efaf000c3263 (table default.Table_edc4c27ed846454288f73f6a0d88cb69 [id=78fd3f516083489ab3b42d90dcef7e4d]). Expected version 1 got 0
I20240430 20:11:55.894127 19942 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 677481aee5fa4eb4b189322d7a3f8f2e (table default.Table_1ce204e15fce4c469816cf4e7f80cac7 [id=d268c31d82df4871bc1a78aea5871b33]). Expected version 1 got 0
I20240430 20:11:55.894181 18988 tablet_bootstrap.cc:492] T b7d8da6e2fe8499d9d53af84a25f2faf P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:55.894356 19942 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 5b26319488644d6a8b6be60460a5e7af (table default.Table_69892a8264a4477db48ee6248758e046 [id=5a1c4a7be2fc465fac89a50854eb3322]). Expected version 3 got 0
I20240430 20:11:55.894565 19942 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet f1073fc792954c5795a05aa7d0ec24df (table default.Table_ba8b218e936b4bfcb073f8182f474cca [id=aedf11fda8914a3b843a57f8a7b69a53]). Expected version 1 got 0
I20240430 20:11:55.894944 19942 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 66dd767bd3f8494f94061fb89e5e5680 (table default.Table_b064d5659f7243f5b0ec167fb6c3f404 [id=5227206f44d04fa89b4842b899f33afb]). Expected version 2 got 0
I20240430 20:11:55.895251 19942 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 81787eccca6c42c1b421377a3bf9ebe6 (table default.Table_0b4ca4cd0af148399dfa21f2d32114e3 [id=7f602584cef9486eb7d9b4389be3f43e]). Expected version 3 got 2
I20240430 20:11:55.895781 19940 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: Sending DeleteTablet for 3 replicas of tablet c6e3d09402574f1e8fe9e4160c70a3ee
I20240430 20:11:55.896440 19940 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:50746 (request call id 79) took 3939 ms. Trace:
I20240430 20:11:55.896556 19940 rpcz_store.cc:276] 0430 20:11:51.957228 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:54.262420 (+2305192us) service_pool.cc:225] Handling call
0430 20:11:54.275110 (+ 12690us) spinlock_profiling.cc:243] Waited 12.2 ms on lock 0x616000070654. stack: 00007f6d23ac8756 00007f6d23ac8510 00007f6d22afb48d 0000000000955804 00000000009557cd 00007f6d3260715d 00007f6d32607253 00007f6d2f339d41 00007f6d36f2534d 00007f6d370b2ca6 00007f6d2de9bf66 00007f6d2de9bc95 00007f6d2a7d004e 00007f6d2a7cef2e 00007f6d2a7d5405 00007f6d2a7d5d2f
0430 20:11:54.500879 (+225769us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0430 20:11:54.501041 (+ 162us) catalog_manager.cc:2749] Modifying in-memory table state
0430 20:11:54.501105 (+ 64us) catalog_manager.cc:2758] Locking tablets
0430 20:11:54.501309 (+ 204us) catalog_manager.cc:2772] Removing table and tablets from system table
0430 20:11:54.526863 (+ 25554us) catalog_manager.cc:2790] Removing table from by-name map
0430 20:11:54.668764 (+141901us) catalog_manager.cc:2806] Committing in-memory state
0430 20:11:55.895692 (+1226928us) catalog_manager.cc:2810] Aborting table tasks
0430 20:11:55.896401 (+ 709us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:54.502088 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:54.502185 (+ 97us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:54.502190 (+ 5us) write_op.cc:427] Acquired schema lock
0430 20:11:54.502192 (+ 2us) tablet.cc:660] Decoding operations
0430 20:11:54.502322 (+ 130us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:54.502334 (+ 12us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:54.502338 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
0430 20:11:54.502434 (+ 96us) tablet.cc:699] Row locks acquired
0430 20:11:54.502436 (+ 2us) write_op.cc:260] PREPARE: finished
0430 20:11:54.502469 (+ 33us) write_op.cc:270] Start()
0430 20:11:54.502511 (+ 42us) write_op.cc:276] Timestamp: P: 1714507914502423 usec, L: 0
0430 20:11:54.502514 (+ 3us) op_driver.cc:331] REPLICATION: starting
0430 20:11:54.502691 (+ 177us) log.cc:842] Serialized 793 byte log entry
0430 20:11:54.515697 (+ 13006us) op_driver.cc:444] REPLICATION: finished
0430 20:11:54.515848 (+ 151us) write_op.cc:301] APPLY: starting
0430 20:11:54.515866 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:54.515914 (+ 48us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:54.515950 (+ 36us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:54.516027 (+ 77us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:54.516062 (+ 35us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0430 20:11:54.516069 (+ 7us) write_op.cc:312] APPLY: finished
0430 20:11:54.516145 (+ 76us) log.cc:842] Serialized 29 byte log entry
0430 20:11:54.516207 (+ 62us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:54.516230 (+ 23us) write_op.cc:446] Released schema lock
0430 20:11:54.516237 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:54.516295 (+ 58us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:54.516298 (+ 3us) write_op.cc:446] Released schema lock
Metrics: {"spinlock_wait_cycles":28099456,"child_traces":[["op",{"apply.queue_time_us":82,"apply.run_cpu_time_us":518,"apply.run_wall_time_us":518,"num_ops":2,"prepare.queue_time_us":55,"prepare.run_cpu_time_us":667,"prepare.run_wall_time_us":667,"replication_time_us":13154}]]}
I20240430 20:11:55.899492 18988 tablet_bootstrap.cc:654] T b7d8da6e2fe8499d9d53af84a25f2faf P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:55.900307 18733 tablet_service.cc:1514] Processing DeleteTablet for tablet c6e3d09402574f1e8fe9e4160c70a3ee with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:54 UTC) from {username='slave'} at 127.0.0.1:56540
I20240430 20:11:55.910537 19937 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: Sending DeleteTablet for 3 replicas of tablet c85a6d11d8af486492d8b843c75632e6
I20240430 20:11:55.911100 19937 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:50746 (request call id 78) took 4121 ms. Trace:
I20240430 20:11:55.911214 19937 rpcz_store.cc:276] 0430 20:11:51.789188 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:54.196544 (+2407356us) service_pool.cc:225] Handling call
0430 20:11:54.209526 (+ 12982us) spinlock_profiling.cc:243] Waited 12.6 ms on lock 0x616000070654. stack: 00007f6d23ac8756 00007f6d23ac8510 00007f6d22afb48d 0000000000955804 00000000009557cd 00007f6d3260715d 00007f6d32607253 00007f6d2f339d41 00007f6d36f2534d 00007f6d370b2ca6 00007f6d2de9bf66 00007f6d2de9bc95 00007f6d2a7d004e 00007f6d2a7cef2e 00007f6d2a7d5405 00007f6d2a7d5d2f
0430 20:11:54.210180 (+ 654us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0430 20:11:54.210330 (+ 150us) catalog_manager.cc:2749] Modifying in-memory table state
0430 20:11:54.210413 (+ 83us) catalog_manager.cc:2758] Locking tablets
0430 20:11:54.210602 (+ 189us) catalog_manager.cc:2772] Removing table and tablets from system table
0430 20:11:54.310496 (+ 99894us) catalog_manager.cc:2790] Removing table from by-name map
0430 20:11:54.500533 (+190037us) catalog_manager.cc:2806] Committing in-memory state
0430 20:11:55.910458 (+1409925us) catalog_manager.cc:2810] Aborting table tasks
0430 20:11:55.911066 (+ 608us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:54.271004 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:54.271181 (+ 177us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:54.271188 (+ 7us) write_op.cc:427] Acquired schema lock
0430 20:11:54.271191 (+ 3us) tablet.cc:660] Decoding operations
0430 20:11:54.271393 (+ 202us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:54.271410 (+ 17us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:54.271416 (+ 6us) tablet.cc:683] Acquiring locks for 2 operations
0430 20:11:54.271583 (+ 167us) tablet.cc:699] Row locks acquired
0430 20:11:54.271586 (+ 3us) write_op.cc:260] PREPARE: finished
0430 20:11:54.271635 (+ 49us) write_op.cc:270] Start()
0430 20:11:54.271683 (+ 48us) write_op.cc:276] Timestamp: P: 1714507914271607 usec, L: 0
0430 20:11:54.271688 (+ 5us) op_driver.cc:331] REPLICATION: starting
0430 20:11:54.285118 (+ 13430us) log.cc:842] Serialized 728 byte log entry
0430 20:11:54.287948 (+ 2830us) spinlock_profiling.cc:243] Waited 13.1 ms on lock 0x616000070654. stack: 00007f6d23ac8756 00007f6d23ac8510 00007f6d22afb48d 0000000000955804 00000000009557cd 000000000095519a 00007f6d2f30d162 00007f6d30675b02 00007f6d30674629 00007f6d3067c53b 00007f6d3067c3b8 0000000000a60bcc 00007f6d23b0eb74 00007f6d23b1219b 00007f6d23b12018 0000000000a60bcc
0430 20:11:54.307702 (+ 19754us) op_driver.cc:444] REPLICATION: finished
0430 20:11:54.308033 (+ 331us) write_op.cc:301] APPLY: starting
0430 20:11:54.308051 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:54.308092 (+ 41us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:54.308095 (+ 3us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:54.308183 (+ 88us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:54.308219 (+ 36us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0430 20:11:54.308226 (+ 7us) write_op.cc:312] APPLY: finished
0430 20:11:54.308315 (+ 89us) log.cc:842] Serialized 29 byte log entry
0430 20:11:54.308387 (+ 72us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:54.308404 (+ 17us) write_op.cc:446] Released schema lock
0430 20:11:54.308410 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:54.308467 (+ 57us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:54.308470 (+ 3us) write_op.cc:446] Released schema lock
Metrics: {"spinlock_wait_cycles":29085056,"child_traces":[["op",{"apply.queue_time_us":241,"apply.run_cpu_time_us":512,"apply.run_wall_time_us":512,"num_ops":2,"prepare.queue_time_us":59508,"prepare.run_cpu_time_us":1582,"prepare.run_wall_time_us":17037,"raft.queue_time_us":1376,"raft.run_cpu_time_us":291,"raft.run_wall_time_us":879,"replication_time_us":35974,"spinlock_wait_cycles":30126336}]]}
I20240430 20:11:55.915942 18733 tablet_service.cc:1514] Processing DeleteTablet for tablet c85a6d11d8af486492d8b843c75632e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:54 UTC) from {username='slave'} at 127.0.0.1:56540
I20240430 20:11:55.917991 21644 raft_consensus.cc:482] T 347a8224114b49e5b7dcb75d9b726e0a P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:55.918200 21644 raft_consensus.cc:504] T 347a8224114b49e5b7dcb75d9b726e0a P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
W20240430 20:11:55.922715 19945 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c4953578de5f45f3a6ba092a13357460 consensus state has no leader
W20240430 20:11:55.923202 19929 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f1073fc792954c5795a05aa7d0ec24df consensus state has no leader
I20240430 20:11:55.927866 21605 tablet.cc:1722] T 6b51cbc731794083a88fe4bed1ddeebf P 2efb542aa55e439b99ccb98d1c5c884a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240430 20:11:55.929900 21436 raft_consensus.cc:482] T dbb427aefa6b47b0805d1751fea4efcb P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:55.930047 21436 raft_consensus.cc:504] T dbb427aefa6b47b0805d1751fea4efcb P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:55.930759 21436 leader_election.cc:288] T dbb427aefa6b47b0805d1751fea4efcb P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:55.931469 21397 ts_tablet_manager.cc:1888] T 898b2c91adf34545808df9c786e218c9 P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:55.932520 21397 log.cc:1196] T 898b2c91adf34545808df9c786e218c9 P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/898b2c91adf34545808df9c786e218c9
I20240430 20:11:55.940038 21397 ts_tablet_manager.cc:1909] T 898b2c91adf34545808df9c786e218c9 P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:55.931639 18453 tablet_service.cc:1514] Processing DeleteTablet for tablet c6e3d09402574f1e8fe9e4160c70a3ee with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:54 UTC) from {username='slave'} at 127.0.0.1:41450
I20240430 20:11:55.941141 18447 tablet_service.cc:1514] Processing DeleteTablet for tablet c85a6d11d8af486492d8b843c75632e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:54 UTC) from {username='slave'} at 127.0.0.1:41450
I20240430 20:11:55.942361 21644 leader_election.cc:288] T 347a8224114b49e5b7dcb75d9b726e0a P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:55.945956 19930 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: Sending DeleteTablet for 3 replicas of tablet 13f0a0a87da242c8acb9892bc2897616
I20240430 20:11:55.946489 19930 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:50746 (request call id 77) took 4170 ms. Trace:
I20240430 20:11:55.946650 19930 rpcz_store.cc:276] 0430 20:11:51.775814 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:54.027389 (+2251575us) service_pool.cc:225] Handling call
0430 20:11:54.028078 (+ 689us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0430 20:11:54.028233 (+ 155us) catalog_manager.cc:2749] Modifying in-memory table state
0430 20:11:54.028299 (+ 66us) catalog_manager.cc:2758] Locking tablets
0430 20:11:54.028460 (+ 161us) catalog_manager.cc:2772] Removing table and tablets from system table
0430 20:11:54.109238 (+ 80778us) catalog_manager.cc:2790] Removing table from by-name map
0430 20:11:54.109341 (+ 103us) catalog_manager.cc:2806] Committing in-memory state
0430 20:11:55.945892 (+1836551us) catalog_manager.cc:2810] Aborting table tasks
0430 20:11:55.946428 (+ 536us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:54.030683 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:54.030830 (+ 147us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:54.030838 (+ 8us) write_op.cc:427] Acquired schema lock
0430 20:11:54.030843 (+ 5us) tablet.cc:660] Decoding operations
0430 20:11:54.031017 (+ 174us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:54.031035 (+ 18us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:54.031043 (+ 8us) tablet.cc:683] Acquiring locks for 2 operations
0430 20:11:54.031185 (+ 142us) tablet.cc:699] Row locks acquired
0430 20:11:54.031189 (+ 4us) write_op.cc:260] PREPARE: finished
0430 20:11:54.031232 (+ 43us) write_op.cc:270] Start()
0430 20:11:54.031275 (+ 43us) write_op.cc:276] Timestamp: P: 1714507914031177 usec, L: 0
0430 20:11:54.031281 (+ 6us) op_driver.cc:331] REPLICATION: starting
0430 20:11:54.031533 (+ 252us) log.cc:842] Serialized 728 byte log entry
0430 20:11:54.039419 (+ 7886us) op_driver.cc:444] REPLICATION: finished
0430 20:11:54.039585 (+ 166us) write_op.cc:301] APPLY: starting
0430 20:11:54.039604 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:54.039655 (+ 51us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:54.039694 (+ 39us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:54.039779 (+ 85us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:54.039817 (+ 38us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0430 20:11:54.039825 (+ 8us) write_op.cc:312] APPLY: finished
0430 20:11:54.039921 (+ 96us) log.cc:842] Serialized 29 byte log entry
0430 20:11:54.039997 (+ 76us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:54.040022 (+ 25us) write_op.cc:446] Released schema lock
0430 20:11:54.040029 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:54.040092 (+ 63us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:54.040096 (+ 4us) write_op.cc:446] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":111,"apply.run_cpu_time_us":565,"apply.run_wall_time_us":564,"num_ops":2,"prepare.queue_time_us":1521,"prepare.run_cpu_time_us":965,"prepare.run_wall_time_us":965,"raft.queue_time_us":459,"raft.run_cpu_time_us":296,"raft.run_wall_time_us":295,"replication_time_us":8107}]]}
W20240430 20:11:55.891023 18557 leader_election.cc:341] T cdeef95b54ca42d38d25b57738d84ef1 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
W20240430 20:11:55.947595 18557 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.177.3:44867, user_credentials={real_user=slave}} blocked reactor thread for 56587us
I20240430 20:11:55.920425 18607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "347a8224114b49e5b7dcb75d9b726e0a" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:55.947769 18607 raft_consensus.cc:2454] T 347a8224114b49e5b7dcb75d9b726e0a P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:55.948865 18600 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b7fdc6ec7e046c5af894f9a6eda17b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:40 UTC) from {username='slave'} at 127.0.0.1:45934
I20240430 20:11:55.949061 18586 tablet_service.cc:1514] Processing DeleteTablet for tablet 489dc838744a49eba542009c86688aa5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:43 UTC) from {username='slave'} at 127.0.0.1:45934
I20240430 20:11:55.949504 18593 tablet_service.cc:1514] Processing DeleteTablet for tablet a663030d37614053a7c0369841403422 with delete_type TABLET_DATA_DELETED (Sending initial creation of tablet) from {username='slave'} at 127.0.0.1:45934
I20240430 20:11:55.949905 18602 tablet_service.cc:1514] Processing DeleteTablet for tablet a37d917bcc45468f9e3ee805640fba98 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:44 UTC) from {username='slave'} at 127.0.0.1:45934
I20240430 20:11:55.949954 18587 tablet_service.cc:1514] Processing DeleteTablet for tablet 7bde1d6934784c9aa5911181e8df9eeb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:50 UTC) from {username='slave'} at 127.0.0.1:45934
I20240430 20:11:55.949914 18586 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e5225f1f5f544ecbf8dfa470af3daca with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:51 UTC) from {username='slave'} at 127.0.0.1:45934
I20240430 20:11:55.950078 18593 tablet_service.cc:1514] Processing DeleteTablet for tablet 024069ab786749028c020abd1c836cf5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:44 UTC) from {username='slave'} at 127.0.0.1:45934
I20240430 20:11:55.950198 18599 tablet_service.cc:1514] Processing DeleteTablet for tablet b7141e215d6d44bc8406c80508190874 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:43 UTC) from {username='slave'} at 127.0.0.1:45934
I20240430 20:11:55.950420 18603 tablet_service.cc:1514] Processing DeleteTablet for tablet fd1b8407071e4e4287fc172009777414 with delete_type TABLET_DATA_DELETED (replaced by tablet 8abf0b43f8a447dc902135ce94e4fc78) from {username='slave'} at 127.0.0.1:45934
I20240430 20:11:55.950551 18586 tablet_service.cc:1514] Processing DeleteTablet for tablet f9bf604e6c234349a9f72f45a4c4aef6 with delete_type TABLET_DATA_DELETED (replaced by tablet 415d99c36b1941e08aa4a16c50483ee4) from {username='slave'} at 127.0.0.1:45934
I20240430 20:11:55.950711 18585 tablet_service.cc:1514] Processing DeleteTablet for tablet 436579f473844c6ca1038b496201a550 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:40 UTC) from {username='slave'} at 127.0.0.1:45934
I20240430 20:11:55.950840 18587 tablet_service.cc:1514] Processing DeleteTablet for tablet dad16ae02b8b4e29a724d8eec67bb4b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:42 UTC) from {username='slave'} at 127.0.0.1:45934
I20240430 20:11:55.951006 18599 tablet_service.cc:1514] Processing DeleteTablet for tablet d7c1fc7e81364fb890bbc13d7fb6848b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:40 UTC) from {username='slave'} at 127.0.0.1:45934
I20240430 20:11:55.951047 18593 tablet_service.cc:1514] Processing DeleteTablet for tablet 2797eed593fb4b06a3c4d6a28e8075a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:42 UTC) from {username='slave'} at 127.0.0.1:45934
I20240430 20:11:55.951166 18602 tablet_service.cc:1514] Processing DeleteTablet for tablet ebf74809b52f47f3b0ed21b6ac0f541a with delete_type TABLET_DATA_DELETED (replaced by tablet 9e8ab75e7d9c4b64a7b7dd2678569490) from {username='slave'} at 127.0.0.1:45934
I20240430 20:11:55.951200 18595 tablet_service.cc:1514] Processing DeleteTablet for tablet c6e3d09402574f1e8fe9e4160c70a3ee with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:54 UTC) from {username='slave'} at 127.0.0.1:45934
I20240430 20:11:55.951387 18592 tablet_service.cc:1514] Processing DeleteTablet for tablet c85a6d11d8af486492d8b843c75632e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:54 UTC) from {username='slave'} at 127.0.0.1:45934
I20240430 20:11:55.951768 18590 tablet_service.cc:1514] Processing DeleteTablet for tablet 2aa2a489116940dea3173cad16d94fa5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:40 UTC) from {username='slave'} at 127.0.0.1:45934
I20240430 20:11:55.952167 18588 tablet_service.cc:1514] Processing DeleteTablet for tablet 6db55bb510b24b0cae897936211e15f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:34 UTC) from {username='slave'} at 127.0.0.1:45934
I20240430 20:11:55.952344 18589 tablet_service.cc:1514] Processing DeleteTablet for tablet 47e98935eb21449abd3d4d24327778d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:45 UTC) from {username='slave'} at 127.0.0.1:45934
I20240430 20:11:55.952492 21436 consensus_queue.cc:1035] T 330453ddeea6456e8ef93fa92608b0d3 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240430 20:11:55.953785 18558 leader_election.cc:341] T cdeef95b54ca42d38d25b57738d84ef1 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:55.954046 18479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbb427aefa6b47b0805d1751fea4efcb" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:55.954241 18558 leader_election.cc:302] T cdeef95b54ca42d38d25b57738d84ef1 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:55.954694 18733 tablet_service.cc:1514] Processing DeleteTablet for tablet 13f0a0a87da242c8acb9892bc2897616 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:54 UTC) from {username='slave'} at 127.0.0.1:56540
I20240430 20:11:55.954725 21638 raft_consensus.cc:2726] T cdeef95b54ca42d38d25b57738d84ef1 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:55.955354 18589 tablet_service.cc:1514] Processing DeleteTablet for tablet 13f0a0a87da242c8acb9892bc2897616 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:54 UTC) from {username='slave'} at 127.0.0.1:45934
I20240430 20:11:55.956370 18696 leader_election.cc:302] T 347a8224114b49e5b7dcb75d9b726e0a P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:55.956655 21436 raft_consensus.cc:2781] T 347a8224114b49e5b7dcb75d9b726e0a P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:55.956816 21436 raft_consensus.cc:482] T 347a8224114b49e5b7dcb75d9b726e0a P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:55.956975 21436 raft_consensus.cc:3037] T 347a8224114b49e5b7dcb75d9b726e0a P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:55.957664 18607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbb427aefa6b47b0805d1751fea4efcb" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
W20240430 20:11:55.958221 18696 leader_election.cc:341] T dbb427aefa6b47b0805d1751fea4efcb P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Illegal state: Tablet not RUNNING: STOPPED
I20240430 20:11:55.889542 21396 ts_tablet_manager.cc:1909] T dbb427aefa6b47b0805d1751fea4efcb P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:55.959830 21436 raft_consensus.cc:504] T 347a8224114b49e5b7dcb75d9b726e0a P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:55.960793 21396 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:45934 (request call id 84) took 2125 ms. Trace:
I20240430 20:11:55.961906 18479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "347a8224114b49e5b7dcb75d9b726e0a" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:55.962134 18479 raft_consensus.cc:2454] T 347a8224114b49e5b7dcb75d9b726e0a P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:55.961453 21644 consensus_queue.cc:1035] T 330453ddeea6456e8ef93fa92608b0d3 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:55.962766 21397 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56540 (request call id 63) took 2226 ms. Trace:
I20240430 20:11:55.964056 21397 rpcz_store.cc:276] 0430 20:11:53.736287 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:53.736333 (+ 46us) service_pool.cc:225] Handling call
0430 20:11:55.787669 (+2051336us) ts_tablet_manager.cc:1120] Deleting tablet 898b2c91adf34545808df9c786e218c9
0430 20:11:55.787683 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:55.896564 (+108881us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:55.922162 (+ 25598us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:55.962748 (+ 40586us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2051187}
I20240430 20:11:55.965221 19900 catalog_manager.cc:4830] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): tablet 898b2c91adf34545808df9c786e218c9 (table default.Table_8e605def35724c5c945b61107596d81a [id=36f6e11d7b1843c3be3e8b8160e06d53]) successfully deleted
W20240430 20:11:55.965871 18699 leader_election.cc:341] T dbb427aefa6b47b0805d1751fea4efcb P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:55.965986 18699 leader_election.cc:302] T dbb427aefa6b47b0805d1751fea4efcb P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710
I20240430 20:11:55.967955 21436 leader_election.cc:288] T 347a8224114b49e5b7dcb75d9b726e0a P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Requested vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:55.972456 21397 tablet_replica.cc:330] T 558e71ec370d46ca8fbdcec5b57eb2ec P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:55.972872 21397 raft_consensus.cc:2229] T 558e71ec370d46ca8fbdcec5b57eb2ec P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:55.962204 18589 tablet_service.cc:1514] Processing DeleteTablet for tablet 04b3dff86d5c48a09d03da58bd48f360 with delete_type TABLET_DATA_DELETED (replaced by tablet 89b774967624498fa2be499878b71124) from {username='slave'} at 127.0.0.1:45934
I20240430 20:11:55.963191 18588 tablet_service.cc:1514] Processing DeleteTablet for tablet c1017a50d1d646c486c0b9a75cb88ce4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:43 UTC) from {username='slave'} at 127.0.0.1:45934
I20240430 20:11:55.964061 18591 tablet_service.cc:1514] Processing DeleteTablet for tablet f940d17b02b541289121463e05b0bb5f with delete_type TABLET_DATA_DELETED (replaced by tablet 53da4b3855344b6d92211af767195dc8) from {username='slave'} at 127.0.0.1:45934
I20240430 20:11:55.964978 18607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "347a8224114b49e5b7dcb75d9b726e0a" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:55.969771 18590 tablet_service.cc:1514] Processing DeleteTablet for tablet 13c2e69ae91f4957887fd64b90f6e435 with delete_type TABLET_DATA_DELETED (replaced by tablet eba4bd4665ae4c04a2cab3ab1a2da780) from {username='slave'} at 127.0.0.1:45934
I20240430 20:11:55.976559 18607 raft_consensus.cc:3037] T 347a8224114b49e5b7dcb75d9b726e0a P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:55.976892 18450 tablet_service.cc:1514] Processing DeleteTablet for tablet 13f0a0a87da242c8acb9892bc2897616 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:54 UTC) from {username='slave'} at 127.0.0.1:41450
I20240430 20:11:55.977214 21454 raft_consensus.cc:2726] T dbb427aefa6b47b0805d1751fea4efcb P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:55.979473 18607 raft_consensus.cc:2454] T 347a8224114b49e5b7dcb75d9b726e0a P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:55.980211 21397 raft_consensus.cc:2258] T 558e71ec370d46ca8fbdcec5b57eb2ec P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:55.980377 18696 leader_election.cc:302] T 347a8224114b49e5b7dcb75d9b726e0a P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:55.980634 21454 raft_consensus.cc:2781] T 347a8224114b49e5b7dcb75d9b726e0a P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:55.980814 21454 raft_consensus.cc:686] T 347a8224114b49e5b7dcb75d9b726e0a P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Becoming Leader. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Running, Role: LEADER
I20240430 20:11:55.980965 21397 ts_tablet_manager.cc:1875] T 558e71ec370d46ca8fbdcec5b57eb2ec P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:55.981117 21454 consensus_queue.cc:237] T 347a8224114b49e5b7dcb75d9b726e0a P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
W20240430 20:11:55.981292 18558 consensus_peers.cc:485] T f7d3dea34cf54fe2b949d1b7ca68577a P 5e1c6397f09747f091534faf93542710 -> Peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Couldn't send request to peer 2efb542aa55e439b99ccb98d1c5c884a. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240430 20:11:55.982470 21396 rpcz_store.cc:276] 0430 20:11:53.835118 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:53.835183 (+ 65us) service_pool.cc:225] Handling call
0430 20:11:55.841279 (+2006096us) ts_tablet_manager.cc:1120] Deleting tablet dbb427aefa6b47b0805d1751fea4efcb
0430 20:11:55.844548 (+ 3269us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:55.851698 (+ 7150us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:55.888595 (+ 36897us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:55.960775 (+ 72180us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2005848}
I20240430 20:11:55.984266 19895 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet dbb427aefa6b47b0805d1751fea4efcb (table default.Table_4130caf534004edfb413056d7f6cfd52 [id=da9ad991953a40c4b77b31317597ad1d]) successfully deleted
I20240430 20:11:55.991194 21396 tablet_replica.cc:330] T 558e71ec370d46ca8fbdcec5b57eb2ec P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:55.991613 21396 raft_consensus.cc:2229] T 558e71ec370d46ca8fbdcec5b57eb2ec P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Raft consensus shutting down.
I20240430 20:11:55.992079 21396 raft_consensus.cc:2258] T 558e71ec370d46ca8fbdcec5b57eb2ec P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:55.995231 21638 raft_consensus.cc:482] T 9e741b00c74d43b5b6243ba7003f8804 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:55.995378 21638 raft_consensus.cc:504] T 9e741b00c74d43b5b6243ba7003f8804 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:55.996150 21638 leader_election.cc:288] T 9e741b00c74d43b5b6243ba7003f8804 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:55.996286 18983 tablet_bootstrap.cc:492] T a37d917bcc45468f9e3ee805640fba98 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:55.996415 18983 ts_tablet_manager.cc:1367] T a37d917bcc45468f9e3ee805640fba98 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.112s user 0.007s sys 0.001s
I20240430 20:11:55.997165 18983 raft_consensus.cc:348] T a37d917bcc45468f9e3ee805640fba98 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:55.997398 18983 raft_consensus.cc:374] T a37d917bcc45468f9e3ee805640fba98 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:55.997455 18983 raft_consensus.cc:729] T a37d917bcc45468f9e3ee805640fba98 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:55.997678 18983 consensus_queue.cc:260] T a37d917bcc45468f9e3ee805640fba98 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:55.999001 18983 ts_tablet_manager.cc:1398] T a37d917bcc45468f9e3ee805640fba98 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20240430 20:11:56.001964 18983 tablet_bootstrap.cc:492] T ef40bfdb5bf84f54a0068f40c7d1617a P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:55.993374 21396 ts_tablet_manager.cc:1875] T 558e71ec370d46ca8fbdcec5b57eb2ec P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:56.016029 18810 heartbeater.cc:347] Connected to a master server at 127.17.177.60:37225
I20240430 20:11:56.016526 18810 heartbeater.cc:503] Master 127.17.177.60:37225 was elected leader, sending a full tablet report...
I20240430 20:11:56.019775 18469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e741b00c74d43b5b6243ba7003f8804" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:56.021682 21638 consensus_queue.cc:1035] T a0d3c4ba64024bccad95ecc909d03ae0 P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:56.022132 18983 tablet_bootstrap.cc:654] T ef40bfdb5bf84f54a0068f40c7d1617a P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:56.023996 18749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e741b00c74d43b5b6243ba7003f8804" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:56.029943 21638 raft_consensus.cc:482] T d34a450a91d542c98261456d1f635e9c P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:56.030215 21638 raft_consensus.cc:504] T d34a450a91d542c98261456d1f635e9c P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:56.030936 21638 leader_election.cc:288] T d34a450a91d542c98261456d1f635e9c P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:56.037829 21396 ts_tablet_manager.cc:1888] T 558e71ec370d46ca8fbdcec5b57eb2ec P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:56.039759 18469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d34a450a91d542c98261456d1f635e9c" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:56.042697 21396 log.cc:1196] T 558e71ec370d46ca8fbdcec5b57eb2ec P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/558e71ec370d46ca8fbdcec5b57eb2ec
I20240430 20:11:56.043526 21396 ts_tablet_manager.cc:1909] T 558e71ec370d46ca8fbdcec5b57eb2ec P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:56.044111 21396 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:45934 (request call id 86) took 2208 ms. Trace:
W20240430 20:11:56.044865 18557 leader_election.cc:341] T 9e741b00c74d43b5b6243ba7003f8804 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:56.045454 21396 rpcz_store.cc:276] 0430 20:11:53.835317 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:53.835360 (+ 43us) service_pool.cc:225] Handling call
0430 20:11:55.991121 (+2155761us) ts_tablet_manager.cc:1120] Deleting tablet 558e71ec370d46ca8fbdcec5b57eb2ec
0430 20:11:55.991134 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:56.020078 (+ 28944us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:56.037708 (+ 17630us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:56.044090 (+ 6382us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2155570}
I20240430 20:11:56.045809 21396 tablet_replica.cc:330] T 898b2c91adf34545808df9c786e218c9 P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:56.046020 21396 raft_consensus.cc:2229] T 898b2c91adf34545808df9c786e218c9 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:56.046090 19895 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet 558e71ec370d46ca8fbdcec5b57eb2ec (table default.Table_408be016c50f4d0fa19189bbaddab089 [id=6b7bd5a6c20e4b43aa62365627503d5b]) successfully deleted
I20240430 20:11:56.046731 21396 raft_consensus.cc:2258] T 898b2c91adf34545808df9c786e218c9 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:56.048791 21396 ts_tablet_manager.cc:1875] T 898b2c91adf34545808df9c786e218c9 P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:56.050531 18756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d34a450a91d542c98261456d1f635e9c" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:56.051147 18469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "347a8224114b49e5b7dcb75d9b726e0a" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:56.051354 18469 raft_consensus.cc:3037] T 347a8224114b49e5b7dcb75d9b726e0a P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:56.054301 18469 raft_consensus.cc:2454] T 347a8224114b49e5b7dcb75d9b726e0a P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:56.056604 18988 tablet_bootstrap.cc:492] T b7d8da6e2fe8499d9d53af84a25f2faf P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:56.056802 18988 ts_tablet_manager.cc:1367] T b7d8da6e2fe8499d9d53af84a25f2faf P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.163s user 0.005s sys 0.009s
I20240430 20:11:56.057646 18988 raft_consensus.cc:348] T b7d8da6e2fe8499d9d53af84a25f2faf P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:56.057942 18988 raft_consensus.cc:374] T b7d8da6e2fe8499d9d53af84a25f2faf P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:56.058036 18988 raft_consensus.cc:729] T b7d8da6e2fe8499d9d53af84a25f2faf P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:56.059882 18988 consensus_queue.cc:260] T b7d8da6e2fe8499d9d53af84a25f2faf P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:56.060493 18988 ts_tablet_manager.cc:1398] T b7d8da6e2fe8499d9d53af84a25f2faf P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240430 20:11:56.060793 18988 tablet_bootstrap.cc:492] T 4391b5227fb64ef481542a962a5e21a5 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:56.062561 18988 tablet_bootstrap.cc:654] T 4391b5227fb64ef481542a962a5e21a5 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
W20240430 20:11:56.063920 18557 leader_election.cc:341] T d34a450a91d542c98261456d1f635e9c P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:56.065079 21396 ts_tablet_manager.cc:1888] T 898b2c91adf34545808df9c786e218c9 P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:56.065358 21396 log.cc:1196] T 898b2c91adf34545808df9c786e218c9 P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/898b2c91adf34545808df9c786e218c9
I20240430 20:11:56.065953 21396 ts_tablet_manager.cc:1909] T 898b2c91adf34545808df9c786e218c9 P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
W20240430 20:11:56.066330 18558 leader_election.cc:341] T 9e741b00c74d43b5b6243ba7003f8804 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:56.066476 18558 leader_election.cc:302] T 9e741b00c74d43b5b6243ba7003f8804 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4
W20240430 20:11:56.067224 18558 leader_election.cc:341] T d34a450a91d542c98261456d1f635e9c P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:56.067322 18558 leader_election.cc:302] T d34a450a91d542c98261456d1f635e9c P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:56.067652 21643 raft_consensus.cc:2726] T 9e741b00c74d43b5b6243ba7003f8804 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:56.067800 21643 consensus_queue.cc:1035] T 086b3dfa84c045a18c8c4f177dce606a P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:56.068527 21643 raft_consensus.cc:2726] T d34a450a91d542c98261456d1f635e9c P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:56.071556 18988 tablet_bootstrap.cc:492] T 4391b5227fb64ef481542a962a5e21a5 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:56.071689 18988 ts_tablet_manager.cc:1367] T 4391b5227fb64ef481542a962a5e21a5 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.001s
I20240430 20:11:56.077853 21454 consensus_queue.cc:1035] T a663030d37614053a7c0369841403422 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:56.078099 21397 ts_tablet_manager.cc:1888] T 558e71ec370d46ca8fbdcec5b57eb2ec P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:56.078384 21397 log.cc:1196] T 558e71ec370d46ca8fbdcec5b57eb2ec P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/558e71ec370d46ca8fbdcec5b57eb2ec
I20240430 20:11:56.078469 18988 raft_consensus.cc:348] T 4391b5227fb64ef481542a962a5e21a5 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:56.079222 18988 raft_consensus.cc:374] T 4391b5227fb64ef481542a962a5e21a5 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:56.079512 18988 raft_consensus.cc:729] T 4391b5227fb64ef481542a962a5e21a5 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:56.080077 21397 ts_tablet_manager.cc:1909] T 558e71ec370d46ca8fbdcec5b57eb2ec P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:56.080662 18988 consensus_queue.cc:260] T 4391b5227fb64ef481542a962a5e21a5 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:56.082273 21396 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:45934 (request call id 87) took 2246 ms. Trace:
I20240430 20:11:56.082409 21396 rpcz_store.cc:276] 0430 20:11:53.835400 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:53.835450 (+ 50us) service_pool.cc:225] Handling call
0430 20:11:56.045736 (+2210286us) ts_tablet_manager.cc:1120] Deleting tablet 898b2c91adf34545808df9c786e218c9
0430 20:11:56.045745 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:56.057239 (+ 11494us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:56.064899 (+ 7660us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:56.082250 (+ 17351us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2210179}
I20240430 20:11:56.082799 18983 tablet_bootstrap.cc:492] T ef40bfdb5bf84f54a0068f40c7d1617a P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:56.082943 18983 ts_tablet_manager.cc:1367] T ef40bfdb5bf84f54a0068f40c7d1617a P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.082s user 0.004s sys 0.003s
I20240430 20:11:56.083144 19895 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet 898b2c91adf34545808df9c786e218c9 (table default.Table_8e605def35724c5c945b61107596d81a [id=36f6e11d7b1843c3be3e8b8160e06d53]) successfully deleted
I20240430 20:11:56.083673 18983 raft_consensus.cc:348] T ef40bfdb5bf84f54a0068f40c7d1617a P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:56.083861 18983 raft_consensus.cc:374] T ef40bfdb5bf84f54a0068f40c7d1617a P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:56.083909 18983 raft_consensus.cc:729] T ef40bfdb5bf84f54a0068f40c7d1617a P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:56.084115 18983 consensus_queue.cc:260] T ef40bfdb5bf84f54a0068f40c7d1617a P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:56.084615 18983 ts_tablet_manager.cc:1398] T ef40bfdb5bf84f54a0068f40c7d1617a P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240430 20:11:56.086467 21397 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56540 (request call id 65) took 2349 ms. Trace:
I20240430 20:11:56.087337 21396 tablet_replica.cc:330] T d34a450a91d542c98261456d1f635e9c P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:56.087589 21396 raft_consensus.cc:2229] T d34a450a91d542c98261456d1f635e9c P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:56.087766 21396 raft_consensus.cc:2258] T d34a450a91d542c98261456d1f635e9c P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:56.095335 21667 consensus_queue.cc:1035] T a0d3c4ba64024bccad95ecc909d03ae0 P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:56.095695 21454 raft_consensus.cc:482] T 4391b5227fb64ef481542a962a5e21a5 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:56.095872 21454 raft_consensus.cc:504] T 4391b5227fb64ef481542a962a5e21a5 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:56.096601 21396 ts_tablet_manager.cc:1875] T d34a450a91d542c98261456d1f635e9c P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:56.096932 18607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4391b5227fb64ef481542a962a5e21a5" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:56.097182 18607 raft_consensus.cc:2454] T 4391b5227fb64ef481542a962a5e21a5 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:56.097663 18696 leader_election.cc:302] T 4391b5227fb64ef481542a962a5e21a5 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:56.097923 21644 raft_consensus.cc:2781] T 4391b5227fb64ef481542a962a5e21a5 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:56.097988 21644 raft_consensus.cc:482] T 4391b5227fb64ef481542a962a5e21a5 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:56.098049 21644 raft_consensus.cc:3037] T 4391b5227fb64ef481542a962a5e21a5 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:56.100674 21644 raft_consensus.cc:504] T 4391b5227fb64ef481542a962a5e21a5 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:56.101260 21644 leader_election.cc:288] T 4391b5227fb64ef481542a962a5e21a5 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Requested vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:56.102815 18607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4391b5227fb64ef481542a962a5e21a5" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:56.102989 18607 raft_consensus.cc:3037] T 4391b5227fb64ef481542a962a5e21a5 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:56.104539 18983 tablet_bootstrap.cc:492] T de546e2d6a5142159f23bd50d78ff735 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:56.086788 21397 rpcz_store.cc:276] 0430 20:11:53.736465 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:53.736498 (+ 33us) service_pool.cc:225] Handling call
0430 20:11:55.965374 (+2228876us) ts_tablet_manager.cc:1120] Deleting tablet 558e71ec370d46ca8fbdcec5b57eb2ec
0430 20:11:55.972379 (+ 7005us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:56.022466 (+ 50087us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:56.077983 (+ 55517us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:56.086361 (+ 8378us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10294,"tablet-delete.queue_time_us":2228725}
I20240430 20:11:56.105655 18607 raft_consensus.cc:2454] T 4391b5227fb64ef481542a962a5e21a5 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:56.105683 19900 catalog_manager.cc:4830] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): tablet 558e71ec370d46ca8fbdcec5b57eb2ec (table default.Table_408be016c50f4d0fa19189bbaddab089 [id=6b7bd5a6c20e4b43aa62365627503d5b]) successfully deleted
I20240430 20:11:56.106171 18696 leader_election.cc:302] T 4391b5227fb64ef481542a962a5e21a5 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:56.106523 21644 raft_consensus.cc:2781] T 4391b5227fb64ef481542a962a5e21a5 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:56.106829 21644 raft_consensus.cc:686] T 4391b5227fb64ef481542a962a5e21a5 P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Becoming Leader. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Running, Role: LEADER
I20240430 20:11:56.107061 18810 heartbeater.cc:503] Master 127.17.177.60:37225 was elected leader, sending a full tablet report...
I20240430 20:11:56.107180 21644 consensus_queue.cc:237] T 4391b5227fb64ef481542a962a5e21a5 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:56.108636 21454 leader_election.cc:288] T 4391b5227fb64ef481542a962a5e21a5 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:56.110570 21054 raft_consensus.cc:482] T e5b85a066cdc46d48ccf8201e5423c29 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:56.110826 18985 tablet_bootstrap.cc:492] T fd1b8407071e4e4287fc172009777414 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:56.110750 21054 raft_consensus.cc:504] T e5b85a066cdc46d48ccf8201e5423c29 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:56.110957 18985 ts_tablet_manager.cc:1367] T fd1b8407071e4e4287fc172009777414 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.284s user 0.006s sys 0.008s
I20240430 20:11:56.111475 21054 leader_election.cc:288] T e5b85a066cdc46d48ccf8201e5423c29 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:56.111604 18983 tablet_bootstrap.cc:654] T de546e2d6a5142159f23bd50d78ff735 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:56.111768 18985 raft_consensus.cc:348] T fd1b8407071e4e4287fc172009777414 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:56.112107 18607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5b85a066cdc46d48ccf8201e5423c29" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:56.112522 18607 raft_consensus.cc:2454] T e5b85a066cdc46d48ccf8201e5423c29 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 0.
I20240430 20:11:56.112977 18415 leader_election.cc:302] T e5b85a066cdc46d48ccf8201e5423c29 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:56.113341 21054 raft_consensus.cc:2781] T e5b85a066cdc46d48ccf8201e5423c29 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:56.113368 18590 tablet_service.cc:1514] Processing DeleteTablet for tablet a692b253db914531b8fc6ffaf3e350b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:51 UTC) from {username='slave'} at 127.0.0.1:45934
I20240430 20:11:56.113445 21054 raft_consensus.cc:482] T e5b85a066cdc46d48ccf8201e5423c29 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:56.113540 21054 raft_consensus.cc:3037] T e5b85a066cdc46d48ccf8201e5423c29 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:56.116284 21054 raft_consensus.cc:504] T e5b85a066cdc46d48ccf8201e5423c29 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:56.117149 21054 leader_election.cc:288] T e5b85a066cdc46d48ccf8201e5423c29 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Requested vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:56.111948 18985 raft_consensus.cc:374] T fd1b8407071e4e4287fc172009777414 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:56.118674 19945 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:50746 (request call id 80) took 3998 ms. Trace:
I20240430 20:11:56.118888 18985 raft_consensus.cc:729] T fd1b8407071e4e4287fc172009777414 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:56.118827 19945 rpcz_store.cc:276] 0430 20:11:52.119897 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:54.264421 (+2144524us) service_pool.cc:225] Handling call
0430 20:11:54.273705 (+ 9284us) spinlock_profiling.cc:243] Waited 8.94 ms on lock 0x616000070654. stack: 00007f6d23ac8756 00007f6d23ac8510 00007f6d22afb48d 0000000000955804 00000000009557cd 00007f6d3260715d 00007f6d32607253 00007f6d2f339d41 00007f6d36f2534d 00007f6d370b3f46 00007f6d2de9dc26 00007f6d2de9d955 00007f6d2a7d004e 00007f6d2a7cef2e 00007f6d2a7d5405 00007f6d2a7d5d2f
0430 20:11:54.274120 (+ 415us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0430 20:11:54.274309 (+ 189us) catalog_manager.cc:3613] Apply alter schema
0430 20:11:54.499673 (+225364us) catalog_manager.cc:3636] Acquired catalog manager lock
0430 20:11:54.499830 (+ 157us) catalog_manager.cc:3783] Updating metadata on disk
0430 20:11:54.506719 (+ 6889us) catalog_manager.cc:3814] Committing alterations to in-memory state
0430 20:11:56.118632 (+1611913us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:54.500482 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:54.500611 (+ 129us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:54.500618 (+ 7us) write_op.cc:427] Acquired schema lock
0430 20:11:54.500620 (+ 2us) tablet.cc:660] Decoding operations
0430 20:11:54.500759 (+ 139us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:54.500776 (+ 17us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:54.500784 (+ 8us) tablet.cc:683] Acquiring locks for 1 operations
0430 20:11:54.500862 (+ 78us) tablet.cc:699] Row locks acquired
0430 20:11:54.500864 (+ 2us) write_op.cc:260] PREPARE: finished
0430 20:11:54.500901 (+ 37us) write_op.cc:270] Start()
0430 20:11:54.500944 (+ 43us) write_op.cc:276] Timestamp: P: 1714507914500853 usec, L: 0
0430 20:11:54.500959 (+ 15us) op_driver.cc:331] REPLICATION: starting
0430 20:11:54.501185 (+ 226us) log.cc:842] Serialized 405 byte log entry
0430 20:11:54.505593 (+ 4408us) op_driver.cc:444] REPLICATION: finished
0430 20:11:54.505768 (+ 175us) write_op.cc:301] APPLY: starting
0430 20:11:54.505788 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:54.505844 (+ 56us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:54.505935 (+ 91us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:54.506025 (+ 90us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:54.506073 (+ 48us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0430 20:11:54.506083 (+ 10us) write_op.cc:312] APPLY: finished
0430 20:11:54.506220 (+ 137us) log.cc:842] Serialized 23 byte log entry
0430 20:11:54.506310 (+ 90us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:54.506331 (+ 21us) write_op.cc:446] Released schema lock
0430 20:11:54.506338 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:54.506393 (+ 55us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:54.506397 (+ 4us) write_op.cc:446] Released schema lock
Metrics: {"spinlock_wait_cycles":20567296,"child_traces":[["op",{"apply.queue_time_us":98,"apply.run_cpu_time_us":685,"apply.run_wall_time_us":685,"num_ops":1,"prepare.queue_time_us":57,"prepare.run_cpu_time_us":817,"prepare.run_wall_time_us":817,"raft.queue_time_us":486,"raft.run_cpu_time_us":361,"raft.run_wall_time_us":361,"replication_time_us":4606}]]}
I20240430 20:11:56.119591 18985 consensus_queue.cc:260] T fd1b8407071e4e4287fc172009777414 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:56.121506 18607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5b85a066cdc46d48ccf8201e5423c29" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:56.121752 18607 raft_consensus.cc:3037] T e5b85a066cdc46d48ccf8201e5423c29 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:56.122108 18985 ts_tablet_manager.cc:1398] T fd1b8407071e4e4287fc172009777414 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.011s user 0.001s sys 0.001s
I20240430 20:11:56.123301 18985 tablet_bootstrap.cc:492] T 00be6ddd91e84865a48743fb4d077eec P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:56.123589 21667 raft_consensus.cc:482] T f9bf604e6c234349a9f72f45a4c4aef6 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:56.123720 21667 raft_consensus.cc:504] T f9bf604e6c234349a9f72f45a4c4aef6 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:56.124408 21396 ts_tablet_manager.cc:1888] T d34a450a91d542c98261456d1f635e9c P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:56.124516 21396 log.cc:1196] T d34a450a91d542c98261456d1f635e9c P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/d34a450a91d542c98261456d1f635e9c
I20240430 20:11:56.127892 18988 ts_tablet_manager.cc:1398] T 4391b5227fb64ef481542a962a5e21a5 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.056s user 0.002s sys 0.000s
I20240430 20:11:56.130313 18985 tablet_bootstrap.cc:654] T 00be6ddd91e84865a48743fb4d077eec P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:56.131242 21667 leader_election.cc:288] T f9bf604e6c234349a9f72f45a4c4aef6 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:56.131822 21396 ts_tablet_manager.cc:1909] T d34a450a91d542c98261456d1f635e9c P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:56.132112 21396 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:45934 (request call id 88) took 2296 ms. Trace:
I20240430 20:11:56.132180 21396 rpcz_store.cc:276] 0430 20:11:53.835698 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:53.835740 (+ 42us) service_pool.cc:225] Handling call
0430 20:11:56.082828 (+2247088us) ts_tablet_manager.cc:1120] Deleting tablet d34a450a91d542c98261456d1f635e9c
0430 20:11:56.087280 (+ 4452us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:56.100091 (+ 12811us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:56.124292 (+ 24201us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:56.132096 (+ 7804us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2246950}
I20240430 20:11:56.132524 21396 tablet_replica.cc:330] T 8668f7a285de43f48d83abecb4b2e96c P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:56.132771 21396 raft_consensus.cc:2229] T 8668f7a285de43f48d83abecb4b2e96c P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:56.132941 21396 raft_consensus.cc:2258] T 8668f7a285de43f48d83abecb4b2e96c P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:56.134063 19895 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet d34a450a91d542c98261456d1f635e9c (table default.Table_f30868d759f64cceb682278f0dbb51ae [id=3291bc6bb4f146ee8f90d89029a70aef]) successfully deleted
I20240430 20:11:56.138906 21396 ts_tablet_manager.cc:1875] T 8668f7a285de43f48d83abecb4b2e96c P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:56.141866 21643 raft_consensus.cc:482] T ef40bfdb5bf84f54a0068f40c7d1617a P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:56.142033 21643 raft_consensus.cc:504] T ef40bfdb5bf84f54a0068f40c7d1617a P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:56.142763 21643 leader_election.cc:288] T ef40bfdb5bf84f54a0068f40c7d1617a P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:56.159518 18607 raft_consensus.cc:2454] T e5b85a066cdc46d48ccf8201e5423c29 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 1.
I20240430 20:11:56.161175 21468 raft_consensus.cc:482] T 9a76db2c7ff146758c54908e428a0e02 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:56.161382 21468 raft_consensus.cc:504] T 9a76db2c7ff146758c54908e428a0e02 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:56.162350 21468 leader_election.cc:288] T 9a76db2c7ff146758c54908e428a0e02 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:56.163220 18415 leader_election.cc:302] T e5b85a066cdc46d48ccf8201e5423c29 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:56.163929 21581 raft_consensus.cc:2781] T e5b85a066cdc46d48ccf8201e5423c29 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:56.164086 21581 raft_consensus.cc:686] T e5b85a066cdc46d48ccf8201e5423c29 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Becoming Leader. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Running, Role: LEADER
I20240430 20:11:56.164340 21581 consensus_queue.cc:237] T e5b85a066cdc46d48ccf8201e5423c29 P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:56.164227 21396 ts_tablet_manager.cc:1888] T 8668f7a285de43f48d83abecb4b2e96c P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20240430 20:11:56.165136 19938 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.17.177.2:50593 (request call id 149) took 4409 ms (client timeout 1000 ms). Trace:
I20240430 20:11:56.165442 21396 log.cc:1196] T 8668f7a285de43f48d83abecb4b2e96c P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/8668f7a285de43f48d83abecb4b2e96c
W20240430 20:11:56.165261 19938 rpcz_store.cc:269] 0430 20:11:51.755403 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:51.919223 (+163820us) service_pool.cc:225] Handling call
0430 20:11:52.075306 (+156083us) spinlock_profiling.cc:243] Waited 66.3 ms on lock 0x616000070654. stack: 00007f6d23ac8756 00007f6d23ac8510 00007f6d22afb48d 0000000000955804 00000000009557cd 00007f6d3260715d 00007f6d32607253 00007f6d2f339d41 00007f6d36f2534d 00007f6d370ac668 00007f6d2de97786 00007f6d2de974b5 00007f6d2a7d004e 00007f6d2a7cef2e 00007f6d2a7d5405 00007f6d2a7d5d2f
0430 20:11:56.165072 (+4089766us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:53.158521 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:53.158672 (+ 151us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:53.158679 (+ 7us) write_op.cc:427] Acquired schema lock
0430 20:11:53.158682 (+ 3us) tablet.cc:660] Decoding operations
0430 20:11:53.158809 (+ 127us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:53.158825 (+ 16us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:53.158829 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0430 20:11:53.158905 (+ 76us) tablet.cc:699] Row locks acquired
0430 20:11:53.158907 (+ 2us) write_op.cc:260] PREPARE: finished
0430 20:11:53.158943 (+ 36us) write_op.cc:270] Start()
0430 20:11:53.158981 (+ 38us) write_op.cc:276] Timestamp: P: 1714507913158883 usec, L: 0
0430 20:11:53.158986 (+ 5us) op_driver.cc:331] REPLICATION: starting
0430 20:11:53.159196 (+ 210us) log.cc:842] Serialized 530 byte log entry
0430 20:11:53.162517 (+ 3321us) op_driver.cc:444] REPLICATION: finished
0430 20:11:53.162660 (+ 143us) write_op.cc:301] APPLY: starting
0430 20:11:53.162678 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:53.162721 (+ 43us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:53.162758 (+ 37us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:53.162814 (+ 56us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:53.162847 (+ 33us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0430 20:11:53.162854 (+ 7us) write_op.cc:312] APPLY: finished
0430 20:11:53.162929 (+ 75us) log.cc:842] Serialized 23 byte log entry
0430 20:11:53.162988 (+ 59us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:53.163009 (+ 21us) write_op.cc:446] Released schema lock
0430 20:11:53.163015 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:53.163067 (+ 52us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:53.163073 (+ 6us) write_op.cc:446] Released schema lock
Related trace 'op':
0430 20:11:54.204138 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:54.204283 (+ 145us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:54.204292 (+ 9us) write_op.cc:427] Acquired schema lock
0430 20:11:54.204296 (+ 4us) tablet.cc:660] Decoding operations
0430 20:11:54.204444 (+ 148us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:54.204455 (+ 11us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:54.204460 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0430 20:11:54.204523 (+ 63us) tablet.cc:699] Row locks acquired
0430 20:11:54.204525 (+ 2us) write_op.cc:260] PREPARE: finished
0430 20:11:54.204562 (+ 37us) write_op.cc:270] Start()
0430 20:11:54.204606 (+ 44us) write_op.cc:276] Timestamp: P: 1714507914204541 usec, L: 0
0430 20:11:54.204611 (+ 5us) op_driver.cc:331] REPLICATION: starting
0430 20:11:54.204805 (+ 194us) log.cc:842] Serialized 356 byte log entry
0430 20:11:54.211800 (+ 6995us) op_driver.cc:444] REPLICATION: finished
0430 20:11:54.212792 (+ 992us) write_op.cc:301] APPLY: starting
0430 20:11:54.212801 (+ 9us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:54.212833 (+ 32us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:54.212871 (+ 38us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:54.212911 (+ 40us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:54.212937 (+ 26us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0430 20:11:54.212944 (+ 7us) write_op.cc:312] APPLY: finished
0430 20:11:54.213011 (+ 67us) log.cc:842] Serialized 23 byte log entry
0430 20:11:54.213048 (+ 37us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:54.213060 (+ 12us) write_op.cc:446] Released schema lock
0430 20:11:54.213065 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:54.213105 (+ 40us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:54.213108 (+ 3us) write_op.cc:446] Released schema lock
Metrics: {"mutex_wait_us":167916,"reported_tablets":201,"spinlock_wait_cycles":152600832,"child_traces":[["op",{"apply.queue_time_us":94,"apply.run_cpu_time_us":463,"apply.run_wall_time_us":462,"num_ops":1,"prepare.queue_time_us":3286,"prepare.run_cpu_time_us":781,"prepare.run_wall_time_us":781,"raft.queue_time_us":438,"raft.run_cpu_time_us":259,"raft.run_wall_time_us":259,"replication_time_us":3508}],["op",{"apply.queue_time_us":958,"apply.run_cpu_time_us":354,"apply.run_wall_time_us":353,"num_ops":1,"prepare.queue_time_us":39526,"prepare.run_cpu_time_us":798,"prepare.run_wall_time_us":798,"raft.queue_time_us":1218,"raft.run_cpu_time_us":4,"raft.run_wall_time_us":4,"replication_time_us":7168}]]}
I20240430 20:11:56.167714 19938 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50746:
table { table_name: "default.Table_326878a52ab041609f3c339cd65b7013" } new_table_name: "default.Table_0d3f914f27df478c8df8cbb69bc0d8b4" modify_external_catalogs: true
I20240430 20:11:56.169191 21396 ts_tablet_manager.cc:1909] T 8668f7a285de43f48d83abecb4b2e96c P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:56.177163 21396 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:45934 (request call id 89) took 2340 ms. Trace:
I20240430 20:11:56.177798 21396 rpcz_store.cc:276] 0430 20:11:53.836072 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:53.836112 (+ 40us) service_pool.cc:225] Handling call
0430 20:11:56.132440 (+2296328us) ts_tablet_manager.cc:1120] Deleting tablet 8668f7a285de43f48d83abecb4b2e96c
0430 20:11:56.132454 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:56.147632 (+ 15178us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:56.160899 (+ 13267us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:56.176910 (+ 16011us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2296134}
I20240430 20:11:56.179170 19895 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet 8668f7a285de43f48d83abecb4b2e96c (table default.Table_6d9a8c8bd13b42eca6834e707653a66b [id=71c1c0c9a8b544a0a9b1b1a414a6f749]) successfully deleted
I20240430 20:11:56.177817 21395 ts_tablet_manager.cc:1888] T f7d3dea34cf54fe2b949d1b7ca68577a P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:56.180083 21396 tablet_replica.cc:330] T aa9998f4e17147aa8f2b35c01980463a P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:56.180635 21395 log.cc:1196] T f7d3dea34cf54fe2b949d1b7ca68577a P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/f7d3dea34cf54fe2b949d1b7ca68577a
I20240430 20:11:56.181229 21396 raft_consensus.cc:2229] T aa9998f4e17147aa8f2b35c01980463a P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:56.181450 19940 catalog_manager.cc:5484] T 2e1cac64cefe4336bd3207b5ca58b631 P 3c51c33d5616482cafb68e26688820f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c51c33d5616482cafb68e26688820f4 (127.17.177.3). New cstate: current_term: 1 leader_uuid: "3c51c33d5616482cafb68e26688820f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } health_report { overall_health: UNKNOWN } } }
I20240430 20:11:56.182078 19940 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 89b774967624498fa2be499878b71124 (table default.Table_2ec9e316bc3347b189a13b7a8bbcf829 [id=9fa22833979f4c2f978de651145acef9]). Expected version 1 got 0
I20240430 20:11:56.182281 21396 raft_consensus.cc:2258] T aa9998f4e17147aa8f2b35c01980463a P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:56.182329 19940 catalog_manager.cc:5484] T 1050ab528c7d41559831652a1437d358 P 3c51c33d5616482cafb68e26688820f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c51c33d5616482cafb68e26688820f4 (127.17.177.3). New cstate: current_term: 1 leader_uuid: "3c51c33d5616482cafb68e26688820f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } health_report { overall_health: UNKNOWN } } }
I20240430 20:11:56.183199 19940 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 1050ab528c7d41559831652a1437d358 (table default.Table_acddc4cbf6334546b5fb3e4752fef2a5 [id=274cb15fd751410e83327ac4e8dc4629]). Expected version 2 got 0
I20240430 20:11:56.184363 19940 catalog_manager.cc:5484] T 3191aa54a9644783a2403991c3be27c8 P 3c51c33d5616482cafb68e26688820f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c51c33d5616482cafb68e26688820f4 (127.17.177.3). New cstate: current_term: 1 leader_uuid: "3c51c33d5616482cafb68e26688820f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } health_report { overall_health: UNKNOWN } } }
I20240430 20:11:56.136718 21454 raft_consensus.cc:482] T e5b85a066cdc46d48ccf8201e5423c29 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:56.190532 21454 raft_consensus.cc:504] T e5b85a066cdc46d48ccf8201e5423c29 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:56.190207 21396 ts_tablet_manager.cc:1875] T aa9998f4e17147aa8f2b35c01980463a P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:56.191435 21454 leader_election.cc:288] T e5b85a066cdc46d48ccf8201e5423c29 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
W20240430 20:11:56.182289 18699 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.177.1:41217, user_credentials={real_user=slave}} blocked reactor thread for 54067.7us
I20240430 20:11:56.187556 21397 tablet_replica.cc:330] T f7d3dea34cf54fe2b949d1b7ca68577a P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:56.191733 21397 raft_consensus.cc:2229] T f7d3dea34cf54fe2b949d1b7ca68577a P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:56.191803 18607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5b85a066cdc46d48ccf8201e5423c29" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:56.191900 21397 raft_consensus.cc:2258] T f7d3dea34cf54fe2b949d1b7ca68577a P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:56.192030 18607 raft_consensus.cc:2379] T e5b85a066cdc46d48ccf8201e5423c29 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c51c33d5616482cafb68e26688820f4 in current term 1: Already voted for candidate 2efb542aa55e439b99ccb98d1c5c884a in this term.
I20240430 20:11:56.192441 21397 ts_tablet_manager.cc:1875] T f7d3dea34cf54fe2b949d1b7ca68577a P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:56.193048 18668 heartbeater.cc:503] Master 127.17.177.62:36441 was elected leader, sending a full tablet report...
I20240430 20:11:56.199692 21397 ts_tablet_manager.cc:1888] T f7d3dea34cf54fe2b949d1b7ca68577a P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:56.199940 21397 log.cc:1196] T f7d3dea34cf54fe2b949d1b7ca68577a P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/f7d3dea34cf54fe2b949d1b7ca68577a
I20240430 20:11:56.200420 21397 ts_tablet_manager.cc:1909] T f7d3dea34cf54fe2b949d1b7ca68577a P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:56.200837 21397 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56540 (request call id 67) took 2464 ms. Trace:
I20240430 20:11:56.200996 21397 rpcz_store.cc:276] 0430 20:11:53.736702 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:53.736843 (+ 141us) service_pool.cc:225] Handling call
0430 20:11:56.106357 (+2369514us) ts_tablet_manager.cc:1120] Deleting tablet f7d3dea34cf54fe2b949d1b7ca68577a
0430 20:11:56.187497 (+ 81140us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:56.196405 (+ 8908us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:56.199574 (+ 3169us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:56.200813 (+ 1239us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2369142}
I20240430 20:11:56.201645 19900 catalog_manager.cc:4830] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): tablet f7d3dea34cf54fe2b949d1b7ca68577a (table default.Table_a4ea055447464ca6af32fe80bab34c42 [id=ea8038670f7c44a9bb20c396b54d6440]) successfully deleted
I20240430 20:11:56.202075 21397 tablet_replica.cc:330] T df92eb85261d47d88cbf16d29d9ad30e P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:56.202317 21397 raft_consensus.cc:2229] T df92eb85261d47d88cbf16d29d9ad30e P 3c51c33d5616482cafb68e26688820f4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:56.202560 21397 raft_consensus.cc:2258] T df92eb85261d47d88cbf16d29d9ad30e P 3c51c33d5616482cafb68e26688820f4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:56.203523 21397 ts_tablet_manager.cc:1875] T df92eb85261d47d88cbf16d29d9ad30e P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:56.208171 21054 consensus_queue.cc:1035] T 13c2e69ae91f4957887fd64b90f6e435 P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:56.209298 18988 tablet_bootstrap.cc:492] T fcc16db4d6be42edb7bd1ce18fe0190d P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:56.209285 19929 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:50746 (request call id 76) took 4441 ms. Trace:
I20240430 20:11:56.209468 19929 rpcz_store.cc:276] 0430 20:11:51.767922 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:54.010203 (+2242281us) service_pool.cc:225] Handling call
0430 20:11:54.074852 (+ 64649us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0430 20:11:54.075154 (+ 302us) catalog_manager.cc:3613] Apply alter schema
0430 20:11:54.075388 (+ 234us) catalog_manager.cc:3636] Acquired catalog manager lock
0430 20:11:54.075500 (+ 112us) catalog_manager.cc:3783] Updating metadata on disk
0430 20:11:54.214794 (+139294us) catalog_manager.cc:3814] Committing alterations to in-memory state
0430 20:11:56.209236 (+1994442us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:54.076194 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:54.076313 (+ 119us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:54.076319 (+ 6us) write_op.cc:427] Acquired schema lock
0430 20:11:54.076321 (+ 2us) tablet.cc:660] Decoding operations
0430 20:11:54.076447 (+ 126us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:54.076466 (+ 19us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:54.076474 (+ 8us) tablet.cc:683] Acquiring locks for 1 operations
0430 20:11:54.076557 (+ 83us) tablet.cc:699] Row locks acquired
0430 20:11:54.076561 (+ 4us) write_op.cc:260] PREPARE: finished
0430 20:11:54.076606 (+ 45us) write_op.cc:270] Start()
0430 20:11:54.076646 (+ 40us) write_op.cc:276] Timestamp: P: 1714507914076546 usec, L: 0
0430 20:11:54.076652 (+ 6us) op_driver.cc:331] REPLICATION: starting
0430 20:11:54.203712 (+127060us) log.cc:842] Serialized 405 byte log entry
0430 20:11:54.204009 (+ 297us) spinlock_profiling.cc:243] Waited 127 ms on lock 0x616000070654. stack: 00007f6d23ac8756 00007f6d23ac8510 00007f6d22afb48d 0000000000955804 00000000009557cd 000000000095519a 00007f6d2f30d162 00007f6d30675b02 00007f6d30674629 00007f6d3067c53b 00007f6d3067c3b8 0000000000a60bcc 00007f6d23b0eb74 00007f6d23b1219b 00007f6d23b12018 0000000000a60bcc
0430 20:11:54.211717 (+ 7708us) op_driver.cc:444] REPLICATION: finished
0430 20:11:54.212255 (+ 538us) write_op.cc:301] APPLY: starting
0430 20:11:54.212277 (+ 22us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:54.212336 (+ 59us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:54.212341 (+ 5us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:54.212414 (+ 73us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:54.212469 (+ 55us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0430 20:11:54.212475 (+ 6us) write_op.cc:312] APPLY: finished
0430 20:11:54.212551 (+ 76us) log.cc:842] Serialized 23 byte log entry
0430 20:11:54.212620 (+ 69us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:54.212638 (+ 18us) write_op.cc:446] Released schema lock
0430 20:11:54.212645 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:54.212703 (+ 58us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:54.212707 (+ 4us) write_op.cc:446] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":468,"apply.run_cpu_time_us":507,"apply.run_wall_time_us":507,"num_ops":1,"prepare.queue_time_us":61,"prepare.run_cpu_time_us":2841,"prepare.run_wall_time_us":127893,"raft.queue_time_us":1683,"raft.run_cpu_time_us":335,"raft.run_wall_time_us":335,"replication_time_us":135034,"spinlock_wait_cycles":291509504}]]}
I20240430 20:11:56.211140 19929 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50746:
table { table_name: "default.Table_21208fd7cdca4c73a4857bc4692187f6" } new_table_name: "default.Table_491c94cc9f9048928da775e9183132b2" modify_external_catalogs: true
I20240430 20:11:56.211472 18481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9bf604e6c234349a9f72f45a4c4aef6" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
W20240430 20:11:56.222258 18415 consensus_peers.cc:485] T e87fb2f0361d4aa3ada27bbdfa88e94c P 2efb542aa55e439b99ccb98d1c5c884a -> Peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Couldn't send request to peer 5e1c6397f09747f091534faf93542710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e87fb2f0361d4aa3ada27bbdfa88e94c. This is attempt 6: this message will repeat every 5th retry.
I20240430 20:11:56.229799 21395 ts_tablet_manager.cc:1909] T f7d3dea34cf54fe2b949d1b7ca68577a P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:56.230394 21395 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41450 (request call id 63) took 2426 ms. Trace:
I20240430 20:11:56.230623 21395 rpcz_store.cc:276] 0430 20:11:53.804242 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:53.804289 (+ 47us) service_pool.cc:225] Handling call
0430 20:11:55.836696 (+2032407us) ts_tablet_manager.cc:1120] Deleting tablet f7d3dea34cf54fe2b949d1b7ca68577a
0430 20:11:55.836705 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:56.128584 (+291879us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:56.177522 (+ 48938us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:56.230343 (+ 52821us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2032245}
I20240430 20:11:56.230976 19940 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 69f88532952a4591bf1577541c3430e6 (table default.Table_1177828cad1f4a24bd397619dad96170 [id=a95fed1bc3fa4865bf9a35dfc51b659c]). Expected version 2 got 0
I20240430 20:11:56.231432 19940 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 086b3dfa84c045a18c8c4f177dce606a (table default.Table_df19cb8a6f4d46699350ec5b69c14e5a [id=197923633fc54a4582207942b51530e2]). Expected version 2 got 0
I20240430 20:11:56.231688 19940 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 3a03b6997921408cb5a65f4cae223e9c (table default.Table_744b5c96ea014bdb8feae740356a790e [id=83b1ac2e9c3d4a10b02f96ebb17faf44]). Expected version 1 got 0
I20240430 20:11:56.232520 19940 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 488d0e609b224674932f890939664df2 (table default.Table_58f35a2120874472978ca0e5f816e86c [id=ff18001c6d814afc8c1768d0c84a6bac]). Expected version 1 got 0
I20240430 20:11:56.233171 21468 raft_consensus.cc:482] T 8e7f08e9ac274bfa86763eec895bbfda P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:56.233362 21468 raft_consensus.cc:504] T 8e7f08e9ac274bfa86763eec895bbfda P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:56.233990 21468 leader_election.cc:288] T 8e7f08e9ac274bfa86763eec895bbfda P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:56.237690 18749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9bf604e6c234349a9f72f45a4c4aef6" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:56.240175 18749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef40bfdb5bf84f54a0068f40c7d1617a" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:56.244916 18749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a76db2c7ff146758c54908e428a0e02" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
W20240430 20:11:56.245548 18558 leader_election.cc:341] T f9bf604e6c234349a9f72f45a4c4aef6 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:56.234427 18481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef40bfdb5bf84f54a0068f40c7d1617a" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:56.234851 18479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a76db2c7ff146758c54908e428a0e02" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:56.247022 18467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4391b5227fb64ef481542a962a5e21a5" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:56.238773 18469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e7f08e9ac274bfa86763eec895bbfda" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:56.248636 18988 tablet_bootstrap.cc:654] T fcc16db4d6be42edb7bd1ce18fe0190d P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:56.251363 18749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e7f08e9ac274bfa86763eec895bbfda" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:56.251330 18751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5b85a066cdc46d48ccf8201e5423c29" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:56.251629 18751 raft_consensus.cc:3037] T e5b85a066cdc46d48ccf8201e5423c29 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
W20240430 20:11:56.252465 18415 consensus_peers.cc:485] T 9c73d0a6bc414d3b9900570848d19a51 P 2efb542aa55e439b99ccb98d1c5c884a -> Peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Couldn't send request to peer 5e1c6397f09747f091534faf93542710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9c73d0a6bc414d3b9900570848d19a51. This is attempt 1: this message will repeat every 5th retry.
I20240430 20:11:56.254161 18751 raft_consensus.cc:2454] T e5b85a066cdc46d48ccf8201e5423c29 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 1.
I20240430 20:11:56.251382 18756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5b85a066cdc46d48ccf8201e5423c29" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:56.254562 18756 raft_consensus.cc:2362] T e5b85a066cdc46d48ccf8201e5423c29 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 1. Re-sending same reply.
I20240430 20:11:56.255304 18469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4391b5227fb64ef481542a962a5e21a5" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:56.257354 18810 heartbeater.cc:503] Master 127.17.177.60:37225 was elected leader, sending a full tablet report...
I20240430 20:11:56.266883 21396 ts_tablet_manager.cc:1888] T aa9998f4e17147aa8f2b35c01980463a P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:56.267015 21396 log.cc:1196] T aa9998f4e17147aa8f2b35c01980463a P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/aa9998f4e17147aa8f2b35c01980463a
I20240430 20:11:56.267354 21396 ts_tablet_manager.cc:1909] T aa9998f4e17147aa8f2b35c01980463a P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:56.268071 21396 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:45934 (request call id 90) took 2431 ms. Trace:
I20240430 20:11:56.268172 21396 rpcz_store.cc:276] 0430 20:11:53.836474 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:53.836513 (+ 39us) service_pool.cc:225] Handling call
0430 20:11:56.179800 (+2343287us) ts_tablet_manager.cc:1120] Deleting tablet aa9998f4e17147aa8f2b35c01980463a
0430 20:11:56.179965 (+ 165us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:56.223152 (+ 43187us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:56.266768 (+ 43616us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:56.268050 (+ 1282us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2343148}
I20240430 20:11:56.268535 21396 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:45934 (request call id 91) took 2431 ms. Trace:
I20240430 20:11:56.268600 21396 rpcz_store.cc:276] 0430 20:11:53.836827 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:53.836865 (+ 38us) service_pool.cc:225] Handling call
0430 20:11:56.268435 (+2431570us) ts_tablet_manager.cc:1120] Deleting tablet 1462ecf1176c4e66be42abddbe36b00c
0430 20:11:56.268444 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:56.268523 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2431445}
I20240430 20:11:56.268811 21396 tablet_replica.cc:330] T fcc16db4d6be42edb7bd1ce18fe0190d P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:56.268975 21396 raft_consensus.cc:2229] T fcc16db4d6be42edb7bd1ce18fe0190d P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:56.269093 21396 raft_consensus.cc:2258] T fcc16db4d6be42edb7bd1ce18fe0190d P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:56.270686 18469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5b85a066cdc46d48ccf8201e5423c29" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
W20240430 20:11:56.272039 18558 leader_election.cc:341] T ef40bfdb5bf84f54a0068f40c7d1617a P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
W20240430 20:11:56.272473 18558 leader_election.cc:341] T 9a76db2c7ff146758c54908e428a0e02 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
W20240430 20:11:56.272943 18558 leader_election.cc:341] T 8e7f08e9ac274bfa86763eec895bbfda P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:56.275470 18983 tablet_bootstrap.cc:492] T de546e2d6a5142159f23bd50d78ff735 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:56.275652 18983 ts_tablet_manager.cc:1367] T de546e2d6a5142159f23bd50d78ff735 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.171s user 0.035s sys 0.029s
I20240430 20:11:56.247329 21395 tablet_replica.cc:330] T a445fdfde01c4371a5e6b3daea5b00ab P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:56.275913 21395 raft_consensus.cc:2229] T a445fdfde01c4371a5e6b3daea5b00ab P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Raft consensus shutting down.
I20240430 20:11:56.276389 21395 raft_consensus.cc:2258] T a445fdfde01c4371a5e6b3daea5b00ab P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:56.276453 18983 raft_consensus.cc:348] T de546e2d6a5142159f23bd50d78ff735 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:56.276737 18983 raft_consensus.cc:374] T de546e2d6a5142159f23bd50d78ff735 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:56.276868 18983 raft_consensus.cc:729] T de546e2d6a5142159f23bd50d78ff735 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:56.277128 18983 consensus_queue.cc:260] T de546e2d6a5142159f23bd50d78ff735 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:56.277206 21395 ts_tablet_manager.cc:1875] T a445fdfde01c4371a5e6b3daea5b00ab P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:56.277879 18983 ts_tablet_manager.cc:1398] T de546e2d6a5142159f23bd50d78ff735 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240430 20:11:56.278173 18983 tablet_bootstrap.cc:492] T 5b26319488644d6a8b6be60460a5e7af P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:56.279559 18983 tablet_bootstrap.cc:654] T 5b26319488644d6a8b6be60460a5e7af P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:56.268651 19895 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet aa9998f4e17147aa8f2b35c01980463a (table default.Table_ffb3192f443d437ab63f56f628e596ad [id=608890834da843fa9af0cfc0d548029e]) successfully deleted
W20240430 20:11:56.284044 19895 catalog_manager.cc:4797] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): delete failed for tablet 1462ecf1176c4e66be42abddbe36b00c because the tablet was not found. No further retry: Not found: Tablet not found: 1462ecf1176c4e66be42abddbe36b00c
I20240430 20:11:56.284113 19940 catalog_manager.cc:5484] T 330453ddeea6456e8ef93fa92608b0d3 P 3c51c33d5616482cafb68e26688820f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c51c33d5616482cafb68e26688820f4 (127.17.177.3). New cstate: current_term: 1 leader_uuid: "3c51c33d5616482cafb68e26688820f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } health_report { overall_health: UNKNOWN } } }
I20240430 20:11:56.272898 19902 catalog_manager.cc:4830] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): tablet f7d3dea34cf54fe2b949d1b7ca68577a (table default.Table_a4ea055447464ca6af32fe80bab34c42 [id=ea8038670f7c44a9bb20c396b54d6440]) successfully deleted
I20240430 20:11:56.284612 19940 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 330453ddeea6456e8ef93fa92608b0d3 (table default.Table_c1f36f7df5ae4e7c8eb8b4e99b89748b [id=a50abf28790d4284a76ac6ee22c02606]). Expected version 1 got 0
I20240430 20:11:56.284966 21450 consensus_queue.cc:1035] T 3b7fdc6ec7e046c5af894f9a6eda17b0 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:56.285698 21396 ts_tablet_manager.cc:1875] T fcc16db4d6be42edb7bd1ce18fe0190d P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:56.290109 18669 heartbeater.cc:347] Connected to a master server at 127.17.177.60:37225
I20240430 20:11:56.291616 18983 tablet_bootstrap.cc:492] T 5b26319488644d6a8b6be60460a5e7af P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:56.291817 18983 ts_tablet_manager.cc:1367] T 5b26319488644d6a8b6be60460a5e7af P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.003s
I20240430 20:11:56.292627 18983 raft_consensus.cc:348] T 5b26319488644d6a8b6be60460a5e7af P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:56.293232 18983 raft_consensus.cc:374] T 5b26319488644d6a8b6be60460a5e7af P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:56.293339 18983 raft_consensus.cc:729] T 5b26319488644d6a8b6be60460a5e7af P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:56.293619 18983 consensus_queue.cc:260] T 5b26319488644d6a8b6be60460a5e7af P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:56.294275 18983 ts_tablet_manager.cc:1398] T 5b26319488644d6a8b6be60460a5e7af P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240430 20:11:56.294574 18983 tablet_bootstrap.cc:492] T 6f3996b7e29941e391ea141dd6cc44b2 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:56.294663 18985 tablet_bootstrap.cc:492] T 00be6ddd91e84865a48743fb4d077eec P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:56.294803 18985 ts_tablet_manager.cc:1367] T 00be6ddd91e84865a48743fb4d077eec P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.172s user 0.006s sys 0.009s
I20240430 20:11:56.295513 18985 raft_consensus.cc:348] T 00be6ddd91e84865a48743fb4d077eec P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:56.295691 18985 raft_consensus.cc:374] T 00be6ddd91e84865a48743fb4d077eec P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:56.295735 18985 raft_consensus.cc:729] T 00be6ddd91e84865a48743fb4d077eec P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:56.296033 18983 tablet_bootstrap.cc:654] T 6f3996b7e29941e391ea141dd6cc44b2 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:56.296093 18985 consensus_queue.cc:260] T 00be6ddd91e84865a48743fb4d077eec P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:56.296727 18985 ts_tablet_manager.cc:1398] T 00be6ddd91e84865a48743fb4d077eec P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240430 20:11:56.297144 21468 raft_consensus.cc:482] T 17ddcaf971154a52b606bbf29a539bf0 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:56.297252 21468 raft_consensus.cc:504] T 17ddcaf971154a52b606bbf29a539bf0 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:56.297888 21468 leader_election.cc:288] T 17ddcaf971154a52b606bbf29a539bf0 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:56.298405 18985 tablet_bootstrap.cc:492] T 4391b5227fb64ef481542a962a5e21a5 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:56.299985 18985 tablet_bootstrap.cc:654] T 4391b5227fb64ef481542a962a5e21a5 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:56.285492 19940 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 8e9711d0a382424f9f758bfa883866a0 (table default.Table_f1dc481c6d8c43f589fe2b1892a4cb10 [id=db696ce66005401fa73746e55c071f44]). Expected version 1 got 0
I20240430 20:11:56.309399 18988 tablet_bootstrap.cc:492] T fcc16db4d6be42edb7bd1ce18fe0190d P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:56.309693 18988 ts_tablet_manager.cc:1367] T fcc16db4d6be42edb7bd1ce18fe0190d P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.100s user 0.003s sys 0.001s
W20240430 20:11:56.315610 18557 consensus_peers.cc:485] T f7d3dea34cf54fe2b949d1b7ca68577a P 5e1c6397f09747f091534faf93542710 -> Peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Couldn't send request to peer 3c51c33d5616482cafb68e26688820f4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f7d3dea34cf54fe2b949d1b7ca68577a. This is attempt 1: this message will repeat every 5th retry.
W20240430 20:11:56.316253 18557 leader_election.cc:341] T f9bf604e6c234349a9f72f45a4c4aef6 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:56.316366 18557 leader_election.cc:302] T f9bf604e6c234349a9f72f45a4c4aef6 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:56.316812 21468 raft_consensus.cc:2726] T f9bf604e6c234349a9f72f45a4c4aef6 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240430 20:11:56.317195 18557 leader_election.cc:341] T ef40bfdb5bf84f54a0068f40c7d1617a P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:56.317281 18557 leader_election.cc:302] T ef40bfdb5bf84f54a0068f40c7d1617a P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:56.317658 21468 raft_consensus.cc:2726] T ef40bfdb5bf84f54a0068f40c7d1617a P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:56.318140 21468 consensus_queue.cc:1035] T 086b3dfa84c045a18c8c4f177dce606a P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240430 20:11:56.318925 18557 leader_election.cc:341] T 9a76db2c7ff146758c54908e428a0e02 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:56.319021 18557 leader_election.cc:302] T 9a76db2c7ff146758c54908e428a0e02 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:56.319398 21468 raft_consensus.cc:2726] T 9a76db2c7ff146758c54908e428a0e02 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240430 20:11:56.320732 18557 leader_election.cc:341] T 8e7f08e9ac274bfa86763eec895bbfda P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:56.320827 18557 leader_election.cc:302] T 8e7f08e9ac274bfa86763eec895bbfda P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:56.321197 21468 raft_consensus.cc:2726] T 8e7f08e9ac274bfa86763eec895bbfda P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:56.330157 18988 raft_consensus.cc:348] T fcc16db4d6be42edb7bd1ce18fe0190d P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:56.330495 18988 raft_consensus.cc:374] T fcc16db4d6be42edb7bd1ce18fe0190d P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:56.330651 18988 raft_consensus.cc:729] T fcc16db4d6be42edb7bd1ce18fe0190d P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:56.334465 21396 ts_tablet_manager.cc:1888] T fcc16db4d6be42edb7bd1ce18fe0190d P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:56.334971 21396 log.cc:1196] T fcc16db4d6be42edb7bd1ce18fe0190d P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/fcc16db4d6be42edb7bd1ce18fe0190d
I20240430 20:11:56.339110 18988 consensus_queue.cc:260] T fcc16db4d6be42edb7bd1ce18fe0190d P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:56.339314 21396 ts_tablet_manager.cc:1909] T fcc16db4d6be42edb7bd1ce18fe0190d P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:56.340562 21396 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:45934 (request call id 92) took 2503 ms. Trace:
I20240430 20:11:56.340984 21396 rpcz_store.cc:276] 0430 20:11:53.837193 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:53.837888 (+ 695us) service_pool.cc:225] Handling call
0430 20:11:56.268772 (+2430884us) ts_tablet_manager.cc:1120] Deleting tablet fcc16db4d6be42edb7bd1ce18fe0190d
0430 20:11:56.268779 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:56.311895 (+ 43116us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:56.334065 (+ 22170us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:56.340439 (+ 6374us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2430757}
I20240430 20:11:56.342044 19895 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet fcc16db4d6be42edb7bd1ce18fe0190d (table default.Table_74b89f328a354c9a937566cdaf2a3377 [id=68c6513144ed425aa70cce409b50e176]) successfully deleted
I20240430 20:11:56.343025 21396 tablet_replica.cc:330] T 4dac8b3838954d839094021f1c0503da P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:56.343729 21396 raft_consensus.cc:2229] T 4dac8b3838954d839094021f1c0503da P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:56.344352 21396 raft_consensus.cc:2258] T 4dac8b3838954d839094021f1c0503da P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:56.345300 18983 tablet_bootstrap.cc:492] T 6f3996b7e29941e391ea141dd6cc44b2 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:56.345485 18983 ts_tablet_manager.cc:1367] T 6f3996b7e29941e391ea141dd6cc44b2 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.051s user 0.002s sys 0.027s
I20240430 20:11:56.346215 18983 raft_consensus.cc:348] T 6f3996b7e29941e391ea141dd6cc44b2 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:56.346455 18983 raft_consensus.cc:374] T 6f3996b7e29941e391ea141dd6cc44b2 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:56.346557 18983 raft_consensus.cc:729] T 6f3996b7e29941e391ea141dd6cc44b2 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:56.346853 18983 consensus_queue.cc:260] T 6f3996b7e29941e391ea141dd6cc44b2 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:56.347549 18983 ts_tablet_manager.cc:1398] T 6f3996b7e29941e391ea141dd6cc44b2 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240430 20:11:56.347875 18983 tablet_bootstrap.cc:492] T 266fd23786bc435db5083370f3330ed0 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:56.349296 18983 tablet_bootstrap.cc:654] T 266fd23786bc435db5083370f3330ed0 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:56.353627 21396 ts_tablet_manager.cc:1875] T 4dac8b3838954d839094021f1c0503da P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:56.364218 21395 ts_tablet_manager.cc:1888] T a445fdfde01c4371a5e6b3daea5b00ab P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:56.364501 21395 log.cc:1196] T a445fdfde01c4371a5e6b3daea5b00ab P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/a445fdfde01c4371a5e6b3daea5b00ab
I20240430 20:11:56.371774 18983 tablet_bootstrap.cc:492] T 266fd23786bc435db5083370f3330ed0 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:56.372020 18983 ts_tablet_manager.cc:1367] T 266fd23786bc435db5083370f3330ed0 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.024s user 0.007s sys 0.000s
I20240430 20:11:56.372746 18983 raft_consensus.cc:348] T 266fd23786bc435db5083370f3330ed0 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:56.373049 18983 raft_consensus.cc:374] T 266fd23786bc435db5083370f3330ed0 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:56.373142 18983 raft_consensus.cc:729] T 266fd23786bc435db5083370f3330ed0 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:56.373390 18983 consensus_queue.cc:260] T 266fd23786bc435db5083370f3330ed0 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:56.374078 18983 ts_tablet_manager.cc:1398] T 266fd23786bc435db5083370f3330ed0 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240430 20:11:56.374388 18983 tablet_bootstrap.cc:492] T 798a13f545fd4d8f8c654eaa2452eda1 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:56.375850 18983 tablet_bootstrap.cc:654] T 798a13f545fd4d8f8c654eaa2452eda1 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:56.382131 18983 tablet_bootstrap.cc:492] T 798a13f545fd4d8f8c654eaa2452eda1 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:56.382359 18983 ts_tablet_manager.cc:1367] T 798a13f545fd4d8f8c654eaa2452eda1 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20240430 20:11:56.387480 18983 raft_consensus.cc:348] T 798a13f545fd4d8f8c654eaa2452eda1 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
W20240430 20:11:56.387702 18699 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.177.1:41217, user_credentials={real_user=slave}} blocked reactor thread for 65128.8us
I20240430 20:11:56.387799 18983 raft_consensus.cc:374] T 798a13f545fd4d8f8c654eaa2452eda1 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:56.388079 18983 raft_consensus.cc:729] T 798a13f545fd4d8f8c654eaa2452eda1 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:56.389143 18983 consensus_queue.cc:260] T 798a13f545fd4d8f8c654eaa2452eda1 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:56.389899 18983 ts_tablet_manager.cc:1398] T 798a13f545fd4d8f8c654eaa2452eda1 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240430 20:11:56.390221 18983 tablet_bootstrap.cc:492] T 772eba5f24c549c791cad75565858285 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:56.392004 18983 tablet_bootstrap.cc:654] T 772eba5f24c549c791cad75565858285 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:56.393733 21395 ts_tablet_manager.cc:1909] T a445fdfde01c4371a5e6b3daea5b00ab P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:56.394562 21395 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41450 (request call id 64) took 2589 ms. Trace:
I20240430 20:11:56.377422 18469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17ddcaf971154a52b606bbf29a539bf0" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:56.382755 18985 tablet_bootstrap.cc:492] T 4391b5227fb64ef481542a962a5e21a5 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:56.395175 18985 ts_tablet_manager.cc:1367] T 4391b5227fb64ef481542a962a5e21a5 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.098s user 0.001s sys 0.007s
I20240430 20:11:56.395838 18985 raft_consensus.cc:348] T 4391b5227fb64ef481542a962a5e21a5 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
W20240430 20:11:56.374771 18417 consensus_peers.cc:485] T df92eb85261d47d88cbf16d29d9ad30e P 2efb542aa55e439b99ccb98d1c5c884a -> Peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Couldn't send request to peer 3c51c33d5616482cafb68e26688820f4. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
W20240430 20:11:56.397439 18699 leader_election.cc:341] T 4391b5227fb64ef481542a962a5e21a5 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
W20240430 20:11:56.397908 18699 leader_election.cc:341] T 4391b5227fb64ef481542a962a5e21a5 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:56.395666 21396 ts_tablet_manager.cc:1888] T 4dac8b3838954d839094021f1c0503da P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20240430 20:11:56.396796 18558 leader_election.cc:341] T 17ddcaf971154a52b606bbf29a539bf0 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:56.398214 21396 log.cc:1196] T 4dac8b3838954d839094021f1c0503da P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/4dac8b3838954d839094021f1c0503da
I20240430 20:11:56.399116 21396 ts_tablet_manager.cc:1909] T 4dac8b3838954d839094021f1c0503da P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:56.399391 21688 raft_consensus.cc:482] T 2aa2a489116940dea3173cad16d94fa5 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5e1c6397f09747f091534faf93542710)
I20240430 20:11:56.399513 21688 raft_consensus.cc:504] T 2aa2a489116940dea3173cad16d94fa5 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:56.399758 18988 ts_tablet_manager.cc:1398] T fcc16db4d6be42edb7bd1ce18fe0190d P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.090s user 0.002s sys 0.000s
I20240430 20:11:56.400089 21688 leader_election.cc:288] T 2aa2a489116940dea3173cad16d94fa5 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:56.398846 21395 rpcz_store.cc:276] 0430 20:11:53.804609 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:53.804650 (+ 41us) service_pool.cc:225] Handling call
0430 20:11:56.247256 (+2442606us) ts_tablet_manager.cc:1120] Deleting tablet a445fdfde01c4371a5e6b3daea5b00ab
0430 20:11:56.247265 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:56.281117 (+ 33852us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:56.364007 (+ 82890us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:56.394539 (+ 30532us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2442452}
I20240430 20:11:56.400482 21395 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41450 (request call id 65) took 2595 ms. Trace:
I20240430 20:11:56.400557 21395 rpcz_store.cc:276] 0430 20:11:53.804988 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:53.805043 (+ 55us) service_pool.cc:225] Handling call
0430 20:11:56.400363 (+2595320us) ts_tablet_manager.cc:1120] Deleting tablet 0eb4f3e7ece6496293e01b63620d644b
0430 20:11:56.400373 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:56.400466 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2595125}
I20240430 20:11:56.400810 21395 tablet_replica.cc:330] T 9c73d0a6bc414d3b9900570848d19a51 P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:56.400990 21395 raft_consensus.cc:2229] T 9c73d0a6bc414d3b9900570848d19a51 P 2efb542aa55e439b99ccb98d1c5c884a [term 2 LEADER]: Raft consensus shutting down.
I20240430 20:11:56.401229 21395 pending_rounds.cc:62] T 9c73d0a6bc414d3b9900570848d19a51 P 2efb542aa55e439b99ccb98d1c5c884a: Trying to abort 1 pending ops.
I20240430 20:11:56.401274 21395 pending_rounds.cc:68] T 9c73d0a6bc414d3b9900570848d19a51 P 2efb542aa55e439b99ccb98d1c5c884a: Aborting op as it isn't in flight: id { term: 2 index: 1 } timestamp: 7022624420598267904 op_type: NO_OP noop_request { }
I20240430 20:11:56.401381 21395 raft_consensus.cc:2866] T 9c73d0a6bc414d3b9900570848d19a51 P 2efb542aa55e439b99ccb98d1c5c884a [term 2 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240430 20:11:56.401450 21395 raft_consensus.cc:2258] T 9c73d0a6bc414d3b9900570848d19a51 P 2efb542aa55e439b99ccb98d1c5c884a [term 2 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:56.402483 19902 catalog_manager.cc:4830] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): tablet a445fdfde01c4371a5e6b3daea5b00ab (table default.Table_83d417131d35409582550b7b8ce4a141 [id=8893c7824ec64fc1961e7c6851278a50]) successfully deleted
I20240430 20:11:56.402717 18985 raft_consensus.cc:374] T 4391b5227fb64ef481542a962a5e21a5 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:56.402792 18985 raft_consensus.cc:729] T 4391b5227fb64ef481542a962a5e21a5 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:56.402985 18985 consensus_queue.cc:260] T 4391b5227fb64ef481542a962a5e21a5 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
W20240430 20:11:56.403116 19902 catalog_manager.cc:4797] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): delete failed for tablet 0eb4f3e7ece6496293e01b63620d644b because the tablet was not found. No further retry: Not found: Tablet not found: 0eb4f3e7ece6496293e01b63620d644b
I20240430 20:11:56.403502 18985 ts_tablet_manager.cc:1398] T 4391b5227fb64ef481542a962a5e21a5 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20240430 20:11:56.403810 18985 tablet_bootstrap.cc:492] T b7d8da6e2fe8499d9d53af84a25f2faf P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:56.405198 18985 tablet_bootstrap.cc:654] T b7d8da6e2fe8499d9d53af84a25f2faf P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:56.404965 21396 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:45934 (request call id 83) took 2570 ms. Trace:
I20240430 20:11:56.405642 21396 rpcz_store.cc:276] 0430 20:11:53.834956 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:53.837854 (+ 2898us) service_pool.cc:225] Handling call
0430 20:11:56.342885 (+2505031us) ts_tablet_manager.cc:1120] Deleting tablet 4dac8b3838954d839094021f1c0503da
0430 20:11:56.342896 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:56.368091 (+ 25195us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:56.395067 (+ 26976us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:56.404941 (+ 9874us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":13,"tablet-delete.queue_time_us":2504774}
W20240430 20:11:56.407280 18415 leader_election.cc:341] T 2aa2a489116940dea3173cad16d94fa5 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Not found: Tablet not found: 2aa2a489116940dea3173cad16d94fa5
I20240430 20:11:56.407895 19895 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet 4dac8b3838954d839094021f1c0503da (table default.Table_7b1809d7d84b4e7896d1f10321e3f20c [id=45839a14bd914828b0fea7675f093233]) successfully deleted
I20240430 20:11:56.408843 21468 raft_consensus.cc:482] T 9cc9251abd4e40d59a2bd2d0fada6165 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:56.408973 21468 raft_consensus.cc:504] T 9cc9251abd4e40d59a2bd2d0fada6165 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:56.409005 21396 tablet_replica.cc:330] T d4d6c37b578c4fe1aa666b0a6b0bc7c9 P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:56.409586 21468 leader_election.cc:288] T 9cc9251abd4e40d59a2bd2d0fada6165 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:56.410111 18528 heartbeater.cc:503] Master 127.17.177.62:36441 was elected leader, sending a full tablet report...
I20240430 20:11:56.412911 21396 raft_consensus.cc:2229] T d4d6c37b578c4fe1aa666b0a6b0bc7c9 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:56.413101 18469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cc9251abd4e40d59a2bd2d0fada6165" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
W20240430 20:11:56.413678 18558 leader_election.cc:341] T 9cc9251abd4e40d59a2bd2d0fada6165 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:56.414889 21397 ts_tablet_manager.cc:1888] T df92eb85261d47d88cbf16d29d9ad30e P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.1
I20240430 20:11:56.415056 21397 log.cc:1196] T df92eb85261d47d88cbf16d29d9ad30e P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/df92eb85261d47d88cbf16d29d9ad30e
I20240430 20:11:56.415006 21396 raft_consensus.cc:2258] T d4d6c37b578c4fe1aa666b0a6b0bc7c9 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:56.415494 21397 ts_tablet_manager.cc:1909] T df92eb85261d47d88cbf16d29d9ad30e P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:56.416028 21395 ts_tablet_manager.cc:1875] T 9c73d0a6bc414d3b9900570848d19a51 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:56.416579 21397 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56540 (request call id 68) took 2679 ms. Trace:
I20240430 20:11:56.416712 21397 rpcz_store.cc:276] 0430 20:11:53.736833 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:53.737072 (+ 239us) service_pool.cc:225] Handling call
0430 20:11:56.202025 (+2464953us) ts_tablet_manager.cc:1120] Deleting tablet df92eb85261d47d88cbf16d29d9ad30e
0430 20:11:56.202036 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:56.274550 (+ 72514us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:56.414692 (+140142us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:56.416557 (+ 1865us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2464807}
I20240430 20:11:56.416802 21396 ts_tablet_manager.cc:1875] T d4d6c37b578c4fe1aa666b0a6b0bc7c9 P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:56.417326 19900 catalog_manager.cc:4830] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): tablet df92eb85261d47d88cbf16d29d9ad30e (table default.Table_1755296c0a63478699d9b8387f606a8e [id=8322f4f67ad546849d72aca9a5e985cf]) successfully deleted
I20240430 20:11:56.418545 21397 tablet_replica.cc:330] T a445fdfde01c4371a5e6b3daea5b00ab P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:56.418902 21397 raft_consensus.cc:2229] T a445fdfde01c4371a5e6b3daea5b00ab P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:56.419173 21397 raft_consensus.cc:2258] T a445fdfde01c4371a5e6b3daea5b00ab P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:56.419998 21397 ts_tablet_manager.cc:1875] T a445fdfde01c4371a5e6b3daea5b00ab P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240430 20:11:56.420768 18415 consensus_peers.cc:485] T 13c2e69ae91f4957887fd64b90f6e435 P 2efb542aa55e439b99ccb98d1c5c884a -> Peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Couldn't send request to peer 5e1c6397f09747f091534faf93542710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 13c2e69ae91f4957887fd64b90f6e435. This is attempt 1: this message will repeat every 5th retry.
I20240430 20:11:56.421059 19937 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 1faa2f2c84184b129ba6e4d2857822f0 (table default.Table_0dd55b391be94213b4af60ea6e5a979f [id=1a382ad8009b4522a50d55c307db8f96]). Expected version 1 got 0
I20240430 20:11:56.435053 18529 heartbeater.cc:347] Connected to a master server at 127.17.177.60:37225
W20240430 20:11:56.438956 18415 consensus_peers.cc:485] T dad16ae02b8b4e29a724d8eec67bb4b8 P 2efb542aa55e439b99ccb98d1c5c884a -> Peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Couldn't send request to peer 5e1c6397f09747f091534faf93542710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dad16ae02b8b4e29a724d8eec67bb4b8. This is attempt 1: this message will repeat every 5th retry.
I20240430 20:11:56.436136 19937 catalog_manager.cc:5484] T a0d3c4ba64024bccad95ecc909d03ae0 P 5e1c6397f09747f091534faf93542710 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e1c6397f09747f091534faf93542710 (127.17.177.2). New cstate: current_term: 1 leader_uuid: "5e1c6397f09747f091534faf93542710" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } health_report { overall_health: UNKNOWN } } }
I20240430 20:11:56.440068 18983 tablet_bootstrap.cc:492] T 772eba5f24c549c791cad75565858285 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:56.440441 18983 ts_tablet_manager.cc:1367] T 772eba5f24c549c791cad75565858285 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.050s user 0.011s sys 0.000s
I20240430 20:11:56.441154 19937 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 266fd23786bc435db5083370f3330ed0 (table default.Table_cf60b652e9dc49e7a948ce66fd1cbffe [id=9734bb8dd05a400ca4244b3829f01dee]). Expected version 1 got 0
I20240430 20:11:56.441262 18983 raft_consensus.cc:348] T 772eba5f24c549c791cad75565858285 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:56.441695 18983 raft_consensus.cc:374] T 772eba5f24c549c791cad75565858285 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:56.441835 18983 raft_consensus.cc:729] T 772eba5f24c549c791cad75565858285 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:56.442184 18983 consensus_queue.cc:260] T 772eba5f24c549c791cad75565858285 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:56.443635 18983 ts_tablet_manager.cc:1398] T 772eba5f24c549c791cad75565858285 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240430 20:11:56.443965 18983 tablet_bootstrap.cc:492] T e28529e1c2ae4c7c9b9b0e4bd709d132 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:56.446197 18983 tablet_bootstrap.cc:654] T e28529e1c2ae4c7c9b9b0e4bd709d132 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:56.446666 21468 raft_consensus.cc:482] T e041756c962b4a9f97c2cb314e7d8c05 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:56.446786 21468 raft_consensus.cc:504] T e041756c962b4a9f97c2cb314e7d8c05 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:56.447438 21468 leader_election.cc:288] T e041756c962b4a9f97c2cb314e7d8c05 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:56.448189 21395 ts_tablet_manager.cc:1888] T 9c73d0a6bc414d3b9900570848d19a51 P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.1
I20240430 20:11:56.447794 19937 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet dd1de890e4394db287cf38bb411f5586 (table default.Table_84084a27f7504dd992fce07448c10df3 [id=6257559b0e9545c58848a76af0c5a5f7]). Expected version 2 got 0
I20240430 20:11:56.448717 21395 log.cc:1196] T 9c73d0a6bc414d3b9900570848d19a51 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/9c73d0a6bc414d3b9900570848d19a51
I20240430 20:11:56.449328 19937 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet dac141d1b70c48d785caddd7182811ee (table default.Table_fc0a4540f98b4eea95099b8ab46ea235 [id=f3c53ccffe64410495cc8c379a05cfae]). Expected version 1 got 0
I20240430 20:11:56.421056 18988 tablet_bootstrap.cc:492] T 17ddcaf971154a52b606bbf29a539bf0 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
W20240430 20:11:56.422823 18696 consensus_peers.cc:485] T fd1b8407071e4e4287fc172009777414 P 3c51c33d5616482cafb68e26688820f4 -> Peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Couldn't send request to peer 5e1c6397f09747f091534faf93542710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fd1b8407071e4e4287fc172009777414. This is attempt 1: this message will repeat every 5th retry.
I20240430 20:11:56.448957 18469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e041756c962b4a9f97c2cb314e7d8c05" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
W20240430 20:11:56.455565 18558 leader_election.cc:341] T e041756c962b4a9f97c2cb314e7d8c05 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:56.455026 18985 tablet_bootstrap.cc:492] T b7d8da6e2fe8499d9d53af84a25f2faf P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:56.455987 18985 ts_tablet_manager.cc:1367] T b7d8da6e2fe8499d9d53af84a25f2faf P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.052s user 0.005s sys 0.000s
I20240430 20:11:56.456133 18988 tablet_bootstrap.cc:654] T 17ddcaf971154a52b606bbf29a539bf0 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:56.456800 18985 raft_consensus.cc:348] T b7d8da6e2fe8499d9d53af84a25f2faf P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:56.457016 18985 raft_consensus.cc:374] T b7d8da6e2fe8499d9d53af84a25f2faf P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:56.457070 18985 raft_consensus.cc:729] T b7d8da6e2fe8499d9d53af84a25f2faf P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:56.457306 18985 consensus_queue.cc:260] T b7d8da6e2fe8499d9d53af84a25f2faf P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:56.460371 18985 ts_tablet_manager.cc:1398] T b7d8da6e2fe8499d9d53af84a25f2faf P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240430 20:11:56.460659 18985 tablet_bootstrap.cc:492] T dbb427aefa6b47b0805d1751fea4efcb P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:56.461532 21395 ts_tablet_manager.cc:1909] T 9c73d0a6bc414d3b9900570848d19a51 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:56.463344 19937 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 330453ddeea6456e8ef93fa92608b0d3 (table default.Table_c1f36f7df5ae4e7c8eb8b4e99b89748b [id=a50abf28790d4284a76ac6ee22c02606]). Expected version 1 got 0
I20240430 20:11:56.463598 19937 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 1050ab528c7d41559831652a1437d358 (table default.Table_acddc4cbf6334546b5fb3e4752fef2a5 [id=274cb15fd751410e83327ac4e8dc4629]). Expected version 2 got 0
I20240430 20:11:56.463802 19937 catalog_manager.cc:5484] T 086b3dfa84c045a18c8c4f177dce606a P 5e1c6397f09747f091534faf93542710 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e1c6397f09747f091534faf93542710 (127.17.177.2). New cstate: current_term: 1 leader_uuid: "5e1c6397f09747f091534faf93542710" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } health_report { overall_health: UNKNOWN } } }
I20240430 20:11:56.464246 19937 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 086b3dfa84c045a18c8c4f177dce606a (table default.Table_df19cb8a6f4d46699350ec5b69c14e5a [id=197923633fc54a4582207942b51530e2]). Expected version 2 got 0
I20240430 20:11:56.464836 19937 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet e041756c962b4a9f97c2cb314e7d8c05 (table default.Table_de88cafd92b149f9b545369bff71033c [id=c4bf4806f3d84579b16a3038d8d982c3]). Expected version 1 got 0
I20240430 20:11:56.465380 19937 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 5df364628561475391c9a5e10ff5a05f (table default.Table_3c8f9f7b22f64fb1bdc3f1550987233a [id=7fd6f8576f4c4cbc8ffcdb4ab95e15dc]). Expected version 1 got 0
I20240430 20:11:56.465555 19937 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 821c55edb3414d72b1489eea97d3f73a (table default.Table_4d68c38d9ba7452a829f8318b2b267b8 [id=d06e3e9045b1429aaeebd1fd67d841de]). Expected version 1 got 0
I20240430 20:11:56.465880 19937 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet c4953578de5f45f3a6ba092a13357460 (table default.Table_e3cea4c635274d37a44427ca00897fb9 [id=a826c08c7eef44ec83feae3a6cb384e5]). Expected version 1 got 0
I20240430 20:11:56.466272 21396 ts_tablet_manager.cc:1888] T d4d6c37b578c4fe1aa666b0a6b0bc7c9 P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:56.467211 21396 log.cc:1196] T d4d6c37b578c4fe1aa666b0a6b0bc7c9 P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/d4d6c37b578c4fe1aa666b0a6b0bc7c9
I20240430 20:11:56.429369 18810 heartbeater.cc:503] Master 127.17.177.60:37225 was elected leader, sending a full tablet report...
I20240430 20:11:56.425345 18699 leader_election.cc:302] T e5b85a066cdc46d48ccf8201e5423c29 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710
W20240430 20:11:56.468899 18699 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.177.1:41217, user_credentials={real_user=slave}} blocked reactor thread for 47120.5us
I20240430 20:11:56.469327 21450 raft_consensus.cc:2726] T e5b85a066cdc46d48ccf8201e5423c29 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:56.470412 21395 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41450 (request call id 67) took 2555 ms. Trace:
I20240430 20:11:56.470517 21395 rpcz_store.cc:276] 0430 20:11:53.914698 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:53.914854 (+ 156us) service_pool.cc:225] Handling call
0430 20:11:56.400763 (+2485909us) ts_tablet_manager.cc:1120] Deleting tablet 9c73d0a6bc414d3b9900570848d19a51
0430 20:11:56.400773 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:56.427173 (+ 26400us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:56.448076 (+ 20903us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:56.470388 (+ 22312us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2485552}
I20240430 20:11:56.471032 19902 catalog_manager.cc:4830] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): tablet 9c73d0a6bc414d3b9900570848d19a51 (table default.Table_49a012a7c4c8457fadb0a8f0fc9654c6 [id=eecb2f8eeef547c5ab1abe6a2430e592]) successfully deleted
I20240430 20:11:56.471088 21396 ts_tablet_manager.cc:1909] T d4d6c37b578c4fe1aa666b0a6b0bc7c9 P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:56.471614 18983 tablet_bootstrap.cc:492] T e28529e1c2ae4c7c9b9b0e4bd709d132 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:56.471760 18983 ts_tablet_manager.cc:1367] T e28529e1c2ae4c7c9b9b0e4bd709d132 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.028s user 0.001s sys 0.005s
I20240430 20:11:56.472538 21395 tablet_replica.cc:330] T 06d28b6f48a34b17b0d7fb3102df2f4c P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:56.472743 21395 raft_consensus.cc:2229] T 06d28b6f48a34b17b0d7fb3102df2f4c P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:56.472920 21395 raft_consensus.cc:2258] T 06d28b6f48a34b17b0d7fb3102df2f4c P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:56.473130 18749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17ddcaf971154a52b606bbf29a539bf0" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:56.473515 21395 ts_tablet_manager.cc:1875] T 06d28b6f48a34b17b0d7fb3102df2f4c P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:56.462523 18985 tablet_bootstrap.cc:654] T dbb427aefa6b47b0805d1751fea4efcb P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:56.474995 21454 raft_consensus.cc:482] T 2aa2a489116940dea3173cad16d94fa5 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5e1c6397f09747f091534faf93542710)
I20240430 20:11:56.475131 21454 raft_consensus.cc:504] T 2aa2a489116940dea3173cad16d94fa5 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:56.475697 21454 leader_election.cc:288] T 2aa2a489116940dea3173cad16d94fa5 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:56.476301 18983 raft_consensus.cc:348] T e28529e1c2ae4c7c9b9b0e4bd709d132 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:56.476492 18983 raft_consensus.cc:374] T e28529e1c2ae4c7c9b9b0e4bd709d132 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:56.476552 18983 raft_consensus.cc:729] T e28529e1c2ae4c7c9b9b0e4bd709d132 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:56.476745 18983 consensus_queue.cc:260] T e28529e1c2ae4c7c9b9b0e4bd709d132 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:56.477366 18983 ts_tablet_manager.cc:1398] T e28529e1c2ae4c7c9b9b0e4bd709d132 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240430 20:11:56.477629 18983 tablet_bootstrap.cc:492] T eb7554e115d84e6ca4f05509c5f33e2a P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:56.478135 21396 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:45934 (request call id 93) took 2639 ms. Trace:
I20240430 20:11:56.479007 18983 tablet_bootstrap.cc:654] T eb7554e115d84e6ca4f05509c5f33e2a P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:56.478696 21396 rpcz_store.cc:276] 0430 20:11:53.838277 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:53.838315 (+ 38us) service_pool.cc:225] Handling call
0430 20:11:56.408707 (+2570392us) ts_tablet_manager.cc:1120] Deleting tablet d4d6c37b578c4fe1aa666b0a6b0bc7c9
0430 20:11:56.408717 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:56.440978 (+ 32261us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:56.465481 (+ 24503us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:56.477894 (+ 12413us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3490,"tablet-delete.queue_time_us":2570043}
I20240430 20:11:56.480778 19895 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet d4d6c37b578c4fe1aa666b0a6b0bc7c9 (table default.Table_423319f67e734a64a731bf33ef868beb [id=44f7bf1df366476292532972544fe690]) successfully deleted
W20240430 20:11:56.481108 18696 leader_election.cc:341] T 2aa2a489116940dea3173cad16d94fa5 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Not found: Tablet not found: 2aa2a489116940dea3173cad16d94fa5
I20240430 20:11:56.481436 21396 tablet_replica.cc:330] T 5bd0362ff41b4faca0205d136d596737 P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:56.483109 21468 consensus_queue.cc:1035] T f8bfbf673ea346529d1b2b376a3da496 P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:56.485102 21468 raft_consensus.cc:482] T 94fbd4a57a1e438891fcb456f6b40be3 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:56.485224 21468 raft_consensus.cc:504] T 94fbd4a57a1e438891fcb456f6b40be3 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:56.485781 21468 leader_election.cc:288] T 94fbd4a57a1e438891fcb456f6b40be3 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:56.486356 21396 raft_consensus.cc:2229] T 5bd0362ff41b4faca0205d136d596737 P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Raft consensus shutting down.
I20240430 20:11:56.488257 18467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94fbd4a57a1e438891fcb456f6b40be3" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
W20240430 20:11:56.488749 18558 leader_election.cc:341] T 94fbd4a57a1e438891fcb456f6b40be3 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:56.488921 19937 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 37b2ea4be7494f0985808ade788dbe2b (table default.Table_5f6e0fce999b46de927f9271e5de5071 [id=0baad83a8c0f4aeebc7ddad068625a69]). Expected version 1 got 0
I20240430 20:11:56.489239 21396 raft_consensus.cc:2258] T 5bd0362ff41b4faca0205d136d596737 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:56.493685 21396 ts_tablet_manager.cc:1875] T 5bd0362ff41b4faca0205d136d596737 P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:56.496337 21644 raft_consensus.cc:482] T b7d8da6e2fe8499d9d53af84a25f2faf P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:56.496542 21644 raft_consensus.cc:504] T b7d8da6e2fe8499d9d53af84a25f2faf P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:56.497246 19937 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 9e8ab75e7d9c4b64a7b7dd2678569490 (table default.Table_b2d07520b9944a89ad62fca70396174e [id=1c8e0a3a44ec4a798770c0165ed3dffa]). Expected version 1 got 0
I20240430 20:11:56.497673 21644 leader_election.cc:288] T b7d8da6e2fe8499d9d53af84a25f2faf P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:56.498095 18607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7d8da6e2fe8499d9d53af84a25f2faf" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:56.498319 18607 raft_consensus.cc:2454] T b7d8da6e2fe8499d9d53af84a25f2faf P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:56.498822 18696 leader_election.cc:302] T b7d8da6e2fe8499d9d53af84a25f2faf P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:56.499281 21644 raft_consensus.cc:2781] T b7d8da6e2fe8499d9d53af84a25f2faf P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:56.499403 21644 raft_consensus.cc:482] T b7d8da6e2fe8499d9d53af84a25f2faf P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:56.499657 21644 raft_consensus.cc:3037] T b7d8da6e2fe8499d9d53af84a25f2faf P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:56.499753 19937 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet aaef85eea87f4443943fd3692bb2a38f (table default.Table_3da5b4c8668c470483ae2a0c7773eaaf [id=e292d818ee3c45df90522f08eb2d6227]). Expected version 1 got 0
I20240430 20:11:56.506402 19937 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 69f88532952a4591bf1577541c3430e6 (table default.Table_1177828cad1f4a24bd397619dad96170 [id=a95fed1bc3fa4865bf9a35dfc51b659c]). Expected version 2 got 0
I20240430 20:11:56.510645 21644 raft_consensus.cc:504] T b7d8da6e2fe8499d9d53af84a25f2faf P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:56.512341 18756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cc9251abd4e40d59a2bd2d0fada6165" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:56.515370 21644 leader_election.cc:288] T b7d8da6e2fe8499d9d53af84a25f2faf P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Requested vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:56.516019 18607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7d8da6e2fe8499d9d53af84a25f2faf" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:56.516337 18607 raft_consensus.cc:3037] T b7d8da6e2fe8499d9d53af84a25f2faf P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:56.521894 21667 raft_consensus.cc:482] T 6f3996b7e29941e391ea141dd6cc44b2 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:56.522018 21667 raft_consensus.cc:504] T 6f3996b7e29941e391ea141dd6cc44b2 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:56.522670 21667 leader_election.cc:288] T 6f3996b7e29941e391ea141dd6cc44b2 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:56.523329 19937 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 7a0176f8a74c47efbbe3d3343b3dc896 (table default.Table_144e5402c1864d818436015b26648024 [id=73e5c9f0470d47849109819c063e1ce4]). Expected version 2 got 0
I20240430 20:11:56.524091 18467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f3996b7e29941e391ea141dd6cc44b2" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
W20240430 20:11:56.524871 18558 leader_election.cc:341] T 6f3996b7e29941e391ea141dd6cc44b2 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:56.525490 21395 ts_tablet_manager.cc:1888] T 06d28b6f48a34b17b0d7fb3102df2f4c P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:56.525609 21395 log.cc:1196] T 06d28b6f48a34b17b0d7fb3102df2f4c P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/06d28b6f48a34b17b0d7fb3102df2f4c
I20240430 20:11:56.527266 19937 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 94fbd4a57a1e438891fcb456f6b40be3 (table default.Table_25ddc3adef6b4ccf8b9c00cc6772031c [id=4fa623ffdcf347639150a355806038af]). Expected version 1 got 0
I20240430 20:11:56.529788 18607 raft_consensus.cc:2454] T b7d8da6e2fe8499d9d53af84a25f2faf P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
W20240430 20:11:56.519871 18696 consensus_peers.cc:485] T 8668f7a285de43f48d83abecb4b2e96c P 3c51c33d5616482cafb68e26688820f4 -> Peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Couldn't send request to peer 5e1c6397f09747f091534faf93542710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8668f7a285de43f48d83abecb4b2e96c. This is attempt 1: this message will repeat every 5th retry.
I20240430 20:11:56.520539 18751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e041756c962b4a9f97c2cb314e7d8c05" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:56.530367 19937 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet fbdeb4714b8c4b93ab72822c43329aae (table default.Table_c12ca6938d424fbfa2e1df5cdb00fdbc [id=0fb93f36bc17408ea56bc4cafcfd8580]). Expected version 1 got 0
I20240430 20:11:56.521385 18756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94fbd4a57a1e438891fcb456f6b40be3" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:56.531167 21395 ts_tablet_manager.cc:1909] T 06d28b6f48a34b17b0d7fb3102df2f4c P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:56.559705 18756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aa2a489116940dea3173cad16d94fa5" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:56.559999 18756 raft_consensus.cc:2454] T 2aa2a489116940dea3173cad16d94fa5 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 1.
I20240430 20:11:56.563620 18477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aa2a489116940dea3173cad16d94fa5" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:56.563925 18477 raft_consensus.cc:2454] T 2aa2a489116940dea3173cad16d94fa5 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:56.563620 18476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7d8da6e2fe8499d9d53af84a25f2faf" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:56.564240 18476 raft_consensus.cc:2454] T b7d8da6e2fe8499d9d53af84a25f2faf P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:56.564575 18473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7d8da6e2fe8499d9d53af84a25f2faf" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:56.564700 18473 raft_consensus.cc:3037] T b7d8da6e2fe8499d9d53af84a25f2faf P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:56.564831 21395 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41450 (request call id 71) took 2637 ms. Trace:
I20240430 20:11:56.564934 21395 rpcz_store.cc:276] 0430 20:11:53.927172 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:53.927226 (+ 54us) service_pool.cc:225] Handling call
0430 20:11:56.472486 (+2545260us) ts_tablet_manager.cc:1120] Deleting tablet 06d28b6f48a34b17b0d7fb3102df2f4c
0430 20:11:56.472496 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:56.487278 (+ 14782us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:56.525372 (+ 38094us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:56.564793 (+ 39421us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2544996}
I20240430 20:11:56.565217 21395 tablet_replica.cc:330] T f940d17b02b541289121463e05b0bb5f P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:56.565423 21395 raft_consensus.cc:2229] T f940d17b02b541289121463e05b0bb5f P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:56.565608 21395 raft_consensus.cc:2258] T f940d17b02b541289121463e05b0bb5f P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:56.566151 21395 ts_tablet_manager.cc:1875] T f940d17b02b541289121463e05b0bb5f P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:56.567497 18473 raft_consensus.cc:2454] T b7d8da6e2fe8499d9d53af84a25f2faf P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:56.567982 19902 catalog_manager.cc:4830] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): tablet 06d28b6f48a34b17b0d7fb3102df2f4c (table default.Table_4a09e48b08fa4c4387262a5b92b2a42d [id=565591f1f0e84bc89c8d366ef53ae969]) successfully deleted
W20240430 20:11:56.548302 18557 leader_election.cc:341] T 17ddcaf971154a52b606bbf29a539bf0 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:56.555275 21396 ts_tablet_manager.cc:1888] T 5bd0362ff41b4faca0205d136d596737 P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:56.568464 18557 leader_election.cc:302] T 17ddcaf971154a52b606bbf29a539bf0 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:56.568548 21396 log.cc:1196] T 5bd0362ff41b4faca0205d136d596737 P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/5bd0362ff41b4faca0205d136d596737
I20240430 20:11:56.568933 21643 raft_consensus.cc:2726] T 17ddcaf971154a52b606bbf29a539bf0 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:56.569648 18696 leader_election.cc:302] T b7d8da6e2fe8499d9d53af84a25f2faf P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:56.570214 21644 raft_consensus.cc:2781] T b7d8da6e2fe8499d9d53af84a25f2faf P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:56.570273 20489 consensus_queue.cc:1035] T f8bfbf673ea346529d1b2b376a3da496 P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:56.570447 21644 raft_consensus.cc:686] T b7d8da6e2fe8499d9d53af84a25f2faf P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Becoming Leader. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Running, Role: LEADER
I20240430 20:11:56.570726 21644 consensus_queue.cc:237] T b7d8da6e2fe8499d9d53af84a25f2faf P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:56.571069 19937 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 772eba5f24c549c791cad75565858285 (table default.Table_5fc77d5f60ae44f4bb4ca9f7f183672c [id=6ca223ca6354449c959d85ed2ac6a9c8]). Expected version 1 got 0
I20240430 20:11:56.571560 19937 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 2cd41b407d6b444d844c4c7c0ebf82e5 (table default.Table_6f8e0eff8f5a484dbd6a9c3fdb5d03f6 [id=3813b5e9778c4bd683a06598cef0d86f]). Expected version 1 got 0
I20240430 20:11:56.572299 18473 raft_consensus.cc:3037] T 00be6ddd91e84865a48743fb4d077eec P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:56.573156 18988 tablet_bootstrap.cc:492] T 17ddcaf971154a52b606bbf29a539bf0 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:56.573334 18988 ts_tablet_manager.cc:1367] T 17ddcaf971154a52b606bbf29a539bf0 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.173s user 0.006s sys 0.000s
I20240430 20:11:56.574098 18988 raft_consensus.cc:348] T 17ddcaf971154a52b606bbf29a539bf0 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:56.574327 18988 raft_consensus.cc:374] T 17ddcaf971154a52b606bbf29a539bf0 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:56.574406 18988 raft_consensus.cc:729] T 17ddcaf971154a52b606bbf29a539bf0 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:56.574664 18988 consensus_queue.cc:260] T 17ddcaf971154a52b606bbf29a539bf0 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:56.575348 18988 ts_tablet_manager.cc:1398] T 17ddcaf971154a52b606bbf29a539bf0 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240430 20:11:56.575652 18988 tablet_bootstrap.cc:492] T cdeef95b54ca42d38d25b57738d84ef1 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:56.577688 21667 consensus_queue.cc:1035] T 00be6ddd91e84865a48743fb4d077eec P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:56.587374 21396 ts_tablet_manager.cc:1909] T 5bd0362ff41b4faca0205d136d596737 P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
W20240430 20:11:56.596011 18557 leader_election.cc:341] T 9cc9251abd4e40d59a2bd2d0fada6165 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:56.596261 18557 leader_election.cc:302] T 9cc9251abd4e40d59a2bd2d0fada6165 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:56.596545 21667 raft_consensus.cc:2726] T 9cc9251abd4e40d59a2bd2d0fada6165 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:56.597436 18749 raft_consensus.cc:1261] T 9c73d0a6bc414d3b9900570848d19a51 P 3c51c33d5616482cafb68e26688820f4 [term 2 FOLLOWER]: Refusing update from remote peer 2efb542aa55e439b99ccb98d1c5c884a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20240430 20:11:56.598553 18985 tablet_bootstrap.cc:492] T dbb427aefa6b47b0805d1751fea4efcb P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:56.598801 18985 ts_tablet_manager.cc:1367] T dbb427aefa6b47b0805d1751fea4efcb P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.138s user 0.007s sys 0.004s
I20240430 20:11:56.599540 18985 raft_consensus.cc:348] T dbb427aefa6b47b0805d1751fea4efcb P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:56.599788 18985 raft_consensus.cc:374] T dbb427aefa6b47b0805d1751fea4efcb P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:56.599884 18985 raft_consensus.cc:729] T dbb427aefa6b47b0805d1751fea4efcb P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:56.600194 18985 consensus_queue.cc:260] T dbb427aefa6b47b0805d1751fea4efcb P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:56.600750 18985 ts_tablet_manager.cc:1398] T dbb427aefa6b47b0805d1751fea4efcb P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240430 20:11:56.597889 19937 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 488d0e609b224674932f890939664df2 (table default.Table_58f35a2120874472978ca0e5f816e86c [id=ff18001c6d814afc8c1768d0c84a6bac]). Expected version 1 got 0
I20240430 20:11:56.601542 18985 tablet_bootstrap.cc:492] T fcc16db4d6be42edb7bd1ce18fe0190d P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:56.603571 19937 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 17ddcaf971154a52b606bbf29a539bf0 (table default.Table_f2f38f5d69e34eaa9e3dbf24cfd1cdd6 [id=872c2faa4a464edd98a51df2e1b3f817]). Expected version 1 got 0
I20240430 20:11:56.604123 18985 tablet_bootstrap.cc:654] T fcc16db4d6be42edb7bd1ce18fe0190d P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:56.608500 18983 tablet_bootstrap.cc:492] T eb7554e115d84e6ca4f05509c5f33e2a P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:56.608657 18983 ts_tablet_manager.cc:1367] T eb7554e115d84e6ca4f05509c5f33e2a P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.131s user 0.019s sys 0.019s
I20240430 20:11:56.609323 18983 raft_consensus.cc:348] T eb7554e115d84e6ca4f05509c5f33e2a P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:56.609519 18983 raft_consensus.cc:374] T eb7554e115d84e6ca4f05509c5f33e2a P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:56.609671 18983 raft_consensus.cc:729] T eb7554e115d84e6ca4f05509c5f33e2a P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:56.609901 19937 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 2da900b6bbfa4fa99fe8494fd2e68da8 (table default.Table_19a91a7f17ff456ebb145e8dd89c0940 [id=67283dd3995a44c6a7a7bc8722e4f955]). Expected version 1 got 0
I20240430 20:11:56.609915 18983 consensus_queue.cc:260] T eb7554e115d84e6ca4f05509c5f33e2a P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:56.610416 18983 ts_tablet_manager.cc:1398] T eb7554e115d84e6ca4f05509c5f33e2a P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240430 20:11:56.610970 19937 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 5b26319488644d6a8b6be60460a5e7af (table default.Table_69892a8264a4477db48ee6248758e046 [id=5a1c4a7be2fc465fac89a50854eb3322]). Expected version 3 got 0
I20240430 20:11:56.613880 21396 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:45934 (request call id 94) took 2775 ms. Trace:
I20240430 20:11:56.613999 21396 rpcz_store.cc:276] 0430 20:11:53.838373 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:53.861543 (+ 23170us) service_pool.cc:225] Handling call
0430 20:11:56.480946 (+2619403us) ts_tablet_manager.cc:1120] Deleting tablet 5bd0362ff41b4faca0205d136d596737
0430 20:11:56.481176 (+ 230us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:56.513556 (+ 32380us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:56.554079 (+ 40523us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:56.613858 (+ 59779us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2619114}
I20240430 20:11:56.614707 21396 tablet_replica.cc:330] T eb76f60448774fe1967fffe7c6873cfc P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:56.614943 21396 raft_consensus.cc:2229] T eb76f60448774fe1967fffe7c6873cfc P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:56.614506 19895 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet 5bd0362ff41b4faca0205d136d596737 (table default.Table_fe1773cf77654e2a86e95444437b7201 [id=fe2222c2b62f4360a5ade622ae8b23e5]) successfully deleted
I20240430 20:11:56.615316 21396 raft_consensus.cc:2258] T eb76f60448774fe1967fffe7c6873cfc P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:56.615525 19937 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet f1073fc792954c5795a05aa7d0ec24df (table default.Table_91417a5439c8457caa8bb0a12221a5dd [id=aedf11fda8914a3b843a57f8a7b69a53]). Expected version 2 got 0
I20240430 20:11:56.615881 21396 ts_tablet_manager.cc:1875] T eb76f60448774fe1967fffe7c6873cfc P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:56.618135 18983 tablet_bootstrap.cc:492] T 4432d2c97b634a36b44e1d61d8bf2458 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:56.619621 18983 tablet_bootstrap.cc:654] T 4432d2c97b634a36b44e1d61d8bf2458 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:56.629318 18988 tablet_bootstrap.cc:654] T cdeef95b54ca42d38d25b57738d84ef1 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:56.638985 21396 ts_tablet_manager.cc:1888] T eb76f60448774fe1967fffe7c6873cfc P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:56.639139 21396 log.cc:1196] T eb76f60448774fe1967fffe7c6873cfc P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/eb76f60448774fe1967fffe7c6873cfc
I20240430 20:11:56.639530 21396 ts_tablet_manager.cc:1909] T eb76f60448774fe1967fffe7c6873cfc P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:56.640405 21396 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:45934 (request call id 95) took 2778 ms. Trace:
I20240430 20:11:56.640516 21396 rpcz_store.cc:276] 0430 20:11:53.862178 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:53.862226 (+ 48us) service_pool.cc:225] Handling call
0430 20:11:56.614328 (+2752102us) ts_tablet_manager.cc:1120] Deleting tablet eb76f60448774fe1967fffe7c6873cfc
0430 20:11:56.614657 (+ 329us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:56.621477 (+ 6820us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:56.638834 (+ 17357us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:56.640378 (+ 1544us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2751924}
I20240430 20:11:56.641027 19895 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet eb76f60448774fe1967fffe7c6873cfc (table default.Table_d3bf86d0283b4227a93a2a8aa241075b [id=4cd17de5c1504aab878e51dea02917e2]) successfully deleted
I20240430 20:11:56.644994 21396 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:45934 (request call id 106) took 2706 ms. Trace:
I20240430 20:11:56.645097 21396 rpcz_store.cc:276] 0430 20:11:53.938437 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:53.938480 (+ 43us) service_pool.cc:225] Handling call
0430 20:11:56.644864 (+2706384us) ts_tablet_manager.cc:1120] Deleting tablet 1701a15b115741aea13f8327e3d26b34
0430 20:11:56.644879 (+ 15us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:56.644971 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2702165}
I20240430 20:11:56.645462 21396 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:45934 (request call id 107) took 2706 ms. Trace:
W20240430 20:11:56.645584 18699 consensus_peers.cc:485] T 06d28b6f48a34b17b0d7fb3102df2f4c P 3c51c33d5616482cafb68e26688820f4 -> Peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Couldn't send request to peer 2efb542aa55e439b99ccb98d1c5c884a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 06d28b6f48a34b17b0d7fb3102df2f4c. This is attempt 1: this message will repeat every 5th retry.
I20240430 20:11:56.645547 21396 rpcz_store.cc:276] 0430 20:11:53.938542 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:53.938616 (+ 74us) service_pool.cc:225] Handling call
0430 20:11:56.645360 (+2706744us) ts_tablet_manager.cc:1120] Deleting tablet 9c73d0a6bc414d3b9900570848d19a51
0430 20:11:56.645370 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:56.645445 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2706625}
I20240430 20:11:56.645792 21396 tablet_replica.cc:330] T e4cffa295d8c401aa8415ce7e8f56526 P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:56.645833 21454 consensus_queue.cc:1035] T 8668f7a285de43f48d83abecb4b2e96c P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240430 20:11:56.646119 19895 catalog_manager.cc:4797] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): delete failed for tablet 1701a15b115741aea13f8327e3d26b34 because the tablet was not found. No further retry: Not found: Tablet not found: 1701a15b115741aea13f8327e3d26b34
W20240430 20:11:56.646124 18557 leader_election.cc:341] T e041756c962b4a9f97c2cb314e7d8c05 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:56.646235 18557 leader_election.cc:302] T e041756c962b4a9f97c2cb314e7d8c05 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4
W20240430 20:11:56.646400 19895 catalog_manager.cc:4797] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): delete failed for tablet 9c73d0a6bc414d3b9900570848d19a51 because the tablet was not found. No further retry: Not found: Tablet not found: 9c73d0a6bc414d3b9900570848d19a51
W20240430 20:11:56.646643 18557 leader_election.cc:341] T 94fbd4a57a1e438891fcb456f6b40be3 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:56.646736 18557 leader_election.cc:302] T 94fbd4a57a1e438891fcb456f6b40be3 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:56.646935 20489 raft_consensus.cc:2726] T e041756c962b4a9f97c2cb314e7d8c05 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:56.647076 20489 raft_consensus.cc:2726] T 94fbd4a57a1e438891fcb456f6b40be3 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:56.647486 18417 leader_election.cc:302] T 2aa2a489116940dea3173cad16d94fa5 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters: 5e1c6397f09747f091534faf93542710
I20240430 20:11:56.648532 21581 raft_consensus.cc:2781] T 2aa2a489116940dea3173cad16d94fa5 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240430 20:11:56.648609 21581 raft_consensus.cc:482] T 2aa2a489116940dea3173cad16d94fa5 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 5e1c6397f09747f091534faf93542710)
I20240430 20:11:56.648674 21581 raft_consensus.cc:3037] T 2aa2a489116940dea3173cad16d94fa5 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Advancing to term 2
I20240430 20:11:56.650058 21396 raft_consensus.cc:2229] T e4cffa295d8c401aa8415ce7e8f56526 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:56.650698 21396 raft_consensus.cc:2258] T e4cffa295d8c401aa8415ce7e8f56526 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:56.651271 21396 ts_tablet_manager.cc:1875] T e4cffa295d8c401aa8415ce7e8f56526 P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:56.651459 21581 raft_consensus.cc:504] T 2aa2a489116940dea3173cad16d94fa5 P 2efb542aa55e439b99ccb98d1c5c884a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
W20240430 20:11:56.653972 18415 leader_election.cc:341] T 2aa2a489116940dea3173cad16d94fa5 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Not found: Tablet not found: 2aa2a489116940dea3173cad16d94fa5
I20240430 20:11:56.654201 21581 leader_election.cc:288] T 2aa2a489116940dea3173cad16d94fa5 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:56.645846 18699 leader_election.cc:302] T 2aa2a489116940dea3173cad16d94fa5 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters: 5e1c6397f09747f091534faf93542710
I20240430 20:11:56.656227 21436 raft_consensus.cc:2781] T 2aa2a489116940dea3173cad16d94fa5 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240430 20:11:56.656311 21436 raft_consensus.cc:482] T 2aa2a489116940dea3173cad16d94fa5 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 5e1c6397f09747f091534faf93542710)
I20240430 20:11:56.656375 21436 raft_consensus.cc:3037] T 2aa2a489116940dea3173cad16d94fa5 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Advancing to term 2
I20240430 20:11:56.656781 21450 consensus_queue.cc:1035] T 3b7fdc6ec7e046c5af894f9a6eda17b0 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:56.662725 21397 ts_tablet_manager.cc:1888] T a445fdfde01c4371a5e6b3daea5b00ab P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:56.662895 21397 log.cc:1196] T a445fdfde01c4371a5e6b3daea5b00ab P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/a445fdfde01c4371a5e6b3daea5b00ab
I20240430 20:11:56.663297 21397 ts_tablet_manager.cc:1909] T a445fdfde01c4371a5e6b3daea5b00ab P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:56.666339 21701 raft_consensus.cc:482] T a37d917bcc45468f9e3ee805640fba98 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:56.666501 21701 raft_consensus.cc:504] T a37d917bcc45468f9e3ee805640fba98 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:56.667186 21701 leader_election.cc:288] T a37d917bcc45468f9e3ee805640fba98 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:56.668049 18483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a37d917bcc45468f9e3ee805640fba98" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
W20240430 20:11:56.668717 18558 leader_election.cc:341] T a37d917bcc45468f9e3ee805640fba98 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:56.672569 19945 catalog_manager.cc:5484] T 347a8224114b49e5b7dcb75d9b726e0a P 3c51c33d5616482cafb68e26688820f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c51c33d5616482cafb68e26688820f4 (127.17.177.3). New cstate: current_term: 1 leader_uuid: "3c51c33d5616482cafb68e26688820f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } health_report { overall_health: UNKNOWN } } }
I20240430 20:11:56.673345 21397 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56540 (request call id 73) took 2935 ms. Trace:
I20240430 20:11:56.673453 21397 rpcz_store.cc:276] 0430 20:11:53.737491 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:53.737532 (+ 41us) service_pool.cc:225] Handling call
0430 20:11:56.418473 (+2680941us) ts_tablet_manager.cc:1120] Deleting tablet a445fdfde01c4371a5e6b3daea5b00ab
0430 20:11:56.418492 (+ 19us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:56.591140 (+172648us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:56.658727 (+ 67587us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:56.673325 (+ 14598us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2680786}
I20240430 20:11:56.674060 19900 catalog_manager.cc:4830] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): tablet a445fdfde01c4371a5e6b3daea5b00ab (table default.Table_83d417131d35409582550b7b8ce4a141 [id=8893c7824ec64fc1961e7c6851278a50]) successfully deleted
I20240430 20:11:56.674206 19945 catalog_manager.cc:5484] T 1fe7c6f1ab54499facd4d01573d3e7bc P 3c51c33d5616482cafb68e26688820f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1). New cstate: current_term: 1 leader_uuid: "2efb542aa55e439b99ccb98d1c5c884a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } }
I20240430 20:11:56.674938 19945 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 1050ab528c7d41559831652a1437d358 (table default.Table_acddc4cbf6334546b5fb3e4752fef2a5 [id=274cb15fd751410e83327ac4e8dc4629]). Expected version 2 got 0
I20240430 20:11:56.675065 21396 ts_tablet_manager.cc:1888] T e4cffa295d8c401aa8415ce7e8f56526 P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:56.675169 21396 log.cc:1196] T e4cffa295d8c401aa8415ce7e8f56526 P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/e4cffa295d8c401aa8415ce7e8f56526
I20240430 20:11:56.675514 21396 ts_tablet_manager.cc:1909] T e4cffa295d8c401aa8415ce7e8f56526 P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:56.676337 18983 tablet_bootstrap.cc:492] T 4432d2c97b634a36b44e1d61d8bf2458 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:56.676461 18983 ts_tablet_manager.cc:1367] T 4432d2c97b634a36b44e1d61d8bf2458 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.058s user 0.001s sys 0.009s
I20240430 20:11:56.677052 18985 tablet_bootstrap.cc:492] T fcc16db4d6be42edb7bd1ce18fe0190d P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:56.677119 18983 raft_consensus.cc:348] T 4432d2c97b634a36b44e1d61d8bf2458 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:56.677302 18983 raft_consensus.cc:374] T 4432d2c97b634a36b44e1d61d8bf2458 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:56.677242 18985 ts_tablet_manager.cc:1367] T fcc16db4d6be42edb7bd1ce18fe0190d P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.076s user 0.003s sys 0.001s
I20240430 20:11:56.677361 18983 raft_consensus.cc:729] T 4432d2c97b634a36b44e1d61d8bf2458 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:56.677732 18983 consensus_queue.cc:260] T 4432d2c97b634a36b44e1d61d8bf2458 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:56.678429 18983 ts_tablet_manager.cc:1398] T 4432d2c97b634a36b44e1d61d8bf2458 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240430 20:11:56.678975 18983 tablet_bootstrap.cc:492] T ffd6e420e6d44c039832de38de001c1b P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:56.679620 21397 tablet_replica.cc:330] T 8901327b3efb4201adce61a9266142ea P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:56.679833 21397 raft_consensus.cc:2229] T 8901327b3efb4201adce61a9266142ea P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:56.680008 21397 raft_consensus.cc:2258] T 8901327b3efb4201adce61a9266142ea P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:56.680546 21397 ts_tablet_manager.cc:1875] T 8901327b3efb4201adce61a9266142ea P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:56.680716 18983 tablet_bootstrap.cc:654] T ffd6e420e6d44c039832de38de001c1b P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:56.678284 18985 raft_consensus.cc:348] T fcc16db4d6be42edb7bd1ce18fe0190d P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:56.681668 18985 raft_consensus.cc:374] T fcc16db4d6be42edb7bd1ce18fe0190d P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:56.681773 18985 raft_consensus.cc:729] T fcc16db4d6be42edb7bd1ce18fe0190d P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:56.682051 18985 consensus_queue.cc:260] T fcc16db4d6be42edb7bd1ce18fe0190d P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:56.685648 18985 ts_tablet_manager.cc:1398] T fcc16db4d6be42edb7bd1ce18fe0190d P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20240430 20:11:56.686038 18985 tablet_bootstrap.cc:492] T 9e741b00c74d43b5b6243ba7003f8804 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:56.686321 21396 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:45934 (request call id 103) took 2748 ms. Trace:
I20240430 20:11:56.681816 21436 raft_consensus.cc:504] T 2aa2a489116940dea3173cad16d94fa5 P 3c51c33d5616482cafb68e26688820f4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:56.687584 21436 leader_election.cc:288] T 2aa2a489116940dea3173cad16d94fa5 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 2 election: Requested vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:56.687983 18985 tablet_bootstrap.cc:654] T 9e741b00c74d43b5b6243ba7003f8804 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:56.688022 21438 raft_consensus.cc:482] T c1017a50d1d646c486c0b9a75cb88ce4 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5e1c6397f09747f091534faf93542710)
I20240430 20:11:56.688175 21438 raft_consensus.cc:504] T c1017a50d1d646c486c0b9a75cb88ce4 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:56.688853 21438 leader_election.cc:288] T c1017a50d1d646c486c0b9a75cb88ce4 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:56.690302 21395 ts_tablet_manager.cc:1888] T f940d17b02b541289121463e05b0bb5f P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:56.690397 21395 log.cc:1196] T f940d17b02b541289121463e05b0bb5f P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/f940d17b02b541289121463e05b0bb5f
W20240430 20:11:56.690724 18415 leader_election.cc:341] T c1017a50d1d646c486c0b9a75cb88ce4 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Not found: Tablet not found: c1017a50d1d646c486c0b9a75cb88ce4
W20240430 20:11:56.690986 18415 consensus_peers.cc:485] T 4dac8b3838954d839094021f1c0503da P 2efb542aa55e439b99ccb98d1c5c884a -> Peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Couldn't send request to peer 5e1c6397f09747f091534faf93542710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4dac8b3838954d839094021f1c0503da. This is attempt 1: this message will repeat every 5th retry.
W20240430 20:11:56.691689 18696 leader_election.cc:341] T 2aa2a489116940dea3173cad16d94fa5 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Not found: Tablet not found: 2aa2a489116940dea3173cad16d94fa5
I20240430 20:11:56.688184 21396 rpcz_store.cc:276] 0430 20:11:53.938098 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:53.938145 (+ 47us) service_pool.cc:225] Handling call
0430 20:11:56.645746 (+2707601us) ts_tablet_manager.cc:1120] Deleting tablet e4cffa295d8c401aa8415ce7e8f56526
0430 20:11:56.645754 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:56.671655 (+ 25901us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:56.674961 (+ 3306us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:56.686295 (+ 11334us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2706982}
I20240430 20:11:56.694504 19895 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet e4cffa295d8c401aa8415ce7e8f56526 (table default.Table_81547db3074240c98cdd95ed3b2dd813 [id=e0ea901452334b36a8493baed478cc1a]) successfully deleted
I20240430 20:11:56.695844 19945 catalog_manager.cc:5484] T 69f88532952a4591bf1577541c3430e6 P 3c51c33d5616482cafb68e26688820f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c51c33d5616482cafb68e26688820f4 (127.17.177.3). New cstate: current_term: 1 leader_uuid: "3c51c33d5616482cafb68e26688820f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } health_report { overall_health: UNKNOWN } } }
I20240430 20:11:56.696310 18983 tablet_bootstrap.cc:492] T ffd6e420e6d44c039832de38de001c1b P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:56.696290 19945 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 69f88532952a4591bf1577541c3430e6 (table default.Table_1177828cad1f4a24bd397619dad96170 [id=a95fed1bc3fa4865bf9a35dfc51b659c]). Expected version 2 got 0
I20240430 20:11:56.696533 18983 ts_tablet_manager.cc:1367] T ffd6e420e6d44c039832de38de001c1b P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.004s
I20240430 20:11:56.696799 19945 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 086b3dfa84c045a18c8c4f177dce606a (table default.Table_df19cb8a6f4d46699350ec5b69c14e5a [id=197923633fc54a4582207942b51530e2]). Expected version 2 got 0
I20240430 20:11:56.697206 19945 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 330453ddeea6456e8ef93fa92608b0d3 (table default.Table_c1f36f7df5ae4e7c8eb8b4e99b89748b [id=a50abf28790d4284a76ac6ee22c02606]). Expected version 1 got 0
I20240430 20:11:56.697360 21397 ts_tablet_manager.cc:1888] T 8901327b3efb4201adce61a9266142ea P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:56.697495 21397 log.cc:1196] T 8901327b3efb4201adce61a9266142ea P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/8901327b3efb4201adce61a9266142ea
I20240430 20:11:56.697873 21397 ts_tablet_manager.cc:1909] T 8901327b3efb4201adce61a9266142ea P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:56.697721 18983 raft_consensus.cc:348] T ffd6e420e6d44c039832de38de001c1b P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:56.698015 18983 raft_consensus.cc:374] T ffd6e420e6d44c039832de38de001c1b P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:56.698134 18983 raft_consensus.cc:729] T ffd6e420e6d44c039832de38de001c1b P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:56.698716 21395 ts_tablet_manager.cc:1909] T f940d17b02b541289121463e05b0bb5f P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:56.698439 18983 consensus_queue.cc:260] T ffd6e420e6d44c039832de38de001c1b P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:56.700075 18983 ts_tablet_manager.cc:1398] T ffd6e420e6d44c039832de38de001c1b P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240430 20:11:56.714763 21468 raft_consensus.cc:482] T 772eba5f24c549c791cad75565858285 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:56.714967 21468 raft_consensus.cc:504] T 772eba5f24c549c791cad75565858285 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:56.715566 21468 leader_election.cc:288] T 772eba5f24c549c791cad75565858285 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:56.716097 18483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "772eba5f24c549c791cad75565858285" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
W20240430 20:11:56.716647 18558 leader_election.cc:341] T 772eba5f24c549c791cad75565858285 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:56.717897 18985 tablet_bootstrap.cc:492] T 9e741b00c74d43b5b6243ba7003f8804 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:56.718067 18985 ts_tablet_manager.cc:1367] T 9e741b00c74d43b5b6243ba7003f8804 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.032s user 0.003s sys 0.000s
I20240430 20:11:56.719040 18985 raft_consensus.cc:348] T 9e741b00c74d43b5b6243ba7003f8804 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:56.719429 18985 raft_consensus.cc:374] T 9e741b00c74d43b5b6243ba7003f8804 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:56.719553 18985 raft_consensus.cc:729] T 9e741b00c74d43b5b6243ba7003f8804 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:56.719897 18985 consensus_queue.cc:260] T 9e741b00c74d43b5b6243ba7003f8804 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
W20240430 20:11:56.720199 18699 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.177.1:41217, user_credentials={real_user=slave}} blocked reactor thread for 62439.7us
I20240430 20:11:56.720623 18985 ts_tablet_manager.cc:1398] T 9e741b00c74d43b5b6243ba7003f8804 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240430 20:11:56.721554 20489 raft_consensus.cc:482] T de546e2d6a5142159f23bd50d78ff735 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:56.721700 20489 raft_consensus.cc:504] T de546e2d6a5142159f23bd50d78ff735 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:56.722393 20489 leader_election.cc:288] T de546e2d6a5142159f23bd50d78ff735 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:56.722856 18483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de546e2d6a5142159f23bd50d78ff735" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
W20240430 20:11:56.723436 18558 leader_election.cc:341] T de546e2d6a5142159f23bd50d78ff735 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:56.724210 18988 tablet_bootstrap.cc:492] T cdeef95b54ca42d38d25b57738d84ef1 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:56.724603 18988 ts_tablet_manager.cc:1367] T cdeef95b54ca42d38d25b57738d84ef1 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.149s user 0.009s sys 0.000s
I20240430 20:11:56.725734 18475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aa2a489116940dea3173cad16d94fa5" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:56.725960 18475 raft_consensus.cc:2379] T 2aa2a489116940dea3173cad16d94fa5 P 2efb542aa55e439b99ccb98d1c5c884a [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 3c51c33d5616482cafb68e26688820f4 in current term 2: Already voted for candidate 2efb542aa55e439b99ccb98d1c5c884a in this term.
I20240430 20:11:56.726248 18474 raft_consensus.cc:3037] T fd1b8407071e4e4287fc172009777414 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:56.730221 21438 raft_consensus.cc:482] T fd1b8407071e4e4287fc172009777414 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:56.730378 21438 raft_consensus.cc:504] T fd1b8407071e4e4287fc172009777414 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:56.731212 21438 leader_election.cc:288] T fd1b8407071e4e4287fc172009777414 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
W20240430 20:11:56.732985 18415 leader_election.cc:341] T fd1b8407071e4e4287fc172009777414 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Not found: Tablet not found: fd1b8407071e4e4287fc172009777414
I20240430 20:11:56.746448 18988 raft_consensus.cc:348] T cdeef95b54ca42d38d25b57738d84ef1 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:56.746819 18988 raft_consensus.cc:374] T cdeef95b54ca42d38d25b57738d84ef1 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:56.746939 18988 raft_consensus.cc:729] T cdeef95b54ca42d38d25b57738d84ef1 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:56.747223 18988 consensus_queue.cc:260] T cdeef95b54ca42d38d25b57738d84ef1 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:56.747809 18988 ts_tablet_manager.cc:1398] T cdeef95b54ca42d38d25b57738d84ef1 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.023s user 0.002s sys 0.000s
I20240430 20:11:56.751137 18764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de546e2d6a5142159f23bd50d78ff735" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:56.751579 18755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aa2a489116940dea3173cad16d94fa5" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:56.751772 18755 raft_consensus.cc:2379] T 2aa2a489116940dea3173cad16d94fa5 P 3c51c33d5616482cafb68e26688820f4 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 2efb542aa55e439b99ccb98d1c5c884a in current term 2: Already voted for candidate 3c51c33d5616482cafb68e26688820f4 in this term.
I20240430 20:11:56.752039 18748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a37d917bcc45468f9e3ee805640fba98" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:56.752699 18756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f3996b7e29941e391ea141dd6cc44b2" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:56.753345 18761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "772eba5f24c549c791cad75565858285" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:56.755208 21395 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41450 (request call id 72) took 2827 ms. Trace:
I20240430 20:11:56.755347 21395 rpcz_store.cc:276] 0430 20:11:53.927799 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:53.927847 (+ 48us) service_pool.cc:225] Handling call
0430 20:11:56.565174 (+2637327us) ts_tablet_manager.cc:1120] Deleting tablet f940d17b02b541289121463e05b0bb5f
0430 20:11:56.565184 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:56.612639 (+ 47455us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:56.690197 (+ 77558us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:56.755181 (+ 64984us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2637133}
I20240430 20:11:56.755684 21395 tablet_replica.cc:330] T 04b3dff86d5c48a09d03da58bd48f360 P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:56.755883 21395 raft_consensus.cc:2229] T 04b3dff86d5c48a09d03da58bd48f360 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Raft consensus shutting down.
I20240430 20:11:56.756737 18985 tablet_bootstrap.cc:492] T d34a450a91d542c98261456d1f635e9c P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:56.758929 18985 tablet_bootstrap.cc:654] T d34a450a91d542c98261456d1f635e9c P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:56.760066 21395 raft_consensus.cc:2258] T 04b3dff86d5c48a09d03da58bd48f360 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:56.760643 21395 ts_tablet_manager.cc:1875] T 04b3dff86d5c48a09d03da58bd48f360 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240430 20:11:56.762262 18696 consensus_peers.cc:485] T 2797eed593fb4b06a3c4d6a28e8075a2 P 3c51c33d5616482cafb68e26688820f4 -> Peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Couldn't send request to peer 5e1c6397f09747f091534faf93542710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2797eed593fb4b06a3c4d6a28e8075a2. This is attempt 1: this message will repeat every 5th retry.
I20240430 20:11:56.763726 19902 catalog_manager.cc:4830] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): tablet f940d17b02b541289121463e05b0bb5f (table default.Table_07993f6164104b888058e1b35c9f4ed7 [id=f2e6cac20f6a4261bb60f2e40f210491]) successfully deleted
I20240430 20:11:56.765697 21397 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56540 (request call id 74) took 3028 ms. Trace:
I20240430 20:11:56.766103 21397 rpcz_store.cc:276] 0430 20:11:53.737593 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:53.737704 (+ 111us) service_pool.cc:225] Handling call
0430 20:11:56.673730 (+2936026us) ts_tablet_manager.cc:1120] Deleting tablet 8901327b3efb4201adce61a9266142ea
0430 20:11:56.673743 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:56.684705 (+ 10962us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:56.697243 (+ 12538us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:56.765673 (+ 68430us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2935859}
I20240430 20:11:56.766134 18988 tablet_bootstrap.cc:492] T e041756c962b4a9f97c2cb314e7d8c05 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:56.769790 19900 catalog_manager.cc:4830] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): tablet 8901327b3efb4201adce61a9266142ea (table default.Table_2582fc0f97c1470d8b6b21cd1240697e [id=fc059935f9c943389180fab316c33a05]) successfully deleted
I20240430 20:11:56.770411 21397 tablet_replica.cc:330] T 47e98935eb21449abd3d4d24327778d5 P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:56.770974 21396 tablet_replica.cc:330] T 1f4e860484784aa9a2d28af2e9fb611c P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:56.771236 21396 raft_consensus.cc:2229] T 1f4e860484784aa9a2d28af2e9fb611c P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:56.771453 21396 raft_consensus.cc:2258] T 1f4e860484784aa9a2d28af2e9fb611c P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:56.771994 21396 ts_tablet_manager.cc:1875] T 1f4e860484784aa9a2d28af2e9fb611c P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240430 20:11:56.772403 18557 leader_election.cc:341] T de546e2d6a5142159f23bd50d78ff735 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:56.772522 18557 leader_election.cc:302] T de546e2d6a5142159f23bd50d78ff735 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:56.772737 20489 raft_consensus.cc:2726] T de546e2d6a5142159f23bd50d78ff735 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:56.773176 21397 raft_consensus.cc:2229] T 47e98935eb21449abd3d4d24327778d5 P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Raft consensus shutting down.
I20240430 20:11:56.771281 18988 tablet_bootstrap.cc:654] T e041756c962b4a9f97c2cb314e7d8c05 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:56.773562 21397 raft_consensus.cc:2258] T 47e98935eb21449abd3d4d24327778d5 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:56.773612 21395 ts_tablet_manager.cc:1888] T 04b3dff86d5c48a09d03da58bd48f360 P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:56.773710 21395 log.cc:1196] T 04b3dff86d5c48a09d03da58bd48f360 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/04b3dff86d5c48a09d03da58bd48f360
I20240430 20:11:56.774036 21395 ts_tablet_manager.cc:1909] T 04b3dff86d5c48a09d03da58bd48f360 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:56.774775 21395 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41450 (request call id 69) took 2847 ms. Trace:
I20240430 20:11:56.774869 18983 tablet_bootstrap.cc:492] T 72205028d0654e69a6fa20f105604e1c P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:56.774856 21395 rpcz_store.cc:276] 0430 20:11:53.926953 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:53.928259 (+ 1306us) service_pool.cc:225] Handling call
0430 20:11:56.755635 (+2827376us) ts_tablet_manager.cc:1120] Deleting tablet 04b3dff86d5c48a09d03da58bd48f360
0430 20:11:56.755644 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:56.765253 (+ 9609us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:56.773508 (+ 8255us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:56.774751 (+ 1243us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2827105}
W20240430 20:11:56.776578 18417 leader_election.cc:341] T c1017a50d1d646c486c0b9a75cb88ce4 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Not found: Tablet not found: c1017a50d1d646c486c0b9a75cb88ce4
I20240430 20:11:56.776746 18983 tablet_bootstrap.cc:654] T 72205028d0654e69a6fa20f105604e1c P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:56.776721 18417 leader_election.cc:302] T c1017a50d1d646c486c0b9a75cb88ce4 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a; no voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710
I20240430 20:11:56.777133 21688 raft_consensus.cc:2726] T c1017a50d1d646c486c0b9a75cb88ce4 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240430 20:11:56.777045 20489 consensus_queue.cc:1035] T 00be6ddd91e84865a48743fb4d077eec P 5e1c6397f09747f091534faf93542710 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:56.777405 18417 leader_election.cc:302] T 2aa2a489116940dea3173cad16d94fa5 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a; no voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710
I20240430 20:11:56.777791 21688 raft_consensus.cc:2726] T 2aa2a489116940dea3173cad16d94fa5 P 2efb542aa55e439b99ccb98d1c5c884a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240430 20:11:56.778435 19902 catalog_manager.cc:4830] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): tablet 04b3dff86d5c48a09d03da58bd48f360 (table default.Table_2ec9e316bc3347b189a13b7a8bbcf829 [id=9fa22833979f4c2f978de651145acef9]) successfully deleted
W20240430 20:11:56.780216 18557 leader_election.cc:341] T a37d917bcc45468f9e3ee805640fba98 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:56.780349 18557 leader_election.cc:302] T a37d917bcc45468f9e3ee805640fba98 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:56.780570 20489 raft_consensus.cc:2726] T a37d917bcc45468f9e3ee805640fba98 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:56.781642 21397 ts_tablet_manager.cc:1875] T 47e98935eb21449abd3d4d24327778d5 P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240430 20:11:56.783020 18557 leader_election.cc:341] T 6f3996b7e29941e391ea141dd6cc44b2 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:56.783167 18557 leader_election.cc:302] T 6f3996b7e29941e391ea141dd6cc44b2 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:56.783754 20489 raft_consensus.cc:2726] T 6f3996b7e29941e391ea141dd6cc44b2 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240430 20:11:56.786211 18557 leader_election.cc:341] T 772eba5f24c549c791cad75565858285 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:56.786329 18557 leader_election.cc:302] T 772eba5f24c549c791cad75565858285 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:56.786520 21468 raft_consensus.cc:482] T 5b26319488644d6a8b6be60460a5e7af P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:56.786670 21468 raft_consensus.cc:504] T 5b26319488644d6a8b6be60460a5e7af P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:56.786898 20489 raft_consensus.cc:2726] T 772eba5f24c549c791cad75565858285 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:56.787320 21468 leader_election.cc:288] T 5b26319488644d6a8b6be60460a5e7af P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:56.786231 21396 ts_tablet_manager.cc:1888] T 1f4e860484784aa9a2d28af2e9fb611c P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:56.787808 21396 log.cc:1196] T 1f4e860484784aa9a2d28af2e9fb611c P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/1f4e860484784aa9a2d28af2e9fb611c
I20240430 20:11:56.788147 18468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b26319488644d6a8b6be60460a5e7af" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:56.788395 21396 ts_tablet_manager.cc:1909] T 1f4e860484784aa9a2d28af2e9fb611c P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
W20240430 20:11:56.788903 18558 leader_election.cc:341] T 5b26319488644d6a8b6be60460a5e7af P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:56.792119 18749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd1b8407071e4e4287fc172009777414" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:56.792819 18761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b26319488644d6a8b6be60460a5e7af" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:56.792870 21396 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:45934 (request call id 104) took 2854 ms. Trace:
I20240430 20:11:56.792965 21396 rpcz_store.cc:276] 0430 20:11:53.938208 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:53.938250 (+ 42us) service_pool.cc:225] Handling call
0430 20:11:56.694963 (+2756713us) ts_tablet_manager.cc:1120] Deleting tablet 1f4e860484784aa9a2d28af2e9fb611c
0430 20:11:56.770912 (+ 75949us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:56.779963 (+ 9051us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:56.786115 (+ 6152us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:56.792850 (+ 6735us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3636,"tablet-delete.queue_time_us":2756156}
I20240430 20:11:56.795070 19895 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet 1f4e860484784aa9a2d28af2e9fb611c (table default.Table_45d50fc81325443097375ae6e65a1633 [id=9c861883de1d49c78146d65e9d4dd39f]) successfully deleted
I20240430 20:11:56.795238 21396 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:45934 (request call id 105) took 2856 ms. Trace:
I20240430 20:11:56.795312 21396 rpcz_store.cc:276] 0430 20:11:53.938311 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:53.938350 (+ 39us) service_pool.cc:225] Handling call
0430 20:11:56.795142 (+2856792us) ts_tablet_manager.cc:1120] Deleting tablet 0eb4f3e7ece6496293e01b63620d644b
0430 20:11:56.795152 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:56.795221 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":21,"tablet-delete.queue_time_us":2856337}
I20240430 20:11:56.796090 21395 tablet_replica.cc:330] T d7c1fc7e81364fb890bbc13d7fb6848b P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:56.796316 21395 raft_consensus.cc:2229] T d7c1fc7e81364fb890bbc13d7fb6848b P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Raft consensus shutting down.
W20240430 20:11:56.796393 19895 catalog_manager.cc:4797] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): delete failed for tablet 0eb4f3e7ece6496293e01b63620d644b because the tablet was not found. No further retry: Not found: Tablet not found: 0eb4f3e7ece6496293e01b63620d644b
I20240430 20:11:56.796701 21395 raft_consensus.cc:2258] T d7c1fc7e81364fb890bbc13d7fb6848b P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:56.797334 21395 ts_tablet_manager.cc:1875] T d7c1fc7e81364fb890bbc13d7fb6848b P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:56.800825 21396 tablet_replica.cc:330] T 2873af6da928463f9809864b8eaf0c4a P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:56.801123 21396 raft_consensus.cc:2229] T 2873af6da928463f9809864b8eaf0c4a P 5e1c6397f09747f091534faf93542710 [term 2 LEADER]: Raft consensus shutting down.
I20240430 20:11:56.801481 21396 raft_consensus.cc:2258] T 2873af6da928463f9809864b8eaf0c4a P 5e1c6397f09747f091534faf93542710 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:56.802094 21396 ts_tablet_manager.cc:1875] T 2873af6da928463f9809864b8eaf0c4a P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:56.810992 19930 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 6b51cbc731794083a88fe4bed1ddeebf (table default.Table_c6c962da746e4fd8a4d7200d09f08e9e [id=8e0f5d8c941d423ea7b30a89513d5170]). Expected version 1 got 0
I20240430 20:11:56.811293 19930 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 62732af95c6a4c749c5b726f0460dc72 (table default.Table_e3160f5ced1b4718b479401ca4df675c [id=2d44ec94131148048021b2ee6366ba30]). Expected version 3 got 0
I20240430 20:11:56.811688 19930 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet a0baee91f5b44945b595d0e234af018e (table default.Table_5ed3ea587cf843db9a1ff1ffab40b678 [id=b3d1c2f97c4648518dbea83a46cadd85]). Expected version 2 got 0
I20240430 20:11:56.811863 19930 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 266fd23786bc435db5083370f3330ed0 (table default.Table_cf60b652e9dc49e7a948ce66fd1cbffe [id=9734bb8dd05a400ca4244b3829f01dee]). Expected version 1 got 0
I20240430 20:11:56.812278 19930 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet dd1de890e4394db287cf38bb411f5586 (table default.Table_84084a27f7504dd992fce07448c10df3 [id=6257559b0e9545c58848a76af0c5a5f7]). Expected version 2 got 0
I20240430 20:11:56.813048 19930 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 130007acc8ba4e709c3eb85fabe75903 (table default.Table_78987875650940c293247bc43db345ab [id=36f478a20a1f4a53a52fc29ade62b7f6]). Expected version 1 got 0
I20240430 20:11:56.813241 19930 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet dac141d1b70c48d785caddd7182811ee (table default.Table_fc0a4540f98b4eea95099b8ab46ea235 [id=f3c53ccffe64410495cc8c379a05cfae]). Expected version 1 got 0
I20240430 20:11:56.813406 19930 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet ed0c37a71ede4bfc9eb5ad1a4365f25e (table default.Table_a2e85efc6b454524b0b122102ec07002 [id=86d795a82e674d859104741ebd04f4e8]). Expected version 1 got 0
I20240430 20:11:56.813576 19930 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 12795670f21447a8964fe1efc6e92398 (table default.Table_896bbae1f6944005b0272ca8ebc97218 [id=d5cea90cbc2c48df9d8b80da1eccc949]). Expected version 3 got 0
I20240430 20:11:56.814128 19930 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet a482f4622f774d60b4efd6827b8be7d7 (table default.Table_e686cdfcbcc046749fa5b51578a64794 [id=c3dc9f05316d46d6a3ef4d03f40877ec]). Expected version 1 got 0
I20240430 20:11:56.817471 18983 tablet_bootstrap.cc:492] T 72205028d0654e69a6fa20f105604e1c P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:56.817690 18983 ts_tablet_manager.cc:1367] T 72205028d0654e69a6fa20f105604e1c P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.043s user 0.003s sys 0.007s
I20240430 20:11:56.818454 18699 leader_election.cc:302] T 2aa2a489116940dea3173cad16d94fa5 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710
I20240430 20:11:56.819337 21454 raft_consensus.cc:2726] T 2aa2a489116940dea3173cad16d94fa5 P 3c51c33d5616482cafb68e26688820f4 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240430 20:11:56.819196 18983 raft_consensus.cc:348] T 72205028d0654e69a6fa20f105604e1c P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:56.819460 18983 raft_consensus.cc:374] T 72205028d0654e69a6fa20f105604e1c P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:56.819512 18983 raft_consensus.cc:729] T 72205028d0654e69a6fa20f105604e1c P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:56.819653 21454 consensus_queue.cc:1035] T fd1b8407071e4e4287fc172009777414 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:56.819718 18983 consensus_queue.cc:260] T 72205028d0654e69a6fa20f105604e1c P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:56.820551 18983 ts_tablet_manager.cc:1398] T 72205028d0654e69a6fa20f105604e1c P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240430 20:11:56.820828 18983 tablet_bootstrap.cc:492] T b7141e215d6d44bc8406c80508190874 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:56.821916 18985 tablet_bootstrap.cc:492] T d34a450a91d542c98261456d1f635e9c P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:56.822114 18985 ts_tablet_manager.cc:1367] T d34a450a91d542c98261456d1f635e9c P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.065s user 0.009s sys 0.003s
I20240430 20:11:56.822561 18983 tablet_bootstrap.cc:654] T b7141e215d6d44bc8406c80508190874 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:56.823061 18985 raft_consensus.cc:348] T d34a450a91d542c98261456d1f635e9c P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:56.823333 18985 raft_consensus.cc:374] T d34a450a91d542c98261456d1f635e9c P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:56.823459 18985 raft_consensus.cc:729] T d34a450a91d542c98261456d1f635e9c P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:56.823763 18985 consensus_queue.cc:260] T d34a450a91d542c98261456d1f635e9c P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:56.824371 18985 ts_tablet_manager.cc:1398] T d34a450a91d542c98261456d1f635e9c P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240430 20:11:56.824654 18985 tablet_bootstrap.cc:492] T 17ddcaf971154a52b606bbf29a539bf0 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:56.825958 18988 tablet_bootstrap.cc:492] T e041756c962b4a9f97c2cb314e7d8c05 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:56.826141 18988 ts_tablet_manager.cc:1367] T e041756c962b4a9f97c2cb314e7d8c05 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.060s user 0.012s sys 0.006s
I20240430 20:11:56.827028 18985 tablet_bootstrap.cc:654] T 17ddcaf971154a52b606bbf29a539bf0 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:56.827402 18988 raft_consensus.cc:348] T e041756c962b4a9f97c2cb314e7d8c05 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:56.828425 18988 raft_consensus.cc:374] T e041756c962b4a9f97c2cb314e7d8c05 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:56.828617 18988 raft_consensus.cc:729] T e041756c962b4a9f97c2cb314e7d8c05 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:56.828930 18988 consensus_queue.cc:260] T e041756c962b4a9f97c2cb314e7d8c05 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:56.829283 21468 raft_consensus.cc:482] T 798a13f545fd4d8f8c654eaa2452eda1 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:56.829453 21468 raft_consensus.cc:504] T 798a13f545fd4d8f8c654eaa2452eda1 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:56.830103 21468 leader_election.cc:288] T 798a13f545fd4d8f8c654eaa2452eda1 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:56.831018 18988 ts_tablet_manager.cc:1398] T e041756c962b4a9f97c2cb314e7d8c05 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240430 20:11:56.831344 18988 tablet_bootstrap.cc:492] T 9a76db2c7ff146758c54908e428a0e02 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:56.836742 18988 tablet_bootstrap.cc:654] T 9a76db2c7ff146758c54908e428a0e02 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:56.833016 18810 heartbeater.cc:503] Master 127.17.177.60:37225 was elected leader, sending a full tablet report...
I20240430 20:11:56.830749 18468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "798a13f545fd4d8f8c654eaa2452eda1" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:56.840764 18983 tablet_bootstrap.cc:492] T b7141e215d6d44bc8406c80508190874 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:56.840971 18983 ts_tablet_manager.cc:1367] T b7141e215d6d44bc8406c80508190874 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.020s user 0.001s sys 0.003s
W20240430 20:11:56.841537 18558 leader_election.cc:341] T 798a13f545fd4d8f8c654eaa2452eda1 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:56.841713 18983 raft_consensus.cc:348] T b7141e215d6d44bc8406c80508190874 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:56.841910 18983 raft_consensus.cc:374] T b7141e215d6d44bc8406c80508190874 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:56.841974 18983 raft_consensus.cc:729] T b7141e215d6d44bc8406c80508190874 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:56.842196 18983 consensus_queue.cc:260] T b7141e215d6d44bc8406c80508190874 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:56.842725 18983 ts_tablet_manager.cc:1398] T b7141e215d6d44bc8406c80508190874 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240430 20:11:56.843032 18983 tablet_bootstrap.cc:492] T bb471d33b0784c9c9de1e5425cdd1fd7 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:56.844638 18983 tablet_bootstrap.cc:654] T bb471d33b0784c9c9de1e5425cdd1fd7 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:56.845106 19930 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 5b26319488644d6a8b6be60460a5e7af (table default.Table_69892a8264a4477db48ee6248758e046 [id=5a1c4a7be2fc465fac89a50854eb3322]). Expected version 3 got 0
W20240430 20:11:56.846633 18417 consensus_peers.cc:485] T d61c279d20b0471a967942525ceb4fcb P 2efb542aa55e439b99ccb98d1c5c884a -> Peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Couldn't send request to peer 3c51c33d5616482cafb68e26688820f4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d61c279d20b0471a967942525ceb4fcb. This is attempt 6: this message will repeat every 5th retry.
I20240430 20:11:56.847191 18417 leader_election.cc:302] T fd1b8407071e4e4287fc172009777414 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a; no voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710
I20240430 20:11:56.847445 21688 raft_consensus.cc:2726] T fd1b8407071e4e4287fc172009777414 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240430 20:11:56.848592 18557 leader_election.cc:341] T 5b26319488644d6a8b6be60460a5e7af P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:56.848728 18557 leader_election.cc:302] T 5b26319488644d6a8b6be60460a5e7af P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:56.849175 21468 raft_consensus.cc:2726] T 5b26319488644d6a8b6be60460a5e7af P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:56.850559 19930 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 0368d37dd9e2452982fb81f539c6805b (table default.Table_404b186c544e4b30bb36cb77d1a14120 [id=4330626e342a48699ea939add5c91b11]). Expected version 1 got 0
I20240430 20:11:56.851738 19930 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 2da900b6bbfa4fa99fe8494fd2e68da8 (table default.Table_19a91a7f17ff456ebb145e8dd89c0940 [id=67283dd3995a44c6a7a7bc8722e4f955]). Expected version 1 got 0
I20240430 20:11:56.852669 21396 ts_tablet_manager.cc:1888] T 2873af6da928463f9809864b8eaf0c4a P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.2
I20240430 20:11:56.851951 19930 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 17ddcaf971154a52b606bbf29a539bf0 (table default.Table_f2f38f5d69e34eaa9e3dbf24cfd1cdd6 [id=872c2faa4a464edd98a51df2e1b3f817]). Expected version 1 got 0
I20240430 20:11:56.853314 19930 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 086b3dfa84c045a18c8c4f177dce606a (table default.Table_df19cb8a6f4d46699350ec5b69c14e5a [id=197923633fc54a4582207942b51530e2]). Expected version 2 got 0
I20240430 20:11:56.853564 19930 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 1050ab528c7d41559831652a1437d358 (table default.Table_acddc4cbf6334546b5fb3e4752fef2a5 [id=274cb15fd751410e83327ac4e8dc4629]). Expected version 2 got 0
I20240430 20:11:56.853724 21396 log.cc:1196] T 2873af6da928463f9809864b8eaf0c4a P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/2873af6da928463f9809864b8eaf0c4a
I20240430 20:11:56.853960 21395 ts_tablet_manager.cc:1888] T d7c1fc7e81364fb890bbc13d7fb6848b P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:56.854451 21396 ts_tablet_manager.cc:1909] T 2873af6da928463f9809864b8eaf0c4a P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:56.854857 19930 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet e041756c962b4a9f97c2cb314e7d8c05 (table default.Table_de88cafd92b149f9b545369bff71033c [id=c4bf4806f3d84579b16a3038d8d982c3]). Expected version 1 got 0
I20240430 20:11:56.855036 21395 log.cc:1196] T d7c1fc7e81364fb890bbc13d7fb6848b P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/d7c1fc7e81364fb890bbc13d7fb6848b
I20240430 20:11:56.855464 21395 ts_tablet_manager.cc:1909] T d7c1fc7e81364fb890bbc13d7fb6848b P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:56.859082 21468 raft_consensus.cc:482] T ffd6e420e6d44c039832de38de001c1b P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:56.859207 21468 raft_consensus.cc:504] T ffd6e420e6d44c039832de38de001c1b P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:56.859486 18983 tablet_bootstrap.cc:492] T bb471d33b0784c9c9de1e5425cdd1fd7 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:56.859607 18983 ts_tablet_manager.cc:1367] T bb471d33b0784c9c9de1e5425cdd1fd7 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.000s
I20240430 20:11:56.859784 21468 leader_election.cc:288] T ffd6e420e6d44c039832de38de001c1b P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:56.860450 18983 raft_consensus.cc:348] T bb471d33b0784c9c9de1e5425cdd1fd7 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:56.860627 18983 raft_consensus.cc:374] T bb471d33b0784c9c9de1e5425cdd1fd7 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:56.860672 18983 raft_consensus.cc:729] T bb471d33b0784c9c9de1e5425cdd1fd7 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:56.861013 18483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffd6e420e6d44c039832de38de001c1b" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:56.860893 18983 consensus_queue.cc:260] T bb471d33b0784c9c9de1e5425cdd1fd7 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:56.861495 18983 ts_tablet_manager.cc:1398] T bb471d33b0784c9c9de1e5425cdd1fd7 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
W20240430 20:11:56.861548 18558 leader_election.cc:341] T ffd6e420e6d44c039832de38de001c1b P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:56.863802 19930 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet f5eaf2413e1d4e69b56fae61f768f0ce (table default.Table_1280cfe32d5b41abb4d3eb1bd70641bf [id=8489b872074a42b6abb713e420968141]). Expected version 1 got 0
I20240430 20:11:56.864087 19930 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 5df364628561475391c9a5e10ff5a05f (table default.Table_3c8f9f7b22f64fb1bdc3f1550987233a [id=7fd6f8576f4c4cbc8ffcdb4ab95e15dc]). Expected version 1 got 0
I20240430 20:11:56.864428 19930 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet c4953578de5f45f3a6ba092a13357460 (table default.Table_e3cea4c635274d37a44427ca00897fb9 [id=a826c08c7eef44ec83feae3a6cb384e5]). Expected version 1 got 0
I20240430 20:11:56.864593 19930 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 821c55edb3414d72b1489eea97d3f73a (table default.Table_4d68c38d9ba7452a829f8318b2b267b8 [id=d06e3e9045b1429aaeebd1fd67d841de]). Expected version 1 got 0
I20240430 20:11:56.865693 19930 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 37b2ea4be7494f0985808ade788dbe2b (table default.Table_5f6e0fce999b46de927f9271e5de5071 [id=0baad83a8c0f4aeebc7ddad068625a69]). Expected version 1 got 0
I20240430 20:11:56.866189 19930 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 9e8ab75e7d9c4b64a7b7dd2678569490 (table default.Table_b2d07520b9944a89ad62fca70396174e [id=1c8e0a3a44ec4a798770c0165ed3dffa]). Expected version 1 got 0
I20240430 20:11:56.866856 18983 tablet_bootstrap.cc:492] T 024069ab786749028c020abd1c836cf5 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:56.867790 18985 tablet_bootstrap.cc:492] T 17ddcaf971154a52b606bbf29a539bf0 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:56.868014 18985 ts_tablet_manager.cc:1367] T 17ddcaf971154a52b606bbf29a539bf0 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.043s user 0.003s sys 0.000s
I20240430 20:11:56.868855 21396 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:45934 (request call id 99) took 2931 ms. Trace:
I20240430 20:11:56.868921 18985 raft_consensus.cc:348] T 17ddcaf971154a52b606bbf29a539bf0 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:56.869001 21396 rpcz_store.cc:276] 0430 20:11:53.937722 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:53.938224 (+ 502us) service_pool.cc:225] Handling call
0430 20:11:56.800748 (+2862524us) ts_tablet_manager.cc:1120] Deleting tablet 2873af6da928463f9809864b8eaf0c4a
0430 20:11:56.800765 (+ 17us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:56.827302 (+ 26537us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:56.852564 (+ 25262us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:56.868833 (+ 16269us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2861682}
I20240430 20:11:56.869159 18985 raft_consensus.cc:374] T 17ddcaf971154a52b606bbf29a539bf0 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:56.869225 18985 raft_consensus.cc:729] T 17ddcaf971154a52b606bbf29a539bf0 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:56.869613 21396 tablet_replica.cc:330] T de1dfe6c82d34a4d8d02e466c55327ff P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:56.869566 19895 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet 2873af6da928463f9809864b8eaf0c4a (table default.Table_9dc9f3b0b0c8452a9da7c85e62d87bdc [id=b7706c6195364115b5ba240651c22940]) successfully deleted
I20240430 20:11:56.869823 21396 raft_consensus.cc:2229] T de1dfe6c82d34a4d8d02e466c55327ff P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Raft consensus shutting down.
I20240430 20:11:56.869871 18985 consensus_queue.cc:260] T 17ddcaf971154a52b606bbf29a539bf0 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:56.870208 21396 raft_consensus.cc:2258] T de1dfe6c82d34a4d8d02e466c55327ff P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:56.870451 18985 ts_tablet_manager.cc:1398] T 17ddcaf971154a52b606bbf29a539bf0 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240430 20:11:56.870831 21396 ts_tablet_manager.cc:1875] T de1dfe6c82d34a4d8d02e466c55327ff P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:56.874890 21395 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41450 (request call id 70) took 2947 ms. Trace:
I20240430 20:11:56.875061 21395 rpcz_store.cc:276] 0430 20:11:53.927059 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:53.928230 (+ 1171us) service_pool.cc:225] Handling call
0430 20:11:56.796031 (+2867801us) ts_tablet_manager.cc:1120] Deleting tablet d7c1fc7e81364fb890bbc13d7fb6848b
0430 20:11:56.796044 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:56.817146 (+ 21102us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:56.853864 (+ 36718us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:56.874869 (+ 21005us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2795281}
I20240430 20:11:56.875707 21395 tablet_replica.cc:330] T c1017a50d1d646c486c0b9a75cb88ce4 P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:56.875684 19902 catalog_manager.cc:4830] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): tablet d7c1fc7e81364fb890bbc13d7fb6848b (table default.Table_fa64149f685c4784bda92ae4f578dd92 [id=3d6e1fcb9c0f403394d8b436d100b9d2]) successfully deleted
I20240430 20:11:56.875952 21395 raft_consensus.cc:2229] T c1017a50d1d646c486c0b9a75cb88ce4 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:56.876123 21395 raft_consensus.cc:2258] T c1017a50d1d646c486c0b9a75cb88ce4 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:56.876698 21395 ts_tablet_manager.cc:1875] T c1017a50d1d646c486c0b9a75cb88ce4 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:56.878821 18983 tablet_bootstrap.cc:654] T 024069ab786749028c020abd1c836cf5 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:56.879338 21468 raft_consensus.cc:482] T 266fd23786bc435db5083370f3330ed0 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:56.879451 21468 raft_consensus.cc:504] T 266fd23786bc435db5083370f3330ed0 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:56.880129 21468 leader_election.cc:288] T 266fd23786bc435db5083370f3330ed0 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:56.880388 18985 tablet_bootstrap.cc:492] T e041756c962b4a9f97c2cb314e7d8c05 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:56.880800 18468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "266fd23786bc435db5083370f3330ed0" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
W20240430 20:11:56.881346 18558 leader_election.cc:341] T 266fd23786bc435db5083370f3330ed0 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:56.885586 18988 tablet_bootstrap.cc:492] T 9a76db2c7ff146758c54908e428a0e02 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:56.885756 18988 ts_tablet_manager.cc:1367] T 9a76db2c7ff146758c54908e428a0e02 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.054s user 0.002s sys 0.004s
I20240430 20:11:56.881466 19930 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet aaef85eea87f4443943fd3692bb2a38f (table default.Table_3da5b4c8668c470483ae2a0c7773eaaf [id=e292d818ee3c45df90522f08eb2d6227]). Expected version 1 got 0
I20240430 20:11:56.886535 18988 raft_consensus.cc:348] T 9a76db2c7ff146758c54908e428a0e02 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:56.886857 18988 raft_consensus.cc:374] T 9a76db2c7ff146758c54908e428a0e02 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:56.886934 18988 raft_consensus.cc:729] T 9a76db2c7ff146758c54908e428a0e02 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:56.886955 19930 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet a59ef6c545dd41a28265a4c1b8ce10e0 (table default.Table_07e40af8f1c546048975c8edf88170d7 [id=86df5d161d8641c89070e7eed9a39360]). Expected version 2 got 0
I20240430 20:11:56.887133 18988 consensus_queue.cc:260] T 9a76db2c7ff146758c54908e428a0e02 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:56.887465 19930 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 330453ddeea6456e8ef93fa92608b0d3 (table default.Table_c1f36f7df5ae4e7c8eb8b4e99b89748b [id=a50abf28790d4284a76ac6ee22c02606]). Expected version 1 got 0
I20240430 20:11:56.887610 18988 ts_tablet_manager.cc:1398] T 9a76db2c7ff146758c54908e428a0e02 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240430 20:11:56.888835 19930 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 22fef8bb6b034db098c3372ae992f7a7 (table default.Table_4440df38610340758047115845b42b95 [id=7d186ca20c094cc19aef792fe1c094e9]). Expected version 1 got 0
I20240430 20:11:56.889110 19930 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 69f88532952a4591bf1577541c3430e6 (table default.Table_1177828cad1f4a24bd397619dad96170 [id=a95fed1bc3fa4865bf9a35dfc51b659c]). Expected version 2 got 0
I20240430 20:11:56.889449 19930 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet eb6c96047b114ebba73db7a371f2da8e (table default.Table_7d5f6f18cbd64c869739872a7dff9be2 [id=e67d619e36c0485f8f22b4a81cb4cb6c]). Expected version 1 got 0
I20240430 20:11:56.889614 19930 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 7a0176f8a74c47efbbe3d3343b3dc896 (table default.Table_144e5402c1864d818436015b26648024 [id=73e5c9f0470d47849109819c063e1ce4]). Expected version 2 got 0
I20240430 20:11:56.889920 19930 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 94fbd4a57a1e438891fcb456f6b40be3 (table default.Table_25ddc3adef6b4ccf8b9c00cc6772031c [id=4fa623ffdcf347639150a355806038af]). Expected version 1 got 0
I20240430 20:11:56.890170 19930 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet acd23f0a63d5448aaa7ed78422e67bad (table default.Table_972f13be4dca4cd19b9a04534dc59f51 [id=69f37db4de634a3185613e40013e0ed7]). Expected version 1 got 0
I20240430 20:11:56.890821 19930 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 81787eccca6c42c1b421377a3bf9ebe6 (table default.Table_0b4ca4cd0af148399dfa21f2d32114e3 [id=7f602584cef9486eb7d9b4389be3f43e]). Expected version 3 got 2
I20240430 20:11:56.891166 19930 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet f1073fc792954c5795a05aa7d0ec24df (table default.Table_91417a5439c8457caa8bb0a12221a5dd [id=aedf11fda8914a3b843a57f8a7b69a53]). Expected version 2 got 0
I20240430 20:11:56.891456 19930 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 331c031d44854d679a27d10d9151d0ea (table default.Table_763f0800e8bb4fa69976886416250f42 [id=61c8a5b99ef34a0baeeb343fef7cfd05]). Expected version 1 got 0
I20240430 20:11:56.891912 18988 tablet_bootstrap.cc:492] T 9cc9251abd4e40d59a2bd2d0fada6165 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:56.895671 18988 tablet_bootstrap.cc:654] T 9cc9251abd4e40d59a2bd2d0fada6165 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:56.896132 21395 ts_tablet_manager.cc:1888] T c1017a50d1d646c486c0b9a75cb88ce4 P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:56.896245 21395 log.cc:1196] T c1017a50d1d646c486c0b9a75cb88ce4 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/c1017a50d1d646c486c0b9a75cb88ce4
I20240430 20:11:56.896582 21395 ts_tablet_manager.cc:1909] T c1017a50d1d646c486c0b9a75cb88ce4 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:56.897548 21395 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41450 (request call id 74) took 2889 ms. Trace:
I20240430 20:11:56.897644 21395 rpcz_store.cc:276] 0430 20:11:54.007768 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:54.007894 (+ 126us) service_pool.cc:225] Handling call
0430 20:11:56.875658 (+2867764us) ts_tablet_manager.cc:1120] Deleting tablet c1017a50d1d646c486c0b9a75cb88ce4
0430 20:11:56.875668 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:56.879733 (+ 4065us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:56.896019 (+ 16286us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:56.897528 (+ 1509us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2862178}
I20240430 20:11:56.898105 19902 catalog_manager.cc:4830] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): tablet c1017a50d1d646c486c0b9a75cb88ce4 (table default.Table_d95c70be474e415cba6a345438fad1ba [id=095b8b568ebc475aa3c21e0c03ef5d79]) successfully deleted
I20240430 20:11:56.907169 19930 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 7ce09fcbe1ac457a9c05efaf000c3263 (table default.Table_edc4c27ed846454288f73f6a0d88cb69 [id=78fd3f516083489ab3b42d90dcef7e4d]). Expected version 1 got 0
I20240430 20:11:56.907853 19930 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet fbdeb4714b8c4b93ab72822c43329aae (table default.Table_c12ca6938d424fbfa2e1df5cdb00fdbc [id=0fb93f36bc17408ea56bc4cafcfd8580]). Expected version 1 got 0
I20240430 20:11:56.907932 21493 raft_consensus.cc:482] T 3e5225f1f5f544ecbf8dfa470af3daca P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5e1c6397f09747f091534faf93542710)
I20240430 20:11:56.908083 21493 raft_consensus.cc:504] T 3e5225f1f5f544ecbf8dfa470af3daca P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:56.908394 19930 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 488d0e609b224674932f890939664df2 (table default.Table_58f35a2120874472978ca0e5f816e86c [id=ff18001c6d814afc8c1768d0c84a6bac]). Expected version 1 got 0
I20240430 20:11:56.908726 21493 leader_election.cc:288] T 3e5225f1f5f544ecbf8dfa470af3daca P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:56.908903 19930 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 2cd41b407d6b444d844c4c7c0ebf82e5 (table default.Table_6f8e0eff8f5a484dbd6a9c3fdb5d03f6 [id=3813b5e9778c4bd683a06598cef0d86f]). Expected version 1 got 0
I20240430 20:11:56.909045 18760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "798a13f545fd4d8f8c654eaa2452eda1" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:56.909245 19930 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 772eba5f24c549c791cad75565858285 (table default.Table_5fc77d5f60ae44f4bb4ca9f7f183672c [id=6ca223ca6354449c959d85ed2ac6a9c8]). Expected version 1 got 0
I20240430 20:11:56.909482 19930 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 1faa2f2c84184b129ba6e4d2857822f0 (table default.Table_0dd55b391be94213b4af60ea6e5a979f [id=1a382ad8009b4522a50d55c307db8f96]). Expected version 1 got 0
I20240430 20:11:56.909631 19930 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 3a03b6997921408cb5a65f4cae223e9c (table default.Table_744b5c96ea014bdb8feae740356a790e [id=83b1ac2e9c3d4a10b02f96ebb17faf44]). Expected version 1 got 0
I20240430 20:11:56.909811 19930 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet ae20f44df50343289b0a512a4ea53692 (table default.Table_1844f6b1bda84812825384e14a2fd94e [id=71da9acc3cff4d95bbac4a686b708fd5]). Expected version 1 got 0
W20240430 20:11:56.910084 18696 leader_election.cc:341] T 3e5225f1f5f544ecbf8dfa470af3daca P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Not found: Tablet not found: 3e5225f1f5f544ecbf8dfa470af3daca
I20240430 20:11:56.910701 18983 tablet_bootstrap.cc:492] T 024069ab786749028c020abd1c836cf5 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:56.910838 18983 ts_tablet_manager.cc:1367] T 024069ab786749028c020abd1c836cf5 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.044s user 0.003s sys 0.003s
I20240430 20:11:56.911180 18760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffd6e420e6d44c039832de38de001c1b" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:56.911558 18983 raft_consensus.cc:348] T 024069ab786749028c020abd1c836cf5 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:56.911698 21468 raft_consensus.cc:482] T b7141e215d6d44bc8406c80508190874 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:56.911734 18983 raft_consensus.cc:374] T 024069ab786749028c020abd1c836cf5 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:56.911811 18983 raft_consensus.cc:729] T 024069ab786749028c020abd1c836cf5 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:56.911836 21468 raft_consensus.cc:504] T b7141e215d6d44bc8406c80508190874 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:56.912087 18983 consensus_queue.cc:260] T 024069ab786749028c020abd1c836cf5 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:56.912442 21468 leader_election.cc:288] T b7141e215d6d44bc8406c80508190874 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:56.912628 18983 ts_tablet_manager.cc:1398] T 024069ab786749028c020abd1c836cf5 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240430 20:11:56.913249 18468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7141e215d6d44bc8406c80508190874" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
W20240430 20:11:56.913833 18558 leader_election.cc:341] T b7141e215d6d44bc8406c80508190874 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:56.915553 18760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "266fd23786bc435db5083370f3330ed0" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:56.916148 21396 ts_tablet_manager.cc:1888] T de1dfe6c82d34a4d8d02e466c55327ff P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:56.916308 21396 log.cc:1196] T de1dfe6c82d34a4d8d02e466c55327ff P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/de1dfe6c82d34a4d8d02e466c55327ff
I20240430 20:11:56.916849 21396 ts_tablet_manager.cc:1909] T de1dfe6c82d34a4d8d02e466c55327ff P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:56.918459 21395 tablet_replica.cc:330] T 436579f473844c6ca1038b496201a550 P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:56.918700 19940 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: default.Table_404b186c544e4b30bb36cb77d1a14120 [id=4330626e342a48699ea939add5c91b11] alter complete (version 1)
I20240430 20:11:56.921202 18810 heartbeater.cc:503] Master 127.17.177.60:37225 was elected leader, sending a full tablet report...
I20240430 20:11:56.922104 18760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7141e215d6d44bc8406c80508190874" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:56.923434 18988 tablet_bootstrap.cc:492] T 9cc9251abd4e40d59a2bd2d0fada6165 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:56.923573 18988 ts_tablet_manager.cc:1367] T 9cc9251abd4e40d59a2bd2d0fada6165 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.032s user 0.002s sys 0.003s
I20240430 20:11:56.923640 19937 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: default.Table_744b5c96ea014bdb8feae740356a790e [id=83b1ac2e9c3d4a10b02f96ebb17faf44] alter complete (version 1)
I20240430 20:11:56.924351 18988 raft_consensus.cc:348] T 9cc9251abd4e40d59a2bd2d0fada6165 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:56.924554 18988 raft_consensus.cc:374] T 9cc9251abd4e40d59a2bd2d0fada6165 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:56.924611 18988 raft_consensus.cc:729] T 9cc9251abd4e40d59a2bd2d0fada6165 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:56.924794 18988 consensus_queue.cc:260] T 9cc9251abd4e40d59a2bd2d0fada6165 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:56.925458 18988 ts_tablet_manager.cc:1398] T 9cc9251abd4e40d59a2bd2d0fada6165 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240430 20:11:56.925563 19941 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: default.Table_9df4a0b4170f4f28a6c17b0cb052ead6 [id=d1655c507b434f129ad7ae0b04365e51] alter complete (version 2)
W20240430 20:11:56.925629 19938 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 36a07d27145a46e48c3f10e8211a389e consensus state has no leader
I20240430 20:11:56.925812 18988 tablet_bootstrap.cc:492] T de546e2d6a5142159f23bd50d78ff735 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:56.925765 19945 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: default.Table_19a91a7f17ff456ebb145e8dd89c0940 [id=67283dd3995a44c6a7a7bc8722e4f955] alter complete (version 1)
I20240430 20:11:56.925834 19938 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:50746 (request call id 81) took 4801 ms. Trace:
I20240430 20:11:56.925935 19938 rpcz_store.cc:276] 0430 20:11:52.124367 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:56.167320 (+4042953us) service_pool.cc:225] Handling call
0430 20:11:56.220021 (+ 52701us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0430 20:11:56.220276 (+ 255us) catalog_manager.cc:3613] Apply alter schema
0430 20:11:56.228761 (+ 8485us) catalog_manager.cc:3636] Acquired catalog manager lock
0430 20:11:56.228878 (+ 117us) catalog_manager.cc:3783] Updating metadata on disk
0430 20:11:56.256546 (+ 27668us) catalog_manager.cc:3814] Committing alterations to in-memory state
0430 20:11:56.925776 (+669230us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:56.244176 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:56.244340 (+ 164us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:56.244345 (+ 5us) write_op.cc:427] Acquired schema lock
0430 20:11:56.244347 (+ 2us) tablet.cc:660] Decoding operations
0430 20:11:56.244455 (+ 108us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:56.244468 (+ 13us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:56.244473 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0430 20:11:56.244543 (+ 70us) tablet.cc:699] Row locks acquired
0430 20:11:56.244546 (+ 3us) write_op.cc:260] PREPARE: finished
0430 20:11:56.244575 (+ 29us) write_op.cc:270] Start()
0430 20:11:56.244607 (+ 32us) write_op.cc:276] Timestamp: P: 1714507916244518 usec, L: 0
0430 20:11:56.244612 (+ 5us) op_driver.cc:331] REPLICATION: starting
0430 20:11:56.244788 (+ 176us) log.cc:842] Serialized 405 byte log entry
0430 20:11:56.254631 (+ 9843us) op_driver.cc:444] REPLICATION: finished
0430 20:11:56.254744 (+ 113us) write_op.cc:301] APPLY: starting
0430 20:11:56.254759 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:56.254807 (+ 48us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:56.254846 (+ 39us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:56.254901 (+ 55us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:56.254936 (+ 35us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0430 20:11:56.254944 (+ 8us) write_op.cc:312] APPLY: finished
0430 20:11:56.255027 (+ 83us) log.cc:842] Serialized 23 byte log entry
0430 20:11:56.255094 (+ 67us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:56.255113 (+ 19us) write_op.cc:446] Released schema lock
0430 20:11:56.255119 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:56.255179 (+ 60us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:56.255185 (+ 6us) write_op.cc:446] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":62,"apply.run_cpu_time_us":493,"apply.run_wall_time_us":492,"num_ops":1,"prepare.queue_time_us":14742,"prepare.run_cpu_time_us":701,"prepare.run_wall_time_us":2540,"raft.queue_time_us":31,"raft.run_cpu_time_us":208,"raft.run_wall_time_us":208,"replication_time_us":9972}]]}
I20240430 20:11:56.928228 18988 tablet_bootstrap.cc:654] T de546e2d6a5142159f23bd50d78ff735 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:56.929416 21397 ts_tablet_manager.cc:1888] T 47e98935eb21449abd3d4d24327778d5 P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:56.929525 21397 log.cc:1196] T 47e98935eb21449abd3d4d24327778d5 P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/47e98935eb21449abd3d4d24327778d5
I20240430 20:11:56.930895 21397 ts_tablet_manager.cc:1909] T 47e98935eb21449abd3d4d24327778d5 P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:56.931243 21397 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56540 (request call id 71) took 3194 ms. Trace:
I20240430 20:11:56.931325 21397 rpcz_store.cc:276] 0430 20:11:53.737189 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:53.737819 (+ 630us) service_pool.cc:225] Handling call
0430 20:11:56.770339 (+3032520us) ts_tablet_manager.cc:1120] Deleting tablet 47e98935eb21449abd3d4d24327778d5
0430 20:11:56.770357 (+ 18us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:56.850805 (+ 80448us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:56.929279 (+ 78474us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:56.931223 (+ 1944us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3032343}
I20240430 20:11:56.931489 18733 tablet_service.cc:1514] Processing DeleteTablet for tablet dad16ae02b8b4e29a724d8eec67bb4b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:42 UTC) from {username='slave'} at 127.0.0.1:56540
I20240430 20:11:56.931625 21397 tablet_replica.cc:330] T 06d28b6f48a34b17b0d7fb3102df2f4c P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:56.931725 18983 tablet_bootstrap.cc:492] T 72d316f424214792ac1d5029ffb1b04a P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:56.931790 21397 raft_consensus.cc:2229] T 06d28b6f48a34b17b0d7fb3102df2f4c P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Raft consensus shutting down.
I20240430 20:11:56.932291 19900 catalog_manager.cc:4830] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): tablet 47e98935eb21449abd3d4d24327778d5 (table default.Table_67d51131549440cc99629dda924693f7 [id=842d2b7e31cb4e7793d88103997a88d8]) successfully deleted
I20240430 20:11:56.933368 21397 raft_consensus.cc:2258] T 06d28b6f48a34b17b0d7fb3102df2f4c P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:56.935168 21397 ts_tablet_manager.cc:1875] T 06d28b6f48a34b17b0d7fb3102df2f4c P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:56.936275 18983 tablet_bootstrap.cc:654] T 72d316f424214792ac1d5029ffb1b04a P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:56.939296 21468 raft_consensus.cc:482] T bb471d33b0784c9c9de1e5425cdd1fd7 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:56.939441 21468 raft_consensus.cc:504] T bb471d33b0784c9c9de1e5425cdd1fd7 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:56.940047 21468 leader_election.cc:288] T bb471d33b0784c9c9de1e5425cdd1fd7 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:56.945804 18985 tablet_bootstrap.cc:654] T e041756c962b4a9f97c2cb314e7d8c05 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:56.948418 18730 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f4e860484784aa9a2d28af2e9fb611c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:39 UTC) from {username='slave'} at 127.0.0.1:56540
I20240430 20:11:56.949097 18739 tablet_service.cc:1514] Processing DeleteTablet for tablet d4d6c37b578c4fe1aa666b0a6b0bc7c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:40 UTC) from {username='slave'} at 127.0.0.1:56540
I20240430 20:11:56.949327 18731 tablet_service.cc:1514] Processing DeleteTablet for tablet 90a694bd5db649afa50f3c4e752939c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:39 UTC) from {username='slave'} at 127.0.0.1:56540
I20240430 20:11:56.949638 18738 tablet_service.cc:1514] Processing DeleteTablet for tablet 6db55bb510b24b0cae897936211e15f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:34 UTC) from {username='slave'} at 127.0.0.1:56540
I20240430 20:11:56.949806 18728 tablet_service.cc:1514] Processing DeleteTablet for tablet a445fdfde01c4371a5e6b3daea5b00ab with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:44 UTC) from {username='slave'} at 127.0.0.1:56540
I20240430 20:11:56.949862 18725 tablet_service.cc:1514] Processing DeleteTablet for tablet 8901327b3efb4201adce61a9266142ea with delete_type TABLET_DATA_DELETED (replaced by tablet b1277899f0c040cea7dc2b12f3fd68de) from {username='slave'} at 127.0.0.1:56540
I20240430 20:11:56.950452 18726 tablet_service.cc:1514] Processing DeleteTablet for tablet c1017a50d1d646c486c0b9a75cb88ce4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:43 UTC) from {username='slave'} at 127.0.0.1:56540
I20240430 20:11:56.950716 18744 tablet_service.cc:1514] Processing DeleteTablet for tablet f7d3dea34cf54fe2b949d1b7ca68577a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:40 UTC) from {username='slave'} at 127.0.0.1:56540
I20240430 20:11:56.951218 18735 tablet_service.cc:1514] Processing DeleteTablet for tablet 898b2c91adf34545808df9c786e218c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:40 UTC) from {username='slave'} at 127.0.0.1:56540
I20240430 20:11:56.951411 18736 tablet_service.cc:1514] Processing DeleteTablet for tablet 2aa2a489116940dea3173cad16d94fa5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:40 UTC) from {username='slave'} at 127.0.0.1:56540
I20240430 20:11:56.951606 18740 tablet_service.cc:1514] Processing DeleteTablet for tablet aa9998f4e17147aa8f2b35c01980463a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:42 UTC) from {username='slave'} at 127.0.0.1:56540
I20240430 20:11:56.952000 18742 tablet_service.cc:1514] Processing DeleteTablet for tablet 489dc838744a49eba542009c86688aa5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:43 UTC) from {username='slave'} at 127.0.0.1:56540
I20240430 20:11:56.952047 21396 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:45934 (request call id 98) took 3014 ms. Trace:
I20240430 20:11:56.952133 21396 rpcz_store.cc:276] 0430 20:11:53.937641 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:53.937696 (+ 55us) service_pool.cc:225] Handling call
0430 20:11:56.869560 (+2931864us) ts_tablet_manager.cc:1120] Deleting tablet de1dfe6c82d34a4d8d02e466c55327ff
0430 20:11:56.869571 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:56.884415 (+ 14844us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:56.916039 (+ 31624us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:56.952025 (+ 35986us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2930465}
I20240430 20:11:56.949806 18729 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f0b341a31224bf1be525b84b9500304 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:44 UTC) from {username='slave'} at 127.0.0.1:56540
I20240430 20:11:56.952437 21396 tablet_replica.cc:330] T f7d3dea34cf54fe2b949d1b7ca68577a P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:56.952617 21396 raft_consensus.cc:2229] T f7d3dea34cf54fe2b949d1b7ca68577a P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Raft consensus shutting down.
I20240430 20:11:56.952684 18741 tablet_service.cc:1514] Processing DeleteTablet for tablet 558e71ec370d46ca8fbdcec5b57eb2ec with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:37 UTC) from {username='slave'} at 127.0.0.1:56540
I20240430 20:11:56.953264 18731 tablet_service.cc:1514] Processing DeleteTablet for tablet df92eb85261d47d88cbf16d29d9ad30e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:45 UTC) from {username='slave'} at 127.0.0.1:56540
I20240430 20:11:56.953716 18741 tablet_service.cc:1514] Processing DeleteTablet for tablet 47e98935eb21449abd3d4d24327778d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:45 UTC) from {username='slave'} at 127.0.0.1:56540
I20240430 20:11:56.953912 18730 tablet_service.cc:1514] Processing DeleteTablet for tablet 2797eed593fb4b06a3c4d6a28e8075a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:42 UTC) from {username='slave'} at 127.0.0.1:56540
I20240430 20:11:56.955189 21396 raft_consensus.cc:2258] T f7d3dea34cf54fe2b949d1b7ca68577a P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:56.956279 19940 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: default.Table_896bbae1f6944005b0272ca8ebc97218 [id=d5cea90cbc2c48df9d8b80da1eccc949] alter complete (version 3)
I20240430 20:11:56.957185 21397 ts_tablet_manager.cc:1888] T 06d28b6f48a34b17b0d7fb3102df2f4c P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:56.957284 21397 log.cc:1196] T 06d28b6f48a34b17b0d7fb3102df2f4c P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/06d28b6f48a34b17b0d7fb3102df2f4c
I20240430 20:11:56.957659 21397 ts_tablet_manager.cc:1909] T 06d28b6f48a34b17b0d7fb3102df2f4c P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:56.958005 21397 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56540 (request call id 72) took 3220 ms. Trace:
I20240430 20:11:56.959697 18983 tablet_bootstrap.cc:492] T 72d316f424214792ac1d5029ffb1b04a P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:56.959851 18983 ts_tablet_manager.cc:1367] T 72d316f424214792ac1d5029ffb1b04a P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.028s user 0.005s sys 0.001s
I20240430 20:11:56.960760 18983 raft_consensus.cc:348] T 72d316f424214792ac1d5029ffb1b04a P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:56.960974 18983 raft_consensus.cc:374] T 72d316f424214792ac1d5029ffb1b04a P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:56.961033 18983 raft_consensus.cc:729] T 72d316f424214792ac1d5029ffb1b04a P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:56.961061 19895 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet de1dfe6c82d34a4d8d02e466c55327ff (table default.Table_3ec50a6d9cd84592becff23d2cfa0c03 [id=1e785b6c159b4312bd1b064271f55a49]) successfully deleted
I20240430 20:11:56.961277 18983 consensus_queue.cc:260] T 72d316f424214792ac1d5029ffb1b04a P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:56.961956 18983 ts_tablet_manager.cc:1398] T 72d316f424214792ac1d5029ffb1b04a P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240430 20:11:56.962247 18983 tablet_bootstrap.cc:492] T 4fa9575d96a84bef876467e21dada803 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:56.962296 21395 raft_consensus.cc:2229] T 436579f473844c6ca1038b496201a550 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:56.964311 21395 raft_consensus.cc:2258] T 436579f473844c6ca1038b496201a550 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:56.965094 21396 ts_tablet_manager.cc:1875] T f7d3dea34cf54fe2b949d1b7ca68577a P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:56.966888 18468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb471d33b0784c9c9de1e5425cdd1fd7" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:56.967662 18983 tablet_bootstrap.cc:654] T 4fa9575d96a84bef876467e21dada803 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
W20240430 20:11:56.967830 18558 leader_election.cc:341] T bb471d33b0784c9c9de1e5425cdd1fd7 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:56.970633 21397 rpcz_store.cc:276] 0430 20:11:53.737342 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:53.737996 (+ 654us) service_pool.cc:225] Handling call
0430 20:11:56.931579 (+3193583us) ts_tablet_manager.cc:1120] Deleting tablet 06d28b6f48a34b17b0d7fb3102df2f4c
0430 20:11:56.931589 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:56.948112 (+ 16523us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:56.957071 (+ 8959us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:56.957986 (+ 915us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3193468}
I20240430 20:11:56.970899 19937 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: default.Table_1844f6b1bda84812825384e14a2fd94e [id=71da9acc3cff4d95bbac4a686b708fd5] alter complete (version 1)
I20240430 20:11:56.971433 19945 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: default.Table_b9983e101a6547649eb44ee04b5e1147 [id=29769be2b9ec4ecba9de5c0a65692f85] alter complete (version 1)
I20240430 20:11:56.971793 21397 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56540 (request call id 70) took 3234 ms. Trace:
I20240430 20:11:56.971907 21397 rpcz_store.cc:276] 0430 20:11:53.737084 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:53.738195 (+ 1111us) service_pool.cc:225] Handling call
0430 20:11:56.971630 (+3233435us) ts_tablet_manager.cc:1120] Deleting tablet 1701a15b115741aea13f8327e3d26b34
0430 20:11:56.971647 (+ 17us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:56.971767 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3233261}
I20240430 20:11:56.972998 21397 tablet_replica.cc:330] T 283e5e73cbc04daba0de19f4541f9fd0 P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:56.973448 21397 raft_consensus.cc:2229] T 283e5e73cbc04daba0de19f4541f9fd0 P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Raft consensus shutting down.
I20240430 20:11:56.974553 21397 raft_consensus.cc:2258] T 283e5e73cbc04daba0de19f4541f9fd0 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:56.975409 21397 ts_tablet_manager.cc:1875] T 283e5e73cbc04daba0de19f4541f9fd0 P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:56.975762 21395 ts_tablet_manager.cc:1875] T 436579f473844c6ca1038b496201a550 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240430 20:11:56.978579 19895 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 36a07d27145a46e48c3f10e8211a389e (table default.Table_0d3f914f27df478c8df8cbb69bc0d8b4 [id=d4763eaeacb546ae94b77b5a78fef986], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 36a07d27145a46e48c3f10e8211a389e consensus state has no leader
W20240430 20:11:56.983052 18415 consensus_peers.cc:485] T d61c279d20b0471a967942525ceb4fcb P 2efb542aa55e439b99ccb98d1c5c884a -> Peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Couldn't send request to peer 5e1c6397f09747f091534faf93542710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d61c279d20b0471a967942525ceb4fcb. This is attempt 6: this message will repeat every 5th retry.
I20240430 20:11:56.983690 21493 consensus_queue.cc:1035] T 347a8224114b49e5b7dcb75d9b726e0a P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:56.989173 19900 catalog_manager.cc:4830] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): tablet 06d28b6f48a34b17b0d7fb3102df2f4c (table default.Table_4a09e48b08fa4c4387262a5b92b2a42d [id=565591f1f0e84bc89c8d366ef53ae969]) successfully deleted
W20240430 20:11:56.937453 19941 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.17.177.3:39881 (request call id 138) took 3805 ms (client timeout 1000 ms). Trace:
W20240430 20:11:56.993005 19941 rpcz_store.cc:269] 0430 20:11:53.131924 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:53.712631 (+580707us) service_pool.cc:225] Handling call
0430 20:11:56.937402 (+3224771us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:54.204969 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:54.205087 (+ 118us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:54.205095 (+ 8us) write_op.cc:427] Acquired schema lock
0430 20:11:54.205098 (+ 3us) tablet.cc:660] Decoding operations
0430 20:11:54.205182 (+ 84us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:54.205189 (+ 7us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:54.205192 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0430 20:11:54.205241 (+ 49us) tablet.cc:699] Row locks acquired
0430 20:11:54.205246 (+ 5us) write_op.cc:260] PREPARE: finished
0430 20:11:54.205278 (+ 32us) write_op.cc:270] Start()
0430 20:11:54.205307 (+ 29us) write_op.cc:276] Timestamp: P: 1714507914205257 usec, L: 0
0430 20:11:54.205313 (+ 6us) op_driver.cc:331] REPLICATION: starting
0430 20:11:54.205498 (+ 185us) log.cc:842] Serialized 530 byte log entry
0430 20:11:54.211856 (+ 6358us) op_driver.cc:444] REPLICATION: finished
0430 20:11:54.213173 (+ 1317us) write_op.cc:301] APPLY: starting
0430 20:11:54.213182 (+ 9us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:54.213209 (+ 27us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:54.213244 (+ 35us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:54.213286 (+ 42us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:54.213310 (+ 24us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0430 20:11:54.213319 (+ 9us) write_op.cc:312] APPLY: finished
0430 20:11:54.213368 (+ 49us) log.cc:842] Serialized 23 byte log entry
0430 20:11:54.213405 (+ 37us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:54.213420 (+ 15us) write_op.cc:446] Released schema lock
0430 20:11:54.213427 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:54.213479 (+ 52us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:54.213484 (+ 5us) write_op.cc:446] Released schema lock
Related trace 'op':
0430 20:11:54.288085 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:54.288238 (+ 153us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:54.288245 (+ 7us) write_op.cc:427] Acquired schema lock
0430 20:11:54.288249 (+ 4us) tablet.cc:660] Decoding operations
0430 20:11:54.288384 (+ 135us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:54.288397 (+ 13us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:54.288406 (+ 9us) tablet.cc:683] Acquiring locks for 1 operations
0430 20:11:54.288490 (+ 84us) tablet.cc:699] Row locks acquired
0430 20:11:54.288494 (+ 4us) write_op.cc:260] PREPARE: finished
0430 20:11:54.288534 (+ 40us) write_op.cc:270] Start()
0430 20:11:54.288579 (+ 45us) write_op.cc:276] Timestamp: P: 1714507914288506 usec, L: 0
0430 20:11:54.288585 (+ 6us) op_driver.cc:331] REPLICATION: starting
0430 20:11:54.288823 (+ 238us) log.cc:842] Serialized 356 byte log entry
0430 20:11:54.307772 (+ 18949us) op_driver.cc:444] REPLICATION: finished
0430 20:11:54.308557 (+ 785us) write_op.cc:301] APPLY: starting
0430 20:11:54.308568 (+ 11us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:54.308600 (+ 32us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:54.308665 (+ 65us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:54.308708 (+ 43us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:54.308734 (+ 26us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0430 20:11:54.308740 (+ 6us) write_op.cc:312] APPLY: finished
0430 20:11:54.308803 (+ 63us) log.cc:842] Serialized 23 byte log entry
0430 20:11:54.308843 (+ 40us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:54.308857 (+ 14us) write_op.cc:446] Released schema lock
0430 20:11:54.308862 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:54.308910 (+ 48us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:54.308916 (+ 6us) write_op.cc:446] Released schema lock
Metrics: {"mutex_wait_us":27101,"reported_tablets":56,"child_traces":[["op",{"apply.queue_time_us":1287,"apply.run_cpu_time_us":349,"apply.run_wall_time_us":348,"num_ops":1,"prepare.queue_time_us":27845,"prepare.run_cpu_time_us":9481,"prepare.run_wall_time_us":65961,"raft.queue_time_us":609,"raft.run_cpu_time_us":3,"raft.run_wall_time_us":3,"replication_time_us":6525,"thread_start_us":125295,"threads_started":2}],["op",{"apply.queue_time_us":755,"apply.run_cpu_time_us":413,"apply.run_wall_time_us":412,"num_ops":1,"prepare.queue_time_us":14644,"prepare.run_cpu_time_us":854,"prepare.run_wall_time_us":1234,"raft.queue_time_us":913,"raft.run_cpu_time_us":6,"raft.run_wall_time_us":7,"replication_time_us":19171}]]}
I20240430 20:11:57.016724 21397 ts_tablet_manager.cc:1888] T 283e5e73cbc04daba0de19f4541f9fd0 P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:57.016871 21397 log.cc:1196] T 283e5e73cbc04daba0de19f4541f9fd0 P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/283e5e73cbc04daba0de19f4541f9fd0
I20240430 20:11:57.017298 21397 ts_tablet_manager.cc:1909] T 283e5e73cbc04daba0de19f4541f9fd0 P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
W20240430 20:11:57.019107 18415 consensus_peers.cc:485] T aa9998f4e17147aa8f2b35c01980463a P 2efb542aa55e439b99ccb98d1c5c884a -> Peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Couldn't send request to peer 5e1c6397f09747f091534faf93542710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: aa9998f4e17147aa8f2b35c01980463a. This is attempt 1: this message will repeat every 5th retry.
I20240430 20:11:57.019501 18810 heartbeater.cc:503] Master 127.17.177.60:37225 was elected leader, sending a full tablet report...
I20240430 20:11:57.025130 20489 raft_consensus.cc:482] T e28529e1c2ae4c7c9b9b0e4bd709d132 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:57.025286 20489 raft_consensus.cc:504] T e28529e1c2ae4c7c9b9b0e4bd709d132 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:57.025899 20489 leader_election.cc:288] T e28529e1c2ae4c7c9b9b0e4bd709d132 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:57.029145 18468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e28529e1c2ae4c7c9b9b0e4bd709d132" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
W20240430 20:11:57.030005 18558 leader_election.cc:341] T e28529e1c2ae4c7c9b9b0e4bd709d132 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:57.037233 21397 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56540 (request call id 64) took 3300 ms. Trace:
I20240430 20:11:57.037434 21397 rpcz_store.cc:276] 0430 20:11:53.736377 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:53.740181 (+ 3804us) service_pool.cc:225] Handling call
0430 20:11:56.972948 (+3232767us) ts_tablet_manager.cc:1120] Deleting tablet 283e5e73cbc04daba0de19f4541f9fd0
0430 20:11:56.972958 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:57.000086 (+ 27128us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:57.016601 (+ 16515us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:57.037202 (+ 20601us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3232578}
I20240430 20:11:57.037896 21397 tablet_replica.cc:330] T fdf53061efd74fa29a9d8208d1fd210e P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:57.038151 21397 raft_consensus.cc:2229] T fdf53061efd74fa29a9d8208d1fd210e P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Raft consensus shutting down.
I20240430 20:11:57.038550 21397 raft_consensus.cc:2258] T fdf53061efd74fa29a9d8208d1fd210e P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:57.039292 21397 ts_tablet_manager.cc:1875] T fdf53061efd74fa29a9d8208d1fd210e P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:57.045724 21397 ts_tablet_manager.cc:1888] T fdf53061efd74fa29a9d8208d1fd210e P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:57.055742 18985 tablet_bootstrap.cc:492] T e041756c962b4a9f97c2cb314e7d8c05 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:57.055936 18985 ts_tablet_manager.cc:1367] T e041756c962b4a9f97c2cb314e7d8c05 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.176s user 0.008s sys 0.003s
I20240430 20:11:57.056828 18985 raft_consensus.cc:348] T e041756c962b4a9f97c2cb314e7d8c05 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:57.057026 18985 raft_consensus.cc:374] T e041756c962b4a9f97c2cb314e7d8c05 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:57.057077 18985 raft_consensus.cc:729] T e041756c962b4a9f97c2cb314e7d8c05 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:57.057265 18985 consensus_queue.cc:260] T e041756c962b4a9f97c2cb314e7d8c05 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:57.058050 18985 ts_tablet_manager.cc:1398] T e041756c962b4a9f97c2cb314e7d8c05 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240430 20:11:57.058472 18985 tablet_bootstrap.cc:492] T 8e7f08e9ac274bfa86763eec895bbfda P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:57.060449 19940 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: default.Table_78987875650940c293247bc43db345ab [id=36f478a20a1f4a53a52fc29ade62b7f6] alter complete (version 1)
I20240430 20:11:57.060900 19937 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: default.Table_7d5f6f18cbd64c869739872a7dff9be2 [id=e67d619e36c0485f8f22b4a81cb4cb6c] alter complete (version 1)
I20240430 20:11:57.061100 19945 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: default.Table_2ec9e316bc3347b189a13b7a8bbcf829 [id=9fa22833979f4c2f978de651145acef9] alter complete (version 1)
I20240430 20:11:57.063104 18985 tablet_bootstrap.cc:654] T 8e7f08e9ac274bfa86763eec895bbfda P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:57.067395 19929 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:50746 (request call id 82) took 4939 ms. Trace:
I20240430 20:11:57.067584 19929 rpcz_store.cc:276] 0430 20:11:52.128279 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:56.210911 (+4082632us) service_pool.cc:225] Handling call
0430 20:11:56.211427 (+ 516us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0430 20:11:56.211650 (+ 223us) catalog_manager.cc:3613] Apply alter schema
0430 20:11:56.211850 (+ 200us) catalog_manager.cc:3636] Acquired catalog manager lock
0430 20:11:56.211962 (+ 112us) catalog_manager.cc:3783] Updating metadata on disk
0430 20:11:56.242411 (+ 30449us) catalog_manager.cc:3814] Committing alterations to in-memory state
0430 20:11:57.067322 (+824911us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:56.234215 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:56.234366 (+ 151us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:56.234374 (+ 8us) write_op.cc:427] Acquired schema lock
0430 20:11:56.234376 (+ 2us) tablet.cc:660] Decoding operations
0430 20:11:56.234512 (+ 136us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:56.234526 (+ 14us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:56.234531 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0430 20:11:56.234631 (+ 100us) tablet.cc:699] Row locks acquired
0430 20:11:56.234634 (+ 3us) write_op.cc:260] PREPARE: finished
0430 20:11:56.234674 (+ 40us) write_op.cc:270] Start()
0430 20:11:56.234710 (+ 36us) write_op.cc:276] Timestamp: P: 1714507916234614 usec, L: 0
0430 20:11:56.234713 (+ 3us) op_driver.cc:331] REPLICATION: starting
0430 20:11:56.235255 (+ 542us) log.cc:842] Serialized 405 byte log entry
0430 20:11:56.240615 (+ 5360us) op_driver.cc:444] REPLICATION: finished
0430 20:11:56.240841 (+ 226us) write_op.cc:301] APPLY: starting
0430 20:11:56.240859 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:56.240915 (+ 56us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:56.240946 (+ 31us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:56.241006 (+ 60us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:56.241040 (+ 34us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0430 20:11:56.241047 (+ 7us) write_op.cc:312] APPLY: finished
0430 20:11:56.241145 (+ 98us) log.cc:842] Serialized 23 byte log entry
0430 20:11:56.241213 (+ 68us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:56.241237 (+ 24us) write_op.cc:446] Released schema lock
0430 20:11:56.241245 (+ 8us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:56.241307 (+ 62us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:56.241314 (+ 7us) write_op.cc:446] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":170,"apply.run_cpu_time_us":527,"apply.run_wall_time_us":527,"num_ops":1,"prepare.queue_time_us":21624,"prepare.run_cpu_time_us":893,"prepare.run_wall_time_us":9964,"raft.queue_time_us":41,"raft.run_cpu_time_us":278,"raft.run_wall_time_us":278,"replication_time_us":5877,"thread_start_us":22498,"threads_started":1}]]}
I20240430 20:11:57.069590 19929 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50746:
name: "default.Table_ed5eedd0ae824a329615cc8cbaa90ab0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240430 20:11:57.070282 19929 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ed5eedd0ae824a329615cc8cbaa90ab0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240430 20:11:57.071242 21397 log.cc:1196] T fdf53061efd74fa29a9d8208d1fd210e P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/fdf53061efd74fa29a9d8208d1fd210e
I20240430 20:11:57.071966 21397 ts_tablet_manager.cc:1909] T fdf53061efd74fa29a9d8208d1fd210e P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:57.075801 21397 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56540 (request call id 79) took 3293 ms. Trace:
I20240430 20:11:57.076867 21397 rpcz_store.cc:276] 0430 20:11:53.782283 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:53.782396 (+ 113us) service_pool.cc:225] Handling call
0430 20:11:57.037846 (+3255450us) ts_tablet_manager.cc:1120] Deleting tablet fdf53061efd74fa29a9d8208d1fd210e
0430 20:11:57.037857 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:57.042508 (+ 4651us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:57.045620 (+ 3112us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:57.075776 (+ 30156us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3255143}
I20240430 20:11:57.077579 21397 tablet_replica.cc:330] T f3711610610542128c4a84e17bce4a3e P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:57.078048 21397 raft_consensus.cc:2229] T f3711610610542128c4a84e17bce4a3e P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:57.078461 21397 raft_consensus.cc:2258] T f3711610610542128c4a84e17bce4a3e P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:57.079175 21397 ts_tablet_manager.cc:1875] T f3711610610542128c4a84e17bce4a3e P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:57.082737 19937 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: default.Table_c6c962da746e4fd8a4d7200d09f08e9e [id=8e0f5d8c941d423ea7b30a89513d5170] alter complete (version 1)
I20240430 20:11:57.083752 19945 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: default.Table_0b4ca4cd0af148399dfa21f2d32114e3 [id=7f602584cef9486eb7d9b4389be3f43e] alter complete (version 3)
I20240430 20:11:57.086760 19940 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: default.Table_a2e85efc6b454524b0b122102ec07002 [id=86d795a82e674d859104741ebd04f4e8] alter complete (version 1)
I20240430 20:11:57.102315 19945 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: default.Table_f1ad29d5986d4b799cfc67afb2b0dfdf [id=838fc8b61a864dea8877a6f144e4cb4f] alter complete (version 2)
I20240430 20:11:57.108775 21688 consensus_queue.cc:1035] T e5b85a066cdc46d48ccf8201e5423c29 P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:57.112740 18810 heartbeater.cc:503] Master 127.17.177.60:37225 was elected leader, sending a full tablet report...
I20240430 20:11:57.114235 21395 ts_tablet_manager.cc:1888] T 436579f473844c6ca1038b496201a550 P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:57.114406 21395 log.cc:1196] T 436579f473844c6ca1038b496201a550 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/436579f473844c6ca1038b496201a550
I20240430 20:11:57.114794 21395 ts_tablet_manager.cc:1909] T 436579f473844c6ca1038b496201a550 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:57.115128 21395 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41450 (request call id 75) took 3101 ms. Trace:
I20240430 20:11:57.115218 21395 rpcz_store.cc:276] 0430 20:11:54.013621 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:54.013739 (+ 118us) service_pool.cc:225] Handling call
0430 20:11:56.918401 (+2904662us) ts_tablet_manager.cc:1120] Deleting tablet 436579f473844c6ca1038b496201a550
0430 20:11:56.918414 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:57.103401 (+184987us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:57.114099 (+ 10698us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:57.115107 (+ 1008us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2902954}
I20240430 20:11:57.115511 21581 raft_consensus.cc:482] T dbb427aefa6b47b0805d1751fea4efcb P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:57.115655 21581 raft_consensus.cc:504] T dbb427aefa6b47b0805d1751fea4efcb P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:57.116267 21581 leader_election.cc:288] T dbb427aefa6b47b0805d1751fea4efcb P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:57.116760 19902 catalog_manager.cc:4830] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): tablet 436579f473844c6ca1038b496201a550 (table default.Table_273caa9a42a9474e8787c37032368654 [id=d89426aa652c42e49f956ea85efb0d43]) successfully deleted
W20240430 20:11:57.119259 18415 leader_election.cc:341] T dbb427aefa6b47b0805d1751fea4efcb P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Not found: Tablet not found: dbb427aefa6b47b0805d1751fea4efcb
I20240430 20:11:57.116552 21397 ts_tablet_manager.cc:1888] T f3711610610542128c4a84e17bce4a3e P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:57.124794 21397 log.cc:1196] T f3711610610542128c4a84e17bce4a3e P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/f3711610610542128c4a84e17bce4a3e
I20240430 20:11:57.128295 21396 ts_tablet_manager.cc:1888] T f7d3dea34cf54fe2b949d1b7ca68577a P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:57.128537 21396 log.cc:1196] T f7d3dea34cf54fe2b949d1b7ca68577a P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/f7d3dea34cf54fe2b949d1b7ca68577a
I20240430 20:11:57.128728 21397 ts_tablet_manager.cc:1909] T f3711610610542128c4a84e17bce4a3e P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:57.129180 21396 ts_tablet_manager.cc:1909] T f7d3dea34cf54fe2b949d1b7ca68577a P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:57.129223 21397 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56540 (request call id 81) took 3345 ms. Trace:
I20240430 20:11:57.129331 21397 rpcz_store.cc:276] 0430 20:11:53.783828 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:53.783888 (+ 60us) service_pool.cc:225] Handling call
0430 20:11:57.077527 (+3293639us) ts_tablet_manager.cc:1120] Deleting tablet f3711610610542128c4a84e17bce4a3e
0430 20:11:57.077538 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:57.085858 (+ 8320us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:57.116442 (+ 30584us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:57.129194 (+ 12752us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3293405}
I20240430 20:11:57.129752 21397 tablet_replica.cc:330] T 3e5225f1f5f544ecbf8dfa470af3daca P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:57.130013 21397 raft_consensus.cc:2229] T 3e5225f1f5f544ecbf8dfa470af3daca P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:57.130185 21397 pending_rounds.cc:62] T 3e5225f1f5f544ecbf8dfa470af3daca P 3c51c33d5616482cafb68e26688820f4: Trying to abort 1 pending ops.
I20240430 20:11:57.130237 21397 pending_rounds.cc:68] T 3e5225f1f5f544ecbf8dfa470af3daca P 3c51c33d5616482cafb68e26688820f4: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7022624414983913472 op_type: ALTER_SCHEMA_OP alter_schema_request { tablet_id: "3e5225f1f5f544ecbf8dfa470af3daca" schema { columns { id: 10 name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } schema_version: 1 new_table_name: "default.Table_c7c87a8e019944cab3ec99e9563ac01a" dest_uuid: "5e1c6397f09747f091534faf93542710" new_extra_config { } }
I20240430 20:11:57.130739 21397 raft_consensus.cc:2258] T 3e5225f1f5f544ecbf8dfa470af3daca P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:57.133417 21397 ts_tablet_manager.cc:1875] T 3e5225f1f5f544ecbf8dfa470af3daca P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:57.136580 21581 raft_consensus.cc:482] T fcc16db4d6be42edb7bd1ce18fe0190d P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:57.136735 21581 raft_consensus.cc:504] T fcc16db4d6be42edb7bd1ce18fe0190d P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:57.137382 21581 leader_election.cc:288] T fcc16db4d6be42edb7bd1ce18fe0190d P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
W20240430 20:11:57.138485 18415 leader_election.cc:341] T fcc16db4d6be42edb7bd1ce18fe0190d P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Not found: Tablet not found: fcc16db4d6be42edb7bd1ce18fe0190d
I20240430 20:11:57.146231 21436 consensus_queue.cc:1035] T 4391b5227fb64ef481542a962a5e21a5 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240430 20:11:57.146931 18415 consensus_peers.cc:485] T d4d6c37b578c4fe1aa666b0a6b0bc7c9 P 2efb542aa55e439b99ccb98d1c5c884a -> Peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Couldn't send request to peer 5e1c6397f09747f091534faf93542710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d4d6c37b578c4fe1aa666b0a6b0bc7c9. This is attempt 1: this message will repeat every 5th retry.
I20240430 20:11:57.147264 21581 raft_consensus.cc:482] T d34a450a91d542c98261456d1f635e9c P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:57.147387 21581 raft_consensus.cc:504] T d34a450a91d542c98261456d1f635e9c P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:57.148208 21581 leader_election.cc:288] T d34a450a91d542c98261456d1f635e9c P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
W20240430 20:11:57.149024 18415 leader_election.cc:341] T d34a450a91d542c98261456d1f635e9c P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Not found: Tablet not found: d34a450a91d542c98261456d1f635e9c
I20240430 20:11:57.156487 18988 tablet_bootstrap.cc:492] T de546e2d6a5142159f23bd50d78ff735 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:57.156843 18988 ts_tablet_manager.cc:1367] T de546e2d6a5142159f23bd50d78ff735 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.231s user 0.014s sys 0.028s
I20240430 20:11:57.158094 18988 raft_consensus.cc:348] T de546e2d6a5142159f23bd50d78ff735 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:57.158571 21396 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:45934 (request call id 101) took 3220 ms. Trace:
I20240430 20:11:57.158746 21396 rpcz_store.cc:276] 0430 20:11:53.937961 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:53.938048 (+ 87us) service_pool.cc:225] Handling call
0430 20:11:56.952391 (+3014343us) ts_tablet_manager.cc:1120] Deleting tablet f7d3dea34cf54fe2b949d1b7ca68577a
0430 20:11:56.952400 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:57.075936 (+123536us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:57.128149 (+ 52213us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:57.158545 (+ 30396us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":422,"tablet-delete.queue_time_us":3013705}
I20240430 20:11:57.159163 18988 raft_consensus.cc:374] T de546e2d6a5142159f23bd50d78ff735 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:57.159405 18988 raft_consensus.cc:729] T de546e2d6a5142159f23bd50d78ff735 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:57.159431 21396 tablet_replica.cc:330] T 8901327b3efb4201adce61a9266142ea P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:57.159451 19895 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet f7d3dea34cf54fe2b949d1b7ca68577a (table default.Table_a4ea055447464ca6af32fe80bab34c42 [id=ea8038670f7c44a9bb20c396b54d6440]) successfully deleted
I20240430 20:11:57.159627 21396 raft_consensus.cc:2229] T 8901327b3efb4201adce61a9266142ea P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:57.159828 18988 consensus_queue.cc:260] T de546e2d6a5142159f23bd50d78ff735 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:57.163046 21493 raft_consensus.cc:482] T 9cc9251abd4e40d59a2bd2d0fada6165 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:57.164574 21493 raft_consensus.cc:504] T 9cc9251abd4e40d59a2bd2d0fada6165 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:57.165225 21493 leader_election.cc:288] T 9cc9251abd4e40d59a2bd2d0fada6165 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:57.166541 18988 ts_tablet_manager.cc:1398] T de546e2d6a5142159f23bd50d78ff735 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20240430 20:11:57.166936 18988 tablet_bootstrap.cc:492] T a37d917bcc45468f9e3ee805640fba98 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:57.169076 18985 tablet_bootstrap.cc:492] T 8e7f08e9ac274bfa86763eec895bbfda P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:57.169292 18985 ts_tablet_manager.cc:1367] T 8e7f08e9ac274bfa86763eec895bbfda P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.111s user 0.000s sys 0.004s
I20240430 20:11:57.170099 18988 tablet_bootstrap.cc:654] T a37d917bcc45468f9e3ee805640fba98 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:57.170078 18985 raft_consensus.cc:348] T 8e7f08e9ac274bfa86763eec895bbfda P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:57.170351 18985 raft_consensus.cc:374] T 8e7f08e9ac274bfa86763eec895bbfda P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:57.170450 18985 raft_consensus.cc:729] T 8e7f08e9ac274bfa86763eec895bbfda P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:57.170802 18985 consensus_queue.cc:260] T 8e7f08e9ac274bfa86763eec895bbfda P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:57.171593 18985 ts_tablet_manager.cc:1398] T 8e7f08e9ac274bfa86763eec895bbfda P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240430 20:11:57.171924 18985 tablet_bootstrap.cc:492] T f9bf604e6c234349a9f72f45a4c4aef6 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:57.173755 18985 tablet_bootstrap.cc:654] T f9bf604e6c234349a9f72f45a4c4aef6 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:57.165887 18607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cc9251abd4e40d59a2bd2d0fada6165" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:57.176920 18607 raft_consensus.cc:2454] T 9cc9251abd4e40d59a2bd2d0fada6165 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:57.177671 18696 leader_election.cc:302] T 9cc9251abd4e40d59a2bd2d0fada6165 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:57.177978 21493 raft_consensus.cc:2781] T 9cc9251abd4e40d59a2bd2d0fada6165 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:57.178046 21493 raft_consensus.cc:482] T 9cc9251abd4e40d59a2bd2d0fada6165 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:57.178105 21493 raft_consensus.cc:3037] T 9cc9251abd4e40d59a2bd2d0fada6165 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:57.178699 18983 tablet_bootstrap.cc:492] T 4fa9575d96a84bef876467e21dada803 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:57.178815 18983 ts_tablet_manager.cc:1367] T 4fa9575d96a84bef876467e21dada803 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.217s user 0.024s sys 0.019s
I20240430 20:11:57.179471 18983 raft_consensus.cc:348] T 4fa9575d96a84bef876467e21dada803 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:57.179649 18983 raft_consensus.cc:374] T 4fa9575d96a84bef876467e21dada803 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:57.179719 18983 raft_consensus.cc:729] T 4fa9575d96a84bef876467e21dada803 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:57.179955 18983 consensus_queue.cc:260] T 4fa9575d96a84bef876467e21dada803 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:57.180665 21493 raft_consensus.cc:504] T 9cc9251abd4e40d59a2bd2d0fada6165 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:57.181262 21493 leader_election.cc:288] T 9cc9251abd4e40d59a2bd2d0fada6165 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Requested vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:57.182178 18617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cc9251abd4e40d59a2bd2d0fada6165" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:57.182346 21395 tablet_replica.cc:330] T a663030d37614053a7c0369841403422 P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:57.182493 18617 raft_consensus.cc:3037] T 9cc9251abd4e40d59a2bd2d0fada6165 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:57.182981 21395 raft_consensus.cc:2229] T a663030d37614053a7c0369841403422 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:57.183202 21395 raft_consensus.cc:2258] T a663030d37614053a7c0369841403422 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:57.185722 18617 raft_consensus.cc:2454] T 9cc9251abd4e40d59a2bd2d0fada6165 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:57.186013 18983 ts_tablet_manager.cc:1398] T 4fa9575d96a84bef876467e21dada803 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240430 20:11:57.186218 18696 leader_election.cc:302] T 9cc9251abd4e40d59a2bd2d0fada6165 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:57.186353 18983 tablet_bootstrap.cc:492] T e848539446a64a968b37045d367bb3ce P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:57.186774 21493 raft_consensus.cc:2781] T 9cc9251abd4e40d59a2bd2d0fada6165 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:57.186899 21493 raft_consensus.cc:686] T 9cc9251abd4e40d59a2bd2d0fada6165 P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Becoming Leader. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Running, Role: LEADER
I20240430 20:11:57.187105 21395 ts_tablet_manager.cc:1875] T a663030d37614053a7c0369841403422 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:57.187129 21493 consensus_queue.cc:237] T 9cc9251abd4e40d59a2bd2d0fada6165 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:57.187785 18983 tablet_bootstrap.cc:654] T e848539446a64a968b37045d367bb3ce P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:57.182180 21396 raft_consensus.cc:2258] T 8901327b3efb4201adce61a9266142ea P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:57.191311 21396 ts_tablet_manager.cc:1875] T 8901327b3efb4201adce61a9266142ea P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:57.194785 21397 ts_tablet_manager.cc:1888] T 3e5225f1f5f544ecbf8dfa470af3daca P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240430 20:11:57.195375 21397 log.cc:1196] T 3e5225f1f5f544ecbf8dfa470af3daca P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/3e5225f1f5f544ecbf8dfa470af3daca
I20240430 20:11:57.195888 21397 ts_tablet_manager.cc:1909] T 3e5225f1f5f544ecbf8dfa470af3daca P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:57.198405 21395 ts_tablet_manager.cc:1888] T a663030d37614053a7c0369841403422 P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:57.206583 21748 tablet.cc:1722] T 8e33df70a1504dbd852fe0b3c3f61622 P 5e1c6397f09747f091534faf93542710: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240430 20:11:57.209698 19929 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:50746 (request call id 83) took 5042 ms. Trace:
I20240430 20:11:57.210213 21397 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56540 (request call id 76) took 3428 ms. Trace:
I20240430 20:11:57.209863 19929 rpcz_store.cc:276] 0430 20:11:52.167223 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:57.068669 (+4901446us) service_pool.cc:225] Handling call
0430 20:11:57.071100 (+ 2431us) catalog_manager.cc:2072] Acquired catalog manager lock
0430 20:11:57.071634 (+ 534us) catalog_manager.cc:2118] Created new table and tablet info
0430 20:11:57.144898 (+ 73264us) catalog_manager.cc:2178] Wrote table and tablets to system table
0430 20:11:57.209504 (+ 64606us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0430 20:11:57.209623 (+ 119us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:57.090711 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:57.090830 (+ 119us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:57.090836 (+ 6us) write_op.cc:427] Acquired schema lock
0430 20:11:57.090839 (+ 3us) tablet.cc:660] Decoding operations
0430 20:11:57.090938 (+ 99us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:57.090951 (+ 13us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:57.090956 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0430 20:11:57.091034 (+ 78us) tablet.cc:699] Row locks acquired
0430 20:11:57.091037 (+ 3us) write_op.cc:260] PREPARE: finished
0430 20:11:57.091079 (+ 42us) write_op.cc:270] Start()
0430 20:11:57.091111 (+ 32us) write_op.cc:276] Timestamp: P: 1714507917091031 usec, L: 0
0430 20:11:57.091115 (+ 4us) op_driver.cc:331] REPLICATION: starting
0430 20:11:57.091634 (+ 519us) log.cc:842] Serialized 433 byte log entry
0430 20:11:57.100549 (+ 8915us) op_driver.cc:444] REPLICATION: finished
0430 20:11:57.100847 (+ 298us) write_op.cc:301] APPLY: starting
0430 20:11:57.100862 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:57.100938 (+ 76us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:57.100975 (+ 37us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:57.101096 (+ 121us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:57.101130 (+ 34us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0430 20:11:57.101138 (+ 8us) write_op.cc:312] APPLY: finished
0430 20:11:57.101213 (+ 75us) log.cc:842] Serialized 29 byte log entry
0430 20:11:57.101271 (+ 58us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:57.101289 (+ 18us) write_op.cc:446] Released schema lock
0430 20:11:57.101295 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:57.101345 (+ 50us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:57.101349 (+ 4us) write_op.cc:446] Released schema lock
Metrics: {"spinlock_wait_cycles":1444736,"child_traces":[["op",{"apply.queue_time_us":249,"apply.run_cpu_time_us":549,"apply.run_wall_time_us":548,"num_ops":2,"prepare.queue_time_us":18626,"prepare.run_cpu_time_us":725,"prepare.run_wall_time_us":1009,"replication_time_us":9410}]]}
I20240430 20:11:57.210376 21397 rpcz_store.cc:276] 0430 20:11:53.781930 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:53.784888 (+ 2958us) service_pool.cc:225] Handling call
0430 20:11:57.129692 (+3344804us) ts_tablet_manager.cc:1120] Deleting tablet 3e5225f1f5f544ecbf8dfa470af3daca
0430 20:11:57.129704 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:57.143784 (+ 14080us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:57.194679 (+ 50895us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:57.210191 (+ 15512us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3344638}
I20240430 20:11:57.211135 21397 tablet_replica.cc:330] T 246fb63a5e9842df8263800e28a5935c P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:57.211392 21397 raft_consensus.cc:2229] T 246fb63a5e9842df8263800e28a5935c P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Raft consensus shutting down.
I20240430 20:11:57.211766 21397 raft_consensus.cc:2258] T 246fb63a5e9842df8263800e28a5935c P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:57.212416 21397 ts_tablet_manager.cc:1875] T 246fb63a5e9842df8263800e28a5935c P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:57.213836 21396 ts_tablet_manager.cc:1888] T 8901327b3efb4201adce61a9266142ea P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:57.206396 21438 raft_consensus.cc:482] T 4391b5227fb64ef481542a962a5e21a5 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:57.218856 21438 raft_consensus.cc:504] T 4391b5227fb64ef481542a962a5e21a5 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:57.219592 21438 leader_election.cc:288] T 4391b5227fb64ef481542a962a5e21a5 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:57.213817 18985 tablet_bootstrap.cc:492] T f9bf604e6c234349a9f72f45a4c4aef6 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:57.223402 18985 ts_tablet_manager.cc:1367] T f9bf604e6c234349a9f72f45a4c4aef6 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.052s user 0.002s sys 0.002s
I20240430 20:11:57.224939 18985 raft_consensus.cc:348] T f9bf604e6c234349a9f72f45a4c4aef6 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:57.225320 18985 raft_consensus.cc:374] T f9bf604e6c234349a9f72f45a4c4aef6 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:57.225510 18985 raft_consensus.cc:729] T f9bf604e6c234349a9f72f45a4c4aef6 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:57.225222 18468 raft_consensus.cc:3037] T 4391b5227fb64ef481542a962a5e21a5 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:57.225533 18475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cc9251abd4e40d59a2bd2d0fada6165" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:57.225976 21395 log.cc:1196] T a663030d37614053a7c0369841403422 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/a663030d37614053a7c0369841403422
I20240430 20:11:57.219964 18760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbb427aefa6b47b0805d1751fea4efcb" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:57.220508 18748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcc16db4d6be42edb7bd1ce18fe0190d" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:57.220788 18761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d34a450a91d542c98261456d1f635e9c" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:57.221339 18745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4391b5227fb64ef481542a962a5e21a5" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:57.226666 18748 raft_consensus.cc:2454] T fcc16db4d6be42edb7bd1ce18fe0190d P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 0.
I20240430 20:11:57.226801 18760 raft_consensus.cc:2454] T dbb427aefa6b47b0805d1751fea4efcb P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 0.
I20240430 20:11:57.226389 21395 ts_tablet_manager.cc:1909] T a663030d37614053a7c0369841403422 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:57.230968 21395 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41450 (request call id 77) took 3187 ms. Trace:
I20240430 20:11:57.231060 21395 rpcz_store.cc:276] 0430 20:11:54.043175 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:54.043238 (+ 63us) service_pool.cc:225] Handling call
0430 20:11:57.182290 (+3139052us) ts_tablet_manager.cc:1120] Deleting tablet a663030d37614053a7c0369841403422
0430 20:11:57.182302 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:57.189997 (+ 7695us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:57.198288 (+ 8291us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:57.230951 (+ 32663us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3130089}
I20240430 20:11:57.231376 21395 tablet_replica.cc:330] T fd1b8407071e4e4287fc172009777414 P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:57.231581 21395 raft_consensus.cc:2229] T fd1b8407071e4e4287fc172009777414 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:57.231566 19902 catalog_manager.cc:4830] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): tablet a663030d37614053a7c0369841403422 (table default.Table_e34613952dcb46618c3bc7cecb31fdaf [id=8f1ad1f8d0224b3daa5d84dc6916227b]) successfully deleted
I20240430 20:11:57.231715 21395 pending_rounds.cc:62] T fd1b8407071e4e4287fc172009777414 P 2efb542aa55e439b99ccb98d1c5c884a: Trying to abort 1 pending ops.
I20240430 20:11:57.231756 21395 pending_rounds.cc:68] T fd1b8407071e4e4287fc172009777414 P 2efb542aa55e439b99ccb98d1c5c884a: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7022624421809082368 op_type: NO_OP noop_request { }
I20240430 20:11:57.231855 21395 raft_consensus.cc:2866] T fd1b8407071e4e4287fc172009777414 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20240430 20:11:57.231958 21395 raft_consensus.cc:2258] T fd1b8407071e4e4287fc172009777414 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:57.231966 18988 tablet_bootstrap.cc:492] T a37d917bcc45468f9e3ee805640fba98 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:57.232107 18988 ts_tablet_manager.cc:1367] T a37d917bcc45468f9e3ee805640fba98 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.065s user 0.005s sys 0.001s
I20240430 20:11:57.233917 21395 ts_tablet_manager.cc:1875] T fd1b8407071e4e4287fc172009777414 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:57.234009 18988 raft_consensus.cc:348] T a37d917bcc45468f9e3ee805640fba98 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:57.234858 18988 raft_consensus.cc:374] T a37d917bcc45468f9e3ee805640fba98 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:57.235145 18988 raft_consensus.cc:729] T a37d917bcc45468f9e3ee805640fba98 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:57.225760 18985 consensus_queue.cc:260] T f9bf604e6c234349a9f72f45a4c4aef6 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:57.235862 18988 consensus_queue.cc:260] T a37d917bcc45468f9e3ee805640fba98 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:57.237730 18985 ts_tablet_manager.cc:1398] T f9bf604e6c234349a9f72f45a4c4aef6 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.014s user 0.001s sys 0.001s
I20240430 20:11:57.237871 18988 ts_tablet_manager.cc:1398] T a37d917bcc45468f9e3ee805640fba98 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.005s user 0.002s sys 0.002s
I20240430 20:11:57.238087 18985 tablet_bootstrap.cc:492] T 94fbd4a57a1e438891fcb456f6b40be3 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:57.238452 18988 tablet_bootstrap.cc:492] T d34a450a91d542c98261456d1f635e9c P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:57.239758 18985 tablet_bootstrap.cc:654] T 94fbd4a57a1e438891fcb456f6b40be3 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:57.248221 18473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cc9251abd4e40d59a2bd2d0fada6165" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
W20240430 20:11:57.222391 18557 leader_election.cc:341] T 798a13f545fd4d8f8c654eaa2452eda1 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:57.248461 18557 leader_election.cc:302] T 798a13f545fd4d8f8c654eaa2452eda1 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4
W20240430 20:11:57.248945 18557 leader_election.cc:341] T ffd6e420e6d44c039832de38de001c1b P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:57.249063 18557 leader_election.cc:302] T ffd6e420e6d44c039832de38de001c1b P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4
W20240430 20:11:57.249408 18557 leader_election.cc:341] T 266fd23786bc435db5083370f3330ed0 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:57.249521 18557 leader_election.cc:302] T 266fd23786bc435db5083370f3330ed0 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:57.250176 20489 raft_consensus.cc:2726] T 798a13f545fd4d8f8c654eaa2452eda1 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:57.250319 20489 raft_consensus.cc:2726] T ffd6e420e6d44c039832de38de001c1b P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:57.250407 20489 raft_consensus.cc:2726] T 266fd23786bc435db5083370f3330ed0 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240430 20:11:57.251691 19900 catalog_manager.cc:4797] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): delete failed for tablet 1701a15b115741aea13f8327e3d26b34 because the tablet was not found. No further retry: Not found: Tablet not found: 1701a15b115741aea13f8327e3d26b34
I20240430 20:11:57.252378 19900 catalog_manager.cc:4830] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): tablet 283e5e73cbc04daba0de19f4541f9fd0 (table default.Table_849e6e38123a4d8cae466d841850d6f9 [id=c1c646a7c27e4d77af05d1ac92aa6ddc]) successfully deleted
I20240430 20:11:57.252853 21438 raft_consensus.cc:482] T 9e741b00c74d43b5b6243ba7003f8804 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:57.252892 19900 catalog_manager.cc:4830] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): tablet fdf53061efd74fa29a9d8208d1fd210e (table default.Table_3c8f9f7b22f64fb1bdc3f1550987233a [id=7fd6f8576f4c4cbc8ffcdb4ab95e15dc]) successfully deleted
I20240430 20:11:57.253003 21438 raft_consensus.cc:504] T 9e741b00c74d43b5b6243ba7003f8804 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:57.253389 19900 catalog_manager.cc:4830] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): tablet f3711610610542128c4a84e17bce4a3e (table default.Table_64d14df4e12146ecb85239e7ca03cf6e [id=c376119401934812aef62a3a18eeb368]) successfully deleted
I20240430 20:11:57.253648 21438 leader_election.cc:288] T 9e741b00c74d43b5b6243ba7003f8804 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:57.254359 19900 catalog_manager.cc:4830] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): tablet 3e5225f1f5f544ecbf8dfa470af3daca (table default.Table_c7c87a8e019944cab3ec99e9563ac01a [id=abf3765074eb4d7d9d2bcc4ab721e6bd]) successfully deleted
I20240430 20:11:57.222697 18983 tablet_bootstrap.cc:492] T e848539446a64a968b37045d367bb3ce P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:57.254909 18983 ts_tablet_manager.cc:1367] T e848539446a64a968b37045d367bb3ce P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.069s user 0.003s sys 0.000s
I20240430 20:11:57.255340 21450 raft_consensus.cc:482] T cdeef95b54ca42d38d25b57738d84ef1 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:57.255503 21450 raft_consensus.cc:504] T cdeef95b54ca42d38d25b57738d84ef1 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:57.255585 18983 raft_consensus.cc:348] T e848539446a64a968b37045d367bb3ce P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:57.255745 18983 raft_consensus.cc:374] T e848539446a64a968b37045d367bb3ce P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:57.255790 18983 raft_consensus.cc:729] T e848539446a64a968b37045d367bb3ce P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:57.255973 18983 consensus_queue.cc:260] T e848539446a64a968b37045d367bb3ce P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:57.256088 21450 leader_election.cc:288] T cdeef95b54ca42d38d25b57738d84ef1 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:57.222968 18617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4391b5227fb64ef481542a962a5e21a5" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:57.256929 18983 ts_tablet_manager.cc:1398] T e848539446a64a968b37045d367bb3ce P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240430 20:11:57.257048 18607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdeef95b54ca42d38d25b57738d84ef1" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:57.258162 18983 tablet_bootstrap.cc:492] T 1faa2f2c84184b129ba6e4d2857822f0 P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:57.258183 18607 raft_consensus.cc:2454] T cdeef95b54ca42d38d25b57738d84ef1 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:57.258971 18696 leader_election.cc:302] T cdeef95b54ca42d38d25b57738d84ef1 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:57.259486 21454 raft_consensus.cc:2781] T cdeef95b54ca42d38d25b57738d84ef1 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:57.259593 21454 raft_consensus.cc:482] T cdeef95b54ca42d38d25b57738d84ef1 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:57.259663 21454 raft_consensus.cc:3037] T cdeef95b54ca42d38d25b57738d84ef1 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:57.260324 18669 heartbeater.cc:347] Connected to a master server at 127.17.177.60:37225
I20240430 20:11:57.263610 21454 raft_consensus.cc:504] T cdeef95b54ca42d38d25b57738d84ef1 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:57.264796 18607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdeef95b54ca42d38d25b57738d84ef1" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:57.265024 18607 raft_consensus.cc:3037] T cdeef95b54ca42d38d25b57738d84ef1 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:57.266361 18983 tablet_bootstrap.cc:654] T 1faa2f2c84184b129ba6e4d2857822f0 P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:57.267611 18607 raft_consensus.cc:2454] T cdeef95b54ca42d38d25b57738d84ef1 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:57.268049 21454 leader_election.cc:288] T cdeef95b54ca42d38d25b57738d84ef1 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Requested vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:57.268214 18696 leader_election.cc:302] T cdeef95b54ca42d38d25b57738d84ef1 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:57.269071 21436 raft_consensus.cc:2781] T cdeef95b54ca42d38d25b57738d84ef1 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:57.269178 21436 raft_consensus.cc:686] T cdeef95b54ca42d38d25b57738d84ef1 P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Becoming Leader. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Running, Role: LEADER
I20240430 20:11:57.269403 21436 consensus_queue.cc:237] T cdeef95b54ca42d38d25b57738d84ef1 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:57.270498 21707 raft_consensus.cc:482] T 17ddcaf971154a52b606bbf29a539bf0 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:57.270620 21707 raft_consensus.cc:504] T 17ddcaf971154a52b606bbf29a539bf0 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:57.270730 18988 tablet_bootstrap.cc:654] T d34a450a91d542c98261456d1f635e9c P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:57.271116 21707 leader_election.cc:288] T 17ddcaf971154a52b606bbf29a539bf0 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:57.271595 21707 raft_consensus.cc:482] T e041756c962b4a9f97c2cb314e7d8c05 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:57.271692 21707 raft_consensus.cc:504] T e041756c962b4a9f97c2cb314e7d8c05 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:57.272138 21707 leader_election.cc:288] T e041756c962b4a9f97c2cb314e7d8c05 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:57.273284 18607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e041756c962b4a9f97c2cb314e7d8c05" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:57.273530 18607 raft_consensus.cc:2454] T e041756c962b4a9f97c2cb314e7d8c05 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:57.273986 18696 leader_election.cc:302] T e041756c962b4a9f97c2cb314e7d8c05 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:57.274303 21493 raft_consensus.cc:2781] T e041756c962b4a9f97c2cb314e7d8c05 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:57.274389 21493 raft_consensus.cc:482] T e041756c962b4a9f97c2cb314e7d8c05 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:57.274456 21493 raft_consensus.cc:3037] T e041756c962b4a9f97c2cb314e7d8c05 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:57.277051 21493 raft_consensus.cc:504] T e041756c962b4a9f97c2cb314e7d8c05 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:57.277495 21396 log.cc:1196] T 8901327b3efb4201adce61a9266142ea P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/8901327b3efb4201adce61a9266142ea
I20240430 20:11:57.277617 21493 leader_election.cc:288] T e041756c962b4a9f97c2cb314e7d8c05 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Requested vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:57.278025 21396 ts_tablet_manager.cc:1909] T 8901327b3efb4201adce61a9266142ea P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:57.278373 21396 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:45934 (request call id 102) took 3340 ms. Trace:
I20240430 20:11:57.278479 21396 rpcz_store.cc:276] 0430 20:11:53.938031 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:53.938994 (+ 963us) service_pool.cc:225] Handling call
0430 20:11:57.159373 (+3220379us) ts_tablet_manager.cc:1120] Deleting tablet 8901327b3efb4201adce61a9266142ea
0430 20:11:57.159386 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:57.200484 (+ 41098us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:57.213720 (+ 13236us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:57.278349 (+ 64629us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3220231}
I20240430 20:11:57.279129 21396 tablet_replica.cc:330] T 246fb63a5e9842df8263800e28a5935c P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:57.279250 18607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e041756c962b4a9f97c2cb314e7d8c05" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:57.279414 21396 raft_consensus.cc:2229] T 246fb63a5e9842df8263800e28a5935c P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:57.279470 18607 raft_consensus.cc:3037] T e041756c962b4a9f97c2cb314e7d8c05 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:57.279610 21396 raft_consensus.cc:2258] T 246fb63a5e9842df8263800e28a5935c P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:57.280234 21396 ts_tablet_manager.cc:1875] T 246fb63a5e9842df8263800e28a5935c P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240430 20:11:57.280591 18699 leader_election.cc:341] T 3e5225f1f5f544ecbf8dfa470af3daca P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Not found: Tablet not found: 3e5225f1f5f544ecbf8dfa470af3daca
I20240430 20:11:57.280699 18699 leader_election.cc:302] T 3e5225f1f5f544ecbf8dfa470af3daca P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710
W20240430 20:11:57.280867 18699 raft_consensus.cc:2670] T 3e5225f1f5f544ecbf8dfa470af3daca P 3c51c33d5616482cafb68e26688820f4: unable to run election callback: Service unavailable: Thread pool token was shut down
I20240430 20:11:57.281488 19938 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 1faa2f2c84184b129ba6e4d2857822f0 (table default.Table_0dd55b391be94213b4af60ea6e5a979f [id=1a382ad8009b4522a50d55c307db8f96]). Expected version 1 got 0
W20240430 20:11:57.281701 18699 leader_election.cc:341] T 9cc9251abd4e40d59a2bd2d0fada6165 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
W20240430 20:11:57.282071 18699 leader_election.cc:341] T 9cc9251abd4e40d59a2bd2d0fada6165 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:57.282195 21450 consensus_queue.cc:1035] T 347a8224114b49e5b7dcb75d9b726e0a P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:57.282328 19938 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 266fd23786bc435db5083370f3330ed0 (table default.Table_cf60b652e9dc49e7a948ce66fd1cbffe [id=9734bb8dd05a400ca4244b3829f01dee]). Expected version 1 got 0
I20240430 20:11:57.282617 18760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb471d33b0784c9c9de1e5425cdd1fd7" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:57.282614 21450 consensus_queue.cc:1035] T 4391b5227fb64ef481542a962a5e21a5 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240430 20:11:57.285902 18619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17ddcaf971154a52b606bbf29a539bf0" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:57.286123 18619 raft_consensus.cc:2454] T 17ddcaf971154a52b606bbf29a539bf0 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:57.286789 18696 leader_election.cc:302] T 17ddcaf971154a52b606bbf29a539bf0 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:57.286988 19938 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet dd1de890e4394db287cf38bb411f5586 (table default.Table_84084a27f7504dd992fce07448c10df3 [id=6257559b0e9545c58848a76af0c5a5f7]). Expected version 2 got 0
I20240430 20:11:57.287524 21707 raft_consensus.cc:2781] T 17ddcaf971154a52b606bbf29a539bf0 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:57.287600 21707 raft_consensus.cc:482] T 17ddcaf971154a52b606bbf29a539bf0 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:57.287657 21707 raft_consensus.cc:3037] T 17ddcaf971154a52b606bbf29a539bf0 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:57.290262 21707 raft_consensus.cc:504] T 17ddcaf971154a52b606bbf29a539bf0 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:57.290333 18748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e28529e1c2ae4c7c9b9b0e4bd709d132" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:57.291062 19938 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet dac141d1b70c48d785caddd7182811ee (table default.Table_fc0a4540f98b4eea95099b8ab46ea235 [id=f3c53ccffe64410495cc8c379a05cfae]). Expected version 1 got 0
I20240430 20:11:57.292127 18619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e741b00c74d43b5b6243ba7003f8804" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:57.292308 18619 raft_consensus.cc:2454] T 9e741b00c74d43b5b6243ba7003f8804 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 0.
I20240430 20:11:57.292893 18988 tablet_bootstrap.cc:492] T d34a450a91d542c98261456d1f635e9c P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:57.293074 18988 ts_tablet_manager.cc:1367] T d34a450a91d542c98261456d1f635e9c P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.055s user 0.002s sys 0.002s
I20240430 20:11:57.293303 21707 leader_election.cc:288] T 17ddcaf971154a52b606bbf29a539bf0 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Requested vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:57.293798 18619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17ddcaf971154a52b606bbf29a539bf0" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:57.293956 18619 raft_consensus.cc:3037] T 17ddcaf971154a52b606bbf29a539bf0 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:57.293951 18988 raft_consensus.cc:348] T d34a450a91d542c98261456d1f635e9c P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:57.294190 18988 raft_consensus.cc:374] T d34a450a91d542c98261456d1f635e9c P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:57.294267 18988 raft_consensus.cc:729] T d34a450a91d542c98261456d1f635e9c P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:57.294504 18988 consensus_queue.cc:260] T d34a450a91d542c98261456d1f635e9c P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
W20240430 20:11:57.295559 18557 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.177.3:44867, user_credentials={real_user=slave}} blocked reactor thread for 45705.3us
W20240430 20:11:57.296545 18557 leader_election.cc:341] T b7141e215d6d44bc8406c80508190874 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:57.296777 18557 leader_election.cc:302] T b7141e215d6d44bc8406c80508190874 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:57.297048 20489 raft_consensus.cc:2726] T b7141e215d6d44bc8406c80508190874 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:57.297013 19938 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 330453ddeea6456e8ef93fa92608b0d3 (table default.Table_c1f36f7df5ae4e7c8eb8b4e99b89748b [id=a50abf28790d4284a76ac6ee22c02606]). Expected version 1 got 0
I20240430 20:11:57.297286 19938 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 1050ab528c7d41559831652a1437d358 (table default.Table_acddc4cbf6334546b5fb3e4752fef2a5 [id=274cb15fd751410e83327ac4e8dc4629]). Expected version 2 got 0
I20240430 20:11:57.296550 18607 raft_consensus.cc:2454] T e041756c962b4a9f97c2cb314e7d8c05 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:57.297209 18988 ts_tablet_manager.cc:1398] T d34a450a91d542c98261456d1f635e9c P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240430 20:11:57.298406 18696 leader_election.cc:302] T e041756c962b4a9f97c2cb314e7d8c05 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:57.297830 19895 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet 8901327b3efb4201adce61a9266142ea (table default.Table_2582fc0f97c1470d8b6b21cd1240697e [id=fc059935f9c943389180fab316c33a05]) successfully deleted
I20240430 20:11:57.298954 21707 raft_consensus.cc:2781] T e041756c962b4a9f97c2cb314e7d8c05 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:57.299028 19938 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 086b3dfa84c045a18c8c4f177dce606a (table default.Table_df19cb8a6f4d46699350ec5b69c14e5a [id=197923633fc54a4582207942b51530e2]). Expected version 2 got 0
I20240430 20:11:57.299176 21707 raft_consensus.cc:686] T e041756c962b4a9f97c2cb314e7d8c05 P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Becoming Leader. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Running, Role: LEADER
I20240430 20:11:57.299474 21707 consensus_queue.cc:237] T e041756c962b4a9f97c2cb314e7d8c05 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:57.300527 18988 tablet_bootstrap.cc:492] T 8e7f08e9ac274bfa86763eec895bbfda P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:57.301064 21397 ts_tablet_manager.cc:1888] T 246fb63a5e9842df8263800e28a5935c P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:57.301503 19938 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet e041756c962b4a9f97c2cb314e7d8c05 (table default.Table_de88cafd92b149f9b545369bff71033c [id=c4bf4806f3d84579b16a3038d8d982c3]). Expected version 1 got 0
I20240430 20:11:57.303345 18619 raft_consensus.cc:2454] T 17ddcaf971154a52b606bbf29a539bf0 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:57.306031 18415 leader_election.cc:302] T 9e741b00c74d43b5b6243ba7003f8804 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:57.310066 18473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdeef95b54ca42d38d25b57738d84ef1" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:57.311439 18473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdeef95b54ca42d38d25b57738d84ef1" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:57.313731 18473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17ddcaf971154a52b606bbf29a539bf0" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:57.313892 18473 raft_consensus.cc:2454] T 17ddcaf971154a52b606bbf29a539bf0 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:57.316537 18473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e041756c962b4a9f97c2cb314e7d8c05" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:57.316699 18473 raft_consensus.cc:2454] T e041756c962b4a9f97c2cb314e7d8c05 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:57.317562 19938 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 5df364628561475391c9a5e10ff5a05f (table default.Table_3c8f9f7b22f64fb1bdc3f1550987233a [id=7fd6f8576f4c4cbc8ffcdb4ab95e15dc]). Expected version 1 got 0
I20240430 20:11:57.306550 18696 leader_election.cc:302] T 17ddcaf971154a52b606bbf29a539bf0 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:57.319272 18473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e041756c962b4a9f97c2cb314e7d8c05" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:57.319440 18473 raft_consensus.cc:3037] T e041756c962b4a9f97c2cb314e7d8c05 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:57.319490 19938 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 821c55edb3414d72b1489eea97d3f73a (table default.Table_4d68c38d9ba7452a829f8318b2b267b8 [id=d06e3e9045b1429aaeebd1fd67d841de]). Expected version 1 got 0
I20240430 20:11:57.320124 19938 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet c4953578de5f45f3a6ba092a13357460 (table default.Table_e3cea4c635274d37a44427ca00897fb9 [id=a826c08c7eef44ec83feae3a6cb384e5]). Expected version 1 got 0
I20240430 20:11:57.321709 21757 raft_consensus.cc:2781] T 17ddcaf971154a52b606bbf29a539bf0 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:57.321915 21757 raft_consensus.cc:686] T 17ddcaf971154a52b606bbf29a539bf0 P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Becoming Leader. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Running, Role: LEADER
I20240430 20:11:57.322135 18473 raft_consensus.cc:2454] T e041756c962b4a9f97c2cb314e7d8c05 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:57.322292 21757 consensus_queue.cc:237] T 17ddcaf971154a52b606bbf29a539bf0 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:57.324178 21397 log.cc:1196] T 246fb63a5e9842df8263800e28a5935c P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/246fb63a5e9842df8263800e28a5935c
I20240430 20:11:57.324566 21397 ts_tablet_manager.cc:1909] T 246fb63a5e9842df8263800e28a5935c P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:57.325546 19938 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 37b2ea4be7494f0985808ade788dbe2b (table default.Table_5f6e0fce999b46de927f9271e5de5071 [id=0baad83a8c0f4aeebc7ddad068625a69]). Expected version 1 got 0
I20240430 20:11:57.327531 19938 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 9e8ab75e7d9c4b64a7b7dd2678569490 (table default.Table_b2d07520b9944a89ad62fca70396174e [id=1c8e0a3a44ec4a798770c0165ed3dffa]). Expected version 1 got 0
W20240430 20:11:57.327551 18557 leader_election.cc:341] T bb471d33b0784c9c9de1e5425cdd1fd7 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:57.327714 18557 leader_election.cc:302] T bb471d33b0784c9c9de1e5425cdd1fd7 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:57.328204 21752 raft_consensus.cc:2726] T bb471d33b0784c9c9de1e5425cdd1fd7 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:57.328800 19938 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet aaef85eea87f4443943fd3692bb2a38f (table default.Table_3da5b4c8668c470483ae2a0c7773eaaf [id=e292d818ee3c45df90522f08eb2d6227]). Expected version 1 got 0
I20240430 20:11:57.329983 18988 tablet_bootstrap.cc:654] T 8e7f08e9ac274bfa86763eec895bbfda P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:57.330469 21395 ts_tablet_manager.cc:1888] T fd1b8407071e4e4287fc172009777414 P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:57.333621 19938 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 36a07d27145a46e48c3f10e8211a389e (table default.Table_0d3f914f27df478c8df8cbb69bc0d8b4 [id=d4763eaeacb546ae94b77b5a78fef986]). Expected version 1 got 0
I20240430 20:11:57.333921 19938 catalog_manager.cc:5484] T 00be6ddd91e84865a48743fb4d077eec P 5e1c6397f09747f091534faf93542710 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e1c6397f09747f091534faf93542710 (127.17.177.2). New cstate: current_term: 1 leader_uuid: "5e1c6397f09747f091534faf93542710" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } health_report { overall_health: UNKNOWN } } }
I20240430 20:11:57.334219 18810 heartbeater.cc:503] Master 127.17.177.60:37225 was elected leader, sending a full tablet report...
I20240430 20:11:57.335340 18668 heartbeater.cc:503] Master 127.17.177.62:36441 was elected leader, sending a full tablet report...
I20240430 20:11:57.338728 19938 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 69f88532952a4591bf1577541c3430e6 (table default.Table_1177828cad1f4a24bd397619dad96170 [id=a95fed1bc3fa4865bf9a35dfc51b659c]). Expected version 2 got 0
I20240430 20:11:57.339407 21752 raft_consensus.cc:482] T 72205028d0654e69a6fa20f105604e1c P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:57.339640 19938 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 8e33df70a1504dbd852fe0b3c3f61622 (table default.Table_491c94cc9f9048928da775e9183132b2 [id=da22cb339ab647e8afda7aa92f3548a6]). Expected version 1 got 0
I20240430 20:11:57.339624 21752 raft_consensus.cc:504] T 72205028d0654e69a6fa20f105604e1c P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:57.340960 19938 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 7a0176f8a74c47efbbe3d3343b3dc896 (table default.Table_144e5402c1864d818436015b26648024 [id=73e5c9f0470d47849109819c063e1ce4]). Expected version 2 got 0
I20240430 20:11:57.341264 21752 leader_election.cc:288] T 72205028d0654e69a6fa20f105604e1c P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:57.342528 21396 ts_tablet_manager.cc:1888] T 246fb63a5e9842df8263800e28a5935c P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:57.345361 19938 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 94fbd4a57a1e438891fcb456f6b40be3 (table default.Table_25ddc3adef6b4ccf8b9c00cc6772031c [id=4fa623ffdcf347639150a355806038af]). Expected version 1 got 0
W20240430 20:11:57.344228 18557 leader_election.cc:341] T e28529e1c2ae4c7c9b9b0e4bd709d132 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:57.347591 18557 leader_election.cc:302] T e28529e1c2ae4c7c9b9b0e4bd709d132 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:57.347422 21396 log.cc:1196] T 246fb63a5e9842df8263800e28a5935c P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/246fb63a5e9842df8263800e28a5935c
I20240430 20:11:57.348187 21753 raft_consensus.cc:2726] T e28529e1c2ae4c7c9b9b0e4bd709d132 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:57.341251 18475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72205028d0654e69a6fa20f105604e1c" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:57.343791 18483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17ddcaf971154a52b606bbf29a539bf0" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:57.348937 18483 raft_consensus.cc:3037] T 17ddcaf971154a52b606bbf29a539bf0 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:57.349092 21395 log.cc:1196] T fd1b8407071e4e4287fc172009777414 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/fd1b8407071e4e4287fc172009777414
I20240430 20:11:57.349651 21395 ts_tablet_manager.cc:1909] T fd1b8407071e4e4287fc172009777414 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:57.349884 21396 ts_tablet_manager.cc:1909] T 246fb63a5e9842df8263800e28a5935c P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:57.350477 19938 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 5b26319488644d6a8b6be60460a5e7af (table default.Table_69892a8264a4477db48ee6248758e046 [id=5a1c4a7be2fc465fac89a50854eb3322]). Expected version 3 got 0
I20240430 20:11:57.351248 21395 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41450 (request call id 79) took 3298 ms. Trace:
I20240430 20:11:57.352037 18483 raft_consensus.cc:2454] T 17ddcaf971154a52b606bbf29a539bf0 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
W20240430 20:11:57.352344 18558 leader_election.cc:341] T 72205028d0654e69a6fa20f105604e1c P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:57.352437 21395 rpcz_store.cc:276] 0430 20:11:54.052938 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:54.053005 (+ 67us) service_pool.cc:225] Handling call
0430 20:11:57.231330 (+3178325us) ts_tablet_manager.cc:1120] Deleting tablet fd1b8407071e4e4287fc172009777414
0430 20:11:57.231339 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:57.327252 (+ 95913us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:57.330354 (+ 3102us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:57.351230 (+ 20876us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3178128}
I20240430 20:11:57.353793 21395 tablet_replica.cc:330] T 2fc6908ecf954020a676b9d10a2b00e7 P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:57.353672 19902 catalog_manager.cc:4830] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): tablet fd1b8407071e4e4287fc172009777414 (table default.Table_5572f0acadaa432e8570b159fd9b76bc [id=d0f3b1709ac94aed8ec568c0327eab34]) successfully deleted
I20240430 20:11:57.359627 19938 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 772eba5f24c549c791cad75565858285 (table default.Table_5fc77d5f60ae44f4bb4ca9f7f183672c [id=6ca223ca6354449c959d85ed2ac6a9c8]). Expected version 1 got 0
I20240430 20:11:57.360211 19938 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 2cd41b407d6b444d844c4c7c0ebf82e5 (table default.Table_6f8e0eff8f5a484dbd6a9c3fdb5d03f6 [id=3813b5e9778c4bd683a06598cef0d86f]). Expected version 1 got 0
I20240430 20:11:57.362816 19938 catalog_manager.cc:5484] T f8bfbf673ea346529d1b2b376a3da496 P 5e1c6397f09747f091534faf93542710 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e1c6397f09747f091534faf93542710 (127.17.177.2). New cstate: current_term: 1 leader_uuid: "5e1c6397f09747f091534faf93542710" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } health_report { overall_health: UNKNOWN } } }
I20240430 20:11:57.363538 21766 raft_consensus.cc:482] T 7683caba80594627900a0185f772c131 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5e1c6397f09747f091534faf93542710)
I20240430 20:11:57.363947 21766 raft_consensus.cc:504] T 7683caba80594627900a0185f772c131 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:57.365391 21766 leader_election.cc:288] T 7683caba80594627900a0185f772c131 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:57.366966 21766 raft_consensus.cc:2781] T 9e741b00c74d43b5b6243ba7003f8804 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:57.367211 19938 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet 17ddcaf971154a52b606bbf29a539bf0 (table default.Table_f2f38f5d69e34eaa9e3dbf24cfd1cdd6 [id=872c2faa4a464edd98a51df2e1b3f817]). Expected version 1 got 0
W20240430 20:11:57.367961 18699 consensus_peers.cc:485] T a663030d37614053a7c0369841403422 P 3c51c33d5616482cafb68e26688820f4 -> Peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Couldn't send request to peer 2efb542aa55e439b99ccb98d1c5c884a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a663030d37614053a7c0369841403422. This is attempt 1: this message will repeat every 5th retry.
I20240430 20:11:57.368156 19938 catalog_manager.cc:5576] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027) does not have the latest schema for tablet f1073fc792954c5795a05aa7d0ec24df (table default.Table_91417a5439c8457caa8bb0a12221a5dd [id=aedf11fda8914a3b843a57f8a7b69a53]). Expected version 2 got 0
W20240430 20:11:57.368368 18699 leader_election.cc:341] T cdeef95b54ca42d38d25b57738d84ef1 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
W20240430 20:11:57.368943 18699 leader_election.cc:341] T cdeef95b54ca42d38d25b57738d84ef1 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:57.374213 18983 tablet_bootstrap.cc:492] T 1faa2f2c84184b129ba6e4d2857822f0 P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:57.374366 18983 ts_tablet_manager.cc:1367] T 1faa2f2c84184b129ba6e4d2857822f0 P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.116s user 0.008s sys 0.012s
I20240430 20:11:57.375104 18983 raft_consensus.cc:348] T 1faa2f2c84184b129ba6e4d2857822f0 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:57.375278 18983 raft_consensus.cc:374] T 1faa2f2c84184b129ba6e4d2857822f0 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:57.375317 18983 raft_consensus.cc:729] T 1faa2f2c84184b129ba6e4d2857822f0 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:57.375499 18983 consensus_queue.cc:260] T 1faa2f2c84184b129ba6e4d2857822f0 P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:57.376675 18983 ts_tablet_manager.cc:1398] T 1faa2f2c84184b129ba6e4d2857822f0 P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240430 20:11:57.379592 18988 tablet_bootstrap.cc:492] T 8e7f08e9ac274bfa86763eec895bbfda P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:57.379776 18988 ts_tablet_manager.cc:1367] T 8e7f08e9ac274bfa86763eec895bbfda P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.080s user 0.005s sys 0.000s
I20240430 20:11:57.379963 21395 raft_consensus.cc:2229] T 2fc6908ecf954020a676b9d10a2b00e7 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:57.380131 21395 raft_consensus.cc:2258] T 2fc6908ecf954020a676b9d10a2b00e7 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:57.380645 21395 ts_tablet_manager.cc:1875] T 2fc6908ecf954020a676b9d10a2b00e7 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:57.381078 18988 raft_consensus.cc:348] T 8e7f08e9ac274bfa86763eec895bbfda P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:57.381529 18988 raft_consensus.cc:374] T 8e7f08e9ac274bfa86763eec895bbfda P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:57.381690 18988 raft_consensus.cc:729] T 8e7f08e9ac274bfa86763eec895bbfda P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:57.382437 21752 raft_consensus.cc:482] T 4fa9575d96a84bef876467e21dada803 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:57.382681 21752 raft_consensus.cc:504] T 4fa9575d96a84bef876467e21dada803 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:57.383993 21752 leader_election.cc:288] T 4fa9575d96a84bef876467e21dada803 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:57.385411 21396 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:45934 (request call id 100) took 3447 ms. Trace:
I20240430 20:11:57.385551 21396 rpcz_store.cc:276] 0430 20:11:53.937842 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:53.938266 (+ 424us) service_pool.cc:225] Handling call
0430 20:11:57.279046 (+3340780us) ts_tablet_manager.cc:1120] Deleting tablet 246fb63a5e9842df8263800e28a5935c
0430 20:11:57.279059 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:57.317228 (+ 38169us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:57.342423 (+ 25195us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:57.385391 (+ 42968us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3339718}
I20240430 20:11:57.385975 21396 tablet_replica.cc:330] T 90a694bd5db649afa50f3c4e752939c1 P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:57.386217 21396 raft_consensus.cc:2229] T 90a694bd5db649afa50f3c4e752939c1 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:57.386443 21396 raft_consensus.cc:2258] T 90a694bd5db649afa50f3c4e752939c1 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:57.386487 19895 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet 246fb63a5e9842df8263800e28a5935c (table default.Table_067c3b36dd2e4cfd828ab8ff6bbcbd2e [id=3a030fd1a2134783a02c9a01911e64e3]) successfully deleted
I20240430 20:11:57.387112 21396 ts_tablet_manager.cc:1875] T 90a694bd5db649afa50f3c4e752939c1 P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240430 20:11:57.387387 18417 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.177.3:44867, user_credentials={real_user=slave}} blocked reactor thread for 134389us
I20240430 20:11:57.390285 21438 consensus_queue.cc:1035] T e5b85a066cdc46d48ccf8201e5423c29 P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:57.382201 18988 consensus_queue.cc:260] T 8e7f08e9ac274bfa86763eec895bbfda P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:57.391858 18468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fa9575d96a84bef876467e21dada803" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
W20240430 20:11:57.392601 18558 leader_election.cc:341] T 4fa9575d96a84bef876467e21dada803 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:57.367547 21766 raft_consensus.cc:482] T 9e741b00c74d43b5b6243ba7003f8804 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:57.393667 21766 raft_consensus.cc:3037] T 9e741b00c74d43b5b6243ba7003f8804 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
W20240430 20:11:57.397048 18699 consensus_peers.cc:485] T fd1b8407071e4e4287fc172009777414 P 3c51c33d5616482cafb68e26688820f4 -> Peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Couldn't send request to peer 2efb542aa55e439b99ccb98d1c5c884a. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240430 20:11:57.398157 18988 ts_tablet_manager.cc:1398] T 8e7f08e9ac274bfa86763eec895bbfda P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.018s user 0.002s sys 0.000s
I20240430 20:11:57.399153 18749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72205028d0654e69a6fa20f105604e1c" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:57.400439 21396 ts_tablet_manager.cc:1888] T 90a694bd5db649afa50f3c4e752939c1 P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:57.400666 21396 log.cc:1196] T 90a694bd5db649afa50f3c4e752939c1 P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/90a694bd5db649afa50f3c4e752939c1
I20240430 20:11:57.402829 21395 ts_tablet_manager.cc:1888] T 2fc6908ecf954020a676b9d10a2b00e7 P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:57.404042 21395 log.cc:1196] T 2fc6908ecf954020a676b9d10a2b00e7 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/2fc6908ecf954020a676b9d10a2b00e7
I20240430 20:11:57.404523 21396 ts_tablet_manager.cc:1909] T 90a694bd5db649afa50f3c4e752939c1 P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:57.404953 21395 ts_tablet_manager.cc:1909] T 2fc6908ecf954020a676b9d10a2b00e7 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:57.406097 21396 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:45934 (request call id 97) took 3468 ms. Trace:
I20240430 20:11:57.406551 21395 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41450 (request call id 80) took 3352 ms. Trace:
I20240430 20:11:57.407058 21395 rpcz_store.cc:276] 0430 20:11:54.054521 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:54.054570 (+ 49us) service_pool.cc:225] Handling call
0430 20:11:57.353737 (+3299167us) ts_tablet_manager.cc:1120] Deleting tablet 2fc6908ecf954020a676b9d10a2b00e7
0430 20:11:57.353748 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:57.394115 (+ 40367us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:57.402713 (+ 8598us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:57.406499 (+ 3786us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":25446,"tablet-delete.queue_time_us":3298965}
I20240430 20:11:57.407917 19902 catalog_manager.cc:4830] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): tablet 2fc6908ecf954020a676b9d10a2b00e7 (table default.Table_766e3df58895427b9e5179eb555fe7e4 [id=5c11311047b34a3eb06cc06e8cc10223]) successfully deleted
I20240430 20:11:57.408119 21396 rpcz_store.cc:276] 0430 20:11:53.937513 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:53.939161 (+ 1648us) service_pool.cc:225] Handling call
0430 20:11:57.385916 (+3446755us) ts_tablet_manager.cc:1120] Deleting tablet 90a694bd5db649afa50f3c4e752939c1
0430 20:11:57.385938 (+ 22us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:57.390241 (+ 4303us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:57.400320 (+ 10079us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:57.406075 (+ 5755us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3444809}
I20240430 20:11:57.408501 21395 tablet_replica.cc:330] T df92eb85261d47d88cbf16d29d9ad30e P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:57.408874 21395 raft_consensus.cc:2229] T df92eb85261d47d88cbf16d29d9ad30e P 2efb542aa55e439b99ccb98d1c5c884a [term 2 LEADER]: Raft consensus shutting down.
W20240430 20:11:57.406734 18557 consensus_peers.cc:485] T f3711610610542128c4a84e17bce4a3e P 5e1c6397f09747f091534faf93542710 -> Peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Couldn't send request to peer 3c51c33d5616482cafb68e26688820f4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f3711610610542128c4a84e17bce4a3e. This is attempt 1: this message will repeat every 5th retry.
W20240430 20:11:57.409870 18557 leader_election.cc:341] T 72205028d0654e69a6fa20f105604e1c P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:57.410048 18557 leader_election.cc:302] T 72205028d0654e69a6fa20f105604e1c P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:57.410478 21753 raft_consensus.cc:2726] T 72205028d0654e69a6fa20f105604e1c P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:57.405298 18983 tablet_bootstrap.cc:492] T 5df364628561475391c9a5e10ff5a05f P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:57.410867 21177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b632d6ef4c104a058a6db208024cab33: Bootstrap replaying log segment 1/1 (135.9K/285.1K this segment, stats: ops{read=424 overwritten=1 applied=422 ignored=0} inserts{seen=380 ignored=0} mutations{seen=483 ignored=0} orphaned_commits=0)
I20240430 20:11:57.413076 18983 tablet_bootstrap.cc:654] T 5df364628561475391c9a5e10ff5a05f P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:57.414305 19895 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet 90a694bd5db649afa50f3c4e752939c1 (table default.Table_97a0991bf9284215b482c532e9dfc113 [id=7e3ac0868c0a4cc581427f121f305323]) successfully deleted
I20240430 20:11:57.415110 21396 tablet_replica.cc:330] T f3711610610542128c4a84e17bce4a3e P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:57.409662 21438 raft_consensus.cc:482] T 90a694bd5db649afa50f3c4e752939c1 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3c51c33d5616482cafb68e26688820f4)
I20240430 20:11:57.415776 21438 raft_consensus.cc:504] T 90a694bd5db649afa50f3c4e752939c1 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:57.416510 21438 leader_election.cc:288] T 90a694bd5db649afa50f3c4e752939c1 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
W20240430 20:11:57.411193 18415 leader_election.cc:341] T 7683caba80594627900a0185f772c131 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Not found: Tablet not found: 7683caba80594627900a0185f772c131
I20240430 20:11:57.411717 21747 tablet.cc:1722] T 8e33df70a1504dbd852fe0b3c3f61622 P 2efb542aa55e439b99ccb98d1c5c884a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240430 20:11:57.418074 21396 raft_consensus.cc:2229] T f3711610610542128c4a84e17bce4a3e P 5e1c6397f09747f091534faf93542710 [term 1 LEADER]: Raft consensus shutting down.
I20240430 20:11:57.418658 21395 raft_consensus.cc:2258] T df92eb85261d47d88cbf16d29d9ad30e P 2efb542aa55e439b99ccb98d1c5c884a [term 2 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:57.419623 18983 tablet_bootstrap.cc:492] T 5df364628561475391c9a5e10ff5a05f P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:57.419740 18985 tablet_bootstrap.cc:492] T 94fbd4a57a1e438891fcb456f6b40be3 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:57.419947 18985 ts_tablet_manager.cc:1367] T 94fbd4a57a1e438891fcb456f6b40be3 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.182s user 0.013s sys 0.005s
W20240430 20:11:57.421013 18417 leader_election.cc:341] T d34a450a91d542c98261456d1f635e9c P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:57.421898 18983 ts_tablet_manager.cc:1367] T 5df364628561475391c9a5e10ff5a05f P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.001s
I20240430 20:11:57.422704 18983 raft_consensus.cc:348] T 5df364628561475391c9a5e10ff5a05f P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:57.421669 21395 ts_tablet_manager.cc:1875] T df92eb85261d47d88cbf16d29d9ad30e P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:57.425611 21766 raft_consensus.cc:504] T 9e741b00c74d43b5b6243ba7003f8804 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:57.426856 18985 raft_consensus.cc:348] T 94fbd4a57a1e438891fcb456f6b40be3 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:57.421733 21396 raft_consensus.cc:2258] T f3711610610542128c4a84e17bce4a3e P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:57.427095 18985 raft_consensus.cc:374] T 94fbd4a57a1e438891fcb456f6b40be3 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:57.427212 18985 raft_consensus.cc:729] T 94fbd4a57a1e438891fcb456f6b40be3 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:57.427021 18983 raft_consensus.cc:374] T 5df364628561475391c9a5e10ff5a05f P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:57.427912 18983 raft_consensus.cc:729] T 5df364628561475391c9a5e10ff5a05f P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:57.427707 21396 ts_tablet_manager.cc:1875] T f3711610610542128c4a84e17bce4a3e P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:57.428051 18985 consensus_queue.cc:260] T 94fbd4a57a1e438891fcb456f6b40be3 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:57.428440 21766 leader_election.cc:288] T 9e741b00c74d43b5b6243ba7003f8804 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:57.428788 18985 ts_tablet_manager.cc:1398] T 94fbd4a57a1e438891fcb456f6b40be3 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
W20240430 20:11:57.429119 18415 leader_election.cc:341] T 90a694bd5db649afa50f3c4e752939c1 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Not found: Tablet not found: 90a694bd5db649afa50f3c4e752939c1
I20240430 20:11:57.430480 18985 tablet_bootstrap.cc:492] T 9a76db2c7ff146758c54908e428a0e02 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:57.430583 18983 consensus_queue.cc:260] T 5df364628561475391c9a5e10ff5a05f P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:57.430727 18606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e741b00c74d43b5b6243ba7003f8804" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:57.431016 18606 raft_consensus.cc:3037] T 9e741b00c74d43b5b6243ba7003f8804 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:57.431489 18983 ts_tablet_manager.cc:1398] T 5df364628561475391c9a5e10ff5a05f P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240430 20:11:57.433600 18606 raft_consensus.cc:2454] T 9e741b00c74d43b5b6243ba7003f8804 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 1.
I20240430 20:11:57.436208 18983 tablet_bootstrap.cc:492] T 716c62f14ca74da9af3282128441b44f P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:57.437057 18415 leader_election.cc:302] T 9e741b00c74d43b5b6243ba7003f8804 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:57.437721 21766 raft_consensus.cc:2781] T 9e741b00c74d43b5b6243ba7003f8804 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:57.438165 21766 raft_consensus.cc:686] T 9e741b00c74d43b5b6243ba7003f8804 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Becoming Leader. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Running, Role: LEADER
I20240430 20:11:57.438642 21766 consensus_queue.cc:237] T 9e741b00c74d43b5b6243ba7003f8804 P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:57.442003 18983 tablet_bootstrap.cc:654] T 716c62f14ca74da9af3282128441b44f P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:57.425459 18417 leader_election.cc:302] T d34a450a91d542c98261456d1f635e9c P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a; no voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710
I20240430 20:11:57.443079 21438 raft_consensus.cc:2726] T d34a450a91d542c98261456d1f635e9c P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:57.443770 18417 leader_election.cc:302] T 4391b5227fb64ef481542a962a5e21a5 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a; no voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710
I20240430 20:11:57.444139 21438 raft_consensus.cc:2726] T 4391b5227fb64ef481542a962a5e21a5 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:57.444569 18417 leader_election.cc:302] T fcc16db4d6be42edb7bd1ce18fe0190d P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters: 5e1c6397f09747f091534faf93542710
I20240430 20:11:57.444979 21438 raft_consensus.cc:2781] T fcc16db4d6be42edb7bd1ce18fe0190d P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:57.445091 21438 raft_consensus.cc:482] T fcc16db4d6be42edb7bd1ce18fe0190d P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:57.445201 21438 raft_consensus.cc:3037] T fcc16db4d6be42edb7bd1ce18fe0190d P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:57.445310 18985 tablet_bootstrap.cc:654] T 9a76db2c7ff146758c54908e428a0e02 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:57.448604 21438 raft_consensus.cc:504] T fcc16db4d6be42edb7bd1ce18fe0190d P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:57.449263 21438 leader_election.cc:288] T fcc16db4d6be42edb7bd1ce18fe0190d P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:57.449587 21395 ts_tablet_manager.cc:1888] T df92eb85261d47d88cbf16d29d9ad30e P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.1
I20240430 20:11:57.449800 21395 log.cc:1196] T df92eb85261d47d88cbf16d29d9ad30e P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/df92eb85261d47d88cbf16d29d9ad30e
I20240430 20:11:57.450647 18760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fa9575d96a84bef876467e21dada803" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
W20240430 20:11:57.450829 18415 leader_election.cc:341] T fcc16db4d6be42edb7bd1ce18fe0190d P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Not found: Tablet not found: fcc16db4d6be42edb7bd1ce18fe0190d
I20240430 20:11:57.451567 21395 ts_tablet_manager.cc:1909] T df92eb85261d47d88cbf16d29d9ad30e P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:57.451867 21395 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41450 (request call id 81) took 3396 ms. Trace:
W20240430 20:11:57.453287 18557 leader_election.cc:341] T 4fa9575d96a84bef876467e21dada803 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:57.453397 18557 leader_election.cc:302] T 4fa9575d96a84bef876467e21dada803 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:57.453756 21752 raft_consensus.cc:2726] T 4fa9575d96a84bef876467e21dada803 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:57.455472 21707 raft_consensus.cc:482] T 8e7f08e9ac274bfa86763eec895bbfda P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:57.455602 21707 raft_consensus.cc:504] T 8e7f08e9ac274bfa86763eec895bbfda P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:57.456202 21707 leader_election.cc:288] T 8e7f08e9ac274bfa86763eec895bbfda P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:57.457254 18607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e7f08e9ac274bfa86763eec895bbfda" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:57.457476 18607 raft_consensus.cc:2454] T 8e7f08e9ac274bfa86763eec895bbfda P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:57.457826 18696 leader_election.cc:302] T 8e7f08e9ac274bfa86763eec895bbfda P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:57.458138 21707 raft_consensus.cc:2781] T 8e7f08e9ac274bfa86763eec895bbfda P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:57.458210 21707 raft_consensus.cc:482] T 8e7f08e9ac274bfa86763eec895bbfda P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:57.458266 21707 raft_consensus.cc:3037] T 8e7f08e9ac274bfa86763eec895bbfda P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:57.458271 21396 ts_tablet_manager.cc:1888] T f3711610610542128c4a84e17bce4a3e P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:57.458364 21396 log.cc:1196] T f3711610610542128c4a84e17bce4a3e P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/f3711610610542128c4a84e17bce4a3e
I20240430 20:11:57.458712 21397 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56540 (request call id 82) took 3674 ms. Trace:
I20240430 20:11:57.458798 21397 rpcz_store.cc:276] 0430 20:11:53.784340 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:53.785084 (+ 744us) service_pool.cc:225] Handling call
0430 20:11:57.211085 (+3426001us) ts_tablet_manager.cc:1120] Deleting tablet 246fb63a5e9842df8263800e28a5935c
0430 20:11:57.211094 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:57.229936 (+ 18842us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:57.300955 (+ 71019us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:57.458691 (+157736us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3425879}
I20240430 20:11:57.459337 19900 catalog_manager.cc:4830] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): tablet 246fb63a5e9842df8263800e28a5935c (table default.Table_067c3b36dd2e4cfd828ab8ff6bbcbd2e [id=3a030fd1a2134783a02c9a01911e64e3]) successfully deleted
I20240430 20:11:57.459683 21397 tablet_replica.cc:330] T e4cffa295d8c401aa8415ce7e8f56526 P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:57.459892 21397 raft_consensus.cc:2229] T e4cffa295d8c401aa8415ce7e8f56526 P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Raft consensus shutting down.
I20240430 20:11:57.460330 21397 raft_consensus.cc:2258] T e4cffa295d8c401aa8415ce7e8f56526 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:57.460881 21397 ts_tablet_manager.cc:1875] T e4cffa295d8c401aa8415ce7e8f56526 P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:57.461589 21707 raft_consensus.cc:504] T 8e7f08e9ac274bfa86763eec895bbfda P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:57.462584 18983 tablet_bootstrap.cc:492] T 716c62f14ca74da9af3282128441b44f P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:57.462805 21707 leader_election.cc:288] T 8e7f08e9ac274bfa86763eec895bbfda P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Requested vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:57.462760 18606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e7f08e9ac274bfa86763eec895bbfda" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:57.462939 18606 raft_consensus.cc:3037] T 8e7f08e9ac274bfa86763eec895bbfda P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:57.463040 21396 ts_tablet_manager.cc:1909] T f3711610610542128c4a84e17bce4a3e P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:57.463670 18468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e7f08e9ac274bfa86763eec895bbfda" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:57.463896 18468 raft_consensus.cc:3037] T 8e7f08e9ac274bfa86763eec895bbfda P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:57.465560 18606 raft_consensus.cc:2454] T 8e7f08e9ac274bfa86763eec895bbfda P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:57.465936 18696 leader_election.cc:302] T 8e7f08e9ac274bfa86763eec895bbfda P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:57.454551 18988 tablet_bootstrap.cc:492] T 94fbd4a57a1e438891fcb456f6b40be3 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:57.466342 21707 raft_consensus.cc:2781] T 8e7f08e9ac274bfa86763eec895bbfda P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:57.466526 21707 raft_consensus.cc:686] T 8e7f08e9ac274bfa86763eec895bbfda P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Becoming Leader. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Running, Role: LEADER
I20240430 20:11:57.466801 18473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e7f08e9ac274bfa86763eec895bbfda" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:57.466840 21707 consensus_queue.cc:237] T 8e7f08e9ac274bfa86763eec895bbfda P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:57.466557 18468 raft_consensus.cc:2454] T 8e7f08e9ac274bfa86763eec895bbfda P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:57.462826 18983 ts_tablet_manager.cc:1367] T 716c62f14ca74da9af3282128441b44f P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.029s user 0.005s sys 0.000s
I20240430 20:11:57.468256 18988 tablet_bootstrap.cc:654] T 94fbd4a57a1e438891fcb456f6b40be3 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:57.469091 18983 raft_consensus.cc:348] T 716c62f14ca74da9af3282128441b44f P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:57.469974 19941 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 266fd23786bc435db5083370f3330ed0 (table default.Table_cf60b652e9dc49e7a948ce66fd1cbffe [id=9734bb8dd05a400ca4244b3829f01dee]). Expected version 1 got 0
I20240430 20:11:57.451969 21395 rpcz_store.cc:276] 0430 20:11:54.055304 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:54.055353 (+ 49us) service_pool.cc:225] Handling call
0430 20:11:57.408014 (+3352661us) ts_tablet_manager.cc:1120] Deleting tablet df92eb85261d47d88cbf16d29d9ad30e
0430 20:11:57.408025 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:57.437660 (+ 29635us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:57.449456 (+ 11796us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:57.451835 (+ 2379us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3352489}
I20240430 20:11:57.470454 18983 raft_consensus.cc:374] T 716c62f14ca74da9af3282128441b44f P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:57.470772 18983 raft_consensus.cc:729] T 716c62f14ca74da9af3282128441b44f P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:57.471071 21395 tablet_replica.cc:330] T dad16ae02b8b4e29a724d8eec67bb4b8 P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:57.469966 21396 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:45934 (request call id 108) took 3531 ms. Trace:
I20240430 20:11:57.471320 21396 rpcz_store.cc:276] 0430 20:11:53.938881 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:53.967414 (+ 28533us) service_pool.cc:225] Handling call
0430 20:11:57.415061 (+3447647us) ts_tablet_manager.cc:1120] Deleting tablet f3711610610542128c4a84e17bce4a3e
0430 20:11:57.415071 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:57.435937 (+ 20866us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:57.458156 (+ 22219us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:57.469939 (+ 11783us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3447327}
I20240430 20:11:57.472018 21396 tablet_replica.cc:330] T a445fdfde01c4371a5e6b3daea5b00ab P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:57.472537 21396 raft_consensus.cc:2229] T a445fdfde01c4371a5e6b3daea5b00ab P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:57.472807 21395 raft_consensus.cc:2229] T dad16ae02b8b4e29a724d8eec67bb4b8 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Raft consensus shutting down.
I20240430 20:11:57.473217 21395 raft_consensus.cc:2258] T dad16ae02b8b4e29a724d8eec67bb4b8 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:57.473426 21396 raft_consensus.cc:2258] T a445fdfde01c4371a5e6b3daea5b00ab P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:57.477872 21396 ts_tablet_manager.cc:1875] T a445fdfde01c4371a5e6b3daea5b00ab P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:57.478634 21395 ts_tablet_manager.cc:1875] T dad16ae02b8b4e29a724d8eec67bb4b8 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:57.472203 18983 consensus_queue.cc:260] T 716c62f14ca74da9af3282128441b44f P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:57.480638 18983 ts_tablet_manager.cc:1398] T 716c62f14ca74da9af3282128441b44f P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20240430 20:11:57.470847 19902 catalog_manager.cc:4830] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): tablet df92eb85261d47d88cbf16d29d9ad30e (table default.Table_1755296c0a63478699d9b8387f606a8e [id=8322f4f67ad546849d72aca9a5e985cf]) successfully deleted
I20240430 20:11:57.471848 19895 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet f3711610610542128c4a84e17bce4a3e (table default.Table_64d14df4e12146ecb85239e7ca03cf6e [id=c376119401934812aef62a3a18eeb368]) successfully deleted
I20240430 20:11:57.481238 19941 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet dd1de890e4394db287cf38bb411f5586 (table default.Table_84084a27f7504dd992fce07448c10df3 [id=6257559b0e9545c58848a76af0c5a5f7]). Expected version 2 got 0
I20240430 20:11:57.482614 19941 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet dac141d1b70c48d785caddd7182811ee (table default.Table_fc0a4540f98b4eea95099b8ab46ea235 [id=f3c53ccffe64410495cc8c379a05cfae]). Expected version 1 got 0
I20240430 20:11:57.494825 19941 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 5b26319488644d6a8b6be60460a5e7af (table default.Table_69892a8264a4477db48ee6248758e046 [id=5a1c4a7be2fc465fac89a50854eb3322]). Expected version 3 got 0
I20240430 20:11:57.495471 19941 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 17ddcaf971154a52b606bbf29a539bf0 (table default.Table_f2f38f5d69e34eaa9e3dbf24cfd1cdd6 [id=872c2faa4a464edd98a51df2e1b3f817]). Expected version 1 got 0
I20240430 20:11:57.496165 21397 ts_tablet_manager.cc:1888] T e4cffa295d8c401aa8415ce7e8f56526 P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:57.496290 19941 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 1050ab528c7d41559831652a1437d358 (table default.Table_acddc4cbf6334546b5fb3e4752fef2a5 [id=274cb15fd751410e83327ac4e8dc4629]). Expected version 2 got 0
I20240430 20:11:57.496728 19941 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet e041756c962b4a9f97c2cb314e7d8c05 (table default.Table_de88cafd92b149f9b545369bff71033c [id=c4bf4806f3d84579b16a3038d8d982c3]). Expected version 1 got 0
I20240430 20:11:57.497612 19941 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 5df364628561475391c9a5e10ff5a05f (table default.Table_3c8f9f7b22f64fb1bdc3f1550987233a [id=7fd6f8576f4c4cbc8ffcdb4ab95e15dc]). Expected version 1 got 0
I20240430 20:11:57.500264 21397 log.cc:1196] T e4cffa295d8c401aa8415ce7e8f56526 P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/e4cffa295d8c401aa8415ce7e8f56526
I20240430 20:11:57.500727 19941 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet c4953578de5f45f3a6ba092a13357460 (table default.Table_e3cea4c635274d37a44427ca00897fb9 [id=a826c08c7eef44ec83feae3a6cb384e5]). Expected version 1 got 0
I20240430 20:11:57.501080 19941 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 821c55edb3414d72b1489eea97d3f73a (table default.Table_4d68c38d9ba7452a829f8318b2b267b8 [id=d06e3e9045b1429aaeebd1fd67d841de]). Expected version 1 got 0
I20240430 20:11:57.501902 19941 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 37b2ea4be7494f0985808ade788dbe2b (table default.Table_5f6e0fce999b46de927f9271e5de5071 [id=0baad83a8c0f4aeebc7ddad068625a69]). Expected version 1 got 0
I20240430 20:11:57.500164 18988 tablet_bootstrap.cc:492] T 94fbd4a57a1e438891fcb456f6b40be3 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:57.502486 18988 ts_tablet_manager.cc:1367] T 94fbd4a57a1e438891fcb456f6b40be3 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.048s user 0.003s sys 0.000s
I20240430 20:11:57.503065 18983 tablet_bootstrap.cc:492] T 9e98e20abc924873853d176455f3678e P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:57.503296 18988 raft_consensus.cc:348] T 94fbd4a57a1e438891fcb456f6b40be3 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:57.503556 18988 raft_consensus.cc:374] T 94fbd4a57a1e438891fcb456f6b40be3 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:57.503652 18988 raft_consensus.cc:729] T 94fbd4a57a1e438891fcb456f6b40be3 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:57.483600 18417 leader_election.cc:302] T dbb427aefa6b47b0805d1751fea4efcb P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters: 5e1c6397f09747f091534faf93542710
I20240430 20:11:57.503793 19941 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 9e8ab75e7d9c4b64a7b7dd2678569490 (table default.Table_b2d07520b9944a89ad62fca70396174e [id=1c8e0a3a44ec4a798770c0165ed3dffa]). Expected version 1 got 0
I20240430 20:11:57.503868 21764 raft_consensus.cc:2781] T dbb427aefa6b47b0805d1751fea4efcb P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:57.503952 21764 raft_consensus.cc:482] T dbb427aefa6b47b0805d1751fea4efcb P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:57.504005 21764 raft_consensus.cc:3037] T dbb427aefa6b47b0805d1751fea4efcb P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:57.503963 18988 consensus_queue.cc:260] T 94fbd4a57a1e438891fcb456f6b40be3 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:57.504544 18988 ts_tablet_manager.cc:1398] T 94fbd4a57a1e438891fcb456f6b40be3 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240430 20:11:57.498206 18810 heartbeater.cc:503] Master 127.17.177.60:37225 was elected leader, sending a full tablet report...
I20240430 20:11:57.498703 21766 raft_consensus.cc:482] T f9bf604e6c234349a9f72f45a4c4aef6 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:57.502236 21754 raft_consensus.cc:482] T 94fbd4a57a1e438891fcb456f6b40be3 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:57.504951 21766 raft_consensus.cc:504] T f9bf604e6c234349a9f72f45a4c4aef6 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:57.504966 21754 raft_consensus.cc:504] T 94fbd4a57a1e438891fcb456f6b40be3 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:57.505548 21766 leader_election.cc:288] T f9bf604e6c234349a9f72f45a4c4aef6 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:57.505558 21754 leader_election.cc:288] T 94fbd4a57a1e438891fcb456f6b40be3 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:57.506186 18606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9bf604e6c234349a9f72f45a4c4aef6" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:57.506407 18606 raft_consensus.cc:2454] T f9bf604e6c234349a9f72f45a4c4aef6 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 0.
I20240430 20:11:57.506502 18607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94fbd4a57a1e438891fcb456f6b40be3" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:57.506729 18607 raft_consensus.cc:2454] T 94fbd4a57a1e438891fcb456f6b40be3 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 0.
I20240430 20:11:57.507229 18415 leader_election.cc:302] T f9bf604e6c234349a9f72f45a4c4aef6 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:57.507289 19941 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet aaef85eea87f4443943fd3692bb2a38f (table default.Table_3da5b4c8668c470483ae2a0c7773eaaf [id=e292d818ee3c45df90522f08eb2d6227]). Expected version 1 got 0
I20240430 20:11:57.507800 21766 raft_consensus.cc:2781] T f9bf604e6c234349a9f72f45a4c4aef6 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:57.507889 21766 raft_consensus.cc:482] T f9bf604e6c234349a9f72f45a4c4aef6 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:57.507962 21766 raft_consensus.cc:3037] T f9bf604e6c234349a9f72f45a4c4aef6 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:57.508769 19941 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 36a07d27145a46e48c3f10e8211a389e (table default.Table_0d3f914f27df478c8df8cbb69bc0d8b4 [id=d4763eaeacb546ae94b77b5a78fef986]). Expected version 1 got 0
I20240430 20:11:57.506554 21764 raft_consensus.cc:504] T dbb427aefa6b47b0805d1751fea4efcb P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:57.510154 21764 leader_election.cc:288] T dbb427aefa6b47b0805d1751fea4efcb P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:57.510257 19941 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 086b3dfa84c045a18c8c4f177dce606a (table default.Table_df19cb8a6f4d46699350ec5b69c14e5a [id=197923633fc54a4582207942b51530e2]). Expected version 2 got 0
I20240430 20:11:57.510673 21766 raft_consensus.cc:504] T f9bf604e6c234349a9f72f45a4c4aef6 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:57.511054 19941 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 330453ddeea6456e8ef93fa92608b0d3 (table default.Table_c1f36f7df5ae4e7c8eb8b4e99b89748b [id=a50abf28790d4284a76ac6ee22c02606]). Expected version 1 got 0
I20240430 20:11:57.511395 21766 leader_election.cc:288] T f9bf604e6c234349a9f72f45a4c4aef6 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:57.511471 21397 ts_tablet_manager.cc:1909] T e4cffa295d8c401aa8415ce7e8f56526 P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:57.511837 18415 leader_election.cc:302] T 94fbd4a57a1e438891fcb456f6b40be3 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:57.512131 21766 raft_consensus.cc:2781] T 94fbd4a57a1e438891fcb456f6b40be3 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:57.512115 19941 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 69f88532952a4591bf1577541c3430e6 (table default.Table_1177828cad1f4a24bd397619dad96170 [id=a95fed1bc3fa4865bf9a35dfc51b659c]). Expected version 2 got 0
I20240430 20:11:57.512334 21766 raft_consensus.cc:482] T 94fbd4a57a1e438891fcb456f6b40be3 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:57.512428 21766 raft_consensus.cc:3037] T 94fbd4a57a1e438891fcb456f6b40be3 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:57.512509 19941 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 7a0176f8a74c47efbbe3d3343b3dc896 (table default.Table_144e5402c1864d818436015b26648024 [id=73e5c9f0470d47849109819c063e1ce4]). Expected version 2 got 0
I20240430 20:11:57.512826 19941 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 94fbd4a57a1e438891fcb456f6b40be3 (table default.Table_25ddc3adef6b4ccf8b9c00cc6772031c [id=4fa623ffdcf347639150a355806038af]). Expected version 1 got 0
I20240430 20:11:57.513263 19941 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 8e33df70a1504dbd852fe0b3c3f61622 (table default.Table_491c94cc9f9048928da775e9183132b2 [id=da22cb339ab647e8afda7aa92f3548a6]). Expected version 1 got 0
I20240430 20:11:57.514628 19941 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet f1073fc792954c5795a05aa7d0ec24df (table default.Table_91417a5439c8457caa8bb0a12221a5dd [id=aedf11fda8914a3b843a57f8a7b69a53]). Expected version 2 got 0
I20240430 20:11:57.515395 18985 tablet_bootstrap.cc:492] T 9a76db2c7ff146758c54908e428a0e02 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:57.515590 18985 ts_tablet_manager.cc:1367] T 9a76db2c7ff146758c54908e428a0e02 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.085s user 0.003s sys 0.002s
I20240430 20:11:57.517217 18985 raft_consensus.cc:348] T 9a76db2c7ff146758c54908e428a0e02 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:57.517467 18985 raft_consensus.cc:374] T 9a76db2c7ff146758c54908e428a0e02 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:57.517575 18985 raft_consensus.cc:729] T 9a76db2c7ff146758c54908e428a0e02 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:57.517655 18607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9bf604e6c234349a9f72f45a4c4aef6" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:57.517839 18607 raft_consensus.cc:3037] T f9bf604e6c234349a9f72f45a4c4aef6 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:57.517829 18985 consensus_queue.cc:260] T 9a76db2c7ff146758c54908e428a0e02 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:57.518457 18985 ts_tablet_manager.cc:1398] T 9a76db2c7ff146758c54908e428a0e02 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240430 20:11:57.518723 18985 tablet_bootstrap.cc:492] T a37d917bcc45468f9e3ee805640fba98 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:57.520026 18985 tablet_bootstrap.cc:654] T a37d917bcc45468f9e3ee805640fba98 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:57.520462 18607 raft_consensus.cc:2454] T f9bf604e6c234349a9f72f45a4c4aef6 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 1.
I20240430 20:11:57.522203 18983 tablet_bootstrap.cc:654] T 9e98e20abc924873853d176455f3678e P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:57.522655 18757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcc16db4d6be42edb7bd1ce18fe0190d" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:57.522852 18757 raft_consensus.cc:3037] T fcc16db4d6be42edb7bd1ce18fe0190d P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:57.523088 18762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9bf604e6c234349a9f72f45a4c4aef6" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:57.523329 18758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbb427aefa6b47b0805d1751fea4efcb" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:57.523495 18758 raft_consensus.cc:3037] T dbb427aefa6b47b0805d1751fea4efcb P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:57.524132 19941 catalog_manager.cc:5484] T e5b85a066cdc46d48ccf8201e5423c29 P 2efb542aa55e439b99ccb98d1c5c884a reported cstate change: term changed from 0 to 1, leader changed from <none> to 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1). New cstate: current_term: 1 leader_uuid: "2efb542aa55e439b99ccb98d1c5c884a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } health_report { overall_health: HEALTHY } } }
I20240430 20:11:57.525254 19941 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 2cd41b407d6b444d844c4c7c0ebf82e5 (table default.Table_6f8e0eff8f5a484dbd6a9c3fdb5d03f6 [id=3813b5e9778c4bd683a06598cef0d86f]). Expected version 1 got 0
I20240430 20:11:57.525514 18757 raft_consensus.cc:2454] T fcc16db4d6be42edb7bd1ce18fe0190d P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 1.
I20240430 20:11:57.525794 18753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94fbd4a57a1e438891fcb456f6b40be3" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:57.525923 18758 raft_consensus.cc:2454] T dbb427aefa6b47b0805d1751fea4efcb P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 1.
I20240430 20:11:57.522692 18763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9bf604e6c234349a9f72f45a4c4aef6" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:57.526203 18749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e741b00c74d43b5b6243ba7003f8804" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:57.522657 18755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e741b00c74d43b5b6243ba7003f8804" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:57.525943 18753 raft_consensus.cc:2454] T 94fbd4a57a1e438891fcb456f6b40be3 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 0.
W20240430 20:11:57.527217 18417 consensus_peers.cc:485] T 8901327b3efb4201adce61a9266142ea P 2efb542aa55e439b99ccb98d1c5c884a -> Peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Couldn't send request to peer 3c51c33d5616482cafb68e26688820f4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8901327b3efb4201adce61a9266142ea. This is attempt 1: this message will repeat every 5th retry.
W20240430 20:11:57.527514 18417 leader_election.cc:341] T 7683caba80594627900a0185f772c131 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Not found: Tablet not found: 7683caba80594627900a0185f772c131
I20240430 20:11:57.527616 18417 leader_election.cc:302] T 7683caba80594627900a0185f772c131 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a; no voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710
I20240430 20:11:57.528115 21438 raft_consensus.cc:2726] T 7683caba80594627900a0185f772c131 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240430 20:11:57.529151 18417 leader_election.cc:341] T 90a694bd5db649afa50f3c4e752939c1 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Not found: Tablet not found: 90a694bd5db649afa50f3c4e752939c1
I20240430 20:11:57.529356 18417 leader_election.cc:302] T 90a694bd5db649afa50f3c4e752939c1 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a; no voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710
I20240430 20:11:57.529659 21438 raft_consensus.cc:2726] T 90a694bd5db649afa50f3c4e752939c1 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240430 20:11:57.531318 18417 leader_election.cc:341] T f9bf604e6c234349a9f72f45a4c4aef6 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:57.531600 18417 leader_election.cc:302] T fcc16db4d6be42edb7bd1ce18fe0190d P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters: 5e1c6397f09747f091534faf93542710
I20240430 20:11:57.531947 21438 raft_consensus.cc:2781] T fcc16db4d6be42edb7bd1ce18fe0190d P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:57.532091 21438 raft_consensus.cc:686] T fcc16db4d6be42edb7bd1ce18fe0190d P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Becoming Leader. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Running, Role: LEADER
I20240430 20:11:57.532388 21438 consensus_queue.cc:237] T fcc16db4d6be42edb7bd1ce18fe0190d P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
W20240430 20:11:57.535307 18417 consensus_peers.cc:485] T a9f37f0f7ce8425f9b24699513d41d61 P 2efb542aa55e439b99ccb98d1c5c884a -> Peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Couldn't send request to peer 3c51c33d5616482cafb68e26688820f4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a9f37f0f7ce8425f9b24699513d41d61. This is attempt 6: this message will repeat every 5th retry.
I20240430 20:11:57.535733 18417 leader_election.cc:302] T dbb427aefa6b47b0805d1751fea4efcb P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters:
I20240430 20:11:57.536365 21754 raft_consensus.cc:2781] T dbb427aefa6b47b0805d1751fea4efcb P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:57.536489 21754 raft_consensus.cc:686] T dbb427aefa6b47b0805d1751fea4efcb P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Becoming Leader. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Running, Role: LEADER
I20240430 20:11:57.536765 21754 consensus_queue.cc:237] T dbb427aefa6b47b0805d1751fea4efcb P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:57.538529 21752 raft_consensus.cc:482] T 72d316f424214792ac1d5029ffb1b04a P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:57.538740 21752 raft_consensus.cc:504] T 72d316f424214792ac1d5029ffb1b04a P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:57.539511 21752 leader_election.cc:288] T 72d316f424214792ac1d5029ffb1b04a P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:57.540381 18750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72d316f424214792ac1d5029ffb1b04a" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
W20240430 20:11:57.540968 18557 leader_election.cc:341] T 72d316f424214792ac1d5029ffb1b04a P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:57.540928 18468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72d316f424214792ac1d5029ffb1b04a" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
W20240430 20:11:57.541529 18558 leader_election.cc:341] T 72d316f424214792ac1d5029ffb1b04a P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:57.541656 18558 leader_election.cc:302] T 72d316f424214792ac1d5029ffb1b04a P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:57.542193 21752 raft_consensus.cc:2726] T 72d316f424214792ac1d5029ffb1b04a P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240430 20:11:57.545209 18417 leader_election.cc:341] T f9bf604e6c234349a9f72f45a4c4aef6 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
W20240430 20:11:57.545621 18417 leader_election.cc:341] T 9e741b00c74d43b5b6243ba7003f8804 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
W20240430 20:11:57.545900 18417 leader_election.cc:341] T 9e741b00c74d43b5b6243ba7003f8804 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
W20240430 20:11:57.546753 18417 consensus_peers.cc:485] T f6cfd089f08b47409fb2a6cf4f6b0385 P 2efb542aa55e439b99ccb98d1c5c884a -> Peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Couldn't send request to peer 3c51c33d5616482cafb68e26688820f4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f6cfd089f08b47409fb2a6cf4f6b0385. This is attempt 6: this message will repeat every 5th retry.
I20240430 20:11:57.548645 19941 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 772eba5f24c549c791cad75565858285 (table default.Table_5fc77d5f60ae44f4bb4ca9f7f183672c [id=6ca223ca6354449c959d85ed2ac6a9c8]). Expected version 1 got 0
I20240430 20:11:57.548919 19941 catalog_manager.cc:5576] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217) does not have the latest schema for tablet 1faa2f2c84184b129ba6e4d2857822f0 (table default.Table_0dd55b391be94213b4af60ea6e5a979f [id=1a382ad8009b4522a50d55c307db8f96]). Expected version 1 got 0
I20240430 20:11:57.567472 21397 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56540 (request call id 84) took 3781 ms. Trace:
I20240430 20:11:57.567632 21397 rpcz_store.cc:276] 0430 20:11:53.785937 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:53.786104 (+ 167us) service_pool.cc:225] Handling call
0430 20:11:57.459635 (+3673531us) ts_tablet_manager.cc:1120] Deleting tablet e4cffa295d8c401aa8415ce7e8f56526
0430 20:11:57.459644 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:57.464568 (+ 4924us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:57.496044 (+ 31476us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:57.567441 (+ 71397us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3673384}
I20240430 20:11:57.568145 21397 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56540 (request call id 93) took 3252 ms. Trace:
I20240430 20:11:57.568226 21397 rpcz_store.cc:276] 0430 20:11:54.316148 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:54.316237 (+ 89us) service_pool.cc:225] Handling call
0430 20:11:57.567976 (+3251739us) ts_tablet_manager.cc:1120] Deleting tablet cefce2f2a392495bb71399ed75db938c
0430 20:11:57.567990 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:57.568127 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3249858}
I20240430 20:11:57.568531 21397 tablet_replica.cc:330] T c6e3d09402574f1e8fe9e4160c70a3ee P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:57.568790 21397 raft_consensus.cc:2229] T c6e3d09402574f1e8fe9e4160c70a3ee P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Raft consensus shutting down.
I20240430 20:11:57.569190 21397 raft_consensus.cc:2258] T c6e3d09402574f1e8fe9e4160c70a3ee P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:57.569801 21397 ts_tablet_manager.cc:1875] T c6e3d09402574f1e8fe9e4160c70a3ee P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:57.569561 18988 tablet_bootstrap.cc:492] T 9e741b00c74d43b5b6243ba7003f8804 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
W20240430 20:11:57.570324 18415 consensus_peers.cc:485] T eb76f60448774fe1967fffe7c6873cfc P 2efb542aa55e439b99ccb98d1c5c884a -> Peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Couldn't send request to peer 5e1c6397f09747f091534faf93542710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: eb76f60448774fe1967fffe7c6873cfc. This is attempt 1: this message will repeat every 5th retry.
I20240430 20:11:57.571434 21395 ts_tablet_manager.cc:1888] T dad16ae02b8b4e29a724d8eec67bb4b8 P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:57.572408 19900 catalog_manager.cc:4830] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): tablet e4cffa295d8c401aa8415ce7e8f56526 (table default.Table_81547db3074240c98cdd95ed3b2dd813 [id=e0ea901452334b36a8493baed478cc1a]) successfully deleted
W20240430 20:11:57.572690 19900 catalog_manager.cc:4810] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): delete failed for tablet cefce2f2a392495bb71399ed75db938c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cefce2f2a392495bb71399ed75db938c already in progress: creating tablet
I20240430 20:11:57.573030 21757 consensus_queue.cc:1035] T b7d8da6e2fe8499d9d53af84a25f2faf P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:57.574514 21757 raft_consensus.cc:482] T d34a450a91d542c98261456d1f635e9c P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:57.574653 21757 raft_consensus.cc:504] T d34a450a91d542c98261456d1f635e9c P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:57.575273 21757 leader_election.cc:288] T d34a450a91d542c98261456d1f635e9c P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:57.577862 21752 raft_consensus.cc:482] T eb7554e115d84e6ca4f05509c5f33e2a P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:57.577992 18988 tablet_bootstrap.cc:654] T 9e741b00c74d43b5b6243ba7003f8804 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:57.577997 21752 raft_consensus.cc:504] T eb7554e115d84e6ca4f05509c5f33e2a P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:57.578576 21752 leader_election.cc:288] T eb7554e115d84e6ca4f05509c5f33e2a P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:57.579226 18760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb7554e115d84e6ca4f05509c5f33e2a" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
W20240430 20:11:57.579497 18696 leader_election.cc:341] T d34a450a91d542c98261456d1f635e9c P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Not found: Tablet not found: d34a450a91d542c98261456d1f635e9c
W20240430 20:11:57.580292 18557 leader_election.cc:341] T eb7554e115d84e6ca4f05509c5f33e2a P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:57.580479 21757 raft_consensus.cc:482] T 9a76db2c7ff146758c54908e428a0e02 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:57.580616 21757 raft_consensus.cc:504] T 9a76db2c7ff146758c54908e428a0e02 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:57.581244 21757 leader_election.cc:288] T 9a76db2c7ff146758c54908e428a0e02 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:57.582314 18606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a76db2c7ff146758c54908e428a0e02" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710" is_pre_election: true
I20240430 20:11:57.582512 18606 raft_consensus.cc:2454] T 9a76db2c7ff146758c54908e428a0e02 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:57.582958 18696 leader_election.cc:302] T 9a76db2c7ff146758c54908e428a0e02 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:57.583220 21707 raft_consensus.cc:2781] T 9a76db2c7ff146758c54908e428a0e02 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:57.583317 21707 raft_consensus.cc:482] T 9a76db2c7ff146758c54908e428a0e02 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:57.583403 21707 raft_consensus.cc:3037] T 9a76db2c7ff146758c54908e428a0e02 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:57.586894 21707 raft_consensus.cc:504] T 9a76db2c7ff146758c54908e428a0e02 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:57.586987 21396 ts_tablet_manager.cc:1888] T a445fdfde01c4371a5e6b3daea5b00ab P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:57.587127 21396 log.cc:1196] T a445fdfde01c4371a5e6b3daea5b00ab P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/a445fdfde01c4371a5e6b3daea5b00ab
I20240430 20:11:57.587483 21396 ts_tablet_manager.cc:1909] T a445fdfde01c4371a5e6b3daea5b00ab P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:57.588291 21396 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:45934 (request call id 109) took 3620 ms. Trace:
I20240430 20:11:57.588378 21396 rpcz_store.cc:276] 0430 20:11:53.967807 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:53.967897 (+ 90us) service_pool.cc:225] Handling call
0430 20:11:57.471691 (+3503794us) ts_tablet_manager.cc:1120] Deleting tablet a445fdfde01c4371a5e6b3daea5b00ab
0430 20:11:57.471979 (+ 288us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:57.526881 (+ 54902us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:57.586862 (+ 59981us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:57.588270 (+ 1408us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3503503}
I20240430 20:11:57.588783 21396 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:45934 (request call id 117) took 3467 ms. Trace:
I20240430 20:11:57.588820 19895 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet a445fdfde01c4371a5e6b3daea5b00ab (table default.Table_83d417131d35409582550b7b8ce4a141 [id=8893c7824ec64fc1961e7c6851278a50]) successfully deleted
I20240430 20:11:57.588930 21396 rpcz_store.cc:276] 0430 20:11:54.121803 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:54.121843 (+ 40us) service_pool.cc:225] Handling call
0430 20:11:57.588657 (+3466814us) ts_tablet_manager.cc:1120] Deleting tablet cefce2f2a392495bb71399ed75db938c
0430 20:11:57.588667 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:57.588765 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3466316}
I20240430 20:11:57.589237 21396 tablet_replica.cc:330] T 3b7fdc6ec7e046c5af894f9a6eda17b0 P 5e1c6397f09747f091534faf93542710: stopping tablet replica
W20240430 20:11:57.589373 19895 catalog_manager.cc:4810] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): delete failed for tablet cefce2f2a392495bb71399ed75db938c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cefce2f2a392495bb71399ed75db938c already in progress: creating tablet
I20240430 20:11:57.589481 21396 raft_consensus.cc:2229] T 3b7fdc6ec7e046c5af894f9a6eda17b0 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:57.589637 21396 raft_consensus.cc:2258] T 3b7fdc6ec7e046c5af894f9a6eda17b0 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:57.589762 21707 leader_election.cc:288] T 9a76db2c7ff146758c54908e428a0e02 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Requested vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:57.590199 21396 ts_tablet_manager.cc:1875] T 3b7fdc6ec7e046c5af894f9a6eda17b0 P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:57.590138 18589 tablet_service.cc:1467] Processing CreateTablet for tablet 4e6ca13dc8eb473e9a764499ec444110 (DEFAULT_TABLE table=default.Table_ed5eedd0ae824a329615cc8cbaa90ab0 [id=cf47f4e403c04cbaa7e384875b56085b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:57.590608 18589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e6ca13dc8eb473e9a764499ec444110. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:57.591122 18606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a76db2c7ff146758c54908e428a0e02" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:57.591284 18606 raft_consensus.cc:3037] T 9a76db2c7ff146758c54908e428a0e02 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:57.593612 18606 raft_consensus.cc:2454] T 9a76db2c7ff146758c54908e428a0e02 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:57.594030 18696 leader_election.cc:302] T 9a76db2c7ff146758c54908e428a0e02 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4, 5e1c6397f09747f091534faf93542710; no voters:
I20240430 20:11:57.594305 21707 raft_consensus.cc:2781] T 9a76db2c7ff146758c54908e428a0e02 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:57.594429 21707 raft_consensus.cc:686] T 9a76db2c7ff146758c54908e428a0e02 P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Becoming Leader. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Running, Role: LEADER
I20240430 20:11:57.594703 21707 consensus_queue.cc:237] T 9a76db2c7ff146758c54908e428a0e02 P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:57.595959 18730 tablet_service.cc:1467] Processing CreateTablet for tablet 4e6ca13dc8eb473e9a764499ec444110 (DEFAULT_TABLE table=default.Table_ed5eedd0ae824a329615cc8cbaa90ab0 [id=cf47f4e403c04cbaa7e384875b56085b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:57.596319 21395 log.cc:1196] T dad16ae02b8b4e29a724d8eec67bb4b8 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/dad16ae02b8b4e29a724d8eec67bb4b8
I20240430 20:11:57.596784 21395 ts_tablet_manager.cc:1909] T dad16ae02b8b4e29a724d8eec67bb4b8 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:57.597219 21395 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41450 (request call id 78) took 3544 ms. Trace:
I20240430 20:11:57.597374 21395 rpcz_store.cc:276] 0430 20:11:54.052786 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:54.053069 (+ 283us) service_pool.cc:225] Handling call
0430 20:11:57.471004 (+3417935us) ts_tablet_manager.cc:1120] Deleting tablet dad16ae02b8b4e29a724d8eec67bb4b8
0430 20:11:57.471016 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:57.525358 (+ 54342us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:57.571322 (+ 45964us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:57.597196 (+ 25874us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3412338}
I20240430 20:11:57.597951 19902 catalog_manager.cc:4830] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): tablet dad16ae02b8b4e29a724d8eec67bb4b8 (table default.Table_8a6b5152d20b4890918aedb454764ee9 [id=121444fdfd2642a38cdbd3a6c411c4fd]) successfully deleted
I20240430 20:11:57.597998 21395 tablet_replica.cc:330] T aa9998f4e17147aa8f2b35c01980463a P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:57.598333 21395 raft_consensus.cc:2229] T aa9998f4e17147aa8f2b35c01980463a P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Raft consensus shutting down.
I20240430 20:11:57.598714 21395 raft_consensus.cc:2258] T aa9998f4e17147aa8f2b35c01980463a P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:57.599414 21395 ts_tablet_manager.cc:1875] T aa9998f4e17147aa8f2b35c01980463a P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:57.600332 19929 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 69f88532952a4591bf1577541c3430e6 (table default.Table_1177828cad1f4a24bd397619dad96170 [id=a95fed1bc3fa4865bf9a35dfc51b659c]). Expected version 2 got 0
W20240430 20:11:57.574071 18415 leader_election.cc:341] T dbb427aefa6b47b0805d1751fea4efcb P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Not found: Tablet not found: dbb427aefa6b47b0805d1751fea4efcb
I20240430 20:11:57.607264 21397 ts_tablet_manager.cc:1888] T c6e3d09402574f1e8fe9e4160c70a3ee P 3c51c33d5616482cafb68e26688820f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240430 20:11:57.607381 21397 log.cc:1196] T c6e3d09402574f1e8fe9e4160c70a3ee P 3c51c33d5616482cafb68e26688820f4: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-2/wal/wals/c6e3d09402574f1e8fe9e4160c70a3ee
I20240430 20:11:57.576186 18473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d34a450a91d542c98261456d1f635e9c" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:57.579941 18483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb7554e115d84e6ca4f05509c5f33e2a" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:57.582090 18475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a76db2c7ff146758c54908e428a0e02" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:57.607540 18473 raft_consensus.cc:2454] T d34a450a91d542c98261456d1f635e9c P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:57.607582 18475 raft_consensus.cc:2454] T 9a76db2c7ff146758c54908e428a0e02 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 0.
I20240430 20:11:57.608036 18415 leader_election.cc:302] T f9bf604e6c234349a9f72f45a4c4aef6 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710; no voters: 3c51c33d5616482cafb68e26688820f4
W20240430 20:11:57.608202 18558 leader_election.cc:341] T eb7554e115d84e6ca4f05509c5f33e2a P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:57.608345 21765 raft_consensus.cc:2781] T f9bf604e6c234349a9f72f45a4c4aef6 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:57.608330 18558 leader_election.cc:302] T eb7554e115d84e6ca4f05509c5f33e2a P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:57.608453 21765 raft_consensus.cc:686] T f9bf604e6c234349a9f72f45a4c4aef6 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Becoming Leader. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Running, Role: LEADER
I20240430 20:11:57.608444 19929 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet e041756c962b4a9f97c2cb314e7d8c05 (table default.Table_de88cafd92b149f9b545369bff71033c [id=c4bf4806f3d84579b16a3038d8d982c3]). Expected version 1 got 0
I20240430 20:11:57.608541 21753 raft_consensus.cc:2726] T eb7554e115d84e6ca4f05509c5f33e2a P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:57.608706 19929 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 17ddcaf971154a52b606bbf29a539bf0 (table default.Table_f2f38f5d69e34eaa9e3dbf24cfd1cdd6 [id=872c2faa4a464edd98a51df2e1b3f817]). Expected version 1 got 0
I20240430 20:11:57.608693 21765 consensus_queue.cc:237] T f9bf604e6c234349a9f72f45a4c4aef6 P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:57.608999 19929 catalog_manager.cc:5484] T b7d8da6e2fe8499d9d53af84a25f2faf P 3c51c33d5616482cafb68e26688820f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c51c33d5616482cafb68e26688820f4 (127.17.177.3). New cstate: current_term: 1 leader_uuid: "3c51c33d5616482cafb68e26688820f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } health_report { overall_health: UNKNOWN } } }
I20240430 20:11:57.608341 18699 leader_election.cc:302] T d34a450a91d542c98261456d1f635e9c P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters: 5e1c6397f09747f091534faf93542710
I20240430 20:11:57.609856 21757 raft_consensus.cc:2781] T d34a450a91d542c98261456d1f635e9c P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240430 20:11:57.609941 21757 raft_consensus.cc:482] T d34a450a91d542c98261456d1f635e9c P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240430 20:11:57.610009 21757 raft_consensus.cc:3037] T d34a450a91d542c98261456d1f635e9c P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:57.582316 18985 tablet_bootstrap.cc:492] T a37d917bcc45468f9e3ee805640fba98 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:57.588428 18474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a76db2c7ff146758c54908e428a0e02" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:57.613135 21757 raft_consensus.cc:504] T d34a450a91d542c98261456d1f635e9c P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:57.613337 18474 raft_consensus.cc:3037] T 9a76db2c7ff146758c54908e428a0e02 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:57.613478 18985 ts_tablet_manager.cc:1367] T a37d917bcc45468f9e3ee805640fba98 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.095s user 0.002s sys 0.003s
I20240430 20:11:57.614423 18473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d34a450a91d542c98261456d1f635e9c" candidate_uuid: "3c51c33d5616482cafb68e26688820f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a"
I20240430 20:11:57.614565 18473 raft_consensus.cc:3037] T d34a450a91d542c98261456d1f635e9c P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:57.614537 21757 leader_election.cc:288] T d34a450a91d542c98261456d1f635e9c P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Requested vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
W20240430 20:11:57.615010 18696 leader_election.cc:341] T d34a450a91d542c98261456d1f635e9c P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Not found: Tablet not found: d34a450a91d542c98261456d1f635e9c
I20240430 20:11:57.615777 19929 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 1050ab528c7d41559831652a1437d358 (table default.Table_acddc4cbf6334546b5fb3e4752fef2a5 [id=274cb15fd751410e83327ac4e8dc4629]). Expected version 2 got 0
I20240430 20:11:57.615927 18474 raft_consensus.cc:2454] T 9a76db2c7ff146758c54908e428a0e02 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:57.590480 18453 tablet_service.cc:1467] Processing CreateTablet for tablet 4e6ca13dc8eb473e9a764499ec444110 (DEFAULT_TABLE table=default.Table_ed5eedd0ae824a329615cc8cbaa90ab0 [id=cf47f4e403c04cbaa7e384875b56085b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240430 20:11:57.616214 19929 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 086b3dfa84c045a18c8c4f177dce606a (table default.Table_df19cb8a6f4d46699350ec5b69c14e5a [id=197923633fc54a4582207942b51530e2]). Expected version 2 got 0
I20240430 20:11:57.617133 18473 raft_consensus.cc:2454] T d34a450a91d542c98261456d1f635e9c P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c51c33d5616482cafb68e26688820f4 in term 1.
I20240430 20:11:57.593277 21766 raft_consensus.cc:504] T 94fbd4a57a1e438891fcb456f6b40be3 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:57.617547 19929 catalog_manager.cc:5576] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867) does not have the latest schema for tablet 330453ddeea6456e8ef93fa92608b0d3 (table default.Table_c1f36f7df5ae4e7c8eb8b4e99b89748b [id=a50abf28790d4284a76ac6ee22c02606]). Expected version 1 got 0
I20240430 20:11:57.617517 18699 leader_election.cc:302] T d34a450a91d542c98261456d1f635e9c P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters: 5e1c6397f09747f091534faf93542710
I20240430 20:11:57.617923 21436 raft_consensus.cc:2781] T d34a450a91d542c98261456d1f635e9c P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:57.618046 21436 raft_consensus.cc:686] T d34a450a91d542c98261456d1f635e9c P 3c51c33d5616482cafb68e26688820f4 [term 1 LEADER]: Becoming Leader. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Running, Role: LEADER
I20240430 20:11:57.618276 21436 consensus_queue.cc:237] T d34a450a91d542c98261456d1f635e9c P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:57.614153 18985 raft_consensus.cc:348] T a37d917bcc45468f9e3ee805640fba98 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:57.618485 18985 raft_consensus.cc:374] T a37d917bcc45468f9e3ee805640fba98 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:57.618716 21766 leader_election.cc:288] T 94fbd4a57a1e438891fcb456f6b40be3 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Requested vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027)
I20240430 20:11:57.618726 18985 raft_consensus.cc:729] T a37d917bcc45468f9e3ee805640fba98 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:57.619396 18985 consensus_queue.cc:260] T a37d917bcc45468f9e3ee805640fba98 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:57.620223 18985 ts_tablet_manager.cc:1398] T a37d917bcc45468f9e3ee805640fba98 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20240430 20:11:57.620513 18760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94fbd4a57a1e438891fcb456f6b40be3" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4"
I20240430 20:11:57.620784 18760 raft_consensus.cc:3037] T 94fbd4a57a1e438891fcb456f6b40be3 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:57.624212 18760 raft_consensus.cc:2454] T 94fbd4a57a1e438891fcb456f6b40be3 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 1.
W20240430 20:11:57.622892 18415 consensus_peers.cc:485] T a9f37f0f7ce8425f9b24699513d41d61 P 2efb542aa55e439b99ccb98d1c5c884a -> Peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Couldn't send request to peer 5e1c6397f09747f091534faf93542710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a9f37f0f7ce8425f9b24699513d41d61. This is attempt 6: this message will repeat every 5th retry.
I20240430 20:11:57.624760 18417 leader_election.cc:302] T 94fbd4a57a1e438891fcb456f6b40be3 P 2efb542aa55e439b99ccb98d1c5c884a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4; no voters:
I20240430 20:11:57.625797 21581 raft_consensus.cc:2781] T 94fbd4a57a1e438891fcb456f6b40be3 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Leader election won for term 1
I20240430 20:11:57.625944 21581 raft_consensus.cc:686] T 94fbd4a57a1e438891fcb456f6b40be3 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 LEADER]: Becoming Leader. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Running, Role: LEADER
I20240430 20:11:57.626202 21581 consensus_queue.cc:237] T 94fbd4a57a1e438891fcb456f6b40be3 P 2efb542aa55e439b99ccb98d1c5c884a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:57.626909 18730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e6ca13dc8eb473e9a764499ec444110. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:57.634253 19929 catalog_manager.cc:5484] T 4391b5227fb64ef481542a962a5e21a5 P 3c51c33d5616482cafb68e26688820f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c51c33d5616482cafb68e26688820f4 (127.17.177.3). New cstate: current_term: 1 leader_uuid: "3c51c33d5616482cafb68e26688820f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } health_report { overall_health: UNKNOWN } } }
I20240430 20:11:57.637476 19937 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: default.Table_e3160f5ced1b4718b479401ca4df675c [id=2d44ec94131148048021b2ee6366ba30] alter complete (version 3)
I20240430 20:11:57.638826 21397 ts_tablet_manager.cc:1909] T c6e3d09402574f1e8fe9e4160c70a3ee P 3c51c33d5616482cafb68e26688820f4: Deleting consensus metadata
I20240430 20:11:57.639789 21397 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56540 (request call id 100) took 1741 ms. Trace:
I20240430 20:11:57.641052 21397 rpcz_store.cc:276] 0430 20:11:55.898328 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:55.900250 (+ 1922us) service_pool.cc:225] Handling call
0430 20:11:57.568480 (+1668230us) ts_tablet_manager.cc:1120] Deleting tablet c6e3d09402574f1e8fe9e4160c70a3ee
0430 20:11:57.568491 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:57.587649 (+ 19158us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:57.607147 (+ 19498us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:57.639761 (+ 32614us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1668036}
I20240430 20:11:57.642851 18607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94fbd4a57a1e438891fcb456f6b40be3" candidate_uuid: "2efb542aa55e439b99ccb98d1c5c884a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e1c6397f09747f091534faf93542710"
I20240430 20:11:57.643096 18607 raft_consensus.cc:3037] T 94fbd4a57a1e438891fcb456f6b40be3 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Advancing to term 1
I20240430 20:11:57.644124 19900 catalog_manager.cc:4830] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): tablet c6e3d09402574f1e8fe9e4160c70a3ee (table default.Table_656db97355f64d7c8aa29b1352dfe9c8 [id=450f5d1c7e084a1bb88d035c0ccfcf82]) successfully deleted
I20240430 20:11:57.644644 21397 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56540 (request call id 101) took 1729 ms. Trace:
I20240430 20:11:57.644723 21397 rpcz_store.cc:276] 0430 20:11:55.915334 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:55.915869 (+ 535us) service_pool.cc:225] Handling call
0430 20:11:57.644468 (+1728599us) ts_tablet_manager.cc:1120] Deleting tablet c85a6d11d8af486492d8b843c75632e6
0430 20:11:57.644478 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:57.644623 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1728370}
W20240430 20:11:57.645205 19900 catalog_manager.cc:4810] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): delete failed for tablet c85a6d11d8af486492d8b843c75632e6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c85a6d11d8af486492d8b843c75632e6 already in progress: creating tablet
I20240430 20:11:57.645869 18607 raft_consensus.cc:2454] T 94fbd4a57a1e438891fcb456f6b40be3 P 5e1c6397f09747f091534faf93542710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2efb542aa55e439b99ccb98d1c5c884a in term 1.
I20240430 20:11:57.647014 18988 tablet_bootstrap.cc:492] T 9e741b00c74d43b5b6243ba7003f8804 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:57.647182 18988 ts_tablet_manager.cc:1367] T 9e741b00c74d43b5b6243ba7003f8804 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.078s user 0.004s sys 0.000s
I20240430 20:11:57.648233 18988 raft_consensus.cc:348] T 9e741b00c74d43b5b6243ba7003f8804 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:57.648469 18988 raft_consensus.cc:374] T 9e741b00c74d43b5b6243ba7003f8804 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:57.648591 18988 raft_consensus.cc:729] T 9e741b00c74d43b5b6243ba7003f8804 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:57.648900 18988 consensus_queue.cc:260] T 9e741b00c74d43b5b6243ba7003f8804 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:57.651871 18528 heartbeater.cc:503] Master 127.17.177.62:36441 was elected leader, sending a full tablet report...
I20240430 20:11:57.643519 21782 tablet.cc:1722] T 8e33df70a1504dbd852fe0b3c3f61622 P 3c51c33d5616482cafb68e26688820f4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240430 20:11:57.659680 18983 tablet_bootstrap.cc:492] T 9e98e20abc924873853d176455f3678e P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:57.659842 18453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e6ca13dc8eb473e9a764499ec444110. 1 dirs total, 0 dirs full, 0 dirs failed
I20240430 20:11:57.660053 18983 ts_tablet_manager.cc:1367] T 9e98e20abc924873853d176455f3678e P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.179s user 0.006s sys 0.003s
I20240430 20:11:57.661249 18983 raft_consensus.cc:348] T 9e98e20abc924873853d176455f3678e P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:57.661538 18983 raft_consensus.cc:374] T 9e98e20abc924873853d176455f3678e P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:57.661692 18983 raft_consensus.cc:729] T 9e98e20abc924873853d176455f3678e P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
I20240430 20:11:57.667218 18985 tablet_bootstrap.cc:492] T cdeef95b54ca42d38d25b57738d84ef1 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:57.668988 18985 tablet_bootstrap.cc:654] T cdeef95b54ca42d38d25b57738d84ef1 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:57.670516 21757 consensus_queue.cc:1035] T b7d8da6e2fe8499d9d53af84a25f2faf P 3c51c33d5616482cafb68e26688820f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240430 20:11:57.670876 21397 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56540 (request call id 107) took 1716 ms. Trace:
I20240430 20:11:57.678701 19945 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: default.Table_763f0800e8bb4fa69976886416250f42 [id=61c8a5b99ef34a0baeeb343fef7cfd05] alter complete (version 1)
I20240430 20:11:57.678974 19940 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: default.Table_4440df38610340758047115845b42b95 [id=7d186ca20c094cc19aef792fe1c094e9] alter complete (version 1)
I20240430 20:11:57.682631 18447 tablet_service.cc:1514] Processing DeleteTablet for tablet 47e98935eb21449abd3d4d24327778d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:45 UTC) from {username='slave'} at 127.0.0.1:41450
I20240430 20:11:57.683743 18447 tablet_service.cc:1514] Processing DeleteTablet for tablet f3711610610542128c4a84e17bce4a3e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:40 UTC) from {username='slave'} at 127.0.0.1:41450
I20240430 20:11:57.683939 18458 tablet_service.cc:1514] Processing DeleteTablet for tablet f7d3dea34cf54fe2b949d1b7ca68577a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:40 UTC) from {username='slave'} at 127.0.0.1:41450
I20240430 20:11:57.684008 18450 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f4e860484784aa9a2d28af2e9fb611c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:39 UTC) from {username='slave'} at 127.0.0.1:41450
I20240430 20:11:57.684453 18450 tablet_service.cc:1514] Processing DeleteTablet for tablet a445fdfde01c4371a5e6b3daea5b00ab with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:44 UTC) from {username='slave'} at 127.0.0.1:41450
I20240430 20:11:57.684808 18453 tablet_service.cc:1514] Processing DeleteTablet for tablet 6db55bb510b24b0cae897936211e15f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:34 UTC) from {username='slave'} at 127.0.0.1:41450
I20240430 20:11:57.685122 19937 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: default.Table_5ed3ea587cf843db9a1ff1ffab40b678 [id=b3d1c2f97c4648518dbea83a46cadd85] alter complete (version 2)
I20240430 20:11:57.689882 18458 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e5225f1f5f544ecbf8dfa470af3daca with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:51 UTC) from {username='slave'} at 127.0.0.1:41450
I20240430 20:11:57.691437 18445 tablet_service.cc:1514] Processing DeleteTablet for tablet 04b3dff86d5c48a09d03da58bd48f360 with delete_type TABLET_DATA_DELETED (replaced by tablet 89b774967624498fa2be499878b71124) from {username='slave'} at 127.0.0.1:41450
I20240430 20:11:57.691660 18453 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c73d0a6bc414d3b9900570848d19a51 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:41 UTC) from {username='slave'} at 127.0.0.1:41450
I20240430 20:11:57.692083 18445 tablet_service.cc:1514] Processing DeleteTablet for tablet fd1b8407071e4e4287fc172009777414 with delete_type TABLET_DATA_DELETED (replaced by tablet 8abf0b43f8a447dc902135ce94e4fc78) from {username='slave'} at 127.0.0.1:41450
I20240430 20:11:57.664875 20489 raft_consensus.cc:482] T 4432d2c97b634a36b44e1d61d8bf2458 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:57.693078 20489 raft_consensus.cc:504] T 4432d2c97b634a36b44e1d61d8bf2458 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:57.693778 20489 leader_election.cc:288] T 4432d2c97b634a36b44e1d61d8bf2458 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:57.694412 18985 tablet_bootstrap.cc:492] T cdeef95b54ca42d38d25b57738d84ef1 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:57.694553 18985 ts_tablet_manager.cc:1367] T cdeef95b54ca42d38d25b57738d84ef1 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.027s user 0.002s sys 0.002s
I20240430 20:11:57.694622 18473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4432d2c97b634a36b44e1d61d8bf2458" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
W20240430 20:11:57.695117 18558 leader_election.cc:341] T 4432d2c97b634a36b44e1d61d8bf2458 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:57.696465 18453 tablet_service.cc:1514] Processing DeleteTablet for tablet dad16ae02b8b4e29a724d8eec67bb4b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:42 UTC) from {username='slave'} at 127.0.0.1:41450
I20240430 20:11:57.696825 18453 tablet_service.cc:1514] Processing DeleteTablet for tablet d7c1fc7e81364fb890bbc13d7fb6848b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:40 UTC) from {username='slave'} at 127.0.0.1:41450
I20240430 20:11:57.697379 18453 tablet_service.cc:1514] Processing DeleteTablet for tablet 2fc6908ecf954020a676b9d10a2b00e7 with delete_type TABLET_DATA_DELETED (replaced by tablet 9a5654fd913f4fbeb9d126f035ac32bf) from {username='slave'} at 127.0.0.1:41450
I20240430 20:11:57.698014 18453 tablet_service.cc:1514] Processing DeleteTablet for tablet 06d28b6f48a34b17b0d7fb3102df2f4c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:33 UTC) from {username='slave'} at 127.0.0.1:41450
I20240430 20:11:57.698524 18456 tablet_service.cc:1514] Processing DeleteTablet for tablet a663030d37614053a7c0369841403422 with delete_type TABLET_DATA_DELETED (Sending initial creation of tablet) from {username='slave'} at 127.0.0.1:41450
I20240430 20:11:57.699226 18447 tablet_service.cc:1514] Processing DeleteTablet for tablet 436579f473844c6ca1038b496201a550 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:40 UTC) from {username='slave'} at 127.0.0.1:41450
I20240430 20:11:57.700455 18983 consensus_queue.cc:260] T 9e98e20abc924873853d176455f3678e P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:57.700824 18447 tablet_service.cc:1514] Processing DeleteTablet for tablet c1017a50d1d646c486c0b9a75cb88ce4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:43 UTC) from {username='slave'} at 127.0.0.1:41450
I20240430 20:11:57.701025 18983 ts_tablet_manager.cc:1398] T 9e98e20abc924873853d176455f3678e P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.041s user 0.001s sys 0.001s
I20240430 20:11:57.701289 18983 tablet_bootstrap.cc:492] T 2fe548ac307d4b159c30659aa83fd76b P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:57.702100 18447 tablet_service.cc:1514] Processing DeleteTablet for tablet f940d17b02b541289121463e05b0bb5f with delete_type TABLET_DATA_DELETED (replaced by tablet 53da4b3855344b6d92211af767195dc8) from {username='slave'} at 127.0.0.1:41450
I20240430 20:11:57.697114 18985 raft_consensus.cc:348] T cdeef95b54ca42d38d25b57738d84ef1 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:57.702312 18985 raft_consensus.cc:374] T cdeef95b54ca42d38d25b57738d84ef1 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:57.702518 18985 raft_consensus.cc:729] T cdeef95b54ca42d38d25b57738d84ef1 P 2efb542aa55e439b99ccb98d1c5c884a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efb542aa55e439b99ccb98d1c5c884a, State: Initialized, Role: FOLLOWER
I20240430 20:11:57.702939 18983 tablet_bootstrap.cc:654] T 2fe548ac307d4b159c30659aa83fd76b P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:57.704839 21397 rpcz_store.cc:276] 0430 20:11:55.954545 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:55.954639 (+ 94us) service_pool.cc:225] Handling call
0430 20:11:57.670723 (+1716084us) ts_tablet_manager.cc:1120] Deleting tablet 13f0a0a87da242c8acb9892bc2897616
0430 20:11:57.670736 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:57.670852 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1715904}
W20240430 20:11:57.705513 19900 catalog_manager.cc:4810] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): delete failed for tablet 13f0a0a87da242c8acb9892bc2897616 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 13f0a0a87da242c8acb9892bc2897616 already in progress: creating tablet
W20240430 20:11:57.706480 19900 catalog_manager.cc:4797] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): delete failed for tablet dad16ae02b8b4e29a724d8eec67bb4b8 because the tablet was not found. No further retry: Not found: Tablet not found: dad16ae02b8b4e29a724d8eec67bb4b8
W20240430 20:11:57.707194 19900 catalog_manager.cc:4797] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): delete failed for tablet 1f4e860484784aa9a2d28af2e9fb611c because the tablet was not found. No further retry: Not found: Tablet not found: 1f4e860484784aa9a2d28af2e9fb611c
I20240430 20:11:57.677256 18988 ts_tablet_manager.cc:1398] T 9e741b00c74d43b5b6243ba7003f8804 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.030s user 0.002s sys 0.000s
I20240430 20:11:57.707770 18988 tablet_bootstrap.cc:492] T f9bf604e6c234349a9f72f45a4c4aef6 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
I20240430 20:11:57.709318 18760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4432d2c97b634a36b44e1d61d8bf2458" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:57.709654 18988 tablet_bootstrap.cc:654] T f9bf604e6c234349a9f72f45a4c4aef6 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
W20240430 20:11:57.709861 18557 leader_election.cc:341] T 4432d2c97b634a36b44e1d61d8bf2458 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:57.702852 18985 consensus_queue.cc:260] T cdeef95b54ca42d38d25b57738d84ef1 P 2efb542aa55e439b99ccb98d1c5c884a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:57.710019 18557 leader_election.cc:302] T 4432d2c97b634a36b44e1d61d8bf2458 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:57.695968 21454 raft_consensus.cc:482] T 50486644ec234933b9a0b45ee39a6685 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:57.708887 18450 tablet_service.cc:1514] Processing DeleteTablet for tablet df92eb85261d47d88cbf16d29d9ad30e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:45 UTC) from {username='slave'} at 127.0.0.1:41450
I20240430 20:11:57.710561 20489 raft_consensus.cc:2726] T 4432d2c97b634a36b44e1d61d8bf2458 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:57.710445 21454 raft_consensus.cc:504] T 50486644ec234933b9a0b45ee39a6685 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:57.711737 21454 leader_election.cc:288] T 50486644ec234933b9a0b45ee39a6685 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:57.711867 18985 ts_tablet_manager.cc:1398] T cdeef95b54ca42d38d25b57738d84ef1 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent starting tablet: real 0.017s user 0.002s sys 0.000s
W20240430 20:11:57.712141 18696 leader_election.cc:341] T 50486644ec234933b9a0b45ee39a6685 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Not found: Tablet not found: 50486644ec234933b9a0b45ee39a6685
W20240430 20:11:57.713469 18699 leader_election.cc:341] T 50486644ec234933b9a0b45ee39a6685 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Not found: Tablet not found: 50486644ec234933b9a0b45ee39a6685
I20240430 20:11:57.713372 18985 tablet_bootstrap.cc:492] T 9cc9251abd4e40d59a2bd2d0fada6165 P 2efb542aa55e439b99ccb98d1c5c884a: Bootstrap starting.
I20240430 20:11:57.713575 18699 leader_election.cc:302] T 50486644ec234933b9a0b45ee39a6685 P 3c51c33d5616482cafb68e26688820f4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c51c33d5616482cafb68e26688820f4; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 5e1c6397f09747f091534faf93542710
I20240430 20:11:57.713939 21707 raft_consensus.cc:2726] T 50486644ec234933b9a0b45ee39a6685 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240430 20:11:57.714308 19930 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: default.Table_1ce204e15fce4c469816cf4e7f80cac7 [id=d268c31d82df4871bc1a78aea5871b33] alter complete (version 1)
I20240430 20:11:57.716043 21395 ts_tablet_manager.cc:1888] T aa9998f4e17147aa8f2b35c01980463a P 2efb542aa55e439b99ccb98d1c5c884a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:57.716158 21395 log.cc:1196] T aa9998f4e17147aa8f2b35c01980463a P 2efb542aa55e439b99ccb98d1c5c884a: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-0/wal/wals/aa9998f4e17147aa8f2b35c01980463a
I20240430 20:11:57.716523 21395 ts_tablet_manager.cc:1909] T aa9998f4e17147aa8f2b35c01980463a P 2efb542aa55e439b99ccb98d1c5c884a: Deleting consensus metadata
I20240430 20:11:57.717326 21395 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41450 (request call id 82) took 3657 ms. Trace:
I20240430 20:11:57.717422 21395 rpcz_store.cc:276] 0430 20:11:54.059479 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:54.059558 (+ 79us) service_pool.cc:225] Handling call
0430 20:11:57.597951 (+3538393us) ts_tablet_manager.cc:1120] Deleting tablet aa9998f4e17147aa8f2b35c01980463a
0430 20:11:57.597961 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:57.712914 (+114953us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:57.715936 (+ 3022us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:57.717305 (+ 1369us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3538229}
I20240430 20:11:57.722721 20489 raft_consensus.cc:482] T 5df364628561475391c9a5e10ff5a05f P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:57.722905 20489 raft_consensus.cc:504] T 5df364628561475391c9a5e10ff5a05f P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:57.723672 20489 leader_election.cc:288] T 5df364628561475391c9a5e10ff5a05f P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867), 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217)
I20240430 20:11:57.723775 18985 tablet_bootstrap.cc:654] T 9cc9251abd4e40d59a2bd2d0fada6165 P 2efb542aa55e439b99ccb98d1c5c884a: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:57.725257 18601 tablet_service.cc:1514] Processing DeleteTablet for tablet f3711610610542128c4a84e17bce4a3e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:40 UTC) from {username='slave'} at 127.0.0.1:45934
I20240430 20:11:57.725461 18592 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f4e860484784aa9a2d28af2e9fb611c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:39 UTC) from {username='slave'} at 127.0.0.1:45934
I20240430 20:11:57.725601 18595 tablet_service.cc:1514] Processing DeleteTablet for tablet 8901327b3efb4201adce61a9266142ea with delete_type TABLET_DATA_DELETED (replaced by tablet b1277899f0c040cea7dc2b12f3fd68de) from {username='slave'} at 127.0.0.1:45934
I20240430 20:11:57.725711 18590 tablet_service.cc:1514] Processing DeleteTablet for tablet e4cffa295d8c401aa8415ce7e8f56526 with delete_type TABLET_DATA_DELETED (replaced by tablet e5b85a066cdc46d48ccf8201e5423c29) from {username='slave'} at 127.0.0.1:45934
I20240430 20:11:57.725838 18595 tablet_service.cc:1514] Processing DeleteTablet for tablet de1dfe6c82d34a4d8d02e466c55327ff with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:39 UTC) from {username='slave'} at 127.0.0.1:45934
I20240430 20:11:57.725948 18592 tablet_service.cc:1514] Processing DeleteTablet for tablet 2873af6da928463f9809864b8eaf0c4a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:32 UTC) from {username='slave'} at 127.0.0.1:45934
I20240430 20:11:57.725683 18602 tablet_service.cc:1514] Processing DeleteTablet for tablet 246fb63a5e9842df8263800e28a5935c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:32 UTC) from {username='slave'} at 127.0.0.1:45934
I20240430 20:11:57.726068 18601 tablet_service.cc:1514] Processing DeleteTablet for tablet f7d3dea34cf54fe2b949d1b7ca68577a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:40 UTC) from {username='slave'} at 127.0.0.1:45934
I20240430 20:11:57.726430 18590 tablet_service.cc:1514] Processing DeleteTablet for tablet dbb427aefa6b47b0805d1751fea4efcb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:45 UTC) from {username='slave'} at 127.0.0.1:45934
I20240430 20:11:57.727159 18745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5df364628561475391c9a5e10ff5a05f" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:57.727540 21396 ts_tablet_manager.cc:1888] T 3b7fdc6ec7e046c5af894f9a6eda17b0 P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240430 20:11:57.727651 21396 log.cc:1196] T 3b7fdc6ec7e046c5af894f9a6eda17b0 P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/3b7fdc6ec7e046c5af894f9a6eda17b0
W20240430 20:11:57.727699 18557 leader_election.cc:341] T 5df364628561475391c9a5e10ff5a05f P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:57.728049 21396 ts_tablet_manager.cc:1909] T 3b7fdc6ec7e046c5af894f9a6eda17b0 P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:57.732187 18592 tablet_service.cc:1514] Processing DeleteTablet for tablet d34a450a91d542c98261456d1f635e9c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:49 UTC) from {username='slave'} at 127.0.0.1:45934
I20240430 20:11:57.732390 18595 tablet_service.cc:1514] Processing DeleteTablet for tablet 8668f7a285de43f48d83abecb4b2e96c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:50 UTC) from {username='slave'} at 127.0.0.1:45934
I20240430 20:11:57.732548 18591 tablet_service.cc:1514] Processing DeleteTablet for tablet 898b2c91adf34545808df9c786e218c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:40 UTC) from {username='slave'} at 127.0.0.1:45934
I20240430 20:11:57.731067 18588 tablet_service.cc:1514] Processing DeleteTablet for tablet a445fdfde01c4371a5e6b3daea5b00ab with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:44 UTC) from {username='slave'} at 127.0.0.1:45934
I20240430 20:11:57.731532 18602 tablet_service.cc:1514] Processing DeleteTablet for tablet 558e71ec370d46ca8fbdcec5b57eb2ec with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:37 UTC) from {username='slave'} at 127.0.0.1:45934
I20240430 20:11:57.732776 18590 tablet_service.cc:1514] Processing DeleteTablet for tablet 4dac8b3838954d839094021f1c0503da with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:40 UTC) from {username='slave'} at 127.0.0.1:45934
I20240430 20:11:57.732944 18601 tablet_service.cc:1514] Processing DeleteTablet for tablet aa9998f4e17147aa8f2b35c01980463a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:42 UTC) from {username='slave'} at 127.0.0.1:45934
I20240430 20:11:57.733810 21396 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:45934 (request call id 131) took 1785 ms. Trace:
I20240430 20:11:57.735459 18473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5df364628561475391c9a5e10ff5a05f" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
W20240430 20:11:57.736707 18558 leader_election.cc:341] T 5df364628561475391c9a5e10ff5a05f P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:57.736819 18558 leader_election.cc:302] T 5df364628561475391c9a5e10ff5a05f P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:57.737448 21752 raft_consensus.cc:2726] T 5df364628561475391c9a5e10ff5a05f P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:57.733893 21396 rpcz_store.cc:276] 0430 20:11:55.948527 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:55.948810 (+ 283us) service_pool.cc:225] Handling call
0430 20:11:57.589190 (+1640380us) ts_tablet_manager.cc:1120] Deleting tablet 3b7fdc6ec7e046c5af894f9a6eda17b0
0430 20:11:57.589198 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:57.705713 (+116515us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:57.727422 (+ 21709us) tablet_metadata.cc:641] Metadata flushed
0430 20:11:57.733792 (+ 6370us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1640192}
I20240430 20:11:57.738476 21395 tablet_replica.cc:330] T 90a694bd5db649afa50f3c4e752939c1 P 2efb542aa55e439b99ccb98d1c5c884a: stopping tablet replica
I20240430 20:11:57.738981 19895 catalog_manager.cc:4830] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): tablet 3b7fdc6ec7e046c5af894f9a6eda17b0 (table default.Table_d441c3bc84ed42358d1bc6e717108efb [id=7b516b6eb3a240a680bf271eab0324f8]) successfully deleted
I20240430 20:11:57.739324 21396 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:45934 (request call id 133) took 1790 ms. Trace:
I20240430 20:11:57.734544 19902 catalog_manager.cc:4830] TS 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): tablet aa9998f4e17147aa8f2b35c01980463a (table default.Table_ffb3192f443d437ab63f56f628e596ad [id=608890834da843fa9af0cfc0d548029e]) successfully deleted
I20240430 20:11:57.739096 21395 raft_consensus.cc:2229] T 90a694bd5db649afa50f3c4e752939c1 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus shutting down.
W20240430 20:11:57.741289 19942 catalog_manager.cc:5676] Failed to send AlterTable RPC for tablet 9e8ab75e7d9c4b64a7b7dd2678569490 (table default.Table_b2d07520b9944a89ad62fca70396174e [id=1c8e0a3a44ec4a798770c0165ed3dffa], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 9e8ab75e7d9c4b64a7b7dd2678569490 consensus state has no leader
I20240430 20:11:57.741631 21395 raft_consensus.cc:2258] T 90a694bd5db649afa50f3c4e752939c1 P 2efb542aa55e439b99ccb98d1c5c884a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:57.739472 21396 rpcz_store.cc:276] 0430 20:11:55.948717 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:55.949032 (+ 315us) service_pool.cc:225] Handling call
0430 20:11:57.738725 (+1789693us) ts_tablet_manager.cc:1120] Deleting tablet 489dc838744a49eba542009c86688aa5
0430 20:11:57.738735 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:57.739308 (+ 573us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1789552}
W20240430 20:11:57.743137 19895 catalog_manager.cc:4797] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): delete failed for tablet 489dc838744a49eba542009c86688aa5 because the tablet was not found. No further retry: Not found: Tablet not found: 489dc838744a49eba542009c86688aa5
I20240430 20:11:57.743611 18601 tablet_service.cc:1514] Processing DeleteTablet for tablet cefce2f2a392495bb71399ed75db938c with delete_type TABLET_DATA_DELETED (replaced by tablet 9e1fed3070794d8da6e31692adfcb8cc) from {username='slave'} at 127.0.0.1:45934
I20240430 20:11:57.745362 21396 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:45934 (request call id 141) took 1795 ms. Trace:
I20240430 20:11:57.745451 21396 rpcz_store.cc:276] 0430 20:11:55.949405 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:55.949450 (+ 45us) service_pool.cc:225] Handling call
0430 20:11:57.743846 (+1794396us) ts_tablet_manager.cc:1120] Deleting tablet a663030d37614053a7c0369841403422
0430 20:11:57.743855 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0430 20:11:57.745343 (+ 1488us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1794237}
I20240430 20:11:57.745734 21396 tablet_replica.cc:330] T a37d917bcc45468f9e3ee805640fba98 P 5e1c6397f09747f091534faf93542710: stopping tablet replica
I20240430 20:11:57.745955 21396 raft_consensus.cc:2229] T a37d917bcc45468f9e3ee805640fba98 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Raft consensus shutting down.
W20240430 20:11:57.746619 19895 catalog_manager.cc:4797] TS 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): delete failed for tablet a663030d37614053a7c0369841403422 because the tablet was not found. No further retry: Not found: Tablet not found: a663030d37614053a7c0369841403422
I20240430 20:11:57.747339 21395 ts_tablet_manager.cc:1875] T 90a694bd5db649afa50f3c4e752939c1 P 2efb542aa55e439b99ccb98d1c5c884a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:57.749455 21752 raft_consensus.cc:482] T 9e98e20abc924873853d176455f3678e P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:57.749572 21752 raft_consensus.cc:504] T 9e98e20abc924873853d176455f3678e P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:57.750192 21752 leader_election.cc:288] T 9e98e20abc924873853d176455f3678e P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:57.750906 18745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e98e20abc924873853d176455f3678e" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:57.750929 18473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e98e20abc924873853d176455f3678e" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
W20240430 20:11:57.751454 18558 leader_election.cc:341] T 9e98e20abc924873853d176455f3678e P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
W20240430 20:11:57.752228 18557 leader_election.cc:341] T 9e98e20abc924873853d176455f3678e P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
W20240430 20:11:57.753242 19900 catalog_manager.cc:4903] TS 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): alter failed for tablet c6e3d09402574f1e8fe9e4160c70a3ee (table default.Table_656db97355f64d7c8aa29b1352dfe9c8 [id=450f5d1c7e084a1bb88d035c0ccfcf82]),no further retry: Not found: Tablet not found: c6e3d09402574f1e8fe9e4160c70a3ee
I20240430 20:11:57.753743 21752 raft_consensus.cc:482] T e848539446a64a968b37045d367bb3ce P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:57.753863 21752 raft_consensus.cc:504] T e848539446a64a968b37045d367bb3ce P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:57.754719 21752 leader_election.cc:288] T e848539446a64a968b37045d367bb3ce P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:57.756583 18473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e848539446a64a968b37045d367bb3ce" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:57.757174 18557 leader_election.cc:302] T 9e98e20abc924873853d176455f3678e P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:57.758040 21752 raft_consensus.cc:2726] T 9e98e20abc924873853d176455f3678e P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:57.758134 18590 tablet_service.cc:1514] Processing DeleteTablet for tablet fcc16db4d6be42edb7bd1ce18fe0190d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:45 UTC) from {username='slave'} at 127.0.0.1:45934
I20240430 20:11:57.758208 18602 tablet_service.cc:1514] Processing DeleteTablet for tablet d4d6c37b578c4fe1aa666b0a6b0bc7c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:40 UTC) from {username='slave'} at 127.0.0.1:45934
W20240430 20:11:57.758458 18558 leader_election.cc:341] T e848539446a64a968b37045d367bb3ce P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:57.758606 18602 tablet_service.cc:1514] Processing DeleteTablet for tablet 5bd0362ff41b4faca0205d136d596737 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:31 UTC) from {username='slave'} at 127.0.0.1:45934
I20240430 20:11:57.758993 18602 tablet_service.cc:1514] Processing DeleteTablet for tablet eb76f60448774fe1967fffe7c6873cfc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:32 UTC) from {username='slave'} at 127.0.0.1:45934
I20240430 20:11:57.760224 18602 tablet_service.cc:1514] Processing DeleteTablet for tablet 90a694bd5db649afa50f3c4e752939c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:39 UTC) from {username='slave'} at 127.0.0.1:45934
I20240430 20:11:57.760469 21396 raft_consensus.cc:2258] T a37d917bcc45468f9e3ee805640fba98 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:57.760444 18745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e848539446a64a968b37045d367bb3ce" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
I20240430 20:11:57.760259 18983 tablet_bootstrap.cc:492] T 2fe548ac307d4b159c30659aa83fd76b P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:57.760848 18983 ts_tablet_manager.cc:1367] T 2fe548ac307d4b159c30659aa83fd76b P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.060s user 0.003s sys 0.000s
W20240430 20:11:57.760916 18557 leader_election.cc:341] T e848539446a64a968b37045d367bb3ce P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:57.761029 18557 leader_election.cc:302] T e848539446a64a968b37045d367bb3ce P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:57.761363 21752 raft_consensus.cc:2726] T e848539446a64a968b37045d367bb3ce P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:57.761627 18983 raft_consensus.cc:348] T 2fe548ac307d4b159c30659aa83fd76b P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
W20240430 20:11:57.761694 19942 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.17.177.2:50593 (request call id 151) took 3640 ms (client timeout 1000 ms). Trace:
I20240430 20:11:57.761793 18983 raft_consensus.cc:374] T 2fe548ac307d4b159c30659aa83fd76b P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:57.761842 18983 raft_consensus.cc:729] T 2fe548ac307d4b159c30659aa83fd76b P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c6397f09747f091534faf93542710, State: Initialized, Role: FOLLOWER
W20240430 20:11:57.761838 19942 rpcz_store.cc:269] 0430 20:11:54.121109 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:54.164529 (+ 43420us) service_pool.cc:225] Handling call
0430 20:11:54.432702 (+268173us) spinlock_profiling.cc:243] Waited 262 ms on lock 0x616000070654. stack: 00007f6d23ac8756 00007f6d23ac8510 00007f6d22afb48d 0000000000955804 00000000009557cd 00007f6d3260715d 00007f6d32607253 00007f6d2f339d41 00007f6d36f2534d 00007f6d370ac668 00007f6d2de97786 00007f6d2de974b5 00007f6d2a7d004e 00007f6d2a7cef2e 00007f6d2a7d5405 00007f6d2a7d5d2f
0430 20:11:57.761604 (+3328902us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":115474,"reported_tablets":217,"spinlock_wait_cycles":603086592}
I20240430 20:11:57.762104 18983 consensus_queue.cc:260] T 2fe548ac307d4b159c30659aa83fd76b P 5e1c6397f09747f091534faf93542710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:57.762676 18983 ts_tablet_manager.cc:1398] T 2fe548ac307d4b159c30659aa83fd76b P 5e1c6397f09747f091534faf93542710: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240430 20:11:57.762954 18983 tablet_bootstrap.cc:492] T 821c55edb3414d72b1489eea97d3f73a P 5e1c6397f09747f091534faf93542710: Bootstrap starting.
I20240430 20:11:57.762302 21396 ts_tablet_manager.cc:1875] T a37d917bcc45468f9e3ee805640fba98 P 5e1c6397f09747f091534faf93542710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:57.764564 18983 tablet_bootstrap.cc:654] T 821c55edb3414d72b1489eea97d3f73a P 5e1c6397f09747f091534faf93542710: Neither blocks nor log segments found. Creating new log.
W20240430 20:11:57.769505 18415 consensus_peers.cc:485] T 8901327b3efb4201adce61a9266142ea P 2efb542aa55e439b99ccb98d1c5c884a -> Peer 5e1c6397f09747f091534faf93542710 (127.17.177.2:46027): Couldn't send request to peer 5e1c6397f09747f091534faf93542710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8901327b3efb4201adce61a9266142ea. This is attempt 1: this message will repeat every 5th retry.
W20240430 20:11:57.770447 19944 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.17.177.2:50593 (request call id 150) took 4949 ms (client timeout 1000 ms). Trace:
W20240430 20:11:57.770735 19944 rpcz_store.cc:269] 0430 20:11:52.821442 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:52.863521 (+ 42079us) service_pool.cc:225] Handling call
0430 20:11:53.953464 (+1089943us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:53.953473 (+ 9us) write_op.cc:446] Released schema lock
0430 20:11:57.770407 (+3816934us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:53.716895 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:53.717041 (+ 146us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:53.717050 (+ 9us) write_op.cc:427] Acquired schema lock
0430 20:11:53.717053 (+ 3us) tablet.cc:660] Decoding operations
0430 20:11:53.717278 (+ 225us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:53.717294 (+ 16us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:53.717302 (+ 8us) tablet.cc:683] Acquiring locks for 5 operations
0430 20:11:53.717491 (+ 189us) tablet.cc:699] Row locks acquired
0430 20:11:53.717493 (+ 2us) write_op.cc:260] PREPARE: finished
0430 20:11:53.717535 (+ 42us) write_op.cc:270] Start()
0430 20:11:53.717580 (+ 45us) write_op.cc:276] Timestamp: P: 1714507913717508 usec, L: 0
0430 20:11:53.717584 (+ 4us) op_driver.cc:331] REPLICATION: starting
0430 20:11:53.717875 (+ 291us) log.cc:842] Serialized 2039 byte log entry
0430 20:11:53.747095 (+ 29220us) op_driver.cc:444] REPLICATION: finished
0430 20:11:53.747231 (+ 136us) write_op.cc:301] APPLY: starting
0430 20:11:53.747254 (+ 23us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:53.747314 (+ 60us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:53.747345 (+ 31us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:53.747451 (+ 106us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:53.747491 (+ 40us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=5
0430 20:11:53.747500 (+ 9us) write_op.cc:312] APPLY: finished
0430 20:11:53.747600 (+ 100us) log.cc:842] Serialized 47 byte log entry
0430 20:11:53.747678 (+ 78us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:53.747708 (+ 30us) write_op.cc:446] Released schema lock
0430 20:11:53.747717 (+ 9us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":73187,"reported_tablets":210,"child_traces":[["op",{"apply.queue_time_us":80,"apply.run_cpu_time_us":550,"apply.run_wall_time_us":550,"num_ops":5,"prepare.queue_time_us":3599,"prepare.run_cpu_time_us":1118,"prepare.run_wall_time_us":1118,"raft.queue_time_us":568,"raft.run_cpu_time_us":345,"raft.run_wall_time_us":345,"replication_time_us":29476}]]}
I20240430 20:11:57.774624 19945 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: default.Table_276bbdc7cfe542d1a2936485a2aad1ec [id=f146a15e6e8a4acbb4c734c1c863da57] alter complete (version 1)
I20240430 20:11:57.775153 19940 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: default.Table_e686cdfcbcc046749fa5b51578a64794 [id=c3dc9f05316d46d6a3ef4d03f40877ec] alter complete (version 1)
I20240430 20:11:57.778851 21397 tablet_replica.cc:330] T d4d6c37b578c4fe1aa666b0a6b0bc7c9 P 3c51c33d5616482cafb68e26688820f4: stopping tablet replica
I20240430 20:11:57.779227 21397 raft_consensus.cc:2229] T d4d6c37b578c4fe1aa666b0a6b0bc7c9 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240430 20:11:57.779532 21397 raft_consensus.cc:2258] T d4d6c37b578c4fe1aa666b0a6b0bc7c9 P 3c51c33d5616482cafb68e26688820f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240430 20:11:57.780328 21397 ts_tablet_manager.cc:1875] T d4d6c37b578c4fe1aa666b0a6b0bc7c9 P 3c51c33d5616482cafb68e26688820f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240430 20:11:57.785441 18445 tablet_service.cc:1514] Processing DeleteTablet for tablet a692b253db914531b8fc6ffaf3e350b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:51 UTC) from {username='slave'} at 127.0.0.1:41450
W20240430 20:11:57.787328 19946 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.17.177.1:47345 (request call id 136) took 4411 ms (client timeout 1000 ms). Trace:
I20240430 20:11:57.792752 18988 tablet_bootstrap.cc:492] T f9bf604e6c234349a9f72f45a4c4aef6 P 3c51c33d5616482cafb68e26688820f4: No bootstrap required, opened a new log
I20240430 20:11:57.793061 18988 ts_tablet_manager.cc:1367] T f9bf604e6c234349a9f72f45a4c4aef6 P 3c51c33d5616482cafb68e26688820f4: Time spent bootstrapping tablet: real 0.085s user 0.003s sys 0.000s
I20240430 20:11:57.793944 18988 raft_consensus.cc:348] T f9bf604e6c234349a9f72f45a4c4aef6 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:57.794273 18988 raft_consensus.cc:374] T f9bf604e6c234349a9f72f45a4c4aef6 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240430 20:11:57.794412 18988 raft_consensus.cc:729] T f9bf604e6c234349a9f72f45a4c4aef6 P 3c51c33d5616482cafb68e26688820f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c51c33d5616482cafb68e26688820f4, State: Initialized, Role: FOLLOWER
I20240430 20:11:57.794864 18988 consensus_queue.cc:260] T f9bf604e6c234349a9f72f45a4c4aef6 P 3c51c33d5616482cafb68e26688820f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } } peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } }
I20240430 20:11:57.795833 18741 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e5225f1f5f544ecbf8dfa470af3daca with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:51 UTC) from {username='slave'} at 127.0.0.1:56540
I20240430 20:11:57.796242 18741 tablet_service.cc:1514] Processing DeleteTablet for tablet 06d28b6f48a34b17b0d7fb3102df2f4c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:33 UTC) from {username='slave'} at 127.0.0.1:56540
I20240430 20:11:57.796619 18741 tablet_service.cc:1514] Processing DeleteTablet for tablet a692b253db914531b8fc6ffaf3e350b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:51 UTC) from {username='slave'} at 127.0.0.1:56540
I20240430 20:11:57.796967 18741 tablet_service.cc:1514] Processing DeleteTablet for tablet 283e5e73cbc04daba0de19f4541f9fd0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:40 UTC) from {username='slave'} at 127.0.0.1:56540
I20240430 20:11:57.797340 18741 tablet_service.cc:1514] Processing DeleteTablet for tablet 13c2e69ae91f4957887fd64b90f6e435 with delete_type TABLET_DATA_DELETED (replaced by tablet eba4bd4665ae4c04a2cab3ab1a2da780) from {username='slave'} at 127.0.0.1:56540
I20240430 20:11:57.797894 18741 tablet_service.cc:1514] Processing DeleteTablet for tablet 4dac8b3838954d839094021f1c0503da with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:40 UTC) from {username='slave'} at 127.0.0.1:56540
I20240430 20:11:57.798102 18733 tablet_service.cc:1514] Processing DeleteTablet for tablet 436579f473844c6ca1038b496201a550 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:40 UTC) from {username='slave'} at 127.0.0.1:56540
I20240430 20:11:57.798470 18733 tablet_service.cc:1514] Processing DeleteTablet for tablet a663030d37614053a7c0369841403422 with delete_type TABLET_DATA_DELETED (Sending initial creation of tablet) from {username='slave'} at 127.0.0.1:56540
I20240430 20:11:57.798646 18983 tablet_bootstrap.cc:492] T 821c55edb3414d72b1489eea97d3f73a P 5e1c6397f09747f091534faf93542710: No bootstrap required, opened a new log
I20240430 20:11:57.798893 18733 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f4e860484784aa9a2d28af2e9fb611c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:39 UTC) from {username='slave'} at 127.0.0.1:56540
I20240430 20:11:57.798909 18983 ts_tablet_manager.cc:1367] T 821c55edb3414d72b1489eea97d3f73a P 5e1c6397f09747f091534faf93542710: Time spent bootstrapping tablet: real 0.036s user 0.001s sys 0.004s
I20240430 20:11:57.799422 18733 tablet_service.cc:1514] Processing DeleteTablet for tablet e4cffa295d8c401aa8415ce7e8f56526 with delete_type TABLET_DATA_DELETED (replaced by tablet e5b85a066cdc46d48ccf8201e5423c29) from {username='slave'} at 127.0.0.1:56540
I20240430 20:11:57.799530 21752 raft_consensus.cc:482] T 024069ab786749028c020abd1c836cf5 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240430 20:11:57.799670 21752 raft_consensus.cc:504] T 024069ab786749028c020abd1c836cf5 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2efb542aa55e439b99ccb98d1c5c884a" member_type: VOTER last_known_addr { host: "127.17.177.1" port: 41217 } } peers { permanent_uuid: "3c51c33d5616482cafb68e26688820f4" member_type: VOTER last_known_addr { host: "127.17.177.3" port: 44867 } } peers { permanent_uuid: "5e1c6397f09747f091534faf93542710" member_type: VOTER last_known_addr { host: "127.17.177.2" port: 46027 } }
I20240430 20:11:57.800192 18730 tablet_service.cc:1514] Processing DeleteTablet for tablet dad16ae02b8b4e29a724d8eec67bb4b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:42 UTC) from {username='slave'} at 127.0.0.1:56540
I20240430 20:11:57.800309 21752 leader_election.cc:288] T 024069ab786749028c020abd1c836cf5 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217), 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867)
I20240430 20:11:57.800730 18988 ts_tablet_manager.cc:1398] T f9bf604e6c234349a9f72f45a4c4aef6 P 3c51c33d5616482cafb68e26688820f4: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240430 20:11:57.801260 18473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "024069ab786749028c020abd1c836cf5" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2efb542aa55e439b99ccb98d1c5c884a" is_pre_election: true
I20240430 20:11:57.801376 18988 tablet_bootstrap.cc:492] T 798a13f545fd4d8f8c654eaa2452eda1 P 3c51c33d5616482cafb68e26688820f4: Bootstrap starting.
W20240430 20:11:57.801803 18558 leader_election.cc:341] T 024069ab786749028c020abd1c836cf5 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2efb542aa55e439b99ccb98d1c5c884a (127.17.177.1:41217): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:57.803197 18988 tablet_bootstrap.cc:654] T 798a13f545fd4d8f8c654eaa2452eda1 P 3c51c33d5616482cafb68e26688820f4: Neither blocks nor log segments found. Creating new log.
I20240430 20:11:57.804294 18733 tablet_service.cc:1514] Processing DeleteTablet for tablet 246fb63a5e9842df8263800e28a5935c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:32 UTC) from {username='slave'} at 127.0.0.1:56540
I20240430 20:11:57.804563 18741 tablet_service.cc:1514] Processing DeleteTablet for tablet fdf53061efd74fa29a9d8208d1fd210e with delete_type TABLET_DATA_DELETED (replaced by tablet 5df364628561475391c9a5e10ff5a05f) from {username='slave'} at 127.0.0.1:56540
I20240430 20:11:57.804857 18741 tablet_service.cc:1514] Processing DeleteTablet for tablet c6e3d09402574f1e8fe9e4160c70a3ee with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:54 UTC) from {username='slave'} at 127.0.0.1:56540
I20240430 20:11:57.805109 21396 ts_tablet_manager.cc:1888] T a37d917bcc45468f9e3ee805640fba98 P 5e1c6397f09747f091534faf93542710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240430 20:11:57.805218 21396 log.cc:1196] T a37d917bcc45468f9e3ee805640fba98 P 5e1c6397f09747f091534faf93542710: Deleting WAL directory at /tmp/dist-test-taskNG3HqZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1714507877523513-18116-0/minicluster-data/ts-1/wal/wals/a37d917bcc45468f9e3ee805640fba98
I20240430 20:11:57.804579 18730 tablet_service.cc:1514] Processing DeleteTablet for tablet f3711610610542128c4a84e17bce4a3e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-04-30 20:11:40 UTC) from {username='slave'} at 127.0.0.1:56540
I20240430 20:11:57.789151 19938 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: default.Table_f1dc481c6d8c43f589fe2b1892a4cb10 [id=db696ce66005401fa73746e55c071f44] alter complete (version 1)
I20240430 20:11:57.805583 21396 ts_tablet_manager.cc:1909] T a37d917bcc45468f9e3ee805640fba98 P 5e1c6397f09747f091534faf93542710: Deleting consensus metadata
I20240430 20:11:57.789113 19930 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 88b3f98793354c7aaa5c106e8ec31ed8: default.Table_cb2a4b9ceaf4431facfff2c9bcf94c5d [id=3f8690a437ae49a3beb009fa7899d5d0] alter complete (version 1)
W20240430 20:11:57.804876 19940 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.17.177.3:39881 (request call id 139) took 2777 ms (client timeout 1000 ms). Trace:
I20240430 20:11:57.811415 18985 tablet_bootstrap.cc:492] T 9cc9251abd4e40d59a2bd2d0fada6165 P 2efb542aa55e439b99ccb98d1c5c884a: No bootstrap required, opened a new log
I20240430 20:11:57.815979 18760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "024069ab786749028c020abd1c836cf5" candidate_uuid: "5e1c6397f09747f091534faf93542710" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c51c33d5616482cafb68e26688820f4" is_pre_election: true
W20240430 20:11:57.817574 18557 leader_election.cc:341] T 024069ab786749028c020abd1c836cf5 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c51c33d5616482cafb68e26688820f4 (127.17.177.3:44867): Illegal state: must be running to vote when last-logged opid is not known
I20240430 20:11:57.817787 18557 leader_election.cc:302] T 024069ab786749028c020abd1c836cf5 P 5e1c6397f09747f091534faf93542710 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e1c6397f09747f091534faf93542710; no voters: 2efb542aa55e439b99ccb98d1c5c884a, 3c51c33d5616482cafb68e26688820f4
I20240430 20:11:57.818275 21752 raft_consensus.cc:2726] T 024069ab786749028c020abd1c836cf5 P 5e1c6397f09747f091534faf93542710 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240430 20:11:57.821123 18985 ts_tablet_manager.cc:1367] T 9cc9251abd4e40d59a2bd2d0fada6165 P 2efb542aa55e439b99ccb98d1c5c884a: Time spent bootstrapping tablet: real 0.108s user 0.003s sys 0.003s
W20240430 20:11:57.811417 19940 rpcz_store.cc:269] 0430 20:11:55.027186 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 20:11:55.897783 (+870597us) service_pool.cc:225] Handling call
0430 20:11:57.804795 (+1907012us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0430 20:11:56.302771 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:56.302932 (+ 161us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:56.302941 (+ 9us) write_op.cc:427] Acquired schema lock
0430 20:11:56.302946 (+ 5us) tablet.cc:660] Decoding operations
0430 20:11:56.303174 (+ 228us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:56.303194 (+ 20us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:56.303201 (+ 7us) tablet.cc:683] Acquiring locks for 4 operations
0430 20:11:56.303358 (+ 157us) tablet.cc:699] Row locks acquired
0430 20:11:56.303362 (+ 4us) write_op.cc:260] PREPARE: finished
0430 20:11:56.303407 (+ 45us) write_op.cc:270] Start()
0430 20:11:56.303451 (+ 44us) write_op.cc:276] Timestamp: P: 1714507916303379 usec, L: 0
0430 20:11:56.303456 (+ 5us) op_driver.cc:331] REPLICATION: starting
0430 20:11:56.303694 (+ 238us) log.cc:842] Serialized 1662 byte log entry
0430 20:11:56.307840 (+ 4146us) op_driver.cc:444] REPLICATION: finished
0430 20:11:56.307985 (+ 145us) write_op.cc:301] APPLY: starting
0430 20:11:56.308006 (+ 21us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:56.308091 (+ 85us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:56.308132 (+ 41us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:56.308249 (+ 117us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:56.308301 (+ 52us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=4
0430 20:11:56.308312 (+ 11us) write_op.cc:312] APPLY: finished
0430 20:11:56.308443 (+ 131us) log.cc:842] Serialized 41 byte log entry
0430 20:11:56.308515 (+ 72us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:56.308545 (+ 30us) write_op.cc:446] Released schema lock
0430 20:11:56.308556 (+ 11us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:56.308625 (+ 69us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:56.308630 (+ 5us) write_op.cc:446] Released schema lock
Related trace 'op':
0430 20:11:56.344503 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 20:11:56.344632 (+ 129us) write_op.cc:424] Acquiring schema lock in shared mode
0430 20:11:56.344639 (+ 7us) write_op.cc:427] Acquired schema lock
0430 20:11:56.344643 (+ 4us) tablet.cc:660] Decoding operations
0430 20:11:56.344761 (+ 118us) write_op.cc:612] Acquiring the partition lock for write op
0430 20:11:56.344775 (+ 14us) write_op.cc:633] Partition lock acquired for write op
0430 20:11:56.344781 (+ 6us) tablet.cc:683] Acquiring locks for 1 operations
0430 20:11:56.344875 (+ 94us) tablet.cc:699] Row locks acquired
0430 20:11:56.344878 (+ 3us) write_op.cc:260] PREPARE: finished
0430 20:11:56.344918 (+ 40us) write_op.cc:270] Start()
0430 20:11:56.344961 (+ 43us) write_op.cc:276] Timestamp: P: 1714507916344888 usec, L: 0
0430 20:11:56.344965 (+ 4us) op_driver.cc:331] REPLICATION: starting
0430 20:11:56.345161 (+ 196us) log.cc:842] Serialized 356 byte log entry
0430 20:11:56.348971 (+ 3810us) op_driver.cc:444] REPLICATION: finished
0430 20:11:56.349087 (+ 116us) write_op.cc:301] APPLY: starting
0430 20:11:56.349106 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0430 20:11:56.349150 (+ 44us) tablet.cc:1368] finished BulkCheckPresence
0430 20:11:56.349179 (+ 29us) tablet.cc:1370] starting ApplyRowOperation cycle
0430 20:11:56.349249 (+ 70us) tablet.cc:1381] finished ApplyRowOperation cycle
0430 20:11:56.349287 (+ 38us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0430 20:11:56.349295 (+ 8us) write_op.cc:312] APPLY: finished
0430 20:11:56.349407 (+ 112us) log.cc:842] Serialized 23 byte log entry
0430 20:11:56.349467 (+ 60us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:56.349488 (+ 21us) write_op.cc:446] Released schema lock
0430 20:11:56.349495 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0430 20:11:56.349556 (+ 61us) write_op.cc:481] Releasing partition, row and schema locks
0430 20:11:56.349560 (+ 4us) write_op.cc:446] Released schema lock
Related trace 'op'